Abusing GitHub as a PyPI server

Published 2024-01-15 on Farid Zakaria's Blog

I did not discover or invent this trick .

I wanted to make available a Python wheel to some developers but I did not want to publish it on PyPI for a variety of reasons.

  1. I am not the original author of the code and I did not want to take credit for it.
  2. I wanted to include the git commit hash in the version number which PyPI does not allow.

The trick is pretty simple but leverages two simple facts:

  1. For a URL to behave similar to PyPI for pip to install a package, it merely must provide an index.html file with links to the wheels. This is the premise of PEP 503 which defines the PyPI Simple Repository API.

     <!DOCTYPE html>
     <html>
     <body>
         <a href="/frob/">frob</a>
         <a href="/spamspamspam/">spamspamspam</a>
     </body>
     </html>
    
  2. GitHub Release Page has a view that includes all the links to all assets in the release. For instance for let’s consider the mlir-wheels repository that uses this trick. It has a single release with over 5,000 “assets”, where each asset is a wheel for a different version and particular platform.

mlir-wheels release page

pip itself cannot use this page unfortunately, because the hyperlinks are loaded via Javascript.

There is however an alternative page that is a basic HTML view of all the assets.

https://github.com/makslevental/mlir-wheels/releases/tag/latest -> https://github.com/makslevental/mlir-wheels/releases/expanded_assets/latest

mlir-wheels expanded assets page

With this page you can easily use pip to install and upgrade the packages.

pip install mlir-python-bindings \
    -f https://github.com/makslevental/mlir-wheels/releases/expanded_assets/latest

Happy hosting. 🎉