Unleashing PyPI support in pixi

Written by Wolf Vollprecht 5 months ago

Many conda and mamba users are mixing conda and pip packages in their environments – because sometimes you just need a package that is not available on conda-forge. The tl;dr: instead of a pip section in your environment.yml file, you can now use the pypi-dependencies section in your pixi.toml file.

For the past 4 months we have spent significant effort on implementing a long-awaited feature: for the first time PyPI and conda packages are deeply integrated in the Pixi package manager! We remain extremely "bullish" on conda packages as a great solution for system packages and anything with compiled code in it. However, we also recognize that there are many Python packages that are only available on PyPI, and that many "real-life" environment.yml files contain a pip section. To solve this pain point we have added native support for PyPI/Python packages in Pixi.

This integration goes way beyond anything other tools like conda or mamba historically attempted. We resolve, install and lock PyPI packages directly in Pixi (no shell invocation to pip!). For this we built a native PyPI package resolver and installer in Rust and integrated it into Pixi where it lives side-by-side with our conda resolver and installer.

This has been a major effort:

We first had to make our SAT solver generic so that it can work with different package ecosystems. We now have implementations for resolvo that support Conda, PyPI and RPM (you see where we're going? 🙂)
Making resolvo generic was not enough -- the metadata in PyPI is inside the wheel files and it's expensive to download a wheel file only to retrieve metadata. Therefore we also needed to make resolvo lazy to fetch as few wheel files as possible (there is still some optimization potential for smart people!)
Lastly we had to implement installation & removal methods for wheel files into generic site-packages folders in the rip tool.

We made rip into a standalone crate that can be used in a variety of situations. For example, one could create Python bindings and try to integrate it into pip itself. We hope that rip provides a useful tool and we would love to see other use cases for it.

Pixi PyPI example

Getting started with PyPI dependencies

If there are packages that you would like to use, but that are not yet on conda-forge – then you can now add them to the pypi-dependencies section in your pixi.toml file.

For example, to use the pyboy GameBoy emulator, you can add the following to your pixi.toml file:

# conda dependencies
[dependencies]
python = "3.11.*"

# extra dependencies from PyPI
[pypi-dependencies]
pyboy = "~=1.6.9"

When you run pixi install pixi will resolve pyboy from PyPI and install it into the existing environment. Afterwards you can use pyboy as if it was a conda package in your pixi tasks or pixi run commands. We've also updated our documentation.

And then they are locked

Just like conda packages, pixi will create entries for PyPI packages in the lockfile. So you know exactly which wheels were installed at a given point in time. We also resolve OS-independently and try to create a solution for your Python dependencies on each supported OS (e.g. we solve for Windows on Linux, and so on). Having a lockfile means that your coworker will get exactly the same set of packages as you. And it means that in the future, when you want to reproduce your results, you can rely on pixi to get the exact same set of packages.

We are not done yet

We are releasing the PyPI integration into pixi as a beta feature that is only enabled when you add a pypi-dependencies section in your pixi.toml file. You will see a little warning about that - but worry not, we are working hard to make this feature stable.

A current limitation is that pure source distribution (SDist) packages cannot be resolved or installed, as we rely on the metadata information contained in wheel files. However, support for SDists is already merged upstream in rip and should come to pixi soon – after we find answers to some open questions regarding the lockfile. For example: should the SDist build environment be part of the lockfile? Feel free to chime in)!

Recently we've picked up work to support resolving RPM packages (used by RedHat, CentOS, Fedora, etc.) – you can follow that development on the resolvo-rpm repository.

We also love to discuss these things on our Discord server, so feel free to join us there!