Let's stop dependency hell

Managing dependencies, creating reproducible development environments, trying to get software to run on multiple machines and operating systems is hard. At prefix.dev, we come from software development backgrounds in different disciplines: Robotics, Data Science and Gaming.

And we've all experienced issues with reproducibility, and dependency management:

• Not being able to install the exact same versions of software after a number of years, making it hard or impossible to re-create machines.

• Environments on CI systems that are completely different from your local machine, causing issues that are hard to debug.

• Dependencies being removed from package repositories, causing builds to fail.

• Semantic versioning not being enforced correctly, causing breaks in the software you are building.

• System libraries differing per platform, causing software failing to build on some machines but not others.

When we started prefix.dev we wanted to tackle these issues, frankly because we are tired of them. And who knows? Maybe so are you.

Pixi: dependency management should feel magical

Managing dependencies and development/virtual environments should be easy. We believe that you shouldn't need to think about it at all. It gets in the way of what you really want to do: develop great software.

That's why we have created pixi — a new package manager for the conda ecosystem, written in Rust, aiming to tackle these issues. It’s based on our open-source conda library rattler. Pixi is a package manager that brings the convenience of modern package managers like Cargo, Poetry or Yarn to the conda ecosystem, and provides the ability to set-up reproducible isolated development environments. We consider it an essential step in order to fix big problems in the package management space.

Pixi builds on top of the conda ecosystem and is a binary package manager, that is language-agnostic, multi-platform (including first-class Windows support). With a specific focus on user-friendliness and being fast.

Why should I care?

Conda has been around for a long time. Modern package managers have innovated with new workflows – but conda, the package manager, has essentially not improved much. We really love working with Cargo for example. We think it can provide for stellar workflows – but it only works for Rust. We wanted to create something that feels like Cargo but that can target all programming languages and platforms.

When working with conda in the past, we’ve noticed that we were missing conveniences and features that we’ve come to expect from more modern package managers. Pixi and conda, like nix are language-agnostic. Pixi is simple to use, with a familiar CLI, similar to Cargo, Poetry and others. To setup a project from scratch:

In the example above, a project called test is initialized, and a virtual environment is created. Cowpy and its dependencies, including Python, are installed into this environment.

A notable workflow change with regards to conda is that there is no explicit environment activation and global environments that live somewhere on the system – it’s all local to the project. A pixi.toml project file is similar in spirit to a Cargo.toml or pyproject.toml file.

Other features that pixi provides out of the box are:

• Automatic cross-platform lock-file generation for reproducibility.

• A cross-platform task system for running project-specific tasks.

• Globally install tools like pipx or condax (using pixi global install …).

• Support for multiple languages in a single project. Combine Python with C++, Rust, Fortran, R, etc.

• Conda supports system-level libraries. Getting the right libssl version for your Rust project can be quite a hassle. Within the conda-ecosystem and pixi it's just another dependency.

Pixi is a single binary that is straightforward to install (no need for a base environment or Python). Just like micromamba you can install it with a “curl” one-liner:

# On UNIX:
curl -fsSL https://pixi.sh/install.sh | bash
# On Windows:
iwr -useb https://pixi.sh/install.ps1 | iex

The source code (BSD-3 License) is on Github.

Pixi is made for collaboration

Configuration is scoped to pixi projects (channels, platforms, and so on). This isolation makes it much easier to share projects. Once you’ve installed pixi, starting with a project is as simple as git-cloning a repo that contains a pixi.toml and you are ready to run a task (build, lint, test, ...):

git clone my-repo
pixi run some-task

When running these two simple commands the following happens:

• Platform specific dependencies are read from the checked-in lock-file (Python, compilers, whatever the project needs)

• Dependencies are downloaded and installed into a project specific virtual environment.

• The environment is activated and the cross-platform command is run (this uses Deno task shell)

Our goal is to make it really easy to share and collaborate on projects. All you should ever need is a git clone and a pixi run start.

Standing on the shoulders of giants

We believe that the conda packaging ecosystem and the excellent conda-forge

channel offer a number of advantages when compared to other packaging distributions. We argue that it is a great ecosystem to build innovative package management solutions on top of.

• A focus on reproducibility (packages are never deleted).

• Support for multiple programming languages.

• Lastly, a strong community aided by an automated CI system and some wonderful bots

• (e.g. automatic bumping and rebuilding of packages).

Conda-based package managers and its ecosystem is already widely used in the scientific, data-science, robotics and fintech communities. That said, we don't want to shut out other ecosystems, we are looking at ways to make it easier to use conda packages in other ecosystems and vice versa. The next thing we are looking into is a tight integration with the PyPI ecosystem.

What's next?

We have ambitious plans for pixi:

1. Building conda packages directly from pixi with a pixi build command.

2. Add source & git dependencies to a pixi.toml.

3. Integration with the PyPI packaging ecosystem – by extending our own solver implementation for PyPI packages.

4. Using pyproject.toml as an alternative to pixi.toml.

5. Providing CI workflows with pixi.

6. Creating OCI images from pixi environments.

7. … and much more, please look at our GitHub issues.

Getting this project into your hands for you to try out was the most important step. The GitHub readme and our documentation should provide you with the instructions to get started. We would love to know what you think about pixi and how we can make it work for you!

We also have a Discord server where we hang out for chats with pixi, mamba and conda enthusiasts! Please drop by for any questions or ideas you might have.