Introducing Pixi's Multiple Environments
Pixi is our cutting-edge package manager for proper cross-platform, binary package management on Linux, macOS and Windows. We're proud to tell you that we have launched the multiple environment feature in pixi. This feature was designed together with multiple contributors and kickstarted by our community.
Developing software usually means running a couple different tasks. You can express these tasks beautifully with pixi: compiling the software, building the docs or running the tests – the pixi task system has got you covered.
However, these tasks might have different requirements, and some users might simply not be interested in building the documentation locally. For this reason, we are excited to ship the new multi-env feature in pixi where tasks can have their own environments. And you can build specialized environments too, for example to run with or without CUDA!
You've built what now?
Environments are pixi's bread and butter.
Before release v0.13.0
we only supported one environment per project.
An environment is a complete installation of all dependencies into a folder (.pixi
) in your project folder.
Pixi links all the required files into the correct location and – badabing badaboom – you have a pixi environment (fun fact: the links are either hard- or reflinks so that an environment doesn't take up any extra space on your disc).
These environments are platform specific and different for linux-64
vs. osx-arm64
.
In theory, you could also call them different environments as they might consist of different packages being installed.
Which means that pixi was already solving multiple environments every time you created a project that supported multiple platforms.
But these were always designed to run on a single platform and look as similar to each other as possible.
With the latest release we allow you to create more than one environment for your project. This works by introducing "features": you can create sets of packages for a feature such as "test", "dev" or any other name you come up with.
An environment can then consist of one or more of these features. This gives the pixi user the real magic-wand to tailor their environments to their specific needs.
A feature can consist of the full description of an environment, including dependencies, platform, channels and even system requirements. More details can be found in the documentation.
Combining these features
allows the user to mix and match environments
.
Let's start with a simple example, adding a test
environment.
This test
environment is only needed for a developer or CI but if you are simply running the environment you don't need to install the test
environment. You can keep using the default environment with no changes!
This is a simple example, but it shows the setup of an environment in the simplest form. You can create environments for different stages of your development process this way. Environments can consist of unlimited features.
Real dependencies stay together 💪
I hear you thinking, but now you might select different versions of python
and numpy
in the test
environment compared to the default
environment.
And you would be right, but pixi has a solution for that as well.
You can define a solve-group
.
This is a group of dependencies that are solved together.
By having the same solve-group
in the default
environment and the test
environment, the exact same dependency versions will be in both environments.
Now the dependencies are all the same only that the test
environment will install more of the total set compared to the default
environment.
This is really powerful where a tester needs extra dependencies but wants to test the exact dependencies that are going to be deployed.
Okay that was easy right, it kinda feels like optional dependencies or features or something that you have heard of before in other tools.
But we allow you to define as much of these features and environments that you want.
Testing 20 different versions of python
? Simply make the environments and let pixi do the heavy lifting.
Note that the key here is that every single environment will be locked in the pixi.lock
file.
This means that you can always go back to the same environment that you had before. On any of the platforms you have defined. Pixi might be the first tool that allows both workflows of solving environments together or separated and then locking them down in a lockfile.
To CUDA or not to CUDA: Mlx in the mix
Let's take a look at a more complex example, one of the more famous issues in the data science world.
Features are not limited to dependencies, you can define complete environments which include system requirements.
Let's say you have a machine learning project that requires CUDA or cpu specific environments based on the machine you are running on.
You can define a pixi.toml
that has two different environments (the default
and cuda
).
The CPU environment will be the default environment and the cuda
will be added on top of the default environment when needed.
This example can then be used to run code in different environments.
If the cuda
environment and the default
environment are both available on the platform you are running on, then pixi
will spawn a selector to let you choose the environment to run the task in.
➜ pixi run train # Spawns the selector for which environment to run in.
? The task 'train' can be run in multiple environments.
Please select an environment to run the task in: ›
❯ default
cuda
pixi run --environment cuda train # Runs the train task directly in the cuda environment
Now let's get more nerdy 🤓
As if it wasn't already nerdy enough, we have a few more tricks up our sleeve.
We've made all the environment solving completely parallel (thanks to resolvo
, our SAT solver).
Parallelism ⚡
Speed definitely sets pixi apart from other tools around. It looks pretty cool if you have 10 environments that support 4 platforms each:
The new pixi.lock
format 🔒
Each dependency corresponds to an entry in the lockfile that specifies exactly the artifact and metadata of the package.
With this information we can statically verify (without an internet connection) that the lockfile satisfies all requirements described in the pixi.toml
file and whether we need to re-solve.
When the requirements of a project change, pixi intelligently updates just the parts of the lockfile that are out-of-date.
It also means that if you have 100 dependencies in one environment shared over 4 platforms and in 10 environments, you will have 4000 entries in the lockfile. Every entry requires a few lines of metadata, so it grows quickly. We have modified the lockfile to deduplicate entries to make the file as small as possible but still human-readable.
This resulted in a file that looks like this pseudo example:
This will still be several thousand lines long for real examples.
But it is a greatly reduced size compared to the previous lockfile style mapped to the multiple environments feature.
With this change we completely let go of the conda-lock
style lockfile unfortunately. We are planning to eventually work on a CEP (Conda Enhancement Proposal) to standardize this new lockfile format.
Let's connect
Do you want a more detailed explanation of the speed improvements and cool graph tricks we built?
Chat with us on Discord, Twitter or Mastodon and we might write a blog about it!