Prefix.dev end of year blog
2023 was the first whole year for prefix.dev, the company! It's time to reflect about what we've been up to. Let me unpack (pun intended) the tools and packages we've been building over the course of last year and show you who we are!
Meet our core team that work full time at prefix.dev of 4 people, Wolf our CEO is joined by Bas, Tim and Ruben.
We also have an awesome developer that started on our team from outside of Europe who started as an open source contributor to
pixi, moved in as an intern and is now a developer on our team, Tarun.
And we've enjoyed, or are currently enjoying the work of multiple freelancers, Adolfo, Eric, Swarnim and Matthias.
We've taken on the big task of re-architecting how people work with conda packages - from solving, building and installing them. Almost all of our tools are - of course - open source and written in Rust. The center piece is the rattler library that we are using in all of our projects.
Rattler is a all-in-one library of functionalities to interact with with the conda ecosystem. Which itself consists of tightly organized “crates” which provide sub-functionalities. You can see the different crates in the image below.
At prefix.dev the project got full time attention from Bas and has since grown into a very capable library and a corner stone of the company. It is used in all of our projects. Rattler is proudly built in the Rust programming language. Rust provides us with an excellent development experience, is cross-platform by design and allows us to build really performant tools - it's a great choice for CLI tools!
We are building a high-performance conda package index. The prefix web server implements a high speed search engine, and indexes all of conda-forge, bioconda and robostack and makes them searchable (much faster than anaconda.org!). We also implemented ways of making your own channels on prefix.dev - public or private. In the second half of 2023 we focused on our open source tools, but we are looking forward to innovate more on our index/platform much more in 2024!
Our main project right now is
pixi and the team is laser focused on it.
Pixi is a cross-platform, multi-language package manager and workflow tool built on the foundation of the conda ecosystem.
It provides developers with an experience similar to popular package managers like
yarn, but for any language and “system-level” packages.
We believe that pixi will solve the biggest hurdles in the package management world for users that just want to develop projects and want to skip the yakshaving that is called package management.
Pixi started as a passion project at prefix.dev - we believe strongly that reproducibility, speed and ease of use should not be mutually exclusive.
We are very happy with the growing interest, measured not only in stars on Github but also in the contributions and questions we are getting from users. Keep them coming!
rip: Rattler Installs Packages
While we were building
pixi we got more and more users that also want
pixi to support the PyPI ecosystem.
While the available tools like
micromamba support the use of
pip in their environments they didn't really work well in unison.
At prefix, we don't like half-baked solutions.
That is why we built a library called
rip which stands for rattler installs packages (and yes, this is a joke on
pip, where it stands for pip installs packages).
Currently, this library is under development and is going to help us use PyPI packages in our
rip is a low-level library like
rattler and not really a replacement for pip.
On that note let's continue with
resolvo as we needed it to build
resolvo, the Package-ecosystem Agnostic Resolver.
A package manager that deals with Conda or PyPI packages needs a SAT solver to figure out what dependencies are compatible and needed to install for your project.
This challenge has been tackled many times in the tech world, but we had some unique needs.
We wanted our solver to be quick, compatible with the conda ecosystem's specifications, and flexible enough to work with more than just conda packages.
Our solution began with transforming
libsolv, an existing and reliable technology used in other package managers like
libsolv is written in very low-level C which makes it hard to read and reason about.
To make it more accessible and maintainable, one of our talented freelancers, Adolfo, ported the parts we needed to Rust (yes the name is a tribute to him).
Afterwards, we integrated the revamped resolver into
rip, thus being the only resolver in
We added some clever techniques like 'incremental/lazy solvin'.
This is necessary because PyPI doesn't have a complete index of all the information needed for the solver.
rip has to smartly fetch a package, figure out what its dependencies are, and then recursively keep fetching those dependencies until it has everything it needs to proceed with the installation.
rattler-build: "Can we fix it? Yes we can!"
Having all these awesome tools to handle packages but no tool to build them would be silly, don't you think.
We're also solving the build problem with our
It's being designed to be used as a standalone binary, e.g.
pixi global install rattler-build gives you the tool ready to use.
But we also plan to use it in
pixi as a library for building pixi projects into conda packages.
rattler-build will stay close to the workflow of
boa as it uses a recipe to create a package.
But we are working on a new recipe format, checkout the CEPs we made for improving the ecosystem and plan to implement in
rattler-build: conda-incubator/ceps/57, conda-incubator/ceps/56.
The new format will be pure
yaml and learn some more lessons from other templating build tools, e.g. GitHub Actions.
Of course we are going to focus on improving
pixi and all its related packages.
But we also want to get into a new stage of our company.
pixi we have some big improvements in mind.
- Full-blown PyPI support: the feature is already partly there but it doesn't support all packages yet. Specifically
sdists aren't supported yet!
- Multiple environments in one project: you probably want to define more than one environment, for example to define test, lint and build-time dependencies. So you can minimize the time needed to install on different systems.
- Building a conda package directly from a
- Workspaces: when using a monorepo you might want to split up the package into sub packages which contain their own manifest file. Which can work together in the main project.
- We want rattler-build to become the default tool in
- Building packages will become much easier and hopefully more fun!
For our website and conda index we have some great ideas in the planning.
- Show descriptions for multiple version pages of a package
- Support for visualization of the run-exports
- Reverse search for files in all packages (e.g. which package contains
- Implement a sparse repodata format so that not the whole repodata needs to be fetched
We are working towards our first commercial products in 2024. If you want to chat with us about our ideas, don't hesitate to reach out: firstname.lastname@example.org. If you want to be a design partner for our products, we are looking for you!
One of our first projects is very likely going to be a "prefix-on-prem" solution.
To make sure your not missing out on any of our developments follow us on any or all these platforms: