Cover image for Prefix.dev end of year blog

Prefix.dev end of year blog

Ruben Arts
Written by Ruben Arts a year ago
2023 was the first whole year for prefix.dev, the company! It's time to reflect about what we've been up to. Let me unpack (pun intended) the tools and packages we've been building over the course of last year and show you who we are!

The Team

Meet our core team that work full time at prefix.dev of 4 people, Wolf our CEO is joined by Bas, Tim and Ruben. We also have an awesome developer that started on our team from outside of Europe who started as an open source contributor to pixi, moved in as an intern and is now a developer on our team, Tarun. And we've enjoyed, or are currently enjoying the work of multiple freelancers, Adolfo, Eric, Swarnim and Matthias.

Projects

We've taken on the big task of re-architecting how people work with conda packages - from solving, building and installing them. Almost all of our tools are - of course - open source and written in Rust. The center piece is the rattler library that we are using in all of our projects.
prefix.dev package layout

The rattler library

Rattler is a all-in-one library of functionalities to interact with with the conda ecosystem. Which itself consists of tightly organized “crates” which provide sub-functionalities. You can see the different crates in the image below.
rattler crate layout
At prefix.dev the project got full time attention from Bas and has since grown into a very capable library and a corner stone of the company. It is used in all of our projects. Rattler is proudly built in the Rust programming language. Rust provides us with an excellent development experience, is cross-platform by design and allows us to build really performant tools - it's a great choice for CLI tools!

The Web Server for prefix.dev

We are building a high-performance conda package index. The prefix web server implements a high speed search engine, and indexes all of conda-forge, bioconda and robostack and makes them searchable (much faster than anaconda.org!). We also implemented ways of making your own channels on prefix.dev - public or private. In the second half of 2023 we focused on our open source tools, but we are looking forward to innovate more on our index/platform much more in 2024!

The pixi Package Manager

Our main project right now is pixi and the team is laser focused on it. Pixi is a cross-platform, multi-language package manager and workflow tool built on the foundation of the conda ecosystem. It provides developers with an experience similar to popular package managers like cargo or yarn, but for any language and “system-level” packages. We believe that pixi will solve the biggest hurdles in the package management world for users that just want to develop projects and want to skip the yakshaving that is called package management.
Pixi started as a passion project at prefix.dev - we believe strongly that reproducibility, speed and ease of use should not be mutually exclusive.
We are very happy with the growing interest, measured not only in stars on Github but also in the contributions and questions we are getting from users. Keep them coming!
Star History Chart

rip: Rattler Installs Packages

While we were building pixi we got more and more users that also want pixi to support the PyPI ecosystem. While the available tools like conda, mamba and micromamba support the use of pip in their environments they didn't really work well in unison. At prefix, we don't like half-baked solutions. That is why we built a library called rip which stands for rattler installs packages (and yes, this is a joke on pip, where it stands for pip installs packages). Currently, this library is under development and is going to help us use PyPI packages in our pixipackage manager. rip is a low-level library like rattler and not really a replacement for pip. On that note let's continue with resolvo as we needed it to build rip

resolvo, the Package-ecosystem Agnostic Resolver.

A package manager that deals with Conda or PyPI packages needs a SAT solver to figure out what dependencies are compatible and needed to install for your project. This challenge has been tackled many times in the tech world, but we had some unique needs. We wanted our solver to be quick, compatible with the conda ecosystem's specifications, and flexible enough to work with more than just conda packages. Our solution began with transforming libsolv, an existing and reliable technology used in other package managers like mamba and rpm. libsolv is written in very low-level C which makes it hard to read and reason about. To make it more accessible and maintainable, one of our talented freelancers, Adolfo, ported the parts we needed to Rust (yes the name is a tribute to him).
Afterwards, we integrated the revamped resolver into rattler and rip, thus being the only resolver in pixi. We added some clever techniques like 'incremental/lazy solvin'. This is necessary because PyPI doesn't have a complete index of all the information needed for the solver. rip has to smartly fetch a package, figure out what its dependencies are, and then recursively keep fetching those dependencies until it has everything it needs to proceed with the installation.

rattler-build: "Can we fix it? Yes we can!"

Having all these awesome tools to handle packages but no tool to build them would be silly, don't you think. We're also solving the build problem with our rattler-build tool. It's being designed to be used as a standalone binary, e.g. pixi global install rattler-build gives you the tool ready to use. But we also plan to use it in pixi as a library for building pixi projects into conda packages.
rattler-build will stay close to the workflow of conda-build and boa as it uses a recipe to create a package. But we are working on a new recipe format, checkout the CEPs we made for improving the ecosystem and plan to implement in rattler-build: conda-incubator/ceps/57, conda-incubator/ceps/56. The new format will be pure yaml and learn some more lessons from other templating build tools, e.g. GitHub Actions.

What is Coming in 2024?

Of course we are going to focus on improving pixi and all its related packages. But we also want to get into a new stage of our company.

Big Features

For pixi we have some big improvements in mind.
  • Full-blown PyPI support: the feature is already partly there but it doesn't support all packages yet. Specifically sdists aren't supported yet!
  • Multiple environments in one project: you probably want to define more than one environment, for example to define test, lint and build-time dependencies. So you can minimize the time needed to install on different systems.
  • Building a conda package directly from a pixi.toml
  • Workspaces: when using a monorepo you might want to split up the package into sub packages which contain their own manifest file. Which can work together in the main project.
For rattler-build
  • We want rattler-build to become the default tool in conda-forge
  • Building packages will become much easier and hopefully more fun!
For our website and conda index we have some great ideas in the planning.
  • Show descriptions for multiple version pages of a package
  • Support for visualization of the run-exports
  • Reverse search for files in all packages (e.g. which package contains blas.h?)
  • Implement a sparse repodata format so that not the whole repodata needs to be fetched

Developing our Commercial Products

We are working towards our first commercial products in 2024. If you want to chat with us about our ideas, don't hesitate to reach out: hi@prefix.dev. If you want to be a design partner for our products, we are looking for you!
One of our first projects is very likely going to be a "prefix-on-prem" solution.

Follow us

To make sure your not missing out on any of our developments follow us on any or all these platforms: