The Rattler Book: Build your own package manager in Rust

Ever wanted to write your own package manager but don't know what to use or how to get started? We wrote a small book that should help with that and is available at: Rattler Book. And the corresponding Github Repo is also open-source, which contains both the rendered source code and literate programming markdown. The result is a readable web-first book based on heavily modified MkDocs theme. This is a small preview:

So why?

A while after we started building our core library, rattler, which powers pixi, rattler-build and prefix.dev, we became pretty bullish on the idea that it should be possible to use rattler to build your own package manager for a programming language, whether one you made yourself or any other! In order to facilitate a workshop with Anaconda, we decided to write a book where we walk through the process of writing such a custom package manager. And we believe it might be interesting for other people as well. rattler is already widely used today, even in Mise, but of course we'd love to see it adopted in other tools as well. We hope this book is a great read and helps with that.

Conda as a Foundation

While the conda ecosystem is sometimes incorrectly identified with python-only tools, the underlying format is very neat: binary packages, platform-specific variants, one-version-per-environment semantics, and a huge existing package set on conda-forge. For example, a lot of programming languages integrate with C libraries, and conda-forge provides a lot of these already!

rattler the library, then extracts the useful parts into composable Rust crates: an extremely fast solver, a parallelized installer, a networking stack, shell activation hooks, and more! Together they make a good combination to build a package manager on top of.

About the Book

The book walks through building "Moonshot", a simple but working package manager for the Lua language that resolves dependencies with a SAT solver, installs binary packages, and activates shell environments.

These are some of the sample commands you can run:

What you see in the code snippet above is lua from conda-forge. imagemagick is a C library with loads of other native dependencies. Adding lua explicitly might seem a bit strange as a lua native package manager, don't worry, there is an exercise in the book to change that! Yes the book has exercises too :)

Lumen as seen above, is a Lua library we built with moonshot using the following command:

What is covered?

A first part has eight chapters that implements one command end-to-end:

1. init: scaffold a project manifest

2. search: query the repodata index

3. add: modify the manifest

4. lock: resolve dependencies via the resolvo SAT solver

5. install: fetch and unpack packages into a prefix

6. shell: generate activation scripts

7. run: execute commands inside the environment

8. build: produce a distributable .conda package

We also have a bunch of exercises per chapter that should help you learn more about each command and potentially more about the conda ecosystem itself

A second part covers the internals, and serves more of a reference:

1. the conda archive format,

2. virtual packages,

3. how the SAT solver works,

4. and the networking stack that handles parallel downloads and CDN mirroring.

5. ... and more

Literate programming

Literate programming a concept introduced by Knuth, and used by excellent books such as Physically Based Rendering. This means that the entire source code, including imports comments etc. Needs to be present in the book.
This means the entire program can and should be generated from the book. We wanted to try this out and its been quite excellent and useful, because it forces the book to always be up-to-date. Entangled the project that we used even provides a way to "stitch" source code back into the markdown. Usually, with literate programming, its one-way.

Some cool extensions we added is the ability to navigate between code blocks, and even view the whole of the rendered source code inside the book:

Conclusions

If we piqued your interest in any way, why don't you start with reading the introduction. If you are specifically interested in adopting it for your language of choice, we have a chapter on that as well. And as always feel free to log on onto discord to chat some more, we always like talking about package managers!

Acknowledgements

The book was made possible by Anaconda! Thank you for generously funding this project.