rattler-build: A new parser
We're working hard on one of the biggest overhauls the conda-world has seen. Conda packages are great -- they work on Windows, macOS and Linux. But it has always been finicky to build them. We want to make the experience of packaging your own (or someone else's) software into a conda package really smooth -- with rattler-build, a new Rust-based CLI tool to create conda-packages!
Generally, the way a conda package is built is by first writing a "recipe" that contains the instructions and metadata that makes up the package. The recipe contains such information as:
- What is the name and version of the package?
- Where should the source code be downloaded from, and what is the checksum of the trusted source code?
- A bash or cmd.exe script to turn the source code into the final (binary) package, executable or shared library
- Some trickery to make dynamic linking work nicely on macOS and Linux
Good parsing is hard
The initial version of rattler-build used serde & serde_yaml to parse the recipe. That worked OK, but was limited because we could not get great error messages out of serde_yaml, since it doesn't report the locations (line & column) of the encountered issues.
Eric Shimizu Karbstein did an excellent job at implementing a more custom YAML parser that supports all the features we wanted: Jinja evaluation and if selectors in the YAML while still providing excellent traceability of encountered errors. The cost is more (mechanical) code to create the parser, but since we do not expect that the recipe format will change much in the future we're very happy with the results. Additionally we are using the excellent miette crate for these beautiful error reports.
A strong schema is key
With the freedom that conda-build recipes gave users, it was not really possible to create a fixed schema, because users could use arbitrary jinja everywhere. Even comments had semantic meaning (as line selectors).
Rattler-build fixes this: only a subset of jinja is allowed and the recipe has to be valid YAML at all times! Thanks to these properties we can create a strict JSON schema which can be used to lint the recipe while writing (look at the red squiggly lines in the screenshot). And later we can lint recipes easily in CI, too. This is a great developer experience enhancement and early rattler-build adopters love it.
Just a single line at the start of the recipe is enough:
# yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json
context: ...Try rattler-build now
It's very easy to get going with rattler-build. You can install it from conda-forge with pixi:
pixi global install rattler-build
It is also already available from homebrew and Arch Linux – or you can build it locally with cargo. To get started, our revamped documentation provides a great introduction to the new recipe format.
rattler-build in Q1 2024
We've been vocal about our goal to get rattler-build into the hands of conda-forge -- by far the largest open source repository of conda packages out there. In fact, a contributor -- Bela Stoyan -- has already managed to upload the first package, built with rattler-build, to conda-forge. The package was produced in a conda-forge CI pipeline and even uploaded with rattler-build (that's right, rattler-build now has a built in upload command that handles prefix.dev, quetz, anaconda.org and Artifactory -- Bela was also instrumental in the PRs for the upload command).
The integration of rattler-build into conda-forge still needs some serious work, that we plan on starting soon:
- Conda-smithy has to understand the new recipe format properly in order to be able to manage the repositories & CI scripts automatically
- The autotick bot also needs to be aware of the new format to send commits with the proper changes
- And finally it would be awesome if grayskull or a Rust based recipe generator could create recipes for the new format
We are looking forward to work on all this in Q1 of 2024.
Acknowledgements
Some very talented people have (or are currently working) on rattler-build: Eric, Swarnim, Orhun and Matthias as well as Bela and Pavel. Thanks!