rattler-build: A new parser
Written by Wolf Vollprecht a year ago
We're working hard on one of the biggest overhauls the conda-world has seen.
Conda packages are great -- they work on Windows, macOS and Linux. But it has
always been finicky to build them. We want to make the experience of packaging
your own (or someone else's) software into a conda package really smooth -- with
rattler-build, a new Rust-based
CLI tool to create conda-packages!
Generally, the way a conda package is built is by first writing a "recipe" that
contains the instructions and metadata that makes up the package. The recipe
contains such information as:
- What is the name and version of the package?
- Where should the source code be downloaded from, and what is the checksum of the trusted source code?
- A bash or cmd.exe script to turn the source code into the final (binary) package, executable or shared library
- Some trickery to make dynamic linking work nicely on macOS and Linux
Good parsing is hard
The initial version of rattler-build used serde & serde_yaml to parse the
recipe. That worked OK, but was limited because we could not get great error
messages out of serde_yaml, since it doesn't report the locations (line &
column) of the encountered issues.
Eric Shimizu Karbstein did an excellent job at implementing a more custom YAML
parser that supports all the features we wanted: Jinja evaluation and if
selectors in the YAML while still providing excellent traceability of
encountered errors. The cost is more (mechanical) code to create the parser, but
since we do not expect that the recipe format will change much in the future
we're very happy with the results. Additionally we are using the excellent
miette crate for these beautiful error reports.
A strong schema is key
With the freedom that conda-build recipes gave users, it was not really possible
to create a fixed schema, because users could use arbitrary jinja everywhere.
Even comments had semantic meaning (as line selectors).
Rattler-build fixes this: only a subset of jinja is allowed and the recipe has
to be valid YAML at all times! Thanks to these properties we can create a strict
JSON schema which can be used to lint the recipe while writing (look at the red
squiggly lines in the screenshot). And later we can lint recipes easily in CI,
too. This is a great developer experience enhancement and early rattler-build
adopters love it.
Just a single line at the start of the recipe is enough:
# yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json
context: ...Try rattler-build now
It's very easy to get going with rattler-build. You can install it from
conda-forge with pixi:
pixi global install rattler-buildIt is also already available from homebrew and Arch Linux – or you can build it
locally with cargo. To get started, our revamped
documentation provides a great
introduction to the new recipe format.
rattler-build in Q1 2024
We've been vocal about our goal to get rattler-build into the hands of
conda-forge -- by far the largest open source repository of conda packages out
there. In fact, a contributor -- Bela Stoyan -- has already managed to upload
the first package, built with rattler-build, to conda-forge. The package was
produced in a conda-forge CI pipeline and even uploaded with rattler-build
(that's right, rattler-build now has a built in upload command that handles
prefix.dev, quetz, anaconda.org and Artifactory -- Bela was also instrumental in
the PRs for the upload command).
The integration of rattler-build into conda-forge still needs some serious work,
that we plan on starting soon:
- Conda-smithy has to understand the new recipe format properly in order to be able to manage the repositories & CI scripts automatically
- The autotick bot also needs to be aware of the new format to send commits with the proper changes
- And finally it would be awesome if grayskull or a Rust based recipe generator could create recipes for the new format
We are looking forward to work on all this in Q1 of 2024.