rattler-build in conda-forge

For the past year, we have been working on a new tool called "rattler-build". We had one goal in mind: improve the experience of building conda packages – and bring those improvements to conda-forge! We are happy to report that we are closer than ever to achieving this goal 🚀

We took the following steps:

1. Designed a new and improved recipe format with the whole conda community. This is the v2 conda-recipe format;
2. Implemented a new build tool in Rust, that uses the new recipe format – that is rattler-build. Rattler-build builds on top of our Rust crates (rattler) that we also use in pixi;
3. Changed the infrastructure in conda-forge to understand not only the original v1 recipe format, but also the new v2 recipe format.

Let's dive into each of these steps in detail.

Improving upon "meta.yaml"

Believe it or not - this process has been over 3 years in the making. Wolf Vollprecht started out with an experimental build tool called boa that is written in Python and makes use of the Python libmamba API. It's where he already tested many of the ideas that are now fully realized in rattler-build:

• A pure YAML format with only a subset of Jinja allowed
• Better support for multi-output recipes
• Extremely fast evaluation speed

The original ideas reaching back to boa were developed more than 4 years ago in 2020! The motivation for the new recipe format has been unchanged, but some details have changed. Let's revisit the changes:

• A recipe is always valid YAML. That makes it much easier to parse and modify it with a YAML parser. It does mean, however, that arbitrary Jinja is not allowed. And also, no more comments with semantic meaning ("selectors").
• For selectors, a special "if/then/else" YAML construct is used instead
• Multiple outputs are defined in a more sane way
• You can have multiple tests per package, install emulators and more improvements

Being valid YAML gives rise to nice benefits - such as a working JSON schema (it's even in schemastore.org – just call your file "recipe.yaml" and it works). Below an example recipe for the boltons package is shown:

context:
  version: "23.0.0"
 
package:
  name: "boltons"
  version: ${{ version }}
 
source:
  url: https://github.com/mahmoud/boltons/archive/refs/tags/${{ version }}.tar.gz
  sha256: 9b2998cd9525ed472079c7dd90fbd216a887202e8729d5969d4f33878f0ff668
 
build:
  noarch: python
  script:
	- python -m pip install . --no-deps -vv
 
requirements:
  host:
	- python
	- pip
	- setuptools
  run:
	- pip
 
about:
  license: BSD-3-Clause
  license_file: LICENSE

Conda Enhancement Proposals

A crucial part of this development process was the use of Conda Enhancement Proposals (CEPs). Similar to Python's PEPs, CEPs allow the community to propose, discuss, and ratify changes to the Conda ecosystem. The new recipe format went through this process, ensuring that it was developed with input from the wider Conda community.

The CEP process involved:

1. Drafting a detailed proposal for the new format
2. Opening it up for community discussion and feedback
3. Iterating on the design based on community input
4. Finally, putting it to a vote for ratification

This community-driven approach has resulted in a format that not only meets the technical requirements but also addresses the needs and concerns of the Conda community at large.

Implementing the new format in rattler-build

First and foremost, we must acknowledge that we're standing on the shoulders of giants. Conda-build has served the community exceptionally well for many years, enabling the creation and distribution of countless packages that have been crucial to scientific computing, data science, and many other fields. Its contributions to the conda ecosystem cannot be overstated, and the work of its maintainers and contributors has been invaluable.

However, as with many long-lived software projects, conda-build has faced challenges as it evolved over time. We did the unthinkable – we rewrote conda-build from scratch. This decision wasn't taken lightly, and it came after careful consideration and attempts to build upon the existing foundation.

Previously, we tried to build on top of conda-build (that was boa). In boa, most of the tasks like packaging the files or indexing the local channel were left to the original conda-build which was imported as a Python library. However, the conda-build codebase has been decaying for quite some time now, and it was clear that it is not a stable foundation for a new tool. In our opinion, Conda-build exhibited a lot of problems that plague large older Python codebases - it has some functions with many parameters that are very hard to trace. A refactor of the project seems impossible due to the intricate ways that state is handled, and the organically grown nature of the project. Because of these issues, we couldn't get the boa developer experience to feel right.

For these reasons, we decided to start from scratch. In addition we had the perfect conditions to build a new, blazing fast and future-proof package build tool in Rust – we already had the foundations in rattler (low-level conda library) and they kept getting better!

We started the development of rattler-build more or less at the same time as prefix.dev was founded – because we realized the seriousness of the problem. What good is a package manager if we cannot build packages for it! We want the conda ecosystem to thrive for another ten years, and a good way to build packages needs to be a part of that.

Over the past year we have been incredibly fortunate to receive a contract from the Sovereign Tech Fund in order to develop rattler-build and integrate it into conda-forge.

Our aim is to make rattler-build both fast and extremely user-friendly. It should give you clear and actionable error messages, for example when parsing goes wrong. Just like the Rust compiler does!

And it should never spin for 5 minutes in an unknown state before starting the actual build process- which is often what we experienced with conda-build.

Performance Improvements

One of the most exciting aspects of rattler-build is its performance. We've seen significant improvements in build times:

• In several cases, rattler-build has reduced build times by 5-6 minutes for processes that previously took 8-10 minutes.
• This represents a 50-60% reduction in build time for some packages.

These improvements are particularly noticeable for packages with complex build processes or multiple outputs. The speed increase not only saves time for package maintainers but also reduces the load on CI systems, allowing for more efficient use of resources across the conda-forge ecosystem.

Implementing support for the new recipe on conda-forge

Of course our work doesn't stop there! We are big fans of conda-forge and want to make sure that as many package builders as possible benefit from the new v2 recipe format! For this reason, we spent the last 6 months working on the support of the new format in various tools of the conda-forge ecosystem:

• conda-smithy: the central tool that ingests a recipe file and creates the contents of the git repository (feedstock). Based on the recipe contents it creates CI pipelines for all platforms, and pinning files with the latest pins for supported packages, compilers, and so on. If you ever manually "re-rendered" a feedstock, or used the "conda-forge bot, please rerender command" – that was smithy! It now supports the new recipe format for all commands. We also implemented the same linting rules as they apply to existing recipes. The conda-forge bot will make sure that your recipe has a license, that you use the right order of keys, and so on and so forth - whether it's the new or the old format!

• cf-graph: software evolves, and cf-graph evolves conda-forge. This repository maintains a large database of metadata for every package in the conda-forge distribution and triggers automatic version updates, migrations to newer dependency versions, and more complicated recipe changes as needed! A very complex but also very central piece of technology in conda-forge, that now understands the new recipe format and inserts the correctly (adjusted) metadata into the graph. It should also trigger automatic version updates!

• staged-recipes: we want users to start out with the new recipe format! Therefore, we implemented support in the staged-recipes repository to build and test packages in the new recipe format – and once the PR is merged, it should also automatically create the feedstock for you. This was the first new style recipe ever merged into conda-forge.

We are working towards a gradual migration of conda-forge towards the new recipe format. It is completely opt-in: you are using conda-build if you have a "meta.yaml" file and rattler-build with the v2 conda recipe format if your feedstock contains a "recipe.yaml" file.

Try it out

Do you have an existing package on conda-forge? You can try rattler-build today. As a first step, you need to convert your meta.yaml. It's easiest to start with the "conda-recipe-manager" tool, created by Schuyler Martin (who works at Anaconda):

pixi global install conda-recipe-manager conda-smithy # if you don't have it already
conda-recipe-manager convert ./recipe/meta.yaml > ./recipe/recipe.yaml
 
# when things work, remove the meta.yaml file
rm ./recipe/meta.yaml
# rerender with conda-smithy
conda-smithy rerender
# and push a PR to your feedstock!

After this, your CI should be triggered as usual, but rattler-build will be used as a build tool. You should see improved logs, and much improved build times (at least for the recipe execution - unfortunately we cannot speed up the Python/C/C++ build process, hehe).

Benefits of Trying rattler-build

By giving rattler-build a try, you can expect:

1. Faster build times, especially for complex packages
2. Clearer error messages when something goes wrong
3. A more maintainable recipe format
4. The opportunity to provide feedback and shape the future of conda package building

Your experience and feedback are crucial for the continued improvement and adoption of rattler-build across the conda ecosystem.

If you have any questions, don't hesitate to reach out to us:

• Read the docs of rattler-build
• Chat with us on Discord
• Write an issue on Github

Looking to the Future

Rattler-build isn't just a new tool—it's a step towards future-proofing the conda ecosystem. We're committed to ensuring that conda remains a robust, efficient, and user-friendly solution for package management and building.

By adopting Rust for rattler-build, we're laying the groundwork for ongoing performance enhancements and maintainability. By involving the community through the Conda Enhancement Proposals (CEP) process, we're ensuring that the tools we develop meet the real needs of conda users and package maintainers.

We encourage you to try rattler-build, push it to its limits, and share your experience with us. Your feedback is crucial in refining the tool and shaping the future of conda package building. Let’s work together to build a faster, more reliable conda ecosystem.

Acknowledgments

Our progress with rattler-build wouldn't have been possible without the support and contributions from several key players:

• Sovereign Tech Fund: For the invaluable contract that funded the development of rattler-build, the v2 recipe format, and the enhancements to the conda-forge infrastructure.
• Nichita Morcotilo, Bas Zalmstra, Julian Hofer, Tim de Jager: For their tireless work on refining the conda-forge infrastructure and fixing critical bugs in rattler-build.
• Matthew Becker and Isuru Fernando: For their insightful guidance and patient reviews of our pull requests, ensuring that every change was both thoughtful and effective.
• The Conda Community: For your continuous input, feedback, and unwavering support throughout this process. Your involvement has been instrumental in the development and improvement of rattler-build.

Together, we're building a stronger, faster, and more efficient conda ecosystem. Let’s continue to push the boundaries of what's possible in package management and building!