Building a C++ package with rattler-build

Written by Wolf Vollprecht a year ago

Conda packages are not only for Python! In fact, one of the strong points of conda packages (and why we are betting on them) is that they are completely language agnostic. This means that you can ship your software in a conda-package, whether it is written in Python, C++ or C, Rust, Zig, or any other language.

In this blog post, we will show you how to build a C++ package with conda-forge and the new rattler-build tool. rattler-build is a tool that reads a "recipe" file and then creates a package from the recipe. It is a modern replacement for the old conda-build tool, and it is much faster and easier to use.

What is a recipe?

A recipe is a YAML file that contains the instructions for the build. It usually contains the following sections:

package: This section contains the name and the version of the package
source: This section contains the URL to the source code and a checksum (SHA256 or MD5).
build: This section contains the instructions to build the package and some additional configuration options.
tests: This section contains the instructions to run the tests after the package has been built.
about: This section contains the metadata about the package, such as the description, the license, and the home page.

Let's build a package

There are some hot new C++ libraries out there. One that I have personally worked on is called "xtensor". It is a C++ library for numerical computing with multi-dimensional arrays which implements many of the ideas of NumPy in C++. Since it is a header-only library the build process is also pretty quick.

Here is a recipe for building the xtensor package:

# yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json

context:
  name: xtensor
  version: "0.24.6"

package:
  name: ${{ name|lower }}
  version: ${{ version }}

source:
  url: https://github.com/xtensor-stack/xtensor/archive/${{ version }}.tar.gz
  sha256: f87259b51aabafdd1183947747edfff4cff75d55375334f2e81cee6dc68ef655

build:
  # you can increment the build number to correct mistakes in the recipe
  number: 0
  script:
    - if: win
      then: |
        cmake -GNinja -D BUILD_TESTS=OFF -D CMAKE_INSTALL_PREFIX=%LIBRARY_PREFIX% %SRC_DIR%
        ninja install
      else: |
        cmake -GNinja -DBUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX=$PREFIX $SRC_DIR -DCMAKE_INSTALL_LIBDIR=lib
        ninja install

requirements:
  build:
    - ${{ compiler('cxx') }}
    - cmake
    - ninja
  host:
    - xtl >=0.7,<0.8
  run:
    - xtl >=0.7,<0.8
  run_constraints:
    - xsimd >=8.0.3,<10

tests:
  - package_contents:
      include:
        - xtensor/xarray.hpp
      files:
        - share/cmake/xtensor/xtensorConfig.cmake
        - share/cmake/xtensor/xtensorConfigVersion.cmake

about:
  homepage: https://github.com/xtensor-stack/xtensor
  license: BSD-3-Clause
  license_file: LICENSE
  summary: The C++ tensor algebra library
  description: Multi dimensional arrays with broadcasting and lazy computing
  documentation: https://xtensor.readthedocs.io
  repository: https://github.com/xtensor-stack/xtensor

Let's go through the recipe step by step:

The context section contains the name and the version of the package. These are used to fill in the variables in the rest of the recipe.
The package section contains the name and the version of the package. These will be used if other packages depend on this package (as input to the SAT solver) and if you use the package in a pixi.toml file.
The source section contains the URL to the source code and a checksum (SHA256 or MD5). This is used to download the source code and to verify that it has not been tampered with.
The build section contains the instructions to build the package and some additional configuration options. In this case, we use cmake to build the package and ninja to run the build. We also set some CMake variables to configure the build. The main trick is that we install the files into the $PREFIX (or %LIBRARY_PREFIX% on Windows) directory. Any files that are added during the build process to the $PREFIX directory will be included in the package.
The requirements section contains the list of dependencies that are needed to build the package. In this case, we need a C++ compiler, cmake, and ninja. We also need the xtl and xsimd libraries to run the package. We use a special Jinja function for the compiler requirement, which will automatically select the right compiler for the target_platform (this also works for cross-compilation).
The tests section contains the instructions to run the tests after the package has been built. In this case, we check that the package contains the right files by using a package_contents test. You can also write tests that run a script and execute any binaries that are included in the package.
The about section contains the metadata about the package, such as the description, the license, and the home page. This is used to display informative metadata in the repository.

Building the package

To build this package, you need rattler-build. The easiest way to get rattler-build on your system is of course with pixi:

pixi global install rattler-build

Will make the rattler-build command globally available on your system.

Once installed you can execute the build with the following command:

rattler-build build --recipe ./xtensor-recipe.yaml --output-dir ~/bld-folder

This will place the new package into the ~/bld-folder/osx-arm64/<package>.tar.bz2 folder. Note that the package will be built for your current platform, so if you are running Linux or Windows, the folder will be win-64 or linux-64 etc.

Using the package from a local source in pixi

To use the package from the local build folder in pixi you just need to point the pixi.toml to that folder:

[project]
channels = ["/home/user/bld-folder", "conda-forge"]
...

[dependencies]
xtensor = "0.24.6"

This will make pixi install the package from the local folder instead of the conda-forge repository (any run dependencies of xtensor will still come from conda-forge, though).

With pixi you will get all the same benefits of reproducibility and lock files to build your own C++ projects on top.

Automate your package building

To build your packages for multiple platforms you can use the new rattler-build-action Github Action (or just use rattler-build on your preferred CI provider).

jobs:
  build:
    name: Build package
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - name: Build conda package with `rattler-build`
      uses: prefix-dev/rattler-build-action@v0.1.1

After the packages are built, you can upload them to your own package server or a channel on prefix.dev (you can choose between private and public channels). To learn more about channels on prefix, read this blog post.

If you want to learn more about rattler-build and how to use it, you can check the rattler-build documentation. To get inspiration on how to write recipes, you can check the conda-forge documentation or the conda-forge feedstock repositories on GitHub. They currently use the old format (meta.yaml) that is used by conda-build but for most recipes it is pretty straightforward to convert them to the new format. We are in the process of adding support for rattler-build and the new recipe format to conda-forge, so be excited for the future!

Building a C++ package with rattler-build

What is a recipe?

Let's build a package

Building the package

Using the package from a local source in pixi

Automate your package building

Read more