Cover image for Building From Source Shouldn't Be This Hard
Written on

Building From Source Shouldn't Be This Hard

Ruben Arts
Ruben Arts

You need to build something from source. Maybe it's a C++ library with custom compile flags, a Python package with some C extensions, or you need to use a patched version of an upstream dependency. You start by installing the right compilers, some new system libraries, setting environment variables, and hoping nothing breaks your existing setup. This often is not just a single iteration and will take a few tries to get right. And even when you do, it's not portable, how do you share this custom build with your colleagues or CI?

Because you build with a very specific configuration, you're going to try to use a Docker image for it. Now your quick local build has turned into a full-blown container, and the only way to share it is to let everyone use this image. This whole experience was cool when it worked but not fun, you rather avoid doing this again.

What you really want is a easy to create, portable, built artifact you can hand to anyone and they can install it without rebuilding. That's exactly what conda packages are built for, self-contained, platform-specific binaries that control their dependencies. But creating them means learning rattler-build, writing recipe.yaml files, and maintaining yet another layer of configuration separate from your project. While powerful, the overhead of learning a new tool and maintaining separate build files is a barrier to building from source.

That is why we're building pixi-build. It's a build system built on top of rattler-build, designed to make building from source easy.

Pixi is already a package manager, but it now also has the ability to build from source. If you're not familiar with Pixi, check out the documentation.

Making source building as easy as installing a package

With pixi-build, your pixi.toml is all you need. It defines your package metadata, dependencies, and how to build. No need for separate recipe files. Pixi sets up the entire build environment in isolation: compilers, toolchains, libraries. It builds the package with standardized scripts and produces a .conda artifact you can share or upload to a channel.

We've taken inspiration from the Python ecosystem. For example, we use build backends that can understand different types of projects, allowing the build configuration knowledge to live in a shared backend instead of long READMEs or custom scripts. This means you can build a Python package with pixi-build-python and a C++ project with pixi-build-cmake without needing to know the details of how to set up compilers, environment variables, or build commands.

Here's a Python package using a standard pyproject.toml, the name and version come from your [project] table, and pixi-build configuration lives under [tool.pixi]:

pyproject.toml:

[project]
name = "my-python-lib"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = ["rich>=13"]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.pixi.workspace]
channels = ["conda-forge"]
preview = ["pixi-build"]

[tool.pixi.package.build]
backend = { name= "pixi-build-python", version = "0.4.*" }
config = { ignore-pypi-mapping = false }

And here's a CMake/C++ project:

pixi.toml:

[workspace]
channels = ["conda-forge"]
preview = ["pixi-build"]

[package]
name = "cpp-example"
version = "0.1.0"

[package.build.backend]
name = "pixi-build-cmake"
version = "0.3.*"

[package.build.config]
extra-args = ["-DCMAKE_BUILD_TYPE=Release"]

[package.host-dependencies]
sdl2 = ">=2.26"

Run pixi build and you get a .conda package. That's it. Pixi pulled in the right compilers for your platform, resolved all dependencies from conda-forge, built everything in a clean environment, and produced an artifact you can share or upload to a channel.

Build From Any Source

One of the most powerful features is remote source builds. You can point the package source at a git repository or URL, and you can do the configuration locally. This makes it really easy to update the build source when new versions come out, or to build from a specific branch or commit.

pixi.toml:

[package.build.source]
git = "https://github.com/astral-sh/ruff.git"
branch = "main"

[package.build]
backend = { name= "pixi-build-python", version = "0.4.*" }
config = { ignore-pypi-mapping = false }

[workspace]
channels = ["conda-forge"]
preview = ["pixi-build"]

This works with git, url, and path dependencies, making it flexible to build packages that you don't have control over.

ruff is a Rust project with a Python package wrapper. Because it uses maturin as a build backend, pixi-build-python automatically picks up that it needs a Rust compiler and sets up the build environment accordingly. This is the power of build backends, they can understand the project structure and configure the build without you needing to know the details.

More Than Just Python and C++

pixi-build supports a growing list of backends:

And the backend system is extensible via JSON-RPC, you can build your own backend for any language or build system.

Already Used in the Wild

pixi-build is still in preview, but it's already being used by some major projects:

  • CPython uses pixi-build-rattler-build to build Python itself in multiple variants: default, free-threading, and sanitizer builds. All from a shared recipe template.

  • NumPy builds with pixi-build-python and Meson, supporting the same multi-variant approach with debug builds and explicit compiler configuration.

  • NVIDIA cuda-python uses pixi features and solve-groups to manage CUDA 12/13 variants in a single workspace.

  • xarray uses pixi-build-python to build their source against different python versions for reproducible testing setups.

These aren't toy examples, they're real build setups for foundational open-source projects where they've been able to simplify their build configuration, reduce maintenance overhead, and provide an easy way to manage source builds for their users and contributors.

Try It Today

  • Install pixi if you haven't already:

curl -fsSL https://pixi.sh/install.sh | sh
# or on Windows:
powershell -ExecutionPolicy ByPass -c "irm -useb https://pixi.sh/install.ps1 | iex"

  • Start a new terminal.

  • Initialize a workspace.

pixi init test-pixi-build 
cd test-pixi-build

  • Add rattler-build from conda-forge:

pixi add rattler-build

  • Now you have rattler-build installed and you can run:

pixi run rattler-build --help

Now lets see if we can use a specific version of rattler-build that you need to build from source to test an upstream change. Due to the Pixi package definition in the repo it is as easy as depending on the git source.

  • Add the pixi-build preview to your workspace:

[workspace]
preview = ["pixi-build"]

  • Change the dependency on rattler-build to a git dependency:

[dependencies]
rattler-build = { git = "https://github.com/prefix-dev/rattler-build", branch = "main" }

  • Let Pixi build, install and run that custom version:

pixi run rattler-build --help

If that was to many steps for you, pixi global install also supports git dependencies now, so you can run:

pixi global install --git https://github.com/prefix-dev/rattler-build

rattler-build

For more details, check out the getting started guide and the examples.

What's Next

The vision is to make pixi-build the go-to solution for building from source, across languages and platforms, making it as easy, reproducible, and portable as installing a package.

Imagine being able to build any open-source project from source with a single command, without needing to learn new tools or maintain separate build configurations.

Where package maintainers can easily provide cross-platform source builds for their users, while staying close to the ecosystem their language provides.

pixi-build is in preview and actively evolving. Here's what's on the horizon:

  • More backends and richer configuration options for existing ones.

  • Improved UX with better error messages, CLI interface improvements, build logs, and debugging tools.

  • Better support for build variants and complex dependency graphs, making it easier to manage multiple builds from the same source.

  • Moving out of preview toward stable.

We're building this in the open, and we want to build it with the community. Dogfooding our own project is not enough to create a robust solution. If you have ideas, feedback, or want to contribute, please join us on GitHub or Discord. We want to make sure pixi-build solves real problems for real users, so your input is invaluable.

Need commercial help?

At Prefix.dev we're building Pixi to be the best tool for package management and building from source, but we know that some organizations have unique needs or want help getting set up. If that's you, please contact us to learn about our commercial offerings, including custom build solutions, consulting, and support.