S3 support in Pixi and rattler-build

Cover image for S3 support in Pixi and rattler-build

At prefix.dev we're building a cross-platform, system-level package and workflow manager called Pixi. We use Conda packages and recently added support for grabbing them straight from an S3 bucket with all the standard credential configuration. Vendor neutrality FTW!

We recently merged S3 support in rattler, pixi and rattler-build – all of our tools now support the most common cloud storage standard. The contribution was made by our friends at QuantCo, across multiple PRs and we were lucky to be joined by Pavel Zwerschke in a community “Show & Tell” where he explained all the details on how to use S3 support across our tools.

At prefix.dev we are excited about S3 support because we strongly believe in the versatility of Conda packages and having vendor agnostic ways of distributing Conda packages is a huge benefit to the ecosystem that will allow it to grow further and get more adoption. At the same time, we obviously hope to offer the superior experience on prefix.dev. Our goal is to make Conda packages as popular and widely used as Docker containers and it's an absolute must to have vendor-neutral options for hosting.

Info

Did you know – the original Conda package manager has supported the S3 protocol since a long time. You just need to install boto3 alongside Conda to enable s3:// channel URLs for Conda. Unfortunately there does not seem to be a lot of documentation for the feature.

Current hosting options for Conda packages

There are a number of hosting options for Conda packages: anaconda.org, prefix.dev, Artifactory or Sonatype Nexus. However, S3 is a bit of a game-changer.

The new S3 support in Pixi offers a compelling alternative with several advantages:

  • Open standard implemented by numerous cloud providers (AWS, Cloudflare R2, Backblaze B2, Hetzner Object Storage, etc.)

  • Built-in authentication through the S3 protocol

  • Reduced vendor lock-in due to wide compatibility

  • Bare-bones, universal solution that works in virtually any environment

We hope that this makes it easier for companies to decide to go all in on pixi & rattler-build.

How Conda Channels Work

A conda channel is essentially a collection of files hosted on a web server. It includes:

  • Platform-specific directories (like noarch) that contain all the packages

  • A repodata.json file containing metadata about packages, versions, and dependencies

Theoretically, even a "flat file server" can run a Conda channel - there is really nothing to it. Keeping the index up to date is the main challenge.

Setting Up S3 Support in Pixi

... and for your own packages. The implementation is straightforward.

1. Add the S3 bucket in your Pixi.toml

In your pixi.toml file you can simply use the s3:// protocol to select a S3 bucket as channel. By default, this will use the various ways of finding standard AWS credentials on your machine.

[workspace]
# ...
channels = ["s3://my-bucket/custom-channel"]

There are a number of additional configuration options that you can supply for broad compatibility with many different S3-storage vendors. All configuration options are listed in the extensive documentation.

# In your pixi.toml file you can set:
[workspace.s3-options.my-bucket]
endpoint-url = "https://my-s3-host"
region = "us-east-1"
force-path-style = false

2. Upload packages

You can use your favorite tool to upload or mirror packages to your S3 bucket. But you can also build a new package with rattler-build and upload it directly with rattler-build itself.

rattler-build upload s3 --channel s3://my-bucket/channel ...

3. Generate the index with rattler-index

Uploading the package(s) was easy – but a working conda channel needs an index. For each subdir (platform) like linux-64, osx-arm64, or noarch, a repodata.json file needs to be created that contains all package file names, dependencies, SHA256 hashes, and so on. This information will be used by the solver to determine which versions to install, and what dependencies to fetch. There are a number of variations to this file – such as zstd compressed or "sharded" across a number of smaller files, and the indexing tool will take care of that.

The rattler-index standalone tool builds on top of the solid Rust foundations of rattler and offers a full-fledged conda package indexing workflow. The latest version supports sharded repodata and repodata patching, which makes it compatible with the latest conda standards. It's a newer alternative to conda-index.

You can install and run rattler-index from conda-forge using the following commands:

# to install rattler-index with pixi
pixi global install rattler-index

# or with conda
conda install rattler-index -c conda-forge

# and then run it using
rattler-index s3 s3://my-bucket/channel --endpoint-url=https://...

If you want to index packages with Python, we also have Python bindings to the functionality in rattler-index as part of py-rattler: Documentation

Watch the show & tell

Pavel presented this information live - and in color at our community show & tell. You can watch it on Youtube:

Still image of the video

Come talk to us

The addition of S3 support to Pixi and Rattler represents a significant step forward in package distribution flexibility. Thanks to the ubiquitous S3 protocol, teams can now host conda packages in virtually any environment with robust authentication and minimal vendor lock-in.

Thanks to Pavel, Moritz and Daniel for working on this feature!

If you want to learn more about hosting Conda packages, talk to us on Discord or reach out via email: hi@prefix.dev

Written on by:
Wolf Vollprecht
Wolf Vollprecht