Introducing channels: host your own packages

Written by Tim de Jager 2 years ago

Today we would like to introduce a new feature, hosting your own package repository using private or public channels. Think something along the lines of a private PyPI or NPM repository, or Shipyard a Rust package registry.

Introduction

During my previous job at Smart Robotics, we utilized mamba/micromamba as our go-to tool for managing developer environments, encompassing everything from installing our essential tooling to running software on physical robots. However, we encountered a challenge as a company: we desired to host our own packages without making them publicly accessible. This predicament is one that many developers using conda may encounter, and at Prefix, we believe it's worth addressing. Now, why would you want to host your own packages? The primary reasons are to enhance software reproducibility and foster seamless collaboration among developers. Let's delve deeper into these aspects.

Developing software together

During my time at Smart Robotics, I collaborated with a diverse group of developers who primarily utilized C++ and Python for various projects and libraries. Fortunately, conda and conda-forge provide robust support for both programming languages. As developers, we faced the challenge of maintaining and versioning our libraries to ensure their compatibility for downstream applications. When it came to using these libraries across different projects, we needed a reliable method to install specific versions. Moreover, considering that each project or robot might require different versions, managing these dependencies presented various options:

Git Submodules (with pinned version, or version branches)
Monorepo (with specific versions/configurations)
Package Manager (with actual versioning)

A package manager gives us the advantage that we can actually constrain versions, and it resolves dependencies for us. Additionally, it tells us when things are not compatible, or even impossible to use together.

Note: Shipyard also expands on this for the Rust use-case. Give it a read as the article makes a pretty good case!

Back to Smart Robotics: using micromamba, we could make totally reproducible environments with properly versioned dependencies that we could reinstall years later. This was only possible by using a package registry. Having a channel with our own private packages allowed us to use the full-fledged solver of micromamba to resolve dependencies, including libraries that we built internally. This enabled the use of open-source and private packages together.

Aside: Conda Channels

I've been using both the words channel and registry throughout this post but let me clarify what I mean by these terms. For that you'll need a quick introduction about conda channels because they differ a bit from you may be used to working with other package managers.

To illustrate I've taken PyPI ecosystem as an example on the left and conda ecosystem on the right. Normally, we as developers, are used to install packages from a single package registry. Be it cargo, npm, pypi etc. Only when you have a private registry, something like: Artifactory, you can host our own packages, and build on top of existing open-source registries. The conda ecosystem is a bit different. It is split into channels, each containing packages that may depend on packages from other channels. We like to call these other channels "related channels".

So essentially, when drawing the parallel to other package management systems each channel is its own package registry. The difference being that while there is mostly one open-source registry for a language, there are multiple for conda.

Wait... but why would you want this? Well, within the autonomy of the channel authors can decide how to manage the channel and it's contents. Two real-world examples:

conda-forge is channel managed by a process of feedstocks (recipes that built packages automatically for new versions) each of these feedstocks are maintained by individuals.
While Robostack is a channel for Robotics, that is coupled to ROS distributions, where the packages are built using an overarching application vinca that builds recipes from ROS package.xml files. It is a lot more specific to ROS. But importantly, the robostack channel builds on top of the conda-forge channel and depends on many lower-level packages from conda-forge.

Let's see this in the wild:

An example from the wild

As an example Bioconda is another channel that contains bioinformatics packages, but depends on packages from the conda-forge channel – a more general and thus related channel.

We can see this by navigating to the package ariba which is in the Bioconda channel and navigating to one of the dependencies in this case: beautifulsoup4 which is hosted on conda-forge.

This way channels can work together to provide a complete ecosystem of packages, by building on top of each other. When we install this package from the bioconda channel, micromamba will also install the dependencies from the conda-forge channel, and will do resolving of constraints/versions over channels.

Introducing prefix.dev channels

We believe that to be able to collaborate effectively in a software team or across an open-source project, you need to be able to share libraries with each other, and be able to apply versioning to internal libraries as well. At prefix, it is our mission to make software collaboration as easy as possible.

That's why we think that having the ability to host your own channels is one of the features that eases collaboration and opens up the ability to manage software in the way you think is best.

You can now host your own private or public channels on prefix.dev, without having to worry about running your own infrastructure (which one would need to with quetz). You benefit from all the features we already have (and those that are coming) automatically. The channels integrate directly with our package search, so you can search for packages in your own channel, or in the public channels. We hope that this creates a user experience that is closer to what you are used to from other package managers. And we're going to great lengths to seamlessly integrate prefix.dev with micromamba and our upcoming package management solutions.

How does it work?

When you are logged in you are able to create your own public/private channels. To create a new channel navigate to the channels page and click the New button.

At this point you can upload packages to your own channel. We've provided a simple interface to do this, you can upload both .conda and .tar.bz2 files.

As you can see you can use our regular interface to browse to different package locations. It even integrates into our regular search, this searches between all the channels you have access to. This includes all public and your private channels, or channels you are a member of.

I'm trying to find appdirs one of the packages I uploaded to my channel:

Upload packages

The uploading UI is a simple interface to upload packages to your channel. But it will probably not be what you want if you build your package from the command-line. In that case it's easier to use our API to upload packages.

This is further explained in our channels documentation here.

Install packages

To install packages with micromamba you can use the -c flag and the API key authentication we provide to install your packages. For example to install appdirs from the gif above you can use the following command (assuming you have micromamba installed, and have created an API Key):

# Create a new test environment
micromamba create -n test
# Activate the environment
micromamba activate test
# Authenticate with the prefix.dev server
# You only need to do this once
micromamba auth login https://repo.prefix.dev --bearer $TOKEN
# Install appdirs from your `my-cool-channel` channel
micromamba install -c https://repo.prefix.dev/my-cool-channel -c conda-forge appdirs

replace my-cool-channel with your channel name, in this case it's my private channel. We also need conda-forge as a channel, because appdirs in my channel depends on packages from that channel.

What's next?

We are really excited to see how people will be using this feature – any feedback you have is much appreciated. As always our issue tracker is available for feedback.

Some things that are on our minds that we could do in the future:

Introduce organisations, that you can manage channels membership with.
Channel usage: display statistics for channel usage.
Make related channels more visible in the UI. So you can see what channels you need to install a package.

Pricing

For now, public and private channels are in beta and free to use - in exchange for your feedback! In the future we will come up with fair usage-based pricing.

If you need professional support in the meantime, or on-prem services, do not hesitate to contact us at hi@prefix.dev.

Please tell us what you think, you can reach us on Discord, Twitter @prefix_dev or via email.