Introducing channels: host your own packages
Today we would like to introduce a new feature, hosting your own package repository using private or public channels. Think something along the lines of a private PyPI or NPM repository, or Shipyard a Rust package registry.
During my previous job at Smart Robotics, we utilized mamba/micromamba as our go-to tool for managing developer environments, encompassing everything from installing our essential tooling to running software on physical robots. However, we encountered a challenge as a company: we desired to host our own packages without making them publicly accessible. This predicament is one that many developers using conda may encounter, and at Prefix, we believe it's worth addressing. Now, why would you want to host your own packages? The primary reasons are to enhance software reproducibility and foster seamless collaboration among developers. Let's delve deeper into these aspects.
During my time at Smart Robotics, I collaborated with a diverse group of
developers who primarily utilized C++ and Python for various projects and
conda-forge provide robust support for
both programming languages. As developers, we faced the challenge of maintaining
and versioning our libraries to ensure their compatibility for downstream
applications. When it came to using these libraries across different projects,
we needed a reliable method to install specific versions. Moreover, considering
that each project or robot might require different versions, managing these
dependencies presented various options:
- Git Submodules (with pinned version, or version branches)
- Monorepo (with specific versions/configurations)
- Package Manager (with actual versioning)
A package manager gives us the advantage that we can actually constrain versions, and it resolves dependencies for us. Additionally, it tells us when things are not compatible, or even impossible to use together.
Note: Shipyard also expands on this for the Rust use-case. Give it a read as the article makes a pretty good case!
Back to Smart Robotics: using micromamba, we could make totally reproducible environments with properly versioned dependencies that we could reinstall years later. This was only possible by using a package registry. Having a channel with our own private packages allowed us to use the full-fledged solver of micromamba to resolve dependencies, including libraries that we built internally. This enabled the use of open-source and private packages together.
I've been using both the words channel and registry throughout this post but let me clarify what I mean by these terms. For that you'll need a quick introduction about conda channels because they differ a bit from you may be used to working with other package managers.
To illustrate I've taken PyPI ecosystem as an example on the left and conda ecosystem on the right. Normally, we as developers, are used to install packages from a single package registry. Be it cargo, npm, pypi etc. Only when you have a private registry, something like: Artifactory, you can host our own packages, and build on top of existing open-source registries. The conda ecosystem is a bit different. It is split into channels, each containing packages that may depend on packages from other channels. We like to call these other channels "related channels".
So essentially, when drawing the parallel to other package management systems each channel is its own package registry. The difference being that while there is mostly one open-source registry for a language, there are multiple for conda.
Wait... but why would you want this? Well, within the autonomy of the channel authors can decide how to manage the channel and it's contents. Two real-world examples:
- conda-forge is channel managed by a process of feedstocks (recipes that built packages automatically for new versions) each of these feedstocks are maintained by individuals.
- While Robostack is a channel for Robotics,
that is coupled to ROS distributions, where the
packages are built using an overarching application
vinca that builds recipes from ROS
package.xmlfiles. It is a lot more specific to ROS. But importantly, the robostack channel builds on top of the conda-forge channel and depends on many lower-level packages from conda-forge.
Let's see this in the wild:
This way channels can work together to provide a complete ecosystem of packages, by building on top of each other. When we install this package from the bioconda channel, micromamba will also install the dependencies from the conda-forge channel, and will do resolving of constraints/versions over channels.
We believe that to be able to collaborate effectively in a software team or across an open-source project, you need to be able to share libraries with each other, and be able to apply versioning to internal libraries as well. At prefix, it is our mission to make software collaboration as easy as possible.
That's why we think that having the ability to host your own channels is one of the features that eases collaboration and opens up the ability to manage software in the way you think is best.
You can now host your own private or public channels on prefix.dev, without having to worry about running your own infrastructure (which one would need to with quetz). You benefit from all the features we already have (and those that are coming) automatically. The channels integrate directly with our package search, so you can search for packages in your own channel, or in the public channels. We hope that this creates a user experience that is closer to what you are used to from other package managers. And we're going to great lengths to seamlessly integrate prefix.dev with micromamba and our upcoming package management solutions.
When you are logged in you are able to create your own public/private channels.
To create a new channel navigate to the channels page and click the
At this point you can upload packages to your own channel. We've provided a
simple interface to do this, you can upload both
As you can see you can use our regular interface to browse to different package locations. It even integrates into our regular search, this searches between all the channels you have access to. This includes all public and your private channels, or channels you are a member of.
I'm trying to find
appdirs one of the packages I uploaded to my channel:
The uploading UI is a simple interface to upload packages to your channel. But it will probably not be what you want if you build your package from the command-line. In that case it's easier to use our API to upload packages.
This is further explained in our channels documentation here.
To install packages with micromamba you can use the
-c flag and the API key
authentication we provide to install your packages. For example to install
appdirs from the gif above you can use the following command (assuming you
have micromamba installed, and have created an API
# Create a new test environment micromamba create -n test # Activate the environment micromamba activate test # Authenticate with the prefix.dev server # You only need to do this once micromamba auth login https://repo.prefix.dev --bearer $TOKEN # Install appdirs from your `my-cool-channel` channel micromamba install -c https://repo.prefix.dev/my-cool-channel -c conda-forge appdirs
my-cool-channel with your channel name, in this case it's my private
channel. We also need
conda-forge as a channel, because
appdirs in my
channel depends on packages from that channel.
We are really excited to see how people will be using this feature – any feedback you have is much appreciated. As always our issue tracker is available for feedback.
Some things that are on our minds that we could do in the future:
- Introduce organisations, that you can manage channels membership with.
- Channel usage: display statistics for channel usage.
- Make related channels more visible in the UI. So you can see what channels you need to install a package.
For now, public and private channels are in beta and free to use - in exchange for your feedback! In the future we will come up with fair usage-based pricing.
If you need professional support in the meantime, or on-prem services, do not hesitate to contact us at email@example.com.