Cover image for Octoconda - turning Github Releases into Conda Packages
Written on

Octoconda - turning Github Releases into Conda Packages

Tobias Hunger
Tobias Hunger

If you've ever wanted to install a tool from GitHub via Conda, you know the pain. Someone releases a great tool with perfectly working pre-built binaries on Github, but there's no Conda package.
What now? You either download it manually, build from source, or maintains a conda recipe yourself.

Octoconda helps out: It takes the release artifact and re-packages it for use in the Conda eco-system.

What does it do?

Octoconda automates this entire workflow. Simply configure a list of repositories and a Conda channel and it:

  • Fetches all releases via the GitHub API

  • Filters out anything already available in a configured Conda channel

  • Matches platform-specific binaries using configurable patterns. The defaults usually work fine

  • Generates Rattler Build recipes with correct URLs and SHA256 checksums

  • Creates build scripts that handles various archive formats found in Github releases

  • Records all the release metadata in the Conda package

  • Packages and uploads everything into the configured Conda channel

Configure once, run periodically and keep all packages up to date automatically.

Github Actions work well for this: They have more access to Github APIs than other services.

Limitations

Octoconda uses Github specific APIs to fetch release information. This means it will not work with other code forges. Contributions that add support for other forges would be welcome.

Repackaged Github releases will depend on the host environment to provide the needed dependencies (in practice there are few as github releases tend to be built to minimize external dependencies). This is very different from packages found in conda-forge: Those are built to fully work within the Conda environment. Conda-forge packages are independent of the host environment, typically smaller as they share code with other packages and can be tailored to make use of the available hardware way better than the "least-common-denominator" approach binary packages published on github often take.

Security

If you trust the binary release artifacts, you should be able to also trust the Conda packages created by Octoconda.

Octoconda will not add or modify files: All the files in the Conda package will have the same SHAs as the same files in the Github release. Files will be moved around -- so that Conda environment will be able to run them, and some obvious junk files (like .DS_Store) will get removed.

Each package Octoconda generates contains meta-data giving the download URL and SHA of the re-packaged binary.

Octoconda uses attestation and trusted publishing to secure its package generation process.

Where can I see Octoconda in Action?

The experimental github-releases channel is populated by octoconda! I am in the process of adding more and more Github releases into this channel.

Just run e.g.

pixi global install --channel https://prefix.dev/github-releases htop-rs

and a cool new htop implementation written in rust is installed on your machine.

Please report any broken package in the channel and free to suggest more binaries via Github issues or PRs!

Get started re-packaging your own binaries

If you want to run your own instance of Octoconda, this should get you up and running:

Fork the octoconda repository.
Create a channel for your Github releases on prefix.dev. This channel must be configured to allow trusted publishing from the octoconda.yaml workflow in your repository.

Now configure what to build in config.toml :

[conda]
channel = "github-releases"

[[packages]]
repository = "BurntSushi/ripgrep"

[[packages]]
repository = "sharkdp/bat"

Replace the github-releases with your channel in the channel setting under conda

Then add a [[packages]] section with a repository key pointing to the Github repository you want to get releases from. More options are available to tailor the repackaging process, check the README.md file for details.

Enable Github actions for your repository and wait a bit for the Octoconda action to complete.

If you want to repackage binaries in private repositories you will need to configure the Github token available to the action. The default works fine for all public Github repositories.

Please report bugs with the packages in github-releases or with Octoconda and join us on Discord if you have further questions.