Virtual Packages in the Conda ecosystem

Cover image for Virtual Packages in the Conda ecosystem

At prefix.dev we are working on fast tools for the Conda ecosystem. We have rewritten conda-build in Rust to make it 10x faster, and we are building Pixi, an innovative package manager and workflow tool for complex projects. We love to dive deep into Conda internals – here we're talking about "virtual packages".

When working with Conda packages, you've likely encountered situations where a package installs perfectly on one system but fails on another. Or perhaps you've wondered how Conda "knows" whether to install CUDA-enabled packages on your GPU-equipped machine. The answer lies in a small trick: virtual packages.

What Are Virtual Packages?

Virtual packages are Conda's way of representing system features and constraints that exist outside of Conda's direct control. Unlike regular packages that you can install, update, or remove, virtual packages are automatically detected by Conda based on your system's characteristics. They act as invisible dependencies that help the solver make intelligent decisions about package compatibility.

Think of virtual packages as Conda's sensors for your system environment. They detect features like:

  • Operating system type and version

  • System libraries (like glibc on Linux)

  • Hardware capabilities (like CUDA support)

  • The Conda version itself

These packages are denoted by a double underscore prefix (e.g., __linux, __cuda) and never appear in conda list output because they're not "real" packages—they're system properties expressed as package constraints.

The Core Virtual Packages

Let's explore the main virtual packages you'll encounter:

Operating System Detection

  • __linux: Present when running on Linux, contains the kernel version

  • __win: Present when running on Windows, contains the Windows version

  • __osx: Present when running on macOS, includes the OS version

  • __unix: Present on both Linux and macOS (useful for Unix-like system requirements)

System Libraries

  • __glibc: Detects the GNU C Library version on Linux systems

  • __cuda: Identifies the maximum CUDA version supported by your display driver

Conda Environment

  • __conda: Reports the Conda version being used for solving (not present when using pixi or rattler/rattler-build)

Practical Applications

Ensuring System Compatibility

Virtual packages help prevent incompatible installations. For instance, a package requiring CUDA 11.0 won't install on a system with only CUDA 10.2 support, because the __cuda=10.2 virtual package constraint prevents it. This automatic checking saves users from cryptic runtime errors.

When building packages with rattler-build, you can leverage virtual packages to create more robust specifications:

requirements:
  run:
    - __cuda >=11.0
    - ${{ "__glibc >=2.17" if linux }}
    - ${{ "__osx >=10.13" if osx }}

This ensures your package only installs on systems meeting minimum requirements, preventing hard-to-debug issues from incompatible system libraries.

Building Platform-Specific Noarch Packages

One of the most powerful uses of virtual packages is creating noarch packages with OS-specific dependencies. This might sound weird—how can a package be architecture-independent yet have platform-specific requirements? Virtual packages make this possible.

Consider a pure Python package that needs a Windows-specific dependency. Traditionally, you'd need to build separate packages for each platform and Python version combination, potentially resulting in dozens of build variants. With virtual packages, you can reduce this to just two or three builds:

build:
  number: 0
  noarch: python
  string: ${{ hash }}_{{ "unix" if unix_variant == "true" else "win" }}

requirements:
  run:
    - python >=3.8
    - numpy
    - if: unix_variant == "true"
      then:
        - __unix
      else:
        - __win
        - windows-only-dependency

Note that this uses an extra variant that specifies unix_variant: ["true", "false"]. We cannot use the actual unix specifier because these packages can both be built on e.g. Linux. We just produce another noarch package that is the same except for some metadata. It's a trick that is necessary until our conditional dependency proposal is accepted and implemented!

Pixi System Requirements & Virtual Packages

Pixi's system requirements are mapped directly to virtual packages. The system requirements will set the upper bound for these system packages and thus constrain, and possibly deselect, some packages that have higher system requirements then found in the pixi.toml file. For example, if your system-requirements table contains: 

[system-requirements]
linux = "4.18"
libc = { family = "glibc", version = "2.28" }
cuda = "12.0"

Pixi will only select packages that are compatible with glibc 2.28 or below. Packages that require __glibc >=2.32 would not be selected, because the highest "installable" version is 2.28.

The same goes for the CUDA driver. Any package asking for __cuda >=11.0 will be installable with a system-requirement of cuda = 12.0. Packages asking for __cuda >=13.0 would be excluded from the solution.

Debugging Virtual Packages

To see which virtual packages Pixi or Conda detects on your system, simply run:

$ pixi info

System
------------
       Pixi version: 0.47.0
           Platform: osx-arm64
   Virtual packages: __unix=0=0
                   : __osx=15.5=0
                   : __archspec=1=m2

# or when using Conda you can run
conda info

Look for the "virtual packages" section in the output. This information is invaluable when troubleshooting installation issues or understanding why certain packages aren't available for your system.

For testing and debugging, you can override virtual package detection using environment variables:

# Test package behavior without CUDA
export CONDA_OVERRIDE_CUDA=""

# Simulate a different glibc version
export CONDA_OVERRIDE_GLIBC="2.17"

Note that pixi uses the same environment variables as Conda to maximize compatibility.

Best Practices for Package Authors

  • Always include virtual package constraints with selectors: When using virtual packages in your recipe, pair them with platform selectors to ensure proper variant generation.

  • Document system requirements: If your package depends on specific virtual packages, document these requirements clearly for users.

  • Test across platforms: Use virtual package overrides to test how your package behaves under different system configurations without needing multiple physical machines.

  • Consider noarch where possible: Virtual packages enable noarch packages even with platform-specific dependencies, reducing build complexity and storage requirements.

Integration with Modern Tools

Both pixi and rattler-build fully support virtual packages, making them first-class citizens in the modern Conda ecosystem. When using pixi for environment management, virtual packages ensure that environments remain portable across similar systems while preventing installation on incompatible ones.

Looking Forward

Virtual packages represent one of Conda's most elegant solutions to the complex problem of system compatibility. They bridge the gap between Conda's controlled package environment and the diverse system configurations in the wild. As the Conda ecosystem continues to evolve with tools like pixi and rattler-build, virtual packages remain a fundamental building block for creating robust, cross-platform software distributions.

Understanding and properly utilizing virtual packages can significantly improve your packages' reliability and user experience. They're not just a technical detail—they're a key tool for managing the complexity of modern software deployment across diverse computing environments.

Written on by:
Wolf Vollprecht
Wolf Vollprecht