Overview

The supercomputer uses Mamba, a high-performance parallel package manager, to allow users to install the Python modules they need. It also plays a pivotal role in optimizing software environments on supercomputers. In the upcoming instructions, we'll explore the process of loading Mamba modules and delve into creating and loading new environments.

The supercomputers use mamba instead of conda and pip. mamba is a parallel and C++ implementation of conda that provides a much faster experience for setting up Python environments on the supercomputer. If you’re familiar with using conda, all you will need to do is replace the conda command with the mamba command. pip is not stable in a multi-user environment like a supercomputer, so Research Computing discourages the use of pip except when necessary.

Why Create a New Environment?

In a fresh terminal session, python or python3 points to a system-installed copy of Python (typically in /usr/bin). As the operating system heavily depends on this Python instance, the version is fixed and only the most basic, built-in libraries are available.

Creating a new environment allows you to have full control over the Python version, the selection of libraries, and the specific versions, too. Python environments can then be engaged and disengaged freely, enabling a wide-variety of specific uses including CPU compute and even GPU acceleration.

Load the Package and Environment Manager

Load the latest stable version of the Mamba Python manager with:

module load mamba/latest

List Available Environments

Many Python suites, such as Pytorch or Qiime, are commonly-requested and thus are provided by Research Computing staff on the supercomputers already. These environments are version-fixed and read-only, so they may be used freely by any number of users simultaneously without any risk of the environment changing.

All global/admin-maintained Python environments may be found under /rc/packages/envs:

User environments are by default installed to ~/.conda/envs, and after running module load mamba/latest, all available environments may be listed with mamba info --envs.

Load Available Environments

Use the source activate command to load the environment you want.

The name of the environment will appear to the left of the command prompt so that you know what environment is currently active.

Creating Environments

Adding dependencies to existing Environments

To install a new package to a mamba environment, first load the mamba module, then activate the existing environment, and finally install as many new dependencies as required.

Open

Using environments in Jupyter

Once an environment is created, a kernel interface will need to be made to have that environment available in Jupyter. This is as easy as, mkjupy <env_name>. Please review Preparing Python Environments for Jupyterfor additional details.

What if the Package I Need Isn’t in Conda-forge?

Conda-forge is the only pre-approved place to get Python modules right now. Other modules will need to go through an approval process with the KE Secure Cloud team before they can be used. Open a request at https://links.asu.edu/kesc-support to ask for a module.