Overview
The supercomputer uses Mamba, a high-performance parallel package manager, to allow users to install the Python modules they need. It also plays a pivotal role in optimizing software environments on supercomputers. In the upcoming instructions, we'll explore the process of loading Mamba modules and delve into creating and loading new environments.
The supercomputers use mamba instead of conda and pip. mamba is a parallel and C++ implementation of conda that provides a much faster experience for setting up Python environments on the supercomputer. If you’re familiar with using conda, all you will need to do is replace the conda command with the mamba command. pip is not stable in a multi-user environment like a supercomputer, so Research Computing discourages the use of pip except when necessary.
Be very careful with pip, it can easily break a mamba environment!
Load the Package and Environment Manager
Load the latest stable version of the mamba Python manager with:
module load mamba/latest
List Available Environments
All global/admin-maintained python environments may be found under /rc/packages/envs. User environments are by default installed to /home/asurite/.conda/envs, and after running module load mamba/latest, all available environments may be listed with mamba info --envs.
[asureite@login1 ~]$ mamba info --envs
mamba version : 1.5.1
# conda environments:
#
pytorchGPU /home/dmccaff4/.conda/envs/pytorchGPU
testing /home/dmccaff4/.conda/envs/testing
updateTest /home/dmccaff4/.conda/envs/updateTest
base /rc/packages/apps/mamba/1.5.1
pytorch-gpu-2.0.1 /rc/packages/envs/pytorch-gpu-2.0.1
scicomp /rc/packages/envs/scicomp
tensorflow-gpu-2.12.1 /rc/packages/envs/tensorflow-gpu-2.12.1
Load Available Environments
$ module load mamba/latest $ source activate gurobi-9.5.1
Environments may also be activated with a full path, e.g.,
source activate /data/sciencelab/.conda/envs/pysci.
This capability makes /data (project-based storage) an ideal location for groups sharing python environments!
Do not load environments with conda or mamba as the prefix, i.e., mamba activate gurobi-9.5.1, as this injects non-supercomputing friendly cruft into your supercomputing configuration files!
Creating Environments
$ module load mamba/latest $ mamba create -n <environment_name> -c conda-forge [-c <channel>] [packages] $ source activate <environment_name>
It is best to install all necessary packages at this step, as it maximizes environment stability and minimizes total build time to have all major dependencies resolved initially.
Environments may also be created by specifying the path, but be careful as creating environments in non-default locations makes it easy to lose the environment!
When using mamba to install packages or create environments, you may see errors related to opening files in /packages/apps/mamba. These errors are harmless. An example is shown below.
Always verify the Prefix: is pointing where you need it to before proceeding with an installation, but otherwise, errors and warnings made by mamba may be ignored.
It is also good practice to verify what is being installed as a new package, what existing packages are being modified, and what existing packages are being removed before proceeding with the install.
Please review the mamba install section below for a summary of the components of mamba install.
Adding dependencies to existing Environments
The global/admin-maintained environments are read-only and can’t be changed by users. To add packages to one of these environments, you will need to clone it.
mamba create --name <new_environment_name> --clone <environment_to_copy>
This will create an environment called <new_environment_name> that is copied from <environment_to_copy>.
To install a new package to a mamba environment, first load the mamba module, then activate the existing environment, and finally install as many new dependencies as required
$ module load mamba/latest $ source activate <environment_name> $ mamba install -c conda-forge [-c <channel>] <packages>
Please review the screenshot of an example mamba install below before proceeding. Annotations are shown in cyan with a black background and cyan outline.
When using mamba to install packages or create environments, you may see errors related to opening files in /packages/apps/mamba. These errors are harmless. An example is shown below.
Always verify the Prefix: is pointing where you need it to before proceeding with an installation, but otherwise, errors and warnings made by mamba may be ignored.
It is also good practice to verify what is being installed as a new package, what existing packages are being modified, and what existing packages are being removed before proceeding with the install.
Using environments in Jupyter
Once an environment is created, a kernel interface will need to be made to have that environment available in Jupyter. This is as easy as, mkjupy <env_name>. Please review Using Jupyter for additional details.
ADVANCED: Building from GitHub repository
Many python packages are not necessarily available on available mamba channels. It is best to avoid these packages when possible. However, it is possible to integrate them into a workflow. First, clone the git repository into your home directory:
$ git clone <url of github repository>
This URL can be copied from GitHub repository. In the figure below, the blue line indicates the URL of the corresponding repository (repo) page.
The cloned directory should include instructions for installing the Python package.
Be sure that you’re either in an existing mamba environment or create a new one that supports the listed dependencies. TYPICALLY THE DEPENDENCIES ARE OVERSPECIFIED--dependency files are typically very fragile and non-portable, and include precise versions for second-order dependencies. If your build is failing, try to remove all but the first-order dependencies (e.g., installing a versioned pytorch will automatically install the most stable version of numpy).
Once the environment is created and activated, and all dependencies installed, the new repository module may be installed as specified in the README, typically a pip install.
Be very careful with pip, it can easily break a mamba environment!
Never use sudo, which is often provided in instructions for system-wide installations by an administrator. It is unnecessary when installing into your own home directory.
Additional Help
If you require further assistance on this topic, please don't hesitate to contact the Research Computing Team. To create a support ticket, kindly send an email to rtshelp@asu.edu. For quick inquiries, you're welcome to reach out via our #rc-support Slack Channel or attend our office hours for live assistance
We also offer a series of workshops. More information here: Educational Opportunities and Workshops