| Table of Contents |
|---|
Overview
The supercomputer uses Mamba, a high-performance parallel package manager, to allow users to install the Python modules they need. It also plays a pivotal role in optimizing software environments on supercomputers. In the upcoming instructions, we'll explore the process of loading Mamba modules and delve into creating and loading new environments.
...
| Panel | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
Be very careful with |
Why Create a New Environment?
In a fresh terminal session, python or python3 points to a system-installed copy of Python (typically in /usr/bin). As the operating system heavily depends on this python Python instance, the version is fixed and only the most basic, built-in libraries are available.
Creating a new environment allows you to have full control over the Python version, the selection of libraries, and the specific versions, too. Python environments can then be engaged and disengaged freely, enabling a wide-variety of specific uses including CPU compute and even GPU acceleration.
Load the Package and Environment Manager
Load the latest stable version of the mamba Mamba Python manager with:
module load mamba/latest
List Available Environments
Many Python suites, such as Pytorch or Qiime, are commonly-requested and thus are provided by Research Computing staff on the supercomputers already. These environments are version-fixed and read-only, so they may be used freely by any number of users simultaneously without any risk of the environment changing.
All global/admin-maintained python Python environments may be found under /rc/packages/envs:
...
| Code Block |
|---|
[<asurite>@login1 ~]$ mamba info --envs
mamba version : 1.5.1
# conda environments:
#
pytorchGPU /home/<asurite>/.conda/envs/pytorchGPU
testing /home/<asurite>/.conda/envs/testing
updateTest /home/<asurite>/.conda/envs/updateTest
base /rc/packages/apps/mamba/1.5.1
pytorch-gpu-2.0.1 /rc/packages/envs/pytorch-gpu-2.0.1
scicomp /rc/packages/envs/scicomp
... |
Load Available Environments
Use the source activate command to load the environment you want.
...
Environments may also be activated with a full path, e.g.,
source activate /project/sciencelab/.conda/envs/pysci.
This capability makes /project (project storage) an ideal location for groups sharing python Python environments!
| Panel | ||||||
|---|---|---|---|---|---|---|
| ||||||
Do not load environments with |
Creating Environments
| Code Block |
|---|
$ module load mamba/latest $ mamba create -n <environment_name> -c conda-forge [-c <channel>] [packages] $ source activate <environment_name> |
...
| Panel | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
When using mamba to install packages or create environments, you may see errors related to opening files in Always verify the It is also good practice to verify what is being installed as a new package, what existing packages are being modified, and what existing packages are being removed before proceeding with the install. Please review the |
Adding dependencies to existing Environments
| Note |
|---|
The global/admin-maintained environments are read-only and can’t be changed by users. To add packages to one of these environments, you will need to clone it.
This will create an environment called |
...
| Tip |
|---|
Please review the screenshot of an example When using mamba to install packages or create environments, you may see errors related to opening files in Always verify the It is also good practice to verify what is being installed as a new package, what existing packages are being modified, and what existing packages are being removed before proceeding with the install. |
...
Using environments in Jupyter
Once an environment is created, a kernel interface will need to be made to have that environment available in Jupyter. This is as easy as, mkjupy <env_name>. Please review Preparing Python Environments for Jupyterfor additional details.
What if the Package I Need Isn’t in Conda-forge?
Conda-forge is the only pre-approved place to get Python modules right now. Other modules will need to go through an approval process with the KE Secure Cloud team before they can be used. Contact asre-support@asu.edu Open a request at https://links.asu.edu/kesc-support to ask for a module.
| Insert excerpt | ||||||||
|---|---|---|---|---|---|---|---|---|
|