...
| Note |
|---|
Be very careful with |
| Warning |
|---|
Do NOT install any package manager like conda on your account! Do NOT use any conda command except special cases! |
Why
...
Use Python Environments?
In a fresh terminal session, python or python3 points to a system-installed copy of Python (typically in /usr/bin). As the operating system heavily depends on this python instance, the version is fixed and only the most basic, built-in libraries are available.
Creating a new environment allows you to have full control over the Python version, the selection of libraries, and the specific versions, too. Python environments can then be engaged and disengaged freely, enabling a wide-variety of specific uses including CPU compute and even GPU acceleration.
Below is a cartoon created with chatGPT 4o, representing a python environment by a tool shed, and python packages installed in this environment by the various tools.
...
Load the
...
Environment Manager “Mamba”
Load the latest stable version of the mamba Python manager with:
module load mamba/latest
...
Find Available Environments
Many Python suitespackages, such as Pytorch or Qiime, are commonly-requested and thus are provided pre-installed by Research Computing staff on the supercomputers already. These environments are version-fixed and read-only, so they may be used freely by any number of users simultaneously without any risk of the environment changing.
All global/public/admin-maintained python environments may be found under /packages/envs:
...
/packages/envs
...
.
...
User environments are by default installed to ~/.conda/envs, and after running module load mamba/latest, all available environments may should be listed with mamba info --envs.
...
Environments may also be activated with a full path, e.g.,
source activate /data/sciencelab/.conda/envs/pysci.
This capability makes /data (/wiki/spaces/RC/pages/60915741project-based storage) an ideal location for groups sharing python environments!
...
| Tip |
|---|
It is best to install all necessary packages in a single command as line 2 showed above. It maximizes environment stability and minimizes total build time to have all major dependencies resolved initially. |
| Warning |
|---|
DO NOT use the “defaults” channel (source: Mamba official troubleshooting guide)! |
To create an environment with a specific path, i.e. the data directory of a research group, the path of this directory needs to be included with the -p flag in the mamba create command:
...
When using mamba to install packages or create environments, you may see errors related to opening files in /packages/apps/mamba. These errors are harmless. An example is shown below.
Always verify the Prefix: is pointing where you need it to before proceeding with an installation, but otherwise, errors and warnings made by mamba may be ignored.
It is also good practice to verify what is being installed as a new package, what existing packages are being modified, and what existing packages are being removed before proceeding with the install.
Example
Please
...
check here for a brief example: A Brief Example
Adding Packages to Public or Existing Environments
| Note |
|---|
The global/admin-maintained public environments are read-only and can’t be changed by users. To add packages to one of these environments, you will need to clone it first. |
To clone a public environment:
| Code Block |
|---|
$ module load mamba/latest
$ source activate <public_environment_name>
$ mamba env export --from-history --no-builds -n <public_environment_name> > /your/path/to/<public_environment_name>.yaml
$ source deactivate
$ mamba create -n <your_environment_name> python=3
$ mamba env update -n <your_environment_name> --file /your/path/to/<public_environment_name>.yaml |
Line 3 above asks mamba to export the list of packages without the version numbers nor the hashes in this public environment, unless the version numbers were specified during the installation process of this public environment. If you wish to preserve all the version numbers, the --from-history and the --no-builds flag should be removed. Note that some public environments are old, and some version conflicts may arise if you specify the version numbers in the .yaml file.
...
Once an environment is created, a kernel interface will need to be made to have that environment available in Jupyter. This is as easy as, mkjupy <env_name>. Please review /wiki/spaces/RC/pages/1905788308 for additional details.check the next page for details:
Preparing Python Environments for Jupyter
ADVANCED: Building from GitHub repository
...