...
| Panel | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
Be very careful with |
Why Create a New Environment?
In a fresh terminal session, python or python3 points to a system-installed copy of Python (typically in /usr/bin). As the operating system heavily depends on this python instance, the version is fixed and only the most basic, built-in libraries are available.
Creating a new environment allows you to have full control over the Python version, the selection of libraries, and the specific versions, too. Python environments can then be engaged and disengaged freely, enabling a wide-variety of specific uses including CPU compute and even GPU acceleration.
Load the Package and Environment Manager
...
List Available Environments
Many Python suites, such as Pytorch or Qiime, are commonly-requested and thus are provided by Research Computing staff on the supercomputers already. These environments are version-fixed and read-only, so they may be used freely by any number of users simultaneously without any risk of the environment changing.
All global/admin-maintained python environments may be found under /rc/packages/envs:
| Code Block |
|---|
$ ls /rc/packages/envs/ pytorch-gpu-2.0.1/ scicomp/ tensorflow-gpu-2.12.1/ |
User environments are by default installed to /home~/asurite/.conda/envs, and after running module load mamba/latest, all available environments may be listed with mamba info --envs.
| Code Block |
|---|
[asureite@login1<asurite>@login1 ~]$ mamba info --envs mamba version : 1.5.1 # conda environments: # pytorchGPU /home/dmccaff4<asurite>/.conda/envs/pytorchGPU testing /home/dmccaff4<asurite>/.conda/envs/testing updateTest /home/dmccaff4<asurite>/.conda/envs/updateTest base /rc/packages/apps/mamba/1.5.1 pytorch-gpu-2.0.1 /rc/packages/envs/pytorch-gpu-2.0.1 scicomp /rc/packages/envs/scicomp tensorflow-gpu-2.12.1 /rc/packages/envs/tensorflow-gpu-2.12.1 |
Load Available Environments
Use the source activate command to load the environment you want.
| Code Block |
|---|
$ module load mamba/latest $ source activate gurobi-9.5.1pytorch-gpu-2.0.1 |
The name of the environment will appear to the left of the command prompt so that you know what environment is currently active.
| Code Block |
|---|
(pytorch-gpu-2.0.1) $ python nobel_prize.py |
...
| Panel | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
Environments may also be created by specifying the path, but be careful as creating environments in non-default locations makes it easy to lose the environment! |
| Info |
|---|
|
| Panel | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
When using mamba to install packages or create environments, you may see errors related to opening files in Always verify the It is also good practice to verify what is being installed as a new package, what existing packages are being modified, and what existing packages are being removed before proceeding with the install. Please review the |
...
To install a new package to a mamba environment, first load the mamba module, then activate the existing environment, and finally install as many new dependencies as required.
| Code Block |
|---|
$ module load mamba/latest $ source activate <environment_name> $ mamba install -c conda-forge [-c <channel>] <packages> |
...
Once an environment is created, a kernel interface will need to be made to have that environment available in Jupyter. This is as easy as, mkjupy <env_name>. Please review Using Preparing Python Environments for Jupyterfor additional details.
ADVANCED: Building from GitHub repository
Many python packages are not necessarily available on available mamba channels. It is best to avoid these packages when possible. However, it is possible to integrate them into a workflow. First, clone the git repository into your home directory:
$ git clone <url of github repository>
This URL can be copied from GitHub repository. In the figure below, the blue line indicates the URL of the corresponding repository (repo) page.
...
The cloned directory should include instructions for installing the Python package.
| Note |
|---|
Be sure that you’re either in an existing mamba environment or create a new one that supports the listed dependencies. TYPICALLY THE DEPENDENCIES ARE OVERSPECIFIED--dependency files are typically very fragile and non-portable, and include precise versions for second-order dependencies. If your build is failing, try to remove all but the first-order dependencies (e.g., installing a versioned pytorch will automatically install the most stable version of numpy). |
Once the environment is created and activated, and all dependencies installed, the new repository module may be installed as specified in the README, typically a pip install.
| Panel | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
Be very careful with |
| Warning |
|---|
Never use |
Additional Help
If you require further assistance on this topic, please don't hesitate to contact the Research Computing Team. To create a support ticket, kindly send an email to rtshelp@asu.edu. For quick inquiries, you're welcome to reach out via our #rc-support Slack Channel or attend our office hours for live assistance
...
What if the Package I Need Isn’t in Conda-forge?
Conda-forge is the only pre-approved place to get Python modules right now. Other modules will need to go through an approval process with the KE Secure Cloud team before they can be used. Contact asre-support@asu.edu to ask for a module.
| Insert excerpt | ||||||||
|---|---|---|---|---|---|---|---|---|
|