{"type":"doc","content":[{"type":"extension","attrs":{"layout":"default","extensionType":"com.atlassian.confluence.macro.core","extensionKey":"toc","parameters":{"macroParams":{},"macroMetadata":{"macroId":{"value":"c2c7253b-069d-4b9b-940b-43c9c9d27432"},"schemaVersion":{"value":"1"},"title":"Table of Contents"}},"localId":"266655cb-580b-421a-bf52-96846333e9f2"}},{"type":"heading","attrs":{"level":1},"content":[{"text":"Overview","type":"text"}]},{"type":"paragraph","content":[{"text":"The supercomputer uses Mamba, a high-performance parallel package manager, to allow users to install the Python modules they need. It also plays a pivotal role in optimizing software environments on supercomputers. In the upcoming instructions, we'll explore the process of loading Mamba modules and delve into creating and loading new environments.","type":"text"}]},{"type":"paragraph","content":[{"text":"The supercomputers use ","type":"text"},{"text":"mamba","type":"text","marks":[{"type":"code"}]},{"text":" instead of ","type":"text"},{"text":"conda","type":"text","marks":[{"type":"code"}]},{"text":" and ","type":"text"},{"text":"pip","type":"text","marks":[{"type":"code"}]},{"text":". ","type":"text"},{"text":"mamba","type":"text","marks":[{"type":"code"}]},{"text":" is a parallel and C++ implementation of ","type":"text"},{"text":"conda","type":"text","marks":[{"type":"code"}]},{"text":" that provides a much faster experience for setting up Python environments on the supercomputer. If you’re familiar with using ","type":"text"},{"text":"conda","type":"text","marks":[{"type":"code"}]},{"text":", all you will need to do is replace the ","type":"text"},{"text":"conda","type":"text","marks":[{"type":"code"}]},{"text":" command with the ","type":"text"},{"text":"mamba","type":"text","marks":[{"type":"code"}]},{"text":" command. ","type":"text"},{"text":"pip","type":"text","marks":[{"type":"code"}]},{"text":" is not stable in a multi-user environment like a supercomputer, so Research Computing discourages the use of ","type":"text"},{"text":"pip","type":"text","marks":[{"type":"code"}]},{"text":" except when necessary.","type":"text"}]},{"type":"panel","attrs":{"panelIconId":"26a0","panelIcon":":warning:","panelIconText":"⚠️","panelColor":"#FFF0B3","panelType":"custom"},"content":[{"type":"paragraph","content":[{"text":"Be very careful with ","type":"text"},{"text":"pip","type":"text","marks":[{"type":"code"}]},{"text":", it can easily break a mamba environment!","type":"text"}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Why Create a New Environment?","type":"text"}]},{"type":"paragraph","content":[{"text":"In a fresh terminal session, ","type":"text"},{"text":"python","type":"text","marks":[{"type":"code"}]},{"text":" or ","type":"text"},{"text":"python3","type":"text","marks":[{"type":"code"}]},{"text":" points to a system-installed copy of Python (typically in ","type":"text"},{"text":"/usr/bin)","type":"text","marks":[{"type":"code"}]},{"text":". As the operating system heavily depends on this Python instance, the version is fixed and only the most basic, built-in libraries are available.","type":"text"}]},{"type":"paragraph","content":[{"text":"Creating a new environment allows you to have full control over the Python version, the selection of libraries, and the specific versions, too. Python environments can then be engaged and disengaged freely, enabling a wide-variety of specific uses including CPU compute and even GPU acceleration.","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Load the Package and Environment Manager","type":"text"}]},{"type":"paragraph","content":[{"text":"Load the latest stable version of the ","type":"text"},{"text":"Mamba","type":"text","marks":[{"type":"link","attrs":{"href":"https://mamba.readthedocs.io/en/latest/user_guide/mamba.html"}}]},{"text":" Python manager with:","type":"text"}]},{"type":"paragraph","content":[{"text":"module load mamba/latest","type":"text","marks":[{"type":"code"}]}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"List Available Environments","type":"text"}]},{"type":"paragraph","content":[{"text":"Many Python suites, such as ","type":"text"},{"text":"Pytorch","type":"text","marks":[{"type":"code"}]},{"text":" or ","type":"text"},{"text":"Qiime","type":"text","marks":[{"type":"code"}]},{"text":", are commonly-requested and thus are provided by Research Computing staff on the supercomputers already. These environments are version-fixed and read-only, so they may be used freely by any number of users simultaneously without any risk of the environment changing.","type":"text"}]},{"type":"paragraph","content":[{"text":"All global/admin-maintained Python environments may be found under ","type":"text"},{"text":"/rc/packages/envs","type":"text","marks":[{"type":"code"}]},{"text":":","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ ls /rc/packages/envs/\n pytorch-gpu-2.0.1/ scicomp/ tensorflow-gpu-2.12.1/ ","type":"text"}]},{"type":"paragraph","content":[{"text":"User environments are by default installed to ","type":"text"},{"text":"~/.conda/envs","type":"text","marks":[{"type":"code"}]},{"text":", and after running ","type":"text"},{"text":"module load mamba/latest","type":"text","marks":[{"type":"code"}]},{"text":", all available environments may be listed with ","type":"text"},{"text":"mamba info --envs","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"codeBlock","content":[{"text":"[@login1 ~]$ mamba info --envs\n\n mamba version : 1.5.1\n# conda environments:\n#\npytorchGPU /home//.conda/envs/pytorchGPU\ntesting /home//.conda/envs/testing\nupdateTest /home//.conda/envs/updateTest\nbase /rc/packages/apps/mamba/1.5.1\npytorch-gpu-2.0.1 /rc/packages/envs/pytorch-gpu-2.0.1\nscicomp /rc/packages/envs/scicomp\n...","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"Load Available Environments","type":"text"}]},{"type":"paragraph","content":[{"text":"Use the ","type":"text"},{"text":"source activate","type":"text","marks":[{"type":"code"}]},{"text":" command to load the environment you want.","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ module load mamba/latest\n$ source activate pytorch-gpu-2.0.1","type":"text"}]},{"type":"paragraph","content":[{"text":"The name of the environment will appear to the left of the command prompt so that you know what environment is currently active.","type":"text"}]},{"type":"codeBlock","content":[{"text":"(pytorch-gpu-2.0.1) $ python nobel_prize.py","type":"text"}]},{"type":"panel","attrs":{"panelType":"note"},"content":[{"type":"paragraph","content":[{"text":"Environments may also be activated with a full path, e.g.,","type":"text"}]},{"type":"paragraph","content":[{"text":"source activate /project/sciencelab/.conda/envs/pysci","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]},{"type":"paragraph","content":[{"text":"This capability makes ","type":"text"},{"text":"/project","type":"text","marks":[{"type":"code"}]},{"text":" (","type":"text"},{"text":"project storage","type":"text","marks":[{"type":"link","attrs":{"href":"/wiki/spaces/KESC/pages/1914208384"}}]},{"text":") an ideal location for groups sharing Python environments!","type":"text"}]}]},{"type":"panel","attrs":{"panelIconId":"atlassian-cross_mark","panelIcon":":cross_mark:","panelColor":"#FFEBE6","panelType":"custom"},"content":[{"type":"paragraph","content":[{"text":"Do not load environments with ","type":"text"},{"text":"conda","type":"text","marks":[{"type":"code"}]},{"text":" or ","type":"text"},{"text":"mamba","type":"text","marks":[{"type":"code"}]},{"text":" as the prefix, i.e., ","type":"text"},{"text":"mamba activate gurobi-9.5.1","type":"text","marks":[{"type":"code"}]},{"text":", as this injects non-supercomputing friendly cruft into your supercomputing configuration files!","type":"text"}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Creating Environments","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ module load mamba/latest\n$ mamba create -n -c conda-forge [-c ] [packages]\n$ source activate ","type":"text"}]},{"type":"panel","attrs":{"panelType":"success"},"content":[{"type":"paragraph","content":[{"text":"It is best to install all necessary packages at this step, as it maximizes environment stability and minimizes total build time to have all major dependencies resolved initially.","type":"text"}]}]},{"type":"panel","attrs":{"panelIconId":"26a0","panelIcon":":warning:","panelIconText":"⚠","panelColor":"#FFF0B3","panelType":"custom"},"content":[{"type":"paragraph","content":[{"text":"Environments may also be created by specifying the path, but ","type":"text"},{"text":"be careful","type":"text","marks":[{"type":"strong"}]},{"text":" as creating environments in non-default locations makes it easy to lose the environment!","type":"text"}]}]},{"type":"panel","attrs":{"panelType":"info"},"content":[{"type":"paragraph","content":[{"text":"conda-forge","type":"text","marks":[{"type":"code"}]},{"text":" is the only available channel on Aloe. Other channels are blocked.","type":"text"}]}]},{"type":"panel","attrs":{"panelIconId":"1f4d4","panelIcon":":notebook_with_decorative_cover:","panelIconText":"📔","panelColor":"#FFBDAD","panelType":"custom"},"content":[{"type":"paragraph","content":[{"text":"When using mamba to install packages or create environments, you may see errors related to opening files in ","type":"text"},{"text":"/packages/apps/mamba","type":"text","marks":[{"type":"code"}]},{"text":". These errors are harmless. An example is shown below.","type":"text"}]},{"type":"paragraph","content":[{"text":"Always verify the ","type":"text"},{"text":"Prefix: ","type":"text","marks":[{"type":"code"}]},{"text":"is pointing where you need it to before proceeding with an installation, but otherwise, errors and warnings made by ","type":"text"},{"text":"mamba","type":"text","marks":[{"type":"code"}]},{"text":" may be ignored.","type":"text"}]},{"type":"paragraph","content":[{"text":"It is also good practice to verify what is being installed as a new package, what existing packages are being modified, and what existing packages are being removed ","type":"text"},{"text":"before proceeding","type":"text","marks":[{"type":"strong"}]},{"text":" with the install.","type":"text"}]},{"type":"paragraph","content":[{"text":"Please review the ","type":"text"},{"text":"mamba install","type":"text","marks":[{"type":"code"}]},{"text":" section below for a summary of the components of ","type":"text"},{"text":"mamba install","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Adding dependencies to existing Environments","type":"text"}]},{"type":"panel","attrs":{"panelType":"warning"},"content":[{"type":"paragraph","content":[{"text":"The global/admin-maintained environments are read-only and can’t be changed by users. To add packages to one of these environments, you will need to clone it.","type":"text"}]},{"type":"paragraph","content":[{"text":"mamba create --name --clone ","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"This will create an environment called","type":"text"},{"text":" ","type":"text","marks":[{"type":"code"}]},{"text":" that is copied from ","type":"text"},{"text":"","type":"text","marks":[{"type":"code"}]},{"text":".","type":"text"}]}]},{"type":"paragraph","content":[{"text":"To install a new package to a mamba environment, first load the mamba module, then activate the existing environment, and finally install ","type":"text"},{"text":"as many new dependencies","type":"text","marks":[{"type":"strong"}]},{"text":" as required.","type":"text"}]},{"type":"codeBlock","content":[{"text":"$ module load mamba/latest\n$ source activate \n$ mamba install -c conda-forge [-c ] ","type":"text"}]},{"type":"panel","attrs":{"panelType":"success"},"content":[{"type":"paragraph","content":[{"text":"Please review the screenshot of an example ","type":"text"},{"text":"mamba install","type":"text","marks":[{"type":"code"}]},{"text":" below before proceeding. Annotations are shown in cyan with a black background and cyan outline.","type":"text"}]},{"type":"paragraph","content":[{"text":"When using mamba to install packages or create environments, you may see errors related to opening files in ","type":"text"},{"text":"/packages/apps/mamba","type":"text","marks":[{"type":"code"}]},{"text":". These errors are harmless. An example is shown below.","type":"text"}]},{"type":"paragraph","content":[{"text":"Always verify the ","type":"text"},{"text":"Prefix: ","type":"text","marks":[{"type":"code"}]},{"text":"is pointing where you need it to before proceeding with an installation, but otherwise, errors and warnings made by ","type":"text"},{"text":"mamba","type":"text","marks":[{"type":"code"}]},{"text":" may be ignored.","type":"text"}]},{"type":"paragraph","content":[{"text":"It is also good practice to verify what is being installed as a new package, what existing packages are being modified, and what existing packages are being removed ","type":"text"},{"text":"before proceeding","type":"text","marks":[{"type":"strong"}]},{"text":" with the install.","type":"text"}]}]},{"type":"mediaSingle","attrs":{"layout":"center","width":760,"widthType":"pixel"},"content":[{"type":"media","attrs":{"width":1226,"id":"a57543b0-b86f-4194-9ad3-5aec8efa51a0","collection":"contentId-1914667538","type":"file","height":1551}}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"Using environments in Jupyter","type":"text"}]},{"type":"paragraph","content":[{"text":"Once an environment is created, a ","type":"text"},{"text":"kernel interface","type":"text","marks":[{"type":"em"}]},{"text":" will need to be made to have that environment available in Jupyter. This is as easy as, ","type":"text"},{"text":"mkjupy ","type":"text","marks":[{"type":"code"}]},{"text":". Please review ","type":"text"},{"type":"inlineCard","attrs":{"url":"https://asurc.atlassian.net/wiki/spaces/KESC/pages/1915224178"}},{"text":"for additional details.","type":"text"}]},{"type":"heading","attrs":{"level":1},"content":[{"text":"What if the Package I Need Isn’t in Conda-forge?","type":"text"}]},{"type":"paragraph","content":[{"text":"Conda-forge is the only pre-approved place to get Python modules right now. Other modules will need to go through an approval process with the KE Secure Cloud team before they can be used. Open a request at ","type":"text"},{"text":"https://links.asu.edu/kesc-support","type":"text","marks":[{"type":"link","attrs":{"href":"https://links.asu.edu/kesc-support"}}]},{"text":" to ask for a module.","type":"text"}]},{"type":"paragraph","content":[{"type":"inlineExtension","attrs":{"extensionType":"com.atlassian.confluence.macro.core","extensionKey":"excerpt-include","parameters":{"macroParams":{"":{"value":"Additional Help Excerpt"},"name":{"value":"help"},"nopanel":{"value":"true"}},"macroMetadata":{"macroId":{"value":"83a70947-783a-455c-97fd-31153dbb3589"},"schemaVersion":{"value":"1"},"title":"Insert excerpt"}},"localId":"e3988cab-24d5-4887-be31-6fa0c32933db"}}]}],"version":1}

Browser not supported