Using anaconda environments

Owned by Eric Tannehill (Unlicensed)
Last updated: May 23, 2022
8 min read

Python is a very popular language for many scientific computing projects. Unfortunately python applications often have very different, and often conflicting package dependencies. This means that installing one application in python would often break other python applications.

For this reason RC no longer centrally maintains python applications and plugins. Instead we recommend using Anaconda. Anaconda still uses the exact same python so there are no issues with compatibility. Anaconda is a program that helps to manage python and enable use of conflicting python packages.

With Anaconda, it is no longer necessary for users to request the RC support team to install python applications and packages. The user can install the packages without need for root or sudo!

Instructions

Steps to create a new anaconda environment:

  1. Log into agave

  2. You may use the login node to create the environment but not to run the application once installed

  3. find and load an appropriate anaconda or conda module

    1. In the text box below the relevant commands are module avail anaconda and module load anaconda/py3

    2. In the text box below the relevant commands are module avail conda and module load conda/rolling

 

[user@cg1-3:~]$ module avail anaconda ------------------------ /usr/share/Modules/modulefiles ------------------------ anaconda/py2 anaconda2/4.2.0 anaconda2/4.4.0 anaconda3/4.2.0 anaconda3/4.4.0 anaconda/py3 anaconda2/4.3.1 anaconda2/5.2.0 anaconda3/4.3.1 anaconda3/5.3.0 [user@cg1-3:~]$ module load anaconda/py3 [user@cg1-3:~]$

OR

[user@cg1-3:~]$ module avail conda ------------------------ /usr/share/Modules/modulefiles ------------------------ conda/rolling [user@cg1-3:~]$ module load conda/rolling [user@cg1-3:~]$

We can now use anaconda or conda to create a custom environment. As an example only, we’re going to create an environment for an application used by many in biology called kaiju. We will call the environment kaiju_example

To create a conda environment the command is conda create -n ENV_NAME. An example is shown below with an additional -y flag, which automatically installs the standard software for the environment without prompting the user for input.

[user@cg1-3:~]$ conda create -y -n kaiju_example Collecting package metadata (repodata.json): done Solving environment: done ## Package Plan ## environment location: /home/user/.conda/envs/kaiju_example added / updated specs: - python=3 The following packages will be downloaded: package | build ---------------------------|----------------- _libgcc_mutex-0.1 | main 3 KB ... (~20 lines TRUNCATED FOR THIS EXAMPLE) zlib-1.2.11 | h7b6447c_3 120 KB ------------------------------------------------------------ Total: 47.3 MB The following NEW packages will be INSTALLED: _libgcc_mutex pkgs/main/linux-64::_libgcc_mutex-0.1-main ... (~20 lines TRUNCATED FOR THIS EXAMPLE) zlib pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3 Downloading and Extracting Packages tzdata-2020d | 121 KB | ############# | 100% ... (~20 lines TRUNCATED FOR THIS EXAMPLE) libffi-3.3 | 54 KB | ############# | 100% Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use # # $ conda activate kaiju_example # # To deactivate an active environment, use # # $ conda deactivate [user@cg1-3:~]$

Anaconda recommends the newer conda activate <environment> over the older source activate <environment> to load environments. However, the newer method makes permanent changes to the shell environment and should be avoided! In the examples below we will always use source activate <environment> to load a target python environment.

Do not use conda activate <environment> to load python environments!

Instead use source activate <environment> to load python environments!

Now that the environment is initialized, we will need to “activate” the environment and install kaiju.

We recommend using the conda installer where possible but will cover other methods later for when it is not possible. For kaiju, simply Googling conda install kaiju gives us the web page https://anaconda.org/bioconda/kaiju, and at the bottom of that web page, we are given the exact command we need to install kaiju using Anaconda: conda install -c bioconda kaiju.

Many python modules have web pages dedicated to them. These pages will often include detailed instructions on how to install the maintained module with pip and conda. Use conda where possible. Additionally, searching the anaconda cloud (which was what was found using google in the previous example) is a good first step in lieu of a search engine query.

In the below window the important commands are source activate kaiju_example and conda install -y -c bioconda kaiju, where the -y flag is passed again to automatically accept the install manifest.

Kaiju should now be ready to use within the kaiju_example conda environment!

Additional useful conda commands

 

command option

description

example full command

-n/--name

next word in command is the name of the environment you are working with

conda create -n MYENV

remove

remove a conda environment

conda remove -n MYENV --all

export

exports your environment to be copied to another computer (must be used with “env”)

conda env export > filename.yml

import

import environment from exported file

conda env import -n ENV_NAME < filename.yml

--revisions

can be used to show changes that have been made to the environment

conda list --revisions

install --revision=REVNUM

used to rollback to the previous revision number

conda install --revision=2

info --envs

list all environments available to you

conda info --envs

list

show packages in an environment

conda -n ENV_NAME list

--stack

activates a new environment while keeping old variables for nested or stacked environments

conda activate --stack ENV_NAME

--clone

clone an existing environment. often used to make changes without affecting a working environment

conda create --name NEW_ENV --clone OLD_ENV

Make conda environments available in Jupyter

To make your conda environment available in jupyter (accessed through the interactive app on login.rc.asu.edu) a few extras steps are necessary. These steps are simplified by a single provided shell command which is documented here: and .

ADVANCED: Installing packages without conda

The vast majority of packages can be found in the conda installer. Unfortunately there will still be cases where conda is not available. Not to worry though as there are still methods for installing to your custom environment using pip.

The default instructions for many python packages is to use pip. Because we are installing to a conda environment you will need to use your discretion to make changes as needed to the install instructions for your package.

A few notes on using pip to install

  • Try to find what the dependencies for your application are and install them using conda first

  • pip installs should be run with --upgrade-strategy only-if-needed flag

  • Do NOT use the --user flag in an install as this will override conda’s environment isolation

  • Create a text file with package requirements for the application to be installed

  • pass the package requirements to the conda installer with the --file $filename argument

  • the file can also be passed to pip with the -r $filename argument

  • because conda will not be aware of the changes made via pip, if an upgrade is to be done in the future, clone the environment first and perform the upgrades against the clone

To install in conda using pip

  1. activate your environment source activate ENV_NAME

  2. make sure pip is installed conda install -n ENV_NAME pip

  3. run the pip install commands with modifications from above as needed

Filter by label

There are no items with the selected labels at this time.

Arizona State University