Building containers and using Singularity on Agave

Owned by Eric Tannehill (Unlicensed)
Last updated: Oct 13, 2023 by Ian Shaeffer
6 min read

New method available

There is now a free cloud process that can be used to build from a recipe that does not require root.  This process still uses all of the information from this page but adds some additional steps which we have documented at Using the Sylabs Builder tool

What is a container

Containers are similar to virtual machines.  They allow a user to run software that is otherwise incompatible with the native operating system.  For example, some scientific software is written to only be compatible with Ubuntu, but our systems only run CentOS.  Running the software inside a container provides the software an Ubuntu environment without re-imaging a cluster node.

The primary difference between containers and virtual machines is that containers are not completely independent, they don't do hardware virtualization, nor are they full operating system installs.  Instead, they rely on the native OS's handling for many functions such as hardware addressing, file handling, and more.

The advantage of this is that containers tend to be very small.  The base image is generally only a few hundred Mega Bytes plus any additional software that is incorporated into the image.

Containers further have a huge advantage in that they are portable.  Once you have a singularity image created, it will run on any other system that has singularity installed.  A Mac, CentOS, Ubuntu box, even other clusters will run your Singularity container with no modification.  You could take your singularity image and it would run perfectly on any other HPC cluster with Singularity installed.


What is Singularity and why not Docker?

Singularity is a container manager designed with HPC environments such as ASU's in mind.

Docker was designed with personal computers in mind, and as such, requires root access to perform any action.  This makes Docker incompatible with shared resource environments such as HPC.

The good news is that Singularity can run the vast majority of docker images with no issues.  It can even pull directly from Docker Hub

The most important functions of Singularity can generally be run without root access, specifically

  • Images can be pulled from Docker or Singularity Hub 
  • Containers can be run
  • Commands can be sent to the container from the shell
  • Local paths can be exported to a running container (/home/myuser : /scratch : /packages : etc )
  • Access GPU functions (if GPU is present on the selected node)
  • XWindows will parse to the XDisplay from the container
  • Foreign OS's will run properly within the container (Ubuntu, CentOS, SuSe etc)

Certain functions still require root and will require the user to perform these actions from their own PC, or the SYLABS builder tool/cloud prior to running on the cluster

  • Customizing an image
  • Creating a writable image
  • Building empty containers
  • Building from Singularity recipe files

Docker images and limited functionality

  • Singularity does not run the container network transparently like docker.  Network ports must be specifically forwarded on the command line when initiating an image
  • The ASU RC use policy does not permit running servers on nodes, therefore many ports are blocked on the nodes and network.
  • Small exceptions are provided for important functions such as databases to support the analysis software.  Generally, these services cannot bind to all IP's of a compute node, and are limited to only accessing the database server locally (cannot be shared between nodes)
  • The above limitations means images that attempt to run servers or bind ports of the native system may not work

Basic usage

In order to pull and use a container as-is

Step 4 can be skipped if you have an existing image file, such as if you have downloaded it from the internet, previously pulled, or built the image.

The biobakery lefse image is used only as an example, you will need to modify the commands for your image


  1. Log into agave ( ssh asurite@agave.asu.edu )
  2. Start an interactive session or add commands as needed to an SBATCH script
  3. load one of the singularity modules ( module load singularity/3.3.0-1 )
  4. Pull the image e.g.  from docker for the biobakery lefse image https://hub.docker.com/r/biobakery/lefse ( singularity pull docker://biobakery/lefse ) 
  5. run the image ( singularity run lefse_latest.sif ) 
  6. run your commands as you normally would inside the container

Importing a path

We don't want to have to copy all of our files into a singularity image.  Instead, it's often easier to just give Singularity access to a path within the container. 

By default only your home directory is accessible inside the container, however, we can make other paths accessible as well.  We can do this with the -B <source path>:<dest path> option

[user@cg1-2:~]$ module load singularity/3.3.0-1
[user@cg1-2:~]$ singularity run lefse.sif
[user@cg1-2:~]$ cd /scratch/user
bash: cd: /scratch/user: No such file or directory
[user@cg1-2:~]$ exit
[user@cg1-2:~]$ singularity run -B /scratch/user:/scratch/user lefse.sif
[user@cg1-2:~]$ cd /scratch/user
[user@cg1-2:/scratch/user]$ ls -la
total 10774371
drwxrwxr-x 2 user user          3 Dec  2 12:01 .
drwxr-xr-x 3 user user         60 Feb 10 09:40 ..
-rw-r--r-- 1 user user 8157644800 Dec  2 12:04 E6B-113X.tar
-rwxr-xr-x 1 user user  922681375 Sep 25 14:35 lefse.simg
-rw-rw-r-- 1 user user 1952628224 Nov  4 13:01 nagiosxi-5.6.7-64.ova
[user@cg1-2:/scratch/user]$ exit



Connecting to a GPU

Singularity will work with GPU's, however, when running the singularity image you will need to include an extra switch "--nv" 

e.g.
singularity run --nv /packages/7x/vmd/1.9.3/vmd.sif

Advanced usage

In the event you need to work with a container that requires you to customize an image, you will need to perform additional steps.

Because customizing requires root, this cannot be done on the cluster, however, once you have created the image file, you can transfer and run the image on the ASU HPC cluster.


To create a custom image you will first need to install Singularity on your local computer.

Building from a recipe

The below build is provided only as a practical example of a successful Singularity build used on Agave.  

You will need to modify the build script to match the software and container OS you will use

Since Singularity 3.0 previous commands such as the expand and create commands for images have been removed, meaning it is best to manage the image entirely from the build command.  Attempting to extensively modify an image once created while possible, will have less than optimal results.


The below recipe is one that was used to create a docker image for the "visit" software.  This software required a number of OS level improvements we couldn't accommodate as it would adversely affect other users.  Creating a singularity container allowed the user to bypass these limitations.


sample singularity recipe
Bootstrap: docker
From: centos:7

%setup
	## This section will process before an OS is installed and is not used in most cases.  
	## Unix commands generally don't work in this recipe section since there is no OS available to interpret yet

%files
	## The tar.gz file will be copied to /usr/local/bin

    /home/myasurite/visit/visit3_0_0.linux-x86_64-rhel7.tar.gz /usr/local/bin  

%post
	## These commands will be passed to the command line as is, now's your chance to unzip and run those installers

    yum -y install xorg-x11-xauth xterm xclock htop epel-release gcc anaconda mesa-libGL mesa-libGL-devel mesa-libGLU mesa-libGLU-devel which openmpi gcc-c++ wget libXcomposite libXcursor libXi libXtst libXrandr alsa-lib mesa-libEGL libXdamage libXScrnSaver bzip2 git llvm mpich llvm-devel
    yum -y install htop
    cd /usr/local/bin
    tar xvzf visit3_0_0.linux-x86_64-rhel7.tar.gz
    mv /usr/local/bin/visit3_0_0.linux-x86_64 /usr/local/bin/visit
	chmod 755 /usr/local/bin/visit/bin/visit_install.sh
	./visit_install.sh

%environment
	# These environment variables will be -PERMANENTLY- stored in the image and will apply with every use of the image

    export LC_ALL=C
    export PATH=/usr/local/bin/visit/3.0.0/linux-x86_64/bin:/usr/local/bin/visit/bin:$PATH


%runscript
	## This line is optional.  ANYTHING PUT IN HERE WILL RUN AUTOMATICALLY EVERY TIME THE IMAGE IS STARTED
	## Depending on how you configure this, this may lock you out of running anything else or using the command line

	visit &

%labels
	## This is metadata only and won't been seen unless someone specifically requests metadata  information from singularity

	VisIt is an Open Source, interactive, scalable, visualization, animation and analysis tool.


Once the build file is ready, the image will need to be created using the command below

singularity build <image_filename> <recipe_filename>



Once the build is complete, run the container and test on your PC to verify it is working as intended before uploading to our systems


For information on uploading to the cluster, see the wiki for transferring.



This document only covers specifics for ASU HPC and does not constitute complete information on using singularity.  For complete information visit the Singularity documentation at https://sylabs.io/docs/

Using the Sylabs builder tool

A separate write-up for the cloud builder tool can be found here.


Converting Dockerfiles/Images

Please note that depending on how the Docker image was created it may not run perfectly without additional singularity command line options.  These options will be unique to each docker image.  Consult the documentation for the Docker Image and cross reference with Singularity documentation to determine which options to use.

Building from a converted Dockerfile will always require root as it is building from a recipe.  See the information above on how to build from a Singularity recipe.

If you already have a docker image you can copy it and run it directly using:

singularity shell docker://ubuntu


If you want to convert a Dockerfile to build in Singularity, you can download the spython tool into your home directory.  Spython has many more uses but we're only concerned with the below command for converting

spython recipe Dockerfile.in >> Singularity.out


Filter by label

There are no items with the selected labels at this time.



Arizona State University