Building Software for HPC Environments

A High-Performance Computing (HPC) environment like the Agave and Sol supercomputer is vastly different from a desktop environment, like your personal workstation. Building software to run for HPC adds considerations in a few areas: architecture, interconnects, binary compatibility, and permissions, to name a few. These considerations make it more time-consuming and complicated to successfully build software.

Research Computing will make a best-effort attempt to install software upon request. Ultimately, it is the user’s responsibility to find a viable solution for their workflow whether by compiling the code manually, using others' pre-compiled binaries or containers, or identifying alternative software.

Some software is central to a HPC environment, such as compiler suites, Python, R, and numerous other libraries; Research Computing admins will always install and make Environment Modules to allow quick and easy access to these resources. The majority of software install requests can be easily completed by RC Admins and a new module can be created on their behalf, publicly available to all users of the supercomputer.

In some cases, such as extremely domain-specific software, largely-untested code, and development versions of software, it may be imprudent to create public Environment modules. Research Computing admins can still assist by building this software in the user’s HOME directory, or in the group share directory /data/grp_XXXX where XXXX is the user’s group (visible with the command groups). Note, there is no functional difference between an installation in a user’s HOME versus one built on shared storage (e.g., /data/grp_, /packages/) as an environment module simply adds environment variables to your terminal session. This allows convenient access to the binaries and proper environment configuration. All groups have this share and all users in the group share this space; it can be used freely up to the limit of 100GB, which is a hard cap (additional storage must be purchased).

Requesting software to be built

Requests for software installations can be made in a support ticket.

Be sure to include all pertinent information:

software URL
version requirements
plugin requirements
location of downloaded source code (if behind paywall or login)

Remember, the supercomputer is 100% Linux; Windows software is not possible, and software requiring compiling ultimately is the responsibility of the user.

Sol currently uses Rocky Linux 8.x; software explicitly requiring Debian, Ubuntu, etc., may compile and a best-effort attempt will be attempted. Beyond that, seek a container solution.

All systems in Research Computing supercomputers run Rocky Linux, a Red-hat variant and use the package manager dnf/yum. apt/apt-get are not available as these exist for Ubuntu/Debian software. If you have software instructing to use this, check if alternative distributions are supported or else the libraries required by apt/apt-get would need to be installed manually to your own $HOME storage space.

Compiling software from source

Luckily, many of the compilation considerations mentioned above can be handled automatically, with provided build scripts and makefiles from the software author. At the minimum, an understanding of Linux filesystem permissions is required to build software, as most software defaults to installing files in locations not permissible by regular, non-admin users.

Filesystem Permissions & Shared Storage

An abbreviated--but valid--installation of python might go like this:

wget https://www.python.org/ftp/python/3.9.7/Python-3.9.7.tgz
tar -xf Python-3.9.7.tgz
cd Python-3.9.7
./configure
make
make install

These steps will work all the way until make install, when it attempts to install to /usr/local/python-3.9.7 and gets a Permission Denied error.

This error is not an indication of a failed build, but simple a failure to copy the binaries to root-owned paths on a node. Many users may recognize this as a sudo-solvable problem, but in HPC environments, that is not the case.

sudo access

sudo access is reserved explicitly for Research Computing admins. Please note, if you are installing software and you get permission errors trying to install to a local directory: stop. Using sudo to overcome permissions issues would allow you to install to a local directory on a specific node--however, software is not designed to be installed in local directories in HPC environments, as none of these files will be available on your next job allocation, where you are likely to be placed on another node altogether.

sudo is inappropriate for software installations in HPC environments in 100% of cases.

Instead, install software to shared filesystems (filesystems that every node in the supercomputer can reach). Such locations include your $HOME directory, your group shared space, and $SCRATCH filesystems.

Changing build directories

The above installation could therefore be adjusted in a very simple way to install to your HOME directory:

...
$ ./configure --prefix=/home/[asurite]/.local/python-3.9.7 
$ make
$ make install

Container solutions

Apptainer (aka Singularity) is available on the RC supercomputers. Many times, software is distributed in an easy-to-download package (.sif file) which can be uploaded to your HOME directory and used immediately.

In some cases where a pre-configured Singularity is not available, but one can be built with a recipe, the container must be built on your local workstation (rather than the supercomputer), due to sudo/ root requirements.