The Agave and Sol supercomputers both utilize high-speed interconnects supporting MPI. On Sol, the interconnect is called Infiniband.

Software that utilizes MPI must be built using parallelization-aware software compilers, which are ready for use through the module system.

When building software for MPI, the two critical choices for the user are:
a) compiler suite (e.g., gcc, aocc, intel, nvhpc, oneapi) and
b) MPI standard (e.g., OpenMPI, mpich, mvapich, intel).

Your application likely will provide guidance about which compiler to use, but if the preferred MPI is not given, it is reasonable to choose from the above list in order of appearance (with OpenMPI being the most general, compatible use-case).

 Build overview

Instructions vary greatly depending on the software being built.

1. Identify a preferred compiler from the developer documentation

2. Search for the compiler among available modules using the command: module avail

3. Load the compiler modulefile, e.g., module load openmpi/4.1.5

4. Change directory to your build directory and run make (or other provided instruction)

Recommended & Supported Compilers

GOOD MODULES

All usable MPI compilers are the following and can be used with the normal module load syntax:

A quick way to check if it is a recommended and supported compiler is if the module has the format: modulename/version; this is in comparison to modulename-version-compiler-version, which signify all the now-deprecated modules.

Loading Related Modules

Sometimes, you will want to load additional modules to work in concert with these compilers. Listed below is any additional information you might be interested to match if you want to use additional packages:

• module load openmpi/4.1.5
  Recommended openmpi/4.1.5 is compatible with module load gcc-11.2.0-gcc-11.2.0

• module load intel/parallel-studio-2020.4
  Intel Parallel Studio is wholly self-contained. It is not recommended to load other compiler-related modules in addition to this module.

• intel/intel-oneapi-2022.1.0
  Intel One API is also wholly self-contained. It is not recommended to load other compiler-related modules in addition to this module.

• mpich/4.1.2
  MPICH is built with the system-provided GNU GCC 8.5.0. This means that mpich/4.1.2 does not need any additional module loads to compile non-mpi software. One caveat is that no compiler libraries are installed on login nodes; ensure you are on a compute node for this module to work.

• mvapich/3.0b
  MVAPICH is built with the system-provided GNU GCC 8.5.0. This means that mvapich/3.0b does not need any additional module loads to compile non-mpi software. One caveat is that no compiler libraries are installed on login nodes; ensure you are on a compute node for this module to work.

• platform-mpi/09.01.04.03
  Platform-mpi is also wholly self-contained. It is not recommended to load other compiler-related modules in addition to this module. This is a niche compiler, and in most circumstances, other MPI modules will be preferred.

• nvhpc/22.5 (22.7)
  The NVidia HPC Compiler Collection is self-contained. This is a rebranded version of what used to be known as the PGI (Portland Group) compilers. Like with Platform MPI, most software will be compatible with the much-more commonly-known and supported compilers above, and nvhpc fits a narrow use-case.

OBSOLETE MODULES

As an example, you will want to avoid further use of MPI modules matching this format:

Choosing a Compiler

It is almost always preferable for the compiler to be selected based on provided documentation. However, there are reasons why different compilers may be considered:

1. Processor architecture mismatch
   Agave compute nodes are almost exclusively Intel processors where Sol nodes are exclusively AMD Epycs.

2. Compiler is unavailable

3. Compiler is incompatible with dependencies (interconnect or otherwise)

The gcc compiler suite is the most universal option and often the most supported compiler.

Choosing an Interconnect

OpenMPI is the most highly-supported and successful general use case for Infiniband. Unless advised by the software itself, openmpi is almost certainly going to be the most straightforward way to build an app with MPI support.