Overview

The Phoenix and Sol supercomputers both utilize high-speed interconnects supporting MPI. On Sol, the interconnect is called Infiniband.

Software that utilizes MPI must be built using parallelization-aware software compilers, which are ready for use through the module system.

When building software for MPI, the two critical choices for the user are:
a) compiler suite (e.g., gcc, aocc, intel, nvhpc, oneapi) and
b) MPI standard (e.g., OpenMPI, mpich, mvapich, intel-mpi).

Your application likely will provide guidance about which compiler to use, but if the preferred MPI is not given, it is reasonable to choose from the above list in order of appearance (with OpenMPI being the most general, compatible use-case).

Building Software

Instructions vary greatly depending on the software being built.

1. Identify a preferred compiler from the developer documentation

2. Start an interactive session with at least 4 cores

3. Search for the compiler among available modules using the command: module avail

4. Load the compiler module file, e.g., module load openmpi/4.1.5

5. Change directory to your build directory and run make (or other provided instructions)

Recommended & Supported Compilers

MPI Modules

All usable MPI compilers are the following and can be used with the normal module load syntax:

Loading Related Modules

Sometimes, you will want to load additional modules to work in concert with these compilers. Listed below is any additional information you might be interested to match if you want to use additional packages:

• module load openmpi/4.1.5
  Recommended openmpi/4.1.5 is compatible with module load gcc-11.2.0-gcc-11.2.0

• module load intel/parallel-studio-2020.4
  Intel Parallel Studio is wholly self-contained. It is not recommended to load other compiler-related modules in addition to this module.

• intel/intel-oneapi-2021.10.0
  Intel One API is also wholly self-contained. It is not recommended to load other compiler-related modules in addition to this module.

• mpich/4.1.2
  MPICH is built with the system-provided gcc-12.1.0-gcc-11.2.0. This means that mpich/4.1.2 does not need any additional module loads to compile non-mpi software. One caveat is that no compiler libraries are installed on login nodes; ensure you are on a compute node for this module to work.

• platform-mpi/09.01.04.03
  Platform-mpi is also wholly self-contained. It is not recommended to load other compiler-related modules in addition to this module. This is a niche compiler, and in most circumstances, other MPI modules will be preferred.

• nvhpc/22.7
  The NVidia HPC Compiler Collection is self-contained. This is a rebranded version of what used to be known as the PGI (Portland Group) compilers. Like with Platform MPI, most software will be compatible with the much-more commonly-known and supported compilers above, and nvhpc fits a narrow use-case.

• mvapich/3.0b(Not supported on Sol at the moment)
  MVAPICH is built with the system-provided GNU GCC 8.5.0. This means that mvapich/3.0b does not need any additional module loads to compile non-mpi software. One caveat is that no compiler libraries are installed on login nodes; ensure you are on a compute node for this module to work.

Choosing a Compiler

It is almost always preferable for the compiler to be selected based on provided documentation. However, there are reasons why different compilers may be considered:

1. Processor architecture mismatch
   Agave compute nodes are almost exclusively Intel processors where Sol nodes are exclusively AMD Epycs.

2. Compiler is unavailable

3. Compiler is incompatible with dependencies (interconnect or otherwise)

The gcc compiler suite is the most universal option and often the most supported compiler.

Choosing an Interconnect

OpenMPI is the most highly supported and successful general use case for Infiniband. Unless advised by the software itself, openmpi will almost certainly be the most straightforward way to build an app with MPI support.

Additional Help