Depending on the MPI stack used to build the application, the process of running MPI software will differ. Below is a concise guide outlining the various MPI stacks along with the corresponding module and run commands to employ.
openmpi 4.1.5 / hpcx 2.17.1
Load with: module load openmpi/4.1.5 or module load hpcx/2.17.1
...
Example script: /packages/public/phx-sbatch-templates/template/example-mpi-with-hpcx-job/main.sh
mpich 4.1.2
Load with: module load mpich/4.1.2
...
Example script: /packages/public/phx-sbatch-templates/template/example-mpi-with-mpich-job/main.sh
Note: If using the Hydra Process ManagerYou may need to add export HYDRA_LAUNCHER_EXTRA_ARGS="--export=ALL"
intel oneapi 2022.1.0
Load with: module load intel/oneapi
...
Example script: /packages/public/phx-sbatch-templates/template/example-mpi-with-intel-job/main.sh
Note: There have been seen a few cases where users have needed to add the following to their job scripts:
export HYDRA_LAUNCHER_EXTRA_ARGS="--export=ALL"
export FI_PROVIDER=verbs #Multi-node jobs only
export FI_PROVIDER=smh #Single-node jobs only