Using Abaqus on Supercomputers
- Josh Burks
- rahull1
Overview
Abaqus is powerful engineering simulation software and is available for use on supercomputers.
License
Abauqs is proprietary software and requires a license from Engineering Technical Services (ETS) for use by researchers. If you intend to conduct simulations for research, it's essential to obtain a research license from ETS. To request a license, please use this link:https://ets.engineering.asu.edu/software-page/software/abaqus-for-research/
Once a license is obtained, run the following command to enable the research license
echo "academic=RESEARCH" > ~/abaqus_v6.envSerial
Abaqus has the capability to run on a single node, utilizing up to 128 cores on Sol. Below is an example of a single node sbatch script for executing Abaqus:
#!/bin/bash
#SBATCH -N 1
#SBATCH --exclusive
#SBATCH --ntasks-per-node=1
#SBATCH -p htc
#SBATCH -q public
#SBATCH -t 0-4
#SBATCH -o slurm.%j.out # STDOUT (%j = JobId)
#SBATCH -e slurm.%j.err # STDERR (%j = JobId)
#SBATCH --export=NONE
##SBATCH --mail-type=ALL # Send a notification when the job starts, stops, or fails
##SBATCH --mail-user=%u@asu.edu # send-to address
module load abaqus/2023
#Modify this to match your job!
abaqus_opts=(
job=test_50x50x20_f7
input=../50x50x20_f7.inp
user=../coupled_uel9.f
)
abq2023 "${abaqus_opts[@]}" cpus=$SLURM_CPUS_PER_TASK interactiveParallel
Abaqus has the capability to run across multiple nodes, allowing it to utilize more cores than those available on a single node.
Here is an example SBATCH for a multinode Abaqus job:
#!/bin/bash
#SBATCH -N 4
#SBATCH --exclusive
#SBATCH --ntasks-per-node=1
#SBATCH -p htc
#SBATCH -q public
#SBATCH -t 0-4
#SBATCH -o slurm.%j.out # STDOUT (%j = JobId)
#SBATCH -e slurm.%j.err # STDERR (%j = JobId)
#SBATCH --export=NONE
##SBATCH --mail-type=ALL # Send a notification when the job starts, stops, or fails
##SBATCH --mail-user=%u@asu.edu # send-to address
module load abaqus/2023 intel/oneapi
#Modify this to match your job!
abaqus_opts=(
job=test_50x50x20_f7
input=../50x50x20_f7.inp
user=../coupled_uel9.f
mp_mode=mpi
)
##############################################################
# DO NOT MODIFY BEYOND THIS UNLESS YOU KNOW WHAT YOU ARE DOING
##############################################################
unset SLURM_GTIDS
# dump the hosts to a text file
HOSTS_FILE=slurm-hosts-${SLURM_JOBID}.out
NPROC_PER_NODE=$(nproc)
# generate the mp_host_list environment variable
echo '[' > ${HOSTS_FILE}
srun -c $NPROC_PER_NODE /bin/bash -c 'echo [\"$(hostname)\",$(nproc)],' >> ${HOSTS_FILE}
echo ']' >> ${HOSTS_FILE}
mapfile -t mp_host_list < ${HOSTS_FILE}
mp_host_list=$(echo ${mp_host_list[@]})
TOTAL_CPUS=$(
sed -e 's/\],//g' "${HOSTS_FILE}" \
| awk -F ',' '{s+=$NF} END{print s}'
)
# write the abaqus environment file
ABAQUS_ENV_FILE="abaqus_v6.env"
cat > ${ABAQUS_ENV_FILE} << EOF
import os
os.environ['ABA_BATCH_OVERRIDE'] = '1'
verbose=3
mp_host_list=${mp_host_list}
mp_mpi_implementation=IMPI
mp_mpirun_path={IMPI:'$(which mpiexec.hydra)'}
if 'SLURM_PROCID' in os.environ:
del os.environ['SLURM_PROCID']
EOF
cat $ABAQUS_ENV_FILE
export TMPDIR="/scratch/$USER/abaqus-tmp"
mkdir $TMPDIR
export I_MPI_HYDRA_BOOTSTRAP=slurm
# Launch Abaqus
abq2023 "${abaqus_opts[@]}" cpus=$TOTAL_CPUS interactive