Creating SBATCH scripts

sbatch scripts are the normal way to submit a non-interactive job to the supercomputer.

Below is an example of an sbatch script, that should be saved as the file myscript.sh.

This script performs performs the simple task of generating a file of sorted uniformly distributed random numbers with the shell, plotting it with python, and then e-mailing the plot to the script owner.

#!/bin/bash
 
#SBATCH -N 1  # number of nodes
#SBATCH -c 1  # number of cores to allocate
#SBATCH -t 0-00:02:00   # time in d-hh:mm:ss
#SBATCH -p serial       # partition 
#SBATCH -q normal       # QOS
#SBATCH -o slurm.%j.out # file to save job's STDOUT (%j = JobId)
#SBATCH -e slurm.%j.err # file to save job's STDERR (%j = JobId)
#SBATCH --mail-type=ALL # Send an e-mail when a job starts, stops, or fails
#SBATCH --export=NONE   # Purge the job-submitting shell environment

# Always purge modules to ensure consistent environments
module purge    
# Load required modules for job's environment
module load anaconda/py3
##
# Generate 1,000,000 random numbers in bash,
#   then store sorted results 
#   in `Distribution.txt`
##
for i in $(seq 1 1e6); do
  printf "%d\n" $RANDOM
done | sort -n > Distribution.txt
# Plot Histogram using python and a heredoc
python << EOF
import pandas as pd, seaborn as sns
sns.mpl.use('Agg')
sns.set(color_codes=True)
df = pd.read_csv('Distribution.txt',header=None,names=['rand'])
sns.distplot(df,kde=False,rug=True)
sns.mpl.pyplot.xlim(0,df['rand'].max())
sns.mpl.pyplot.xlabel('Integer Result')
sns.mpl.pyplot.ylabel('Count')
sns.mpl.pyplot.title('Sampled Uniform Distribution')
sns.mpl.pyplot.savefig('Histogram.png')
EOF
# E-mail diagnostic results to yourself using mailserver and a heredoc
mail -a 'Histogram.png' -s "Histogram Plot" ${USER}@asu.edu << EOF
Hello me,
See the attached Histogram.png for a visualization of the computed results.
EOF

This script uses the #SBATCH flag to specify a few key options:

The number of tasks the job will create:
- #SBATCH -n 1
The runtime of the job in Days-Hours:Minutes (N.B. max wall time is 7 days):
- #SBATCH -t 0-12:00
A file based on the jobid %j where the normal output of the program (STDOUT) should be saved:
- #SBATCH -o slurm.%j.out
A file based on the jobid %j where the error output of the program (STDERR) should be saved:
- #SBATCH -e slurm.%j.err
That email notifications should be sent out when the job starts, ends, or when it fails:
- #SBATCH --mail-type=ALL
The address where email should be sent (%u is automatically replaced with submitter’s username, which is by default the user’s asurite):
- #SBATCH --mail-user=%u@asu.edu
A purge of the shell environment otherwise inherited from the job-submission shell
- #SBATCH --export=NONE
It is a good practice to always purge any modules that may have been loaded before an sbatch script was submitted as they will still be active:
- module purge
This loads python so that it is the active version:
- module load anaconda/py3

Running this script results in an email to the submitting user with the file Histogram.png attached, which is shown below:

How do I run this script?

Assuming that the above has been saved as the file myscript.sh, and you intend to run it on the conventional compute nodes, you can submit this script with the following command:

sbatch myscript.sh

The job will be run on the first available group partition.

Problems with batch files created under Windows

If you are creating an sbatchscript on a Windows system which you then move to a Linux-based supercomputer, you may see the following error when you try to run it:

sbatch: error: Batch script contains DOS line breaks (\r\n)
sbatch: error: instead of expected UNIX line breaks (\n).

This is because of a difference between the way that Windows and Unix systems define the end of a line in a text file.

If you see this error, the dos2unix utility can convert the script to the proper format.

Adding files to the supercomputer can be found here: https://asurc.atlassian.net/wiki/spaces/RC/pages/45449275

Additional sbatch options

The below options can permit greater control of resource allocation. They are provided here for reference.

`--mem=[0|4G|4000M]`

This option requests a specific amount of memory per node. --mem=0 requests all the memory on the node.

If you request two nodes with -N, each of them will get the requested amount of memory assigned for the job. e.g. -N 2 --mem=4G would allocate a single core on two different nodes for a total of 2 cores and 8G of ram.

This option can override the -c or -n options. For instance, the default in the standard "parallel" partition is ~4.5GiB per core. If you set -c 1 and --mem=6G you will get assigned 2 cores to meet the amount of memory needed for your job

As a general rule only one of --mem, -n, or -c should be used in an sbatch script as if there's a conflict your job will fail without running (and provide a cryptic error) [this is not generally true for hybrid workflows, e.g., MPI + OpenMP.]

`-N X`

Requests exactly X number of nodes for the job. Unless you are running an MPI job this option should not be used. The default is 1 node per job and will not use more than 1 node unless the requested number of cores cannot be met with the other options specified.

ex1 without specifying -N

If you had -c 28 the system would only ever assign you one node as all the nodes in the parallel & serial partitions have at least 28 cores.

If you set -n 30 the system may assign you up to 1 core on as many as 30 different nodes, as a single node (in the parallel partition) cannot supply more than 28 cores.

ex2 specifying -N

If you ran a job with -N 3 -c 10 you would get exactly 3 nodes and a total of 10 cores, however, those cores may not be evenly distributed. In this case, you could end up with Node1=1core, Node2=2cores, Node3=8cores

If you ran a job with -N 2 -n 56 you would get exactly 2 nodes with a total of 56 cores. This is useful for MPI jobs to reduce the amount of potential network traffic and increase overall throughput.

Avoid using -N and -n unless your workflow is capable of utilizing multiple nodes for a distributed memory task.

`--exclusive`

This will allocate all the cores on a node (without having to explicitly ask for the exact number of cores). To allocate all RAM, be sure to include --mem=0.

-C

This option “constrains” your job to only nodes that meet particular “features” as have been designated by the RC team. This option may be useful in the event a job is failing because of incompatible hardware of some type.

Please note that not all partitions have all features available. Specifying a partition and requesting a feature not available in that partition will cause your job to fail.

Forcing specific hardware types may also delay your job from starting by a considerable amount, as the pool of resources is much smaller when requesting specific features

Specified features

PHI
Serial (No MPI available)
OPA (Omnipath adaptors for MPI)
IB (Infiniband adaptors for MPI)
MPI (MPI capable)