Submitting a Jupyter Notebook as an sbatch job

Owned by Juan Jose Garcia Mesa
Last updated: Dec 02, 2024
3 min read

This guide walks you through the process of converting a Jupyter Notebook into a Python
script and running it on Sol. This is useful when you have a long-running, computationally-intensive script that you want to run on Sol.

Step 1: Convert the Jupyter Notebook to a Python script

To convert a Jupyter notebook (e.g, example_notebook.ipynb) to a Python script on Sol, you can request a short Jupyter session and click File -> Save and Export Notebook As ... -> Executable script. After the script is downloaded to your local machine, you can upload it back to Sol by going to https://sol.asu.edu and using the Files tab.

Alternatively, you can convert the notebook to a Python script using the terminal by following these steps:

  1. Open a command line interface on Sol by navigating to sol.asu.edu on your browser and selecting the "Sol Shell Access" option from the "System" menu option or by SSHing into Sol using the command ssh <asurite>@sol.asu.edu.

  2. Request a lightwork compute node by running the command aux-interactive.

  3. Load the Jupyter module by running the command module load jupyter/latest.

  4. Convert the Jupyter Notebook to a Python script by running the command

jupyter nbconvert --to script example_notebook.ipynb

This will create a Python script named example_notebook.py in the same directory. After this, you can exit the lightwork compute node by running the command exit.

Step 2: Prepare the Python script for sbatch submission

Review the generated example_notebook.py script and make any necessary adjustments to ensure compatibility with non-interactive execution. A few common adjustments include:

  • Removing or modifying any interactive elements (e.g., widgets, plots, etc.).

  • Ensuring any file paths, inputs, or outputs are explicitly defined and relative to the working directory where the script will run.

  • If the script includes plotting or graphical outputs, consider saving them to files (e.g., PNG, PDF) instead of displaying them interactively.

  • Removing any cells that are not necessary for the sbatch job.

  • Adding necessary imports and environment setup at the beginning of the script.

Step 3: Create an sbatch script

To submit the Python script as an sbatch job, you need to create an sbatch script that specifies the resources and the commands required to run the job. Below is an example sbatch script (submit_job.sbatch). Note that you must activate the appropriate Mamba environment before running the script. All Jupyter kernels, public and user-created, have an associated Mamba environment.

#!/bin/bash #SBATCH -N 1 # number of nodes #SBATCH -c 8 # number of cores #SBATCH -t 0-01:00:00 # time in d-hh:mm:ss #SBATCH -p general # partition #SBATCH -q public # QOS #SBATCH -o slurm.%j.out # file to save job's STDOUT (%j = JobId) #SBATCH -e slurm.%j.err # file to save job's STDERR (%j = JobId) #SBATCH --mail-type=ALL # Send an e-mail when a job starts, stops, or fails #SBATCH --mail-user="%u@asu.edu" #SBATCH --export=NONE # Purge the job-submitting shell environment #Load Mamba module load mamba/latest #Activate environment source activate myEnv #Run the python script python myscript.py

Step 4: Submit the sbatch job

  1. Use the sbatch command to submit the sbatch script:

sbatch submit_job.sbatch

  1. Monitor the job status using the squeue command:

  1. Once the job completes, review the output and error logs slurm.<jobid>.out and
    slurm.<jobid>.err, respectively.

Additional notes

  • Make sure to save your work in the Jupyter Notebook before converting it to a Python script.

  • Ensure that all necessary files and dependencies are available in the working directory where the sbatch job will run.

  • You can test and troubleshoot the Python script using the debug QOS by running
    interactive -q debug -t 15.

Arizona State University