Building WRF on Sol (pre-compiled libraries)

Owned by William Dizon
Last updated: Jan 19, 2024
9 min read

There are two pages for using WRF on Sol:

The process outlined in this page will utilize pre-compiled libraries vetted by Research Computing staff. Building Sol (fully manual) describes a process in which the libraries are built by the user and placed in user storage (e.g., scratch). When using pre-compiled libraries, the user will only compile the WRF+WPS binaries.

The benefit of the pre-compiled library approach includes many conveniences such as:

  1. libraries known to work

  2. save user from worrying about dependency trees and binary compilation time

  3. user configuration (wrf_variables) is more straightforward

  4. easily compile multiple WRF instances in their own independent directory

WRF can be compiled on Sol without any special permissions. You can build it entirely start-to-finish as your own unprivileged supercomputer user.

The steps outlined in this page has successfully been used to compile numerous versions of WRF. However, to change either the WRF or WPS versions (older or newer)--or changing WRF to a different source tree altogether--may warrant changes this tutorial cannot anticipate.

It is recommended to complete this tutorial with the unchanged files to familiarize yourself with the process and the steps involved.

Setting up the Compilation Process for WRF

We will start by copying over the wrf-building scripts to our own scratch space. This space is designated as /scratch/$USER, such as /scratch/wdizon/. Let’s start by moving to a compute node to copy the scripts:

wdizon@login01 $ interactive -c 20 wdizon@c001 $ cp -R /packages/apps/letsbuild/wrf /scratch/$USER/

-c 20 is chosen as it is the maximum number of cores WRF will compile in-parallel for. This is a separate limitation from how many cores the built binaries can run on--the completed binary will not be limited by the number chosen here.

Let’s review the files we have copied:

$ cd /scratch/$USER/wrf $ ls build* check_sanity@ compiler_variables@ configure* extracted_src/ tarballs/ wrf_variables $ cat wrf_variables #!/bin/bash # WRF source code is located at: export WRF_SOURCE_EXTRACTED=/scratch/$USER/wrf-src # WPS source code is located at: export WPS_SOURCE_EXTRACTED=/scratch/$USER/wps-src # During `configure`, this script will copy the above source # paths into the below directory structure: $WRF_INSTALL/$WRF_DESTDIR # Any changes to source code should happen in $WRF_SOURCE_EXTRACTED. # After ANY changes to source code, `configure` must be repeated # Base directory for all wrf compilations (many may exist in parallel) export WRF_INSTALL=/scratch/$USER/wrf_compiles # Install WRF into $WRF_INSTALL with the directory name $WRF_DESTDIR # Rename this for each compilation you want to exist independently # e.g., WRF-4.1.3-patched # e.g., WRF-4.2.2-custom export WRF_DESTDIR=WRF-4-custom

The file wrf_variables is generally the only file that will require any user editing. The following combinations of WRF+WPS have been tested and confirmed to work on Sol with these scripts:

  • WRF 4.1.3 & WPS 4.2

  • WRF 4.2.2 & WPS 4.2

  • WRF 4.3.3 & WPS 4.3

Extracting WRF & WPS Source Code

Versions aside from these listed are also expected to work. To adjust these scripts for your own installation, modify the following:

# WRF source code is located at: export WRF_SOURCE_EXTRACTED=/scratch/$USER/wrf-src # WPS source code is located at: export WPS_SOURCE_EXTRACTED=/scratch/$USER/wps-src

You can either extract the WRF+WPS source to the above directories, respectively, in your scratch space; or you can modify these lines to point to the directory of the source files, whatever they may be named.

As a simple, working example, I can extract the source to the extracted_src directory in the wrf dir we just copied.

These paths can now be entered into wrf_variables:

Testing the Environment is Sane

Execute the following line:

If there are no issues, you should see SUCCESS in exactly the fashion displayed above.

This means at least the following:

  1. NetCDF and MPICH (pre-compiled) are operational

  2. The compiler was successfully able to compile a c and fortran code, showing readiness to continue to WRF.

If you do not see output matching above, do not continue.

If necessary, start a new terminal session and ensure no conflicting modules are loaded (module purge) and try again.

Configure WRF

Upon running this command, you will be asked about which compiler to use and whether to use nesting. This interactive step cannot be automated, so it is key to ensure proper input here:

In this example, we will select 35 (GNU (gfortran/gcc)) and 1 (nesting basic).

Compiling WRF+WPS

Finally, compile:

WRF will take a short while to build. After an amount of time has passed, you will be prompted again for configuring one portion of WPS:

Select 1, the gfortran serial option. Choose a serial variant, no matter what compiler.

At the end of this step, you will have a working WRF compilation built with MPICH and GCC located at:

WRF => /scratch/$USER/wrf_compiles/WRF-4-custom

WPS => /scratch/$USER/wrf_compiles/WRF-4-custom/WPS

These match the path set in wrf_variables under the name WRF_INSTALL and WRF_DESTDIR. WPS is always saved inside the WRF directory with the name WPS.

Indication of Success

At the end of the script, you should see Compilation Complete and you will be returned to the prompt.

Usage Notes

Once built, you can then use module load letsbuild/wrf as a shorthand to update any terminal session/SBATCH script to use these libraries.

This will allow easy access to binaries such as mpiexec.hydra, which is used to invoke WRF with MPI.

$USER variable

The $USER variable will translate to your login username, which matches your ASURITE ID. The $USER variable therefore is to simplify copy/paste operations, rather than expecting the user to type in, for example, /scratch/wdizon , which is a completely permissible/workable alternative.

Repeated Builds

The primary purpose of this instruction set for WRF is to accommodate repeated builds with different source files. By simply changing the directory paths in wrf_variables, you can have new installations placed into a destination of your choosing.

+New Instance of WRF, Same Source code as last build

+New Instance of WRF, Different Source code from last build

  1. Identify tarball needed for WRF source code, place in /scratch/$USER/some_dir

  2. Extract the source tree from the tarball: tar -xf WRF_2_THERMAL_URBCOL_CBC.tgz

  3. Identify the newly created directory name in /scratch/$USER/some_dir
    In this example, the extracted tarball created a dir called WRF_2_THERMAL_URBCOL_CBC, where standard source files might have created WRF-4.2.2.

  4. In wrf_variables, make the changes to reflect this:

Now that it is built:

WRF’s Files are located at /scratch/$USER/wrf_compiles/*/run.

WRF is built using MPICH, and mpiexec.hydra, e.g., /scratch/$USER/wrf_compiles/LIBRARIES/mpich/bin/mpiexec.hydra -np 12 ./wrf.exe

If you run this interactively, be sure to choose -c <num cores> to match -np <num cores>. If you are submitting this with a batch job, makes sure your #SBATCH -c <num cores> matches.

Arizona State University