This is a working draft guide for setting up WRF/WRF-Chem/WRF-GC on the Harvard Research Computing Cannon (formerly “Odyssey”) cluster.
The guide is being actively updated and polished.
module purge module load git/2.17.0-fasrc01 module load intel/17.0.4-fasrc01 module load openmpi/2.1.0-fasrc02 module load netcdf/4.5.0-fasrc02 module load netcdf-fortran/4.4.4-fasrc06 module load jasper/1.900.1-fasrc02
Note that WRF expects netCDF and netCDF-Fortran to be in the same directory. It also does not like
lib64. Do some trickery in a folder you’d like:
mkdir bin mkdir include mkdir lib ln -sf $NETCDF_HOME/lib64/* lib/ ln -sf $NETCDF_FORTRAN_HOME/lib/* lib/ ln -sf $NETCDF_HOME/bin/* bin/ ln -sf $NETCDF_FORTRAN_HOME/bin/* bin/ ln -sf $NETCDF_HOME/include/* include/ ln -sf $NETCDF_FORTRAN_HOME/include/* include/
Set up an environmental file with the modules above and the following directives:
export NETCDF=$(pwd) export JASPERLIB=$JASPER_HOME/lib64 export JASPERINC=$JASPER_HOME/include export CC=icc export OMPI_CC=$CC export CXX=icpc export OMPI_CXX=$CXX export FC=ifort export F77=$FC export F90=$FC export OMPI_FC=$FC export COMPILER=$FC export ESMF_COMPILER=intel # MPI Communication export ESMF_COMM=openmpi export MPI_ROOT=$MPI_HOME # WRF options export WRF_EM_CORE=1 export WRF_NMM_CORE=0 export WRF_CHEM=1
wget https://github.com/wrf-model/WRF/archive/V184.108.40.206.zip unzip V220.127.116.11.zip mv WRF-18.104.22.168 WRFV3
If you are installing WRF-GC, install WRF-GC:
cd WRFV3/ rm -rf chem
git clone our private repository to the
Configure and build WRF
- Launch an interactive node because the compile process is highly memory intensive:
srun -p test -n 6 --pty --mem 30960 -t 0-06:00 /bin/bash
- Go to the
WRFV3folder and run:
Use 15 (
INTEL (ifort/icc) (dmpar)) and
1 for nesting.
- Install the WRF-GC registry:
cd chem make install_registry
- And compile:
You may want to do this in a
screen so it does not interrupt when you exit the shell. It usually takes about 2-4 hours.
Configure and build WPS
Use similar principles as WRF or refer to my WRF-GC on AWS Guide
Remember that WPS must be built after WRF is installed as it relies on I/O libraries in WRF. Also, WRF must be placed in the directory above WPS with the folder name
./configure, choose option
17. Linux x86_64, Intel compiler (serial).
- Run this:
export MPI_LIB="". See explanation below.
./compile. You should now have all the
$ ls *.exe geogrid.exe metgrid.exe ungrib.exe
Why do I have to set
Otherwise in WPS compile, you may get this error in the compile logs:
/n/helmod/apps/centos7/Comp/intel/17.0.4-fasrc01/mvapich2/2.3b-fasrc02/lib64: file not recognized: Is a directory make: [geogrid.exe] Error 1 (ignored)`.
This is somehow caused by a command like this:
ifort -o geogrid.exe ... -lnetcdff -lnetcdf \ /n/helmod/apps/centos7/Comp/intel/17.0.4-fasrc01/mvapich2/2.3b-fasrc02/lib64
You may wonder why this
lib64 is appearing out of nowhere. Well, it’s because Odyssey’s
lib64 for the MPI path, but WPS also uses this variable internally to store a link command to MPI (i.e.
-L/n/helmod/apps/centos7/Comp/intel/17.0.4-fasrc01/mvapich2/2.3b-fasrc02/lib64). This clashes and blows the world up. We are compiling WPS in
(serial) so we don’t really need
MPI_LIB, just unset it for now.
Sample configuration files
A copy of WPS geographical input files have been prepared at
/n/holyscratch01/jacob_lab/hplin/wps-geog-3.9 if you can access
&share wrf_core = 'ARW', max_dom = 1, start_date = '2019-12-01_00:00:00', end_date = '2020-01-01_00:00:00', interval_seconds = 21600, io_form_geogrid = 2, / &geogrid parent_id = 1, parent_grid_ratio = 1, i_parent_start = 1, j_parent_start = 1, e_we = 245, e_sn = 181, geog_data_res = 'gtopo_2m+usgs_2m+nesdis_greenfrac+2m','default', dx = 27000, dy = 27000, map_proj = 'mercator', ref_lat = 38, ref_lon = 105, truelat1 = 38.0, stand_lon = 105, geog_data_path = '/n/holyscratch01/jacob_lab/hplin/wps-geog-3.9' / &ungrib out_format = 'WPS', prefix = 'FILE', / &metgrid fg_name = 'FILE' io_form_metgrid = 2, /
You have to supply your own WRF input meteorological datasets and use the appropriate
A quick reminder that your simulation’s horizontal resolution is set both here (
namelist.wps) and in the WRF run directory (
WRFV3/run/namelist.input), and that you can only use
lat-lon as these are the only grid projections supported by HEMCO in WRF-GC at this moment.
Tell WRF-GC to use the HEMCO emissions data available on Odyssey, which is at
/n/holylfs/EXTERNAL_REPOS/GEOS-CHEM/gcgrid/data/ExtData and shared by everyone.
Go to the WRF run directory (
WRFV3/run) and edit the following files: Remember that these paths need to be updated every time you recompile the code, because fresh files are installed in their place.
Run directory : ./ Root data directory : /n/holylfs/EXTERNAL_REPOS/GEOS-CHEM/gcgrid/data/ExtData Global offsets I0, J0 : 0 0
All other options in
input.geos are ignored as they are relevant to GEOS-Chem Offline only.
ROOT: /n/holylfs/EXTERNAL_REPOS/GEOS-CHEM/gcgrid/data/ExtData/HEMCO METDIR: /does/not/apply/for/WRFGC
HEMCO_Config.rc is fully operational in WRF-GC and you can use it to set the emission inventories you want to use in your simulation.
The main namelist should be configured according to the WRF-GC User’s Guide: Running WRF-GC.
Running WPS and WRF(-GC)
- Launch an interactive node with sufficient memory. The
testpartition is sufficient, WPS is not very compute intensive:
srun -p test -n 6 --pty --mem 30960 -t 0-06:00 /bin/bash
- Directly run
real.exe (prepare input data for WRF) and
wrf.exe use the following batch script (remember to change the name of the executable).
This is written for 1 node 32 cores on
huce_intel; adjust memory and stuff accordingly if you want to use a different partition:
#!/bin/bash #SBATCH -n 32 #SBATCH -N 1 #SBATCH -t 1-00:00 #SBATCH -p huce_intel #SBATCH --mem=128000 module purge module load git/2.17.0-fasrc01 module load intel/17.0.4-fasrc01 module load openmpi/2.1.0-fasrc02 module load netcdf/4.5.0-fasrc02 module load netcdf-fortran/4.4.4-fasrc06 module load jasper/1.900.1-fasrc02 # srun -n $SLURM_NTASKS --mpi=pmix wrf.exe mpirun -np $SLURM_NTASKS ./wrf.exe
Save it as e.g.
wrf.sh and then
sbatch wrf.sh to submit the batch job.
For roughly 40000 grid boxes (
e_we * e_sn), say China at 27x27km resolution, it takes about 4 hours on 36 cores to run about 48 hours simulation. In order for your run not to be interrupted you will have to consider asking for a little more time as there may be fluctuations.
-n 32for number of cores and
-N 1for number of nodes to use.
Test on the
testpartition first (with a duration of
0-01:00or shorter), then submit to
huce_intelor the main partitions, so you don’t waste core-hours. The major errors always appear in the first hour of the run. If the first hour runs through, it will likely run to completion.