Build and Run

These instructions will help users get started running FastEddy in the NSF NCAR High Performance Computing (HPC) environment, using Derecho and Casper.

Existing Builds

The FastEddy team has installed the FastEddy software on both Casper and Derecho.

If desired, users can skip the compilation step and run the following commands to load FastEddy.

Note

Users need to replace <version> below with the desired X.Y.Z version to load (e.g. 4.0.0, 3.0.0, etc.).

Casper

module use /glade/u/fehelp/casper/installations/modulefiles
module load fasteddy/<version>

Derecho

module use /glade/u/fehelp/derecho/installations/modulefiles
module load fasteddy/<version>

Compilation

FastEddy requires a C-compiler, MPI, and CUDA.

Whether compiling on NSF NCAR HPC Environments or other platforms, users should do the following:

1. Download the source code from the Releases page and unpack the release in the desired location or clone the repository in the desired location.

2. Navigate to the SRC/FEMAIN directory.

Compilation on NSF NCAR HPC Environments

These instructions will help users get started with building FastEddy in the NSF NCAR High Performance Computing (HPC) environment, using Derecho and Casper.

The Makefile-based build system included here assumes deployment on the NSF NCAR HPCs.

Currently, the default modules loaded at login suffice on Casper, however the cuda module will need to be loaded on Derecho by running module load cuda.

After following steps 1 and 2 above, build the FastEddy executable by running:

make

(Optionally, run make clean first if appropriate.)

To enable model extensions, additional compilation flags are required:

• To include the Generalized Actuator Disk (GAD) parameterized model, run:

  make WITH_GAD=1
  

• To include the building-resolving URBAN model extension, run:

  make WITH_URBAN=1
  

• To enable both extensions, run:

  make WITH_GAD=1 WITH_URBAN=1
  

The FastEddy executable will be located in the SRC/FEMAIN directory.

Compilation on AMD GPU Accelerated Platforms

These instructions will help users get started with building FastEddy for systems with AMD GPU accelerators. This can be beneficial for users who have allocations on:

• Oak Ridge National Laboratory’s Frontier

• Pawsey Supercomputing Centre’s Setonix

• CSCS’s Lumi

• Fluid Numerics’ Galapagos

After following steps 1 and 2 above, build the FastEddy executable by running:

make -f Makefile.hip

(Optionally, run make clean first if appropriate.)

To enable model extensions, additional compilation flags are required:

• To include the Generalized Actuator Disk (GAD) parameterized model, run:

  make -f Makefile.hip WITH_GAD=1
  

• To include the building-resolving URBAN model extension, run:

  make -f Makefile.hip WITH_URBAN=1
  

• To enable both extensions, run:

  make -f Makefile.hip WITH_GAD=1 WITH_URBAN=1
  

Users may need to define a few environment variables that influence the build process to properly set the paths to various dependencies and to select the target GPU.

• ROCM_PATH : This is the path to your ROCm installation. This variable defaults to /opt/rocm. However, on some systems, multiple versions of ROCm may be available via environment modules and this variable may need to be adjusted accordingly.

• MPI_ROOT : This is the path to your MPI installation. Since this environment variable is not necessarily defined through an HPC center’s environment modules, it is recommended that you set this variable appropriately.

• NETCDF_C_ROOT: This is the path to your NetCDF-C installation. Since this environment variable is not necessarily defined through an HPC center’s environment modules, it is recommended that you set this variable appropriately.

• GPU_ARCH : This is the AMD GPU architecture code for the target GPU you want to build for. This variable defaults to gfx90a, which corresponds to the MI210, MI250, and MI250X GPUs.

The FastEddy executable will be located in the SRC/FEMAIN directory. To build on other HPC systems with NVIDIA GPUs, check for availability of the aformentioned modules/dependencies. Successful compilation may require modifications to shell environment variable include or library paths, or alternatively minor adjustments to the include or library flags in SRC/FEMAIN/Makefile.hip.

Example PBS Run Scripts

Below is bash-based PBS job submission script for running the model on NSF NCAR’s Casper and Derecho. These example scripts:

• fasteddy_pbs_script_casper.sh

• fasteddy_pbs_script_derecho.sh

can be found in the scripts/batch_jobs subdirectory of the GitHub FastEddy-model repository.

The FastEddy code will write its output to an output directory. Please create an output directory, if one does not already exist, in the same location as this script.

The code will produce an log file with the name FastEddy.o<job_id> (for example, FastEddy.o4960197) in the current working directory.

To submit the script for batch processing, run qsub <name of script>, replacing <name of script> with the name of the script.

Note

In the scripts, users will need to:

• Replace “<ProjectAccount>” below with a valid Project Account.

• Replace “<path to code location>” below with the location that contains the FastEddy-model directory from unpacking the release.

• Replace “<example .in filename>” below with the name of the example .in filename. For example, Example01_NBL.in, Example02_CBL.in, etc.

Casper

#!/bin/bash
#PBS -A <ProjectAccount>
#PBS -N FastEddy 
#PBS -l select=1:ncpus=4:mpiprocs=4:ngpus=4:mem=100GB:gpu_type=a100
#PBS -l walltime=12:00:00
#PBS -q casper
#PBS -j oe
#PBS -l job_priority=economy

export BASEDIR=<path to code location>/FastEddy-model/
export SRCDIR=${BASEDIR}/SRC/FEMAIN
export TUTORIALDIR=${BASEDIR}/tutorials/
export EXAMPLE=<example .in filename>

hostname
module -t list
echo " "

mpirun -np 4 ${SRCDIR}/FastEddy ${TUTORIALDIR}/examples/${EXAMPLE}

Derecho

#!/bin/bash
#PBS -A <ProjectAccount>
#PBS -N FastEddy 
#PBS -l select=1:ncpus=4:mpiprocs=4:ngpus=4:mem=100GB:gpu_type=a100
#PBS -l walltime=12:00:00
#PBS -q main 
#PBS -j oe
#PBS -l job_priority=economy

export BASEDIR=<path to code location>/FastEddy-model/
export SRCDIR=${BASEDIR}/SRC/FEMAIN
export TUTORIALDIR=${BASEDIR}/tutorials/
export EXAMPLE=<example .in filename>

hostname
module -t list
echo " "

mpiexec -n 4 --ppn 4 set_gpu_rank ${SRCDIR}/FastEddy ${TUTORIALDIR}/examples/${EXAMPLE}