FastEddy^® version 1.1.0

FastEddy^® (FE) is a large-eddy simulation (LES) model developed by the Research Applications Laboratory (RAL) at the U.S. National Science Foundation National Center for Atmospheric Research (NSF NCAR) in Boulder, Colorado, USA. The fundamental premise of FastEddy model development is to leverage the accelerated and more power efficient computing capacity of graphics processing units (GPU)s to enable not only more widespread use of LES in research activities but also to pursue the adoption of microscale and multiscale, turbulence-resolving, atmospheric boundary layer modeling into local scale weather prediction or actionable science and engineering applications.

Release Notes

When applicable, release notes are followed by the GitHub issue number which describes the bugfix, enhancement, or new feature (FastEddy-model GitHub issues).

FastEddy-model Version 1.1 Release Notes (20240422)

This is the initial release of the FastEddy, an NSF NCAR developed parallelized and GPU-resident, large-eddy simulation code for accelerated modeling of the atmospheric boundary layer.

In addition to the initial code, this release includes a patch for building the system in the current NSF NCAR high performance computing environment on the Casper and Derecho platforms, along with other changes as detailed below:

Repository, build, and test

• Add templates for Issues and Pull Requests (#1)

• Set up the FastEddy Tutorial documentation (#3)

• Consolidate FastEddy-tutorials content into FastEddy-model (#8)

• Adjust FastEddy-tutorials BOMEX notebook & RTD Moist dynamics instructions for hosting datasets under new repo (#10)

Bugfixes

• None

Enhancements

• None

Downloads

Version	Release Date	Tar File	Release Notes
1.1.0	April 22, 2024	v1.1.0.tar.gz	v1.1.0 Release Notes

Tutorial

Getting Started

Introduction

FastEddy is a resident-GPU large eddy simulation (LES) model owned by the U.S. National Science Foundation National Center for Atmospheric Research (NSF NCAR) Research Applications Laboratory (RAL). It is designed for future turbulence-resolving numerical weather prediction.

This is a tutorial designed so that a user can learn how to execute FastEddy, using the four test cases described in this tutorial.

Software and computing requirements

Computing resources with at least four general purpose graphics processing units are recommended to carry out the test cases. System must be enabled with python and Jupyter notebook packages. Add other requirements (compilers, libraries, etc).

Instructions on how to build and run FastEddy on NSF NCAR’s Casper architecture https://github.com/NCAR/FastEddy-model/blob/main/README.md.

Ideal Test Cases

Four test cases are described:

• Dry neutral boundary layer

• Dry convective boundary layer

• Dry stable boundary layer

• Moist cloud-topped boundary layer

Required tutorial resources including python utilities and Jupyter Notebooks are provided in the tutorials directory of the FastEddy-model GitHub repository with required data for the moist dynamics example available at this Zenodo record. All test cases are idealized setups over flat terrain. For each case, the user will set up the input parameter file, execute FastEddy, visualize the output using a Jupyter notebook, and perform some basic analysis of the output. After examining the test cases, the user will carry out some sensitivity tests by changing various input parameters. The purpose of these tests are for the user to become more familiar with the input parameters, and how changes to those parameters affect the output. After the tutorial, the user is expected to have basic knowledge to carry out LES using FastEddy.

Dry neutral boundary layer

Background

This is a canonical neutral boundary layer scenario. The case is broadly based upon Sauer and Munoz-Esparza (2020) but is not identical. A geostrophic wind is prescribed over ground with a set aerodynamic roughness length under a neutrally stratified boundary layer. The purpose of this test case is to visualize and analyze the resultant flow and turbulence characteristics that develop when the LES reaches statistical steady-state.

Input parameters

• Number of grid points: \([N_x,N_y,N_z]=[640,634,58]\)

• Isotropic grid spacings in the horizontal directions: \([dx,dy]=[15,15]\) m, vertical grid is \(dz=15\) m at the surface and stretched with verticalDeformFactor \(=0.75\)

• Domain size: \([9.6 \times 9.51 \times 1.08]\) km

• Model time step: \(0.04\) s

• Advection scheme: 5th-order upwind

• Time scheme: 3rd-order Runge Kutta

• Geostrophic wind: \([U_g,V_g]=[10,0]\) m/s

• Latitude: \(54.0^{\circ}\) N

• Surface potential temperature: \(300\) K

• Potential temperature profile:

\[\begin{split}\partial{\theta}/\partial z = \begin{cases} 0 & \text{if $z$ $\le$ 500 m}\\ 0.08 & \text{if 500 m < $z$ $\le$ 650 m}\\ 0.003 & \text{if $z$ > 650 m} \end{cases}\end{split}\]

• Surface heat flux: \(0.0\) Km/s

• Surface roughness length: \(z_0=0.1\) m

• Rayleigh damping layer: uppermost \(400\) m of the domain

• Initial perturbations: \(\pm 0.25\) K

• Depth of perturbations: \(375\) m

• Top boundary condition: free slip

• Lateral boundary conditions: periodic

• Time period: \(7\) h

Execute FastEddy

Run FastEddy using the input parameters file /examples/Example01_NBL.in. To execute FastEddy, follow the instructions here: https://github.com/NCAR/FastEddy-model/blob/main/README.md.

Visualize the output

Open the Jupyter notebook entitled “MAKE_FE_TUTORIAL_PLOTS.ipynb” and execute it using setting: case = ‘neutral’.

XY-plane views of instantaneous velocity components at \(t=7\) h (FE_NBL.630000):

XZ-plane views of instantaneous velocity components at \(t=7\) h (FE_NBL.630000):

Mean (domain horizontal average) vertical profiles of state variables at \(t=7\) h (FE_NBL.630000):

Horizontally-averaged vertical profiles of turbulence quantities at \(t=6-7\) h [perturbations are computed at each time instance from horizontal-slab means, then averaged horitontally and over the previous 1-hour mean]:

Analyze the output

• Using the XY and XZ cross sections, discuss the characteristics (scale and magnitude) of the resolved turbulence.

• What is the boundary layer height in the neutral case?

• Using the vertical profile plots, explain why the boundary layer is neutral.

Dry convective boundary layer

This is the convective boundary layer scenario described by Sauer and Munoz-Esparza (2020). This case represents the boundary layer conditions at the SWiFT facility near Lubbock, Texas at 4 July 2012 during the period of 18Z-20Z (12:00–14:00 local time), the strongest period of convection on the day.

Input parameters

• Number of grid points: \([N_x,N_y,N_z]=[600,594,122]\)

• Isotropic grid spacings in the horizontal directions: \([dx,dy]=[20,20]\) m, vertical grid is \(dz=20\) m at the surface and stretched with verticalDeformFactor \(=0.80\)

• Domain size: \([12.0 \times 11.9 \times 3.0]\) km

• Model time step: \(0.05\) s

• Geostrophic wind: \([U_g,V_g]=[9,0]\) m/s

• Advection scheme: Hybrid 5th order upwind

• Time scheme: 3rd-order Runge Kutta

• Latitude: \(33.5^{\circ}\) N

• Surface potential temperature: \(309\) K

• Potential temperature profile:

\[\begin{split}\partial{\theta}/\partial z = \begin{cases} 0 & \text{if $z$ $\le$ 600 m}\\ 0.004 & \text{if $z$ > 600 m} \end{cases}\end{split}\]

• Surface heat flux: \(0.35\) Km/s

• Surface roughness length: \(z_0=0.05\) m

• Rayleigh damping layer: uppermost \(400\) m of the domain

• Initial perturbations: \(\pm 0.25\) K

• Depth of perturbations: \(400\) m

• Top boundary condition: free slip

• Lateral boundary conditions: periodic

• Time period: \(4\) h

Execute FastEddy

Run FastEddy using the input parameters file /examples/Example02_CBL.in. To execute FastEddy, follow the instructions here: https://github.com/NCAR/FastEddy-model/blob/main/README.md.

Visualize the output

Open the Jupyter notebook entitled “MAKE_FE_TUTORIAL_PLOTS.ipynb” and execute it using setting: case = ‘convective’.

XY-plane views of instantaneous velocity components at \(t=4\) h (FE_CBL.288000):

XZ-plane views of instantaneous velocity components at \(t=4\) h (FE_CBL.288000):

Mean (domain horizontal average) vertical profiles of state variables at \(t=4\) h (FE_CBL.288000):

Horizontally-averaged vertical profiles of turbulence quantities \(t=3-4\) h [perturbations are computed at each point relative to the previous 1-hour mean, and then horizontally averaged]:

Analyze the output

• Using the XY and XZ cross sections, discuss the characteristics (scale and magnitude) of the resolved turbulence.

• What is the boundary layer height in the convective case?

• Using the vertical profile plots, explain why the boundary layer is unstable.

Dry stable boundary layer

Background

This is the stable boundary layer scenario described by Sauer and Munoz-Esparza (2020). This the stable boundary layer scenario outlined in Kosovic and Curry (2000).

Input parameters

• Number of grid points: \([N_x,N_y,N_z]=[128,126,122]\)

• Isotropic grid spacings: \([dx,dy,dz]=[3.125,3.125,3.125]\) m

• Domain size: \([0.40 \times 0.39 \times 0.38]\) km

• Model time step: \(0.005\) s

• Geostrophic wind: \([U_g,V_g]=[8,0]\) m/s

• Advection scheme: 5th-order upwind

• Time scheme: 3rd-order Runge Kutta

• Latitude: \(73^{\circ}\) N

• Surface potential temperature: \(265\) K

• Potential temperature profile:

\[\begin{split}\partial{\theta}/\partial z = \begin{cases} 0 & \text{if $z$ $\le$ 100 m}\\ 0.01 & \text{if $z$ > 100 m} \end{cases}\end{split}\]

• Surface heat flux: \(-0.25\) K/h

• Surface roughness length: \(z_0=0.1\) m

• Rayleigh damping layer: uppermost \(75\) m of the domain

• Initial perturbations: \(\pm 0.25\) K

• Top boundary condition: free slip

• Lateral boundary conditions: periodic

• Time period: \(12\) h

Execute FastEddy

Run FastEddy using the input parameters file /examples/Example03_SBL.in. To execute FastEddy, follow the instructions here: https://github.com/NCAR/FastEddy-model/blob/main/README.md.

Visualize the output

Open the Jupyter notebook entitled “MAKE_FE_TUTORIAL_PLOTS.ipynb” and execute it using setting: case = ‘stable’.

XY-plane views of instantaneous velocity components at \(t=12\) h (FE_SBL.8640000):

XZ-plane views of instantaneous velocity components at \(t=12\) h (FE_SBL.8640000):

Mean (domain horizontal average) vertical profiles of state variables at \(t=12\) h (FE_SBL.8640000):

Horizontally-averaged vertical profiles of turbulence quantities at \(t=11-12\) h (FE_TEST.8640000) [perturbations are computed at each point relative to the previous 1-hour mean, and then horizontally averaged]:

Analyze the output

• Using the XY and XZ cross sections, discuss the characteristics (scale and magnitude) of the resolved turbulence.

• What is the boundary layer height in the stable case?

• Using the vertical profile plots, explain why the boundary layer is stable.

Moist cloud-topped boundary layer

This tutorial case is the BOMEX LES intercomparison study from Siebesma et al. (2003), corresponding to a non-precipitating shallow cumulus cloud case informed by the Barbados Oceanographic and Meteorological Experiment (BOMEX, Holland & Rasmusson, 1973). The forcing consists of two different sources including prescribed kinematic surface fluxes of sensible and latent heat and large-scale forcing (LSF) tendencies due to mesoscale horizontal advection of water vapor mixing ratio, liquid potential temperature and horizontal momentum. The LSF includes subsidence to compensate the integrated effect of surface fluxes and advection tendencies, formulated as a prescribed time-invariant subsidence profile multiplied by the vertical gradient of horizontally averaged fields accros the domain. The main settings of this case are listed below and are further detailed in Munoz-Esparza et al. (2022).

Input parameters

• Number of grid points: \([N_x,N_y,N_z]=[152,146,122]\)

• Isotropic grid spacings: \([dx,dy,dz]=[100,100,40]\) m

• Domain size: \([15.2 \times 14.6 \times 4.9]\) km

• Model time step: \(0.075\) s

• Geostrophic wind: \([U_g,V_g]=[10,0]\) m \(\mbox{s}^{-1}\)

• Advection schemes: 5th-order upwind (dry dynamics), 3rd-order upwind (water vapor), and 3rd-order WENO (liquid water)

• Time scheme: 3rd-order Runge Kutta

• Latitude: \(14.94^{\circ}\) N

• Surface potential temperature: \(299.1\) K

• Surface sensible heat flux: \(8 \times 10^{-3}\) K m \(\mbox{s}^{-1}\)

• Surface latent heat flux: \(5.2 \times 10^{-5}\) m \(\mbox{s}^{-1}\)

• Surface roughness length: \(z_0=0.0002\) m

• Rayleigh damping layer: uppermost \(500\) m of the domain

• Initial perturbations: \(\pm 0.1\) K

• Depth of perturbations: \(1600\) m

• Top boundary condition: free slip

• Lateral boundary conditions: periodic

• Time period: \(6\) h

• Initital conditions: vertical profiles of \(u\), \(q_v\), and SGSTKE as specified in Siebesma et al. (2003)

• Large-scale forcings: vertical profiles of subsidence and horizontal advection of potential temperature and water vapor as specified in Siebesma et al. (2003)

Execute FastEddy

Note that this example moist dynamics validation case example requires an additional dataset available as a gzip compressed tape archive file at Zenodo record. The contents of the archive incude an initial conditions file BOMEX_IC/FE_BOMEX.0 which is needed to run FastEddy for this case. The archive dataset also contains results from the 11 models that participated in the original Siebesma et al. 2003 model intercomparison as NetCDF files under BOMEX_Siebesma2003_models/*.nc. Run FastEddy using the input parameters file /tutorials/examples/Example04_BOMEX.in. Be sure to copy the extracted initial conditions file from the archived dataset into the initial subdirectory of your case run directory.

Visualize the output

Open the Jupyter notebook entitled “FE_Postrocessing_Example04_BOMEX.ipynb” and execute it.

Time evolution of domain averaged total cloud cover (\(\alpha_c\)) and liquid water path (LWP):

Vertical profiles of potential temperature (\(\theta\)), water vapor (\(q_v\)), horizontal velocity components (\(u\), \(v\)), and liquid content (\(q_l\)). Thin black dashed lines correspond to the initial conditions. Profiles are averaged for the last 3 hr (\(t = 180-360\) min) and over horizontal domain slabs:

Vertical profiles of turbulence kinetic energy (TKE), vertical velocity variance (\(\sigma^2_w\)), and vertical turbulent fluxes of zonal momentum (\(\langle u'w' \rangle\)), water vapor (\(\langle w'q_v' \rangle\)), liquid cloud, and virtual potential temperature (\(\langle w'q_l' \rangle\)). Profiles are averaged for the last 3 hr (\(t = 180–360\) min) and perturbations are computed as the departure from horizontal slab averages. These turbulence quantities are the sum of resolved and subgrid-scale components.

Analyze the output

• Using the time series of cloud properties, could you identify when the simulated shallow cumulus cloud deck has reached quasi-eqilibrium?

• What is the effect of boundary-layer turbulence to the mean profiles of momentum?

• Identify the vertical extent of the cloud layer.

• Which of the turbulent vertical transport terms is responsible for the resulting vertical liquid cloud distribution?

Sensitivity Tests

Instructions

• Re-run the neutral case with \([N_x,N_y,N_z]=[400,400,122]\) and isotropic grid spacings of \([dx,dy,dz]=[10,10,10]\). Adjust the model time step accordingly. Re-make all plots and discuss the differences between the control case. How much longer did it take to complete the simulation?

• Re-run the convective case with a surface heat flux of \(=+0.70\) Km/s. Re-make all plots and discuss the differences between the control case.

• Re-run the neutral case with \(z_0=0.3\) m. Re-make all plots and discuss the differences between the control case.

• Re-run the neutral case with the first order upwind advection scheme. Re-make all plots and discuss the differences between the control case. Why is the first order scheme a bad choice?

• Re-run the stable case with a surface cooling rate of \(-0.5\) K/h. Re-make all plots and discuss the differences between the control case.

• Re-run the stable case using half of the GPUs used in the control simulation. How much slower does the case run?

• Re-run the BOMEX case with a higher-order advection for water vapor (moistureAdvSelectorQv = 3). What is the impact of the increased effective resolution on dynamical, thermodynamical and microphysical quantities, along with turbulence variability and fluxes? How does that change influce the comparison to the other BOMEX LES models?