Meteorological data

In order to run FALL3D, you need to provide meteorological data as an input. Multiple datasets are supported, including global model forecasts (GFS), global analyses and re-analyses (ERA5), and mesoscale models (WRF-ARW). In this session we'll retrieve global forecasts from two datasets:

  • GFS: The Global Forecast System (global weather forecast model)
  • GEFS: The Global Ensemble Forecast System (global weather forecast model)

The GFS (Global Forecast System) is the most well-known global weather model and it’s updated every six hours (00Z, 06Z, 12Z, 18Z). It's produced by the National Centers for Environmental Prediction (NCEP) of the United States National Oceanic and Atmospheric Administration (NOAA).

In this session you need to copy some files to your projects folder:

cd /gpfs/projects/nct01/$USER
cp -r /gpfs/projects/nct00/nct00014/FALL3D_material/hands-on-1 .
cd hands-on-1

The filtering problem

The filtering problem is the most common estimation problem in geophysical applications, and is characterized by sequential processing in which measurements are utilized as they become available.

Filtering

  • Filtering problem: the information from observation up to the time of the current sample is used (e.g. GFS)
  • Smoothing problem: all observation samples (from future) are used. This approach is used to produce reanalysis datasets

Notes:

  • Atmospheric reanalysis is a method to reconstruct the past weather by combining historical observations with a dynamical model. It provides a physically and dynamically coherent description of the state of the atmosphere

Getting GFS and GEFS data (local machine)

Let's see how to get weather forecast data from GFS using the scripts gfs.py and gefs.py included in the FALL3D distribution under the folder Other/Meteo/Utils.

Notes:

  • Unfortunately, there's no outgoing internet connection from the cluster, which prevents the use of external repositories directly from MN4 machines. Consequently, meteorological data must be downloaded from a remote server to a local machine and then copied to the cluster.

A. Installing Python packages

The Python scripts above require the fall3dutil package, which can be installed on Linux and MacOS systems from the Python Package Index (PyPI) using pip. Python "Virtual Environments" allow Python packages to be installed in an isolated location for a particular application. In order to create a virtual environment on a typical Linux system, use the basic venv command:

python3 -m venv fall3d_env
source fall3d_env/bin/activate

This will create a new virtual environment in the fall3d_env subdirectory, and configure the current shell to use it as the default python environment. For more information, please visit the Python Packaging User Guide.

Once the virtual environment is activated, you can install the last version of the fall3dutil package using:

pip install fall3dutil

B. Configuration

Let's obtain GFS data for two domains centered around Etna (Italy) and Fuego (Guatemala) volcanoes. To this purpose, we define a configuration file config.inp with the downloading parameters:

[etna]
date    = 20240306   # Initial date
lon     = 8  36      # Longitude range
lat     = 20 42      # Latitude range
time    = 0  48      # Time forecast range
step    = 3          # Time resolution
cycle   = 0          # Cycle (00Z, 06Z, 12Z or 18Z)
res     = 0.5        # Resolution in degrees

[fuego]
date    = 20240306   # Initial date
lon     = -130 -52   # Longitude range
lat     = 0 33       # Latitude range
ens     = 1 12       # Ensemble member range
time    = 0 48       # Time forecast range
step    = 3          # Time resolution
cycle   = 0          # Cycle (00Z, 06Z, 12Z or 18Z)
res     = 0.5        # Resolution in degrees

C. Getting GFS data

To obtain GFS data to simulate a volcanic eruption at Etna use the following command:

./gfs.py --input config.inp --block etna --verbose

We get a list of GRIB2 files for every forecast time:

gfs.t00z.pgrb2full.0p50.f000  gfs.t00z.pgrb2full.0p50.f012  gfs.t00z.pgrb2full.0p50.f024  gfs.t00z.pgrb2full.0p50.f036  gfs.t00z.pgrb2full.0p50.f048
gfs.t00z.pgrb2full.0p50.f003  gfs.t00z.pgrb2full.0p50.f015  gfs.t00z.pgrb2full.0p50.f027  gfs.t00z.pgrb2full.0p50.f039
gfs.t00z.pgrb2full.0p50.f006  gfs.t00z.pgrb2full.0p50.f018  gfs.t00z.pgrb2full.0p50.f030  gfs.t00z.pgrb2full.0p50.f042
gfs.t00z.pgrb2full.0p50.f009  gfs.t00z.pgrb2full.0p50.f021  gfs.t00z.pgrb2full.0p50.f033  gfs.t00z.pgrb2full.0p50.f045

D. Getting GEFS data

To obtain GEFS data to simulate a volcanic eruption at Fuego use the following command to retrieve an ensemble of weather forecasts:

./gefs.py --input config.inp --block fuego --verbose

For each ensemble member, we get a list of GRIB2 files (two files for every forecast time). For example, the files for the first ensemble member are:

gep01.t00z.pgrb2a.0p50.f000  gep01.t00z.pgrb2a.0p50.f021  gep01.t00z.pgrb2a.0p50.f042  gep01.t00z.pgrb2b.0p50.f012  gep01.t00z.pgrb2b.0p50.f033
gep01.t00z.pgrb2a.0p50.f003  gep01.t00z.pgrb2a.0p50.f024  gep01.t00z.pgrb2a.0p50.f045  gep01.t00z.pgrb2b.0p50.f015  gep01.t00z.pgrb2b.0p50.f036
gep01.t00z.pgrb2a.0p50.f006  gep01.t00z.pgrb2a.0p50.f027  gep01.t00z.pgrb2a.0p50.f048  gep01.t00z.pgrb2b.0p50.f018  gep01.t00z.pgrb2b.0p50.f039
gep01.t00z.pgrb2a.0p50.f009  gep01.t00z.pgrb2a.0p50.f030  gep01.t00z.pgrb2b.0p50.f000  gep01.t00z.pgrb2b.0p50.f021  gep01.t00z.pgrb2b.0p50.f042
gep01.t00z.pgrb2a.0p50.f012  gep01.t00z.pgrb2a.0p50.f033  gep01.t00z.pgrb2b.0p50.f003  gep01.t00z.pgrb2b.0p50.f024  gep01.t00z.pgrb2b.0p50.f045
gep01.t00z.pgrb2a.0p50.f015  gep01.t00z.pgrb2a.0p50.f036  gep01.t00z.pgrb2b.0p50.f006  gep01.t00z.pgrb2b.0p50.f027  gep01.t00z.pgrb2b.0p50.f048
gep01.t00z.pgrb2a.0p50.f018  gep01.t00z.pgrb2a.0p50.f039  gep01.t00z.pgrb2b.0p50.f009  gep01.t00z.pgrb2b.0p50.f030

Copy the GRIB2 files to the cluster, for example, using scp.

Processing GFS/GEFS data (cluster)

FALL3D requires input meteorological data in netCDF format. Consequently, GFS data must be concatenated and converted from GRIB2 to obtain a single netCDF file. To this purpose, you can use the utility to read and write grib2 files wgrib2, which is available in the cluster loading the corresponding module:

module load wgrib

In addition, the command wgrib2 can be used to display the contents of a grib2 file. For example, run:

wgrib2 gfs.t00z.pgrb2full.0p50.f000

Converting GFS data

The following bash script (grib2nc.sh) invokes wgrib2 and can be used to concatenate and convert a list of GRIB2 files:

########## Edit header ##########
WGRIBEXE=wgrib2
TABLEFILE=TABLES/gfs_0p50.levels
TMIN=0
TMAX=48
STEP=3
CYCLE=0
GRIBPATH=GFS
UPDATE_FNAME (){
    fname=${GRIBPATH}/gfs.t${CYCLE}z.pgrb2full.0p50.f${HOUR}
    OUTPUTFILE=etna.gfs.nc
}
#################################

variables="HGT|TMP|RH|UGRD|VGRD|VVEL|PRES|PRATE|LAND|HPBL|SFCR"
CYCLE=$(printf %02d $CYCLE)

for i in $(seq ${TMIN} ${STEP} ${TMAX})
do
    HOUR=$(printf %03d $i)
    UPDATE_FNAME

    echo "Processing ${fname}..."
    ${WGRIBEXE} "${fname}" \
        -match ":(${variables}):" \
        -match ":(([0-9]*[.])?[0-9]+ mb|surface|2 m above ground|10 m above ground):" \
        -nc_table "${TABLEFILE}" \
        -append \
        -nc3 \
        -netcdf \
        "${OUTPUTFILE}" > wgrib.log
done

A single file etna.gfs.nc will be generated, which can be read by FALL3D.

Converting GEFS data

The following script grib2nc-ens.sh can be used to concatenate and convert GEFS data in grib2 format:

########## Edit header ########## 
WGRIBEXE=wgrib2
TABLEFILE=TABLES/gefs_0p50.levels
ENSMIN=1
ENSMAX=12
TMIN=0
TMAX=48
STEP=3
CYCLE=0
GRIBPATH=GEFS
UPDATE_FNAME (){
    fnameA=${GRIBPATH}/gep${ENS}.t${CYCLE}z.pgrb2a.0p50.f${HOUR}
    fnameB=${GRIBPATH}/gep${ENS}.t${CYCLE}z.pgrb2b.0p50.f${HOUR}
    OUTPUTFILE=fuego.gefs_p${ENS}.nc
}
################################# 

variables="HGT|TMP|RH|UGRD|VGRD|VVEL|PRES|PRATE|LAND|HPBL|SFCR"
CYCLE=$(printf %02d $CYCLE)

#
# Ensemble member loop
#
for iens in $(seq ${ENSMIN} ${ENSMAX}); do 
    ENS=$(printf %02d $iens)
    #
    # Time loop
    #
    for i in $(seq ${TMIN} ${STEP} ${TMAX}); do 
        HOUR=$(printf %03d $i)
        UPDATE_FNAME

        echo "Processing ${fnameA}..."
        ${WGRIBEXE} "${fnameA}" \
            -match ":(${variables}):" \
            -match ":(([0-9]*[.])?[0-9]+ mb|surface|2 m above ground|10 m above ground):" \
            -nc_table "${TABLEFILE}" \
            -append \
            -nc3 \
            -netcdf \
            "${OUTPUTFILE}" > wgrib.log

        echo "Processing ${fnameB}..."
        ${WGRIBEXE} "${fnameB}" \
            -match ":(${variables}):" \
            -match ":(([0-9]*[.])?[0-9]+ mb|surface|2 m above ground|10 m above ground):" \
            -nc_table "${TABLEFILE}" \
            -append \
            -nc3 \
            -netcdf \
            "${OUTPUTFILE}" >> wgrib.log
    done
done

A set of netCDF files will be generated for each ensemble member: fuego.gefs_p01.nc, fuego.gefs_p02.nc, etc...

Quick visualization using ncview

You can use the ncview tool to generate quick view of the netCDF files. First, load the required modules:

module load netcdf
module load udunits
module load gsl
module load nco
module load ncview

You can execute ncview from the cluster now:

ncview etna.gfs.nc

Notes:

  • To run your X apps remotely, log in to the remote server over SSH with the -X option, which will enable X forwarding on the client end.
ssh -X {username}@server