|
Table of contents
Chapter 1. Technical overview
Chapter 2. FULL-POS post-processing
and interpolation
Chapter 3. Parallel implementation
REFERENCES
|
|
Next
Section
Previous Section
The state variable of the model consists of the spectral components of the
pseudo-divergence, pseudo-vorticity and surface pressure (or its logarithm)
which gives 2*NFLEV+1 horizontal fields. In addition, we have p thermodynamic
variables and q dynamically-passive scalar variables. In the 1997
ECMWF operational model, p = 1 (R * virtual temperature). Specific
humidity is usually kept in grid-point space, where also three prognostic
cloud fields are used. No passive scalar variables are used in the model.
The size of the dynamical part of the state variable of the model is then:
|
(NSMAX+1)*(NSMAX+2)*((2+p+q)*NFLEV +1), where
NSMAX is the degree of the truncation
and NFLEV the number of levels. |
The IFS has been designed to the following constraints:
|
1) it supports a direct integration, and also
an adjoint and tangent-linear integration with associated management
of the trajectory-de facto there are three models in one; |
|
2) it supports two- and three-time-level Lagrangian
advection schemes and an Eulerian advection scheme; |
|
3) it supports a variable-resolution (reduced)
grid; |
|
4) the post-processing is built in; |
|
5) possibility of a restart which gives the same
result; |
|
6) a normal-mode initialization process is included; |
|
7) the Legendre functions and the normal modes
are recomputed for each experiment (essential since the geometry may
vary); |
|
8) coding conventions-DOCTOR norms are used; |
|
9) message passing and macro-tasking with reproducibility; |
|
10) flexible placement in memory during grid-point
calculations and transforms, in order to avoid memory bank conflicts; |
|
11) message passing and macro-tasking intended
to be performed at high level; |
|
12) language-fixed format FORTRAN 90; |
|
13) unification of the vertical interpolation
routines; |
|
14) the collocation grid is independent of the
truncation. |
The tangent-linear model and its adjoint are provided by coding the tangent
linear and the adjoint of each subroutine. This is not the most efficient
solution in terms of CPU but it is much easier to debug since it can be
done routine by routine and the maintenance will obviously be greatly simplified.
Implicit in the following is the fact that we have three grid-point arrays
(t-dt and t+dt, with only the fields, and t
with the fields plus their horizontal derivatives). For the integration
of the model alone, we need only one spectral array (for the state variable),
even considering the post-processing time steps. However during NMI and
the adjoint integration, we need a second one.
Note: The physical package is taken in this part as a `black box'. Its input
are fields at time t1 and t2 and its
output are fields at the same times t1 and t2,
only those at time t2 being modified.
Here follows a description of the data and control flow. The displacement
of the routine name and the number in square brackets indicate the nesting
level within IFS.
Read namelist NAMPAR0.
Determine if distributed-memory version or shared-memory version is used
(LMESSP). If LMESSP is TRUE NMESS is defined to be 1, otherwise it is set
to 0. NMESS is used to dimension some arrays used only in the distributed-memory
version.
Processors are logically divided in a 2D decomposition (NPRGPNS*NPRGPEW
for grid-point calculations and NPRTRW*NPRTRV for Fourier
and Legendre transforms). It is possible to supply only NPROC =
(NPRG-PNS*NPRGPEW and NPRTRW*NPRTRV) and let the
code decide how to make the 2D decomposition.
NPROCK is the number of processors to be used simultaneously
to execute NUMKF Kalman filter forecasts.
Decide the amount of diagnostic output (NOUTPUT and LMPDIAG)
Initialize Message Passing Interface (MPI),
Determine MYPROC, A-set (MYSETA) and B-set (MYSETB)
Based on NOUPUT determine which PE's are writing diagnostics,
the other PE's do not write because LOUTPUT = FALSE, or output data is dumped
to /dev/null
Read command line arguments and GRIB headers
Define variables fixed at level zero. If LMESSP = TRUE
we disable Cray macro-tasking and the use of `out-of-core' work files (LCRAYPVP
= FALSE, LMLTSK = FALSE, LIOXXX = FALSE)
Set defaults for NAMPAR1 variables. These control the layout
of data distribution (LSPLIT, LAPPLE, NAPLAT), tuning communication for
specific architectures (LSLSYNC, LBIDIR, NPHASE, NCOMBFLEN, NLAGA, NLAGB,
NVALAG, NCOSTLAG, NLAGBDY), control of I/O (NINSTR1, NINSTR2, NSTRIN, NSTROUT,
NFLDIN, NFLDOUT, LSPLITIN, LSPLITOUT), and for debugging semi-Lagrangian
code (NSLPAD)
Read NAMPAR1 namelist and initialize timing arrays
Set up separate communication tag ranges to be able to
distinguish communication performed by different message passing routines
Set up MPI identifiers (MINTET, MREALT, MLOGIT, MCHART).
Calculate NFLEVL, NPSURF(), NPSP, NFLEVMX and NUMLL(),
based on NFLEVG and NPROCB. NPROMA is allowed to be any value greater than
zero if LMESSP = TRUE
Allocate arrays. At this point only distributed memory
arrays based on NPROCA, NPROCB and NFLEVL can be allocated.
Defines data structures for lat-long geometry that are
independent of distributed memory configuration. Mainly NLOENG(NDGSAG: NDGENG)
and NMENG(NDGSAG: NDGENG).
Master initialization routine for message-passing related
data structures. Data distributions in grid-point space, Fourier space and
spectral space are defined.
Allocate data structures to be defined in sump.
The data distributions are not yet defined, so only NPROCA/B and NFLEVL
can be used to determine dimensions.
Determines partitioning of zonal wave numbers to PEs for
all truncations involved (NSMAX, NCMAX, NTMAX and NXMAX). Derived quantities
are also calculated here (NASM0(:), NSPOLEGL, NPROCM(:), NUMPP(:), NSPEC,
NSPEC2, NSPEC2MX, NPOSSP and IMYMS(:))
Back in sump NUMP is defined. Each PE calculates Legendre
polynomials for a subset of latitudes in the northern hemisphere. The latitude
distribution is determined here.
Calculate latitude distribution in grid-point space and
Fourier space. Define a number of related data structures.
Define the grid-point columns that belong to this processor.
This is most complex if NPROCB >1 where a splitting in east-west direction
is applied. The splitting depends on the LAPPLE and NAPLAT values. The grid-point
column distribution on all other processors are communicated via message
passing to this PE. This information is stored in NSTA(: , :) and NONL(:
, :).
Defines variables that control the handling of the latitude
split among A-set (LSNDS, LSNDN, LRCVS, LRCVN, NSNDS, NSNDN).
Calculates MYSENDA(:) and MYRECVA(:) for recursive A-set
transposition case. Values depend on NPHASE
Calculates MYSENDA(:) and MYRECVA(:) for bi-directional
communication case
If it is a variational job allocate trajectory arrays,
most dimensioned using NSPEC2.
Generate an operator display. Only processor 1 writes status
to the ifs.stat file
Allocate SPA3(: , : , :) in case NCONF=1
Read restart files if they are available. Fields are read
in on processor 1 and the proper parts are distributed to other processors.
Both spectral fields, surface fields and upper-air grid-point fields are
treated
Read initial data files if restart files are not available
Initialize spectral fields
Initialize 2D-model versions
Initialize 3D-model versions. Spectral GRIB files are (usually)
read on PE1 and decoded using the GRIBEX subroutine. The correct spectral
subsets are distributed to the other processors using MPE_SEND and MPE_RECV
GRIB decoding
If required some spectral fields (e.g. humidity) are transformed
from spectral to grid point space because they are represented in grid point
space in the state vector
Initialize grid-point orography and derivatives of orography
from spectral-input orography. This is optional because a grid-point orography
might be available.
Calculate spectral coefficients of `TV' (TV defined as
RT/Rd) based on humidity and, if available, cloud liquid/ice
properties. Calculation for distributed memory version is done in the dm-code
version.
Perform transforms of field to grid point space where RT/Rd
is calculated.
Calculates explicit normal modes (SPHBUF(:), HOUBUF(:))
for the zonal wave numbers treated by this processor, and frequencies required
for tidal-wave initialization (FREQ(: , :), FREQ2(: , :)) for all NXPECG
zonal wave numbers.
Calculate logical arrays defining Rossby and gravity modes
LTYPGR(: , :), diabatic subset LTYPDB(: , :) , tidal wave subset LTYPTD
(: , :). They are dimensioned using the global NXPECG.
Optional writing of normal modes on external files (which is not done if
LMESS = TRUE).
Calculates implicit normal modes (RPINBUF(:) with dimensioning
based on the local NCPEC).
Initialize incremental variational job variables. No dependencies
on distributed memory.
Initialize Lanczos common variables. No dependencies on
distributed memory.
Setup of the background error constraint JB.
No dependencies on distributed memory.
If it is a variational job allocate trajectory arrays,
most dimensioned using NSPEC2.
Generate an operator display. Only processor 1 writes status
to the ifs.stat file
Allocate SPA3(: , : , :) in case NCONF=1.
Read restart files if they are available. Fields are read
in on processor 1 and the proper parts are distributed to other processors.
Both spectral fields, surface fields and upper air grid point fields are
treated.
Read initial data files if restart files are not available.
Initialize spectral fields.
Initialize 2D-model versions.
Initialize 3D-model versions. Spectral GRIB files are (usually)
read on PE1 and decoded using the GRIBEX subroutine. The correct spectral
subsets are distributed to the other processors using MPE_SEND and MPE_RECV.
If required some spectral fields (e.g. humidity) are transformed
from spectral to grid-point space because they are represented in grid-point
space in the state vector.
Initialize grid-point orography and derivatives of orography
from spectral-input orography. This is optional because a grid-point orography
might be available.
Calculate spectral coefficients of `TV' (TV defined as
RT/Rd) based on humidity and, if available, cloud liquid/ice
properties. Calculation for distributed memory version is done in the dm-code
version.
Perform transforms of field to grid-point space where RT/Rd
is calculated.
Calculate R based on humidity and, if available, cloud
liquid/ice properties.
Calculate spectral norm diagnostics. Partial norm contributions
are calculated on each PE and communicated to PE1 (using the subroutine
commspnorm). On PE1 the global norms are calculated in a reproducible manner.
Normal-mode initialization.
Projection onto subsets of normal modes.
Hough filter of increments
Controls integration for digital filter
Deallocates arrays used in normal-mode calculations
Integration at level 4
Basic time-stepping flow control
Inverse Legendre transforms
Recombine symmetric and antisymmetric parts
Transposition from wave to Fourier space
Multitasking interface to SCAN2M
Computations in grid-point space
Control routine for inverse FFTs
Inverse Fourier-transform driver
Fourier transform
Transposition from Fourier to grid-point space
Control of grid-point computations
Computes auxiliary arrays
Computes ES and RH from T and Q
Computation of t and t - dt at grid points
DDH accumulation
Calculation of variables and dynamical flux-tendency diagnostics
for DDH
Computes CP, R and R/CP from Q
Semi-Lagrangian halo communications
Control of radiation calculations
Lagged grid-point computations
Interface for Semi-Lagrangian interpolations
Interface for ECMWF Physics package
Post Physics computations
Final memory transfers in cpglag
Compute full-level pressure
Copy grid-point arrays from buffer
Control for Fourier transforms
Transposition from grid-point to Fourier space
Control for direct Legendre transforms
Transposition from Fourier to spectral wave space
Legendre transforms
Control spectral-space calculations
Transposition from horizontal to vertical spectral coefficients
Transposition from vertical to horizontal spectral coefficients
Compute spectral norms
Parallel-processing timing.
Next Section
Previous Section
|
