ERA-40 Archive Plan

2. Archive of observations

2.1 Available observations for use in ERA-40

The observational data for use in ERA-40 include both data from the operational ECMWF archives and data supplied to ECMWF by external institutions for use in re-analysis. Externally supplied data have been stored in the format as received, with only minor representation changes to eliminate machine-dependent features such as Cray blocking. The complexity of these ascii/binary formats vary, and for older observations decoding has often involved the solving of observation-specific problems. Later versions of some datasets were received, with updated information and converted by NCEP to internal NCEP BUFR structure. Data from all sources and structures have been converted at ECMWF to the FM94 BUFR representation and stored in the mass storage system, ecfs, with the name of the source.

The data sources and types are specified in Table 2.1 for the full period 1957-2001. As is explained in the next section all the non-satellite source data have been inserted into a database called PREODB for use in the production assimilation. Data marked as having public availability will be disseminated subject to the rules of data distribution adopted by the Council of ECMWF. Data marked as having restricted availability have been provided to ECMWF under restrictions as to their use or supply to third parties, and thus cannot be made widely available.

2.2 Archive of input observations to the analysis

As may be inferred from the above table, many observations appear in more than one of the data sources. After organizing the data into six-hourly slots the data have been processed from all the different sources into a helper database called PREODB. At the same time basic date and time checks were carried out and a simulated analysis preprocessing step (MAKECMA) performed. All non-satellite data and satellite cloud motion winds (except the reprocessed Meteosat data) are included in the PREODB.

The merging of the multiple data sources, elimination of duplicate observations and other "cleaning" of the data is carried out before the analysis step using in-house "ODB" database software. The archive of these merged observations is being created cycle by cycle as the production streams proceed.

These data can be retrieved from MARS using the following parameters:

class=e4, expver=xxxx, type=ai, expect=any, date=yyyymmdd, time=hh, stream=DA, target="bufrfile"

All types and subtypes are included in the target file. They can be separated by a filter described in here.

The VTPR, TOVS and Meteosat reprocessed data used in the re-analysis are stored in ecfs.

Time series of observations of selected types may also be generated.

2.3 Archive of analysis feedback information in relation to the use of observations

The ERA-40 data assimilation system produces a range of BUFR-encoded feedback data relating to each observation presented to it. The BUFR feedback data contains, in addition to the original input BUFR data, all relevant observation-related information gathered during the data-assimilation cycle. Each of the items of information forms a separate section. Thus, there may be up to seven main feedback sections, of which some may have a few subsections. Each of these sections/subsections starts by first declaring what it is, by using BUFR (unexpanded) descriptors followed by the actual information.

This information includes in particular the departures of background and analysis values from the observation. They also include information relating to the use of the observation by the system, for example the flags set by possible blacklisting of the data, the flags set by climatological and background screening checks and information fed back by the variational quality control. They thus provide a wealth of information concerning the characteristics of the data assimilation and observing systems. Their technical specification is given in the documentation of the observation-processing component of the data assimilation system. The BUFR feedback data are stored in MARS. Data are organized into sets for each six-hour analysis period, with one set for all conventional observations, and one set for each type of satellite data.

These data can be retrieved using the following parameters:

class=e4, expver=xxxx, type=af, expect=any, date=yyyymmdd, time=hh, stream=DA, obstype="see Table 2.2 below", target="feedbackfile"

All types and subtypes are included in the target file. They can be separated by a filter decribed here.

Table 2.2 - Feedback files: Grouping of observations and obstype identifiers for MARS retrieval

Grouping of observations	Obstype parameter
Non satellite observations: surface radiosonde dropsonde profiler pilot aircraft and Satellite cloud motion winds	fbconv
Radiance data: VTPR1, VTPR2 HIRS, MSU, SSU AMSU-A, AMSU-B	fbrad1c
SSM/I Data: Radiances 1D-Var retrievals of TCWV 1D-Var retrievals of surface wind speed	fbssmi
Scatterometer winds	fbscat
Ozone data: TOMS SBUV	fbreo3

Feedback information is only available from the variational analysis. There is no feedback information related to the use observations in the T2m, RH2m, soil moisture, snow and ocean wave height analysis.

Externally produced analyses of sea-surface temperature and sea-ice fraction are used in ERA-40. These are preserved in their original form, as well as in the processed model forms specified below in Section 3.4.