Xu Li, S. Uppala, J.K. Gibson, R. Saunders

European Centre for Medium-Range Weather Forecasts

Shinfield Park, Reading, RG2 9AX, U.K.

1. Introduction

The Vertical Temperature Profile Radiometer (VTPR) data, available from November 1972 to February 1979, represent a specific developmental stage of the global observation system and they will be of significant interest, if they can be used effectively in a reanalysis project such as ERA-40.

So far, only retrieved VTPR soundings with a horizontal resolution of about 600km have been used in the NCEP/NCAR reanalysis. Based on their experience of this reanalysis, VTPR retrievals are not of sufficient quality to be usable before 1975.

In ERA-40, the aim is to assimilate VTPR raw radiances and the VTPR-1c data, with the observed resolution and through the whole period, directly into a 3-dimensional variational data assimilation system in the same way as for TOVS-1c radiances.

Two pre-processing problems and their solutions are described here, focusing on the VTPR-1c data.

2. Basic characteristics of VTPR

The VTPR is an operational 8-channel sounding system mounted on the NOAA 2 ~ 5 spacecraft. Its main characteristics are summarised in Table 1 and Table 2. Six of the eight channels are used to deduce radiances in the 15m CO₂ band and are very similar to the HIRS/TOVS CO₂ channels. Two other channels are used to deduce radiances at the 18m water vapour absorption band and at 11m in an atmospheric window region; these two radiances are used respectively to determine the atmospheric humidity and to act as a control in the determination of clear radiances for the other seven channels. Figure 1 shows the availability of the VTPR data.

Figure 1. Availability of the VTPR data.

3. Two problems of the VTPR data set

Usually, when the data set and its processing (reading and decoding) software or program are available, we should be able to get all the necessary information and then convert it into BUFR, as required by the data assimilation system. However, there exist additional problems in the pre-processing of the VTPR.

Problem-1: Unreasonable values of the decoded data happen frequently and randomly. This error may extend to all elements of a specific data record.

Problem2: Only part-areas of the pixels are earth-located. In an area with 184 (8 x 23) pixels, only three of them, the "principal points", coloured red (see Figure 2), are available in the original VTPR data set.

Figure 2. Diagram of the VTPR scan pattern and the "principal points". Only the principal points, the pixels in red, are earth-located in original VTPR data set.

4. Solution to Problem 1

To solve problem 1, the first step is to set up rejection criteria for the data records, then try to recover the erroneous data. The format and processing program must, however, first be exactly understood.

4.1 Format of VTPR data

The VTPR data available to ERA-40 are coded as Cray data (64 bit) sets. They are written sequentially in file0, file1,..., file44, orbit by orbit. Each orbit of data is stored as three separate files, corresponding to raw radiance (archive 1), cloud clear radiance (archive 2) and retrievals (archive 3). The coded data are written and therefore read one block at a time. The information necessary for decoding and obtaining a record (header, data or end) may come from a single block or multiple blocks. In the latter case, it means that the record is spanning blocks. For VTPR data, only the data records of archive 1 and archive 3 span blocks.

Table 3 and Table 4 show the format descriptions of the header record and data record of archive 1.

4.2 Processing program available

A program to read and decode the VTPR data is available from NCAR (PROG_1) . Figure 3 is the flowchart:

Figure 3. Flowchart of processing one file of VTPR data in original program.

(1) Read in the first block of a file, regard it automatically as the header of the current file and save it for the decoding of the header record after the type index (itp) of the current file is determined.

(2) Read in the next block, regard it automatically as a data record , determine itp, based on the length of this block (lth bytes ).

(3) Decode the header record.

(4) Fill in the array nlbf, including reading in additional data blocks to handle the data records spanning blocks and ignore the padding blocks.

(5) Decode nlbf and get a data record when lbo = lrth.

(6) When an end record is read in, set numr = 0 for processing the next file.

The meanings of the variables and arrays used in flowcharts and rejection criteria are given in Table 5.

There are three basic steps to obtain a data record:

1. Read the coded data blocks into array nbl.
2. Extract data from nbl and then fill them into nlbf until it contains all the information from a record.
3. Decode nlbf and get a data record.

Figure 4 demonstrates the procedure to get a VTPR-1c data record.

Figure 4. Flowchart of processing one file of VTPR data in the original program.

4.3 Establishment of rejection criteria for a data record

To guarantee data quality, rejection critera must be set up and applied to the full data set. This is especially necessary when the quality control in the data assimilation system is unable to identify the erroneous data.

This needs a detailed inventory, i.e., the full data set is read by the processing program. Every element of a data record, especially of the raw radiance data records, is saved and then checked carefully.

Based on the inventories, the rejection criteria have been set up as follows:

1. Criterion 1: The return information from the reading of a data record is 2 (error).
2. Criterion 2: The ltht (see 4.5) is not equal to 2448.
3. Criterion 3: The time (hour, minute, second) of a data record is unreasonable.
4. Criterion 4: The starting time (year, month, day, hour, minute, second) of an orbit is unreasonable.
5. Criterion 5: The time of a data record is earlier than the starting time of the current orbit.

Here, "unreasonable" means the value is beyond the normal range. Only the time parameters of the elements of a data record are used to set up the criteria.

In practice, criterion 3 is important for archive 1 data. Since the time of a data record is in the third part of nlbf, if it is right, usually everything is right. The exception is when criterion 2 applies.

These five rejections happen independently, therefore, if err, rej1, rej2, rej3, rej4 is the number of the data records rejected by criterion<96>1, 2, 3, 4, 5 respectively, the total number of the rejected data records is:

errt = err + rej1 + rej2 + rej3 + rej4.

If nr is the total number of records, then the error rate is:

err_rate = errt/nr

If the rejection criteria are applied, how many data records will be rejected? To answer this question, Inventory_1 is performed. This can help us understand the quality of the whole of the VTPR data. It also provides a verification of the rejection criteria themselves.

Inventory_1:

A careful check indicates that the rejection criteria work very well. Therefore, the rejection rate is really the error rate.

From Table 6, the average error rate of the VTPR-1c data is 2.95%.

Basically, there are no great problems. The errors happened mainly in three periods:

(1) November 1972 <96> April 1974 (file0 - file9), this corresponds to the first period when NOAA 2 data was available.

(2) January 1976 <96> August 1976 (file2 - file25), this corresponds to the end of NOAA 4 data.

(3) June 1978 <96> October 1978 (file39 - file40), this is in the period of NOAA 5.

In these 3 periods, the error rates are not negligable and can very often reach about 8%.

4.4 Modifying the processing program and data recovery

Reasonably and probably, all the information of the VTPR data should be contained in the original VTPR data set. This means that it should be possible to recover the data through modifying the processing program.

The reasons for the erroneous VTPR data have been revealed by the more complete inventories, e.g., the check of the length of every block.

Generally , the reason is that PROG_1 is unable to handle some unexpected situations in the VTPR data. Although every block in the VTPR data set is correctly formulated, and both step 1 (reading in blocks) and step 3 (decoding) (see Figure 4) are always realised correctly in PROG_1, unfortunately, step 2 (extracting and filling in) occasionally does not interpret the data correctly. This leads to erroneously decoded header or data records.

One example of an unexpected situation is when there is no end record in a data file. If file-1 is an archive 3 file (itp = 3) without an end record, the next file (file-2), which is, for instance, an archive 1 file (itp = 1), will be incorrectly categorised as an archive 3 file by PROG_1. This happens because, in PROG_1, itp is assessed and then updated only if the first block of a file is read in. The decoding routines depend on itp, so when itp is wrong, the decoded record is wrong.

Similarly, if there is no header record in file-2, the first block of file-2, which is actually a data block, will be regarded as a header record.

In order to handle such unexpected situations, PROG_1 has been modified and a new processing program, PROG_2, developed.

PROG_2 is based on the concept that data handing at block level is sufficient for all processes, including all the 3 steps in Figure 4. This make PROG_2 be able to handle the situation in which a file is not complete (no end record or header record).

Figure 5 is the flowchart of PROG_2.

Figure 5. Flowchart of VTPR data processing in the modified program.

The main modifications in PROG_2 include:

(1) Additional assessment is made to determine whether a block is a header record or a data record.

(2) The block type is assessed every time a new block is read in, rather than only when the first block of a file is read in as PROG_1.

(3) For archive 1 data, when a decoded data record is rejected by criterion 3, one data block is skipped.

Inventory_2 is undertaken to evaluate the effects of PROG_2.

From Table 7 and Figure 6, we can see that most of the erroneous VTPR-1c data have been recovered. The number of erroneous VTPR-1c data records has been reduced from 49999, corresponding to 58 days' data, to 14961, corresponding to 17 days' data. The average error rate has reduced from 2.95% to 0.88%. The three periods with a large error rate have been reduced to one period.

Figure 6. The error rates of the VTPR-1c data in file0 -file43.

4.5 An unsolved problem in data recovery

The error rates in Inventory_2 are still about 9% in file39 and file40. The reason is that the length of many archive-1 blocks in these two files is longer than 816 bytes (normal), for instance, 820, 824, 828 and so on, and ltht is larger than 2446 bytes, so rejection criterion 2 occurs very often.

In fact, this means that there is something wrong with the data block. PROG_2 cannot deal with this problem.

5. Solution to Problem 2

The horizontal resolution of the principal points is about 600 km. The cloud cleared radiances and the retrievals are only deduced at these pixels. Since ERA-40 will use the raw radiances with the observed resolution, navigation must be done to earth-locate all the pixels. This has been undertaken with ITPP5.01 software. The recalculated locations are generally consistent with those in the original VTPR data at the same pixels (see Figure 7).

It should be mentioned that in the original VTPR data set the epoch time was sometimes written incorrectly. The files of orbit parameters will therefore be created for NOAA-2 -NOAA-5. The necessary information can be obtained from the results of the inventories.

Figure 7. Results of navigation.

6. Conclusions

(1) The rejection criteria for data records have been set up and applied to the full VTPR data set. The results show that these criteria do indeed act as a quality control to the data in the reading process.

(2) Most of the erroneous VTPR-1c data have been recovered by modifying the processing program.

(3) All the pixels' locations have been re-calculated and are generally consistent with those in the original VTPR data set. This is necessary for the use of the VTPR-1c data in ERA-40.

It is expected that the solutions to the two problems would be useful for the quality and consistency of the reanalysis during the VTPR period (1972 -1978).

McMillin L.M. et al, 1973: Satellite Infrared Soundings from NOAA Spacecraft. NOAA Technical Report NESS 65.