Twenty-one years of wave forecast verification

Routine comparisons of wave forecast data from different models were first informally established in 1995. They were intended to provide a mechanism for assessing the quality of operational wave forecast model output. The comparisons were based on an exchange of model analysis and forecast data at the locations of in-situ observations of significant wave height, wave period and wind speed and direction available via the Global Telecommunication System (GTS). Five European and North American institutions routinely running wave forecast models contributed to that exchange (Bidlot et al., 1998). The Expert Team on Wind Waves and Storm Surges of the Joint Technical Commission for Oceanography and Marine Meteorology (JCOMM) noted the value of the exchange during its first meeting in Halifax, Canada, in June 2003 and endorsed the expansion of the scheme to include other wave forecasting systems. The exchange was subsequently expanded to other global wave forecasting centres and a few regional entities (Table 1).

A review of 21 years of wave verification results shows clear improvements in the quality of wave forecasting, as will be illustrated in this article for significant wave height forecasts. The comparison project has benefitted all participants and should continue to do so. However, the informal character of the exchange prevents a rapid adaptation to new data. For these reasons, the World Meteorological Organization (WMO) is seeking to establish a Lead Centre for Wave Forecast Verification (LC-WFV) with clearly defined interfaces between the participants and the Lead Centre. ECMWF has expressed its interest in becoming the designated Lead Centre.

Data

On a monthly basis, each participating centre provides time series of model data at an agreed list of locations to ECMWF, where the data are collated for subsequent access. Observations are also collected at ECMWF. The combined data are then processed to provide summary statistics. These are made available on the ECMWF website (http://www.ecmwf.int/en/forecasts/charts/) and the JCOMM website (http://www.jcomm.info/). The raw data are also made available to all participants for potential further analysis.

Sea state and ocean surface meteorological in-situ observations are routinely collected by several national organisations via networks of moored buoys or weather ships and fixed platforms deployed in their near-shore and offshore areas of interest. The data are usually exchanged via the GTS. As part of this intercomparison, observations that are not commonly available on the GTS are also gathered on a case-by-case basis. The geographical coverage of the wave data is still very limited. It tends to be limited to areas near the coast and some observations are very close to land. At the present global wave model resolution, only a subset of these locations fall within the wave model grids. Most measurements used in this project are made in the northern hemisphere (see Figure 4 for recent coverage).

Before using observations for verification, care has to be taken to process the data to remove any erroneous observations. It is also necessary to match the temporal scale of model data and observations. This scale matching is achieved by averaging the hourly observation data in time windows centred on verifying times. The original quality control and averaging procedure was discussed in Bidlot et al. (2002). It was extended to include platform data as described in Sætra & Bidlot (2004).

The intercomparison relies on the exchange of model output at a list of locations. Because in-situ networks change over time, updates to the list have been necessary. However, not all participants have been able to update their list at the same time, nor do they provide data for all the same locations. Moreover, some participants only run a limited-area model, use a coarser grid or provide data from a different number of forecasts (Table 1). A fair comparison between the different wave forecasting systems can only be achieved if the same observation–model collocations are used. This constrains the number of systems that can be evaluated at any one time.

Over 20 years of progress

Figure 1a shows the significant wave height forecast skill from September 2015 to August 2016 as measured by the scatter index (Box A) for all systems providing global forecasts from 00 UTC (see Table 1). Figure 1b shows the common locations and the data coverage density (the number of observation-model collocations used relative to the maximum number of possible collocations). This article does not aim to explain why each forecasting system performs differently. Rather, it aims to illustrate the remarkable progress that has been made over the years (Figures 2 and 3). Progress might have come from improvements in atmospheric forcing resulting from a collective effort in developing numerical weather prediction (NWP) systems, and/or advances in the wave model physical parametrizations, numerical methods, data assimilation or improved implementation. It is, however, worth mentioning that METFR and SHOM both use winds from ECMWF, which explains their close similarity with ECMWF in terms of forecast performance.

Figure 2a shows the evolution of 5-year running mean scatter index values for day-5 significant wave height forecasts for an area of the North-East Pacific. The selected offshore buoys have been part of the intercomparison since the early years. The plot was produced with consistent 00 and 12 UTC forecasts at all selected locations. The data coverage density over the full period is also shown (Figure 2b). It is not entirely uniform but the locations have been carefully selected to reflect the wave climate of the area. The decrease in scatter index values is a clear indication of the steady improvements made by all participating centres. There is a degree of convergence in model performance since 2009. Comparable results also hold for shorter forecast ranges (not shown). Similarly, other ocean areas with long-term observational coverage, such as the North-West Atlantic, the North-East Atlantic and the North Sea, generally show the same improving trend for all participants and forecast ranges (Figure 3a–c). However, for enclosed areas such as the Western Mediterranean Sea, progress has been less consistent (Figure 3d).

ECMWF data can be collocated with all available in-situ data. Figure 4 shows that enclosed areas and near-shore locations are indeed much more difficult to model, in particular on the western side of all ocean basins. This is not limited to ECMWF but is a feature of forecasts from all centres (Figure 5). Nonetheless, the quality of wave forecasting as a whole has improved quite dramatically. There is obviously room for further advances. It is believed that institutions engaged in wave forecasting will continue to benefit from this type of inter-validation in the same way as NWP centres have benefitted from the exchange of forecast verification scores under the auspices of the WMO.

Outlook

There has been a slow, yet steady increase in the availability of in-situ wave observations. Space-borne altimeter wave height data have been shown to be of very high quality and are now commonly available (Abdalla & Zuo, 2016). The intercomparison should ideally be extended to include these data. The JCOMM Expert Team on Waves and Coastal Hazards has recommended that the current Wave Forecast Verification project should be formalised by establishing a Lead Centre for Wave Forecast Verification (LC-WFV). ECMWF has responded positively to this request. The designated LC-WFV would coordinate efforts to gather a set of selected model fields relevant to wave forecasting activities under an agreed data exchange protocol. Once the process of gathering the relevant fields is in place, the routine verification against in-situ data will be more flexible and adaptive. Moreover, it will become much easier to include new observational datasets and verification metrics.

The author would like to thank Andy Saulter (UK Met Office), Paul Wittmann (FNMOC), Natacha Bernier (ECCC), Arun Chawla (NCEP), Lotfi Aouf (Météo-France), Thomas Bruns (DWD), Aihong Zhong (BoM), Fabrice Ardhuin (SHOM), Nadao Kohno (JMA), Sanwook Park (KMA), José María García-Valdecasas Bernal (PRTOS), Jacob Woge Nielsen (DMI), Richard Gorman (NIWA), Ana Carrasco (METNO) and Paula Etala (SHNSM) for their contribution to the comparison project and for providing the data that has made this article possible.

Further Reading

Abdalla, S. & H. Zuo, 2016: The use of radar altimeter products at ECMWF. ECMWF Newsletter No. 149, 14–19.

Bidlot, J.-R., M. Holt, P.A. Wittmann, R. Lalbeharry & H.S. Chen, 1998: Towards a systematic verification of operational wave models. Proceedings Third Int. Symposium on WAVES97: November 3-7, 1997, Virginia Beach: American Society of Civil Engineers.

Bidlot, J.-R., D.J. Holmes, P.A. Wittmann, R. Lalbeharry & H.S. Chen, 2002: Intercomparison of the performance of operational ocean wave forecasting systems with buoy data. Wea. Forecasting, 17, 287–310.

Sætra, Ø. & J.-R. Bidlot, 2004: On the potential benefits of using probabilistic forecasts for waves and marine winds based on the ECMWF ensemble prediction system. Wea. Forecasting, 19, 673–689.