ECMWF maintains a comprehensive range of verification statistics to evaluate the accuracy of its forecasts. Each year, a summary of verification results is presented to ECMWF’s Technical Advisory Committee (TAC). Their views about the performance of the operational forecasting system in 2020 are given in the box.
ECMWF’s headline scores are computed as 12‑month running averages to filter out the annual cycle and better identify trends in forecast performance. This means that the beneficial effect of new model cycles is fully visible only 12 months after implementation. The first figure shows the significant improvement of upper-air ensemble forecast (ENS) skill due to Cycle 46r1 of the Integrated Forecasting System (IFS), which was implemented in June 2019. The second figure shows that the beneficial effect of this cycle includes surface parameters, specifically a further reduction of the fraction of large ENS errors in 2 m temperature. Substantial improvements are also seen in the precipitation forecast. Compared to forecasts from other global modelling centres, ECMWF has been able to maintain the overall lead in the medium range, both for upper-air and surface parameters. It is worth noting that the medium-range forecast performance of the IFS did not show any obvious degradation due to reduced aircraft observations from March 2020 onwards as a result of COVID-19. The signal was apparently sufficiently small to get masked by natural performance variations within the annual cycle, year-to-year atmospheric variability, as well as the positive effects of new and additional observations and the most recent model upgrade.
Parallel pre-operational testing showed that Cycle 47r1, which was implemented on 30 June 2020, brings substantial improvements in the stratosphere as well as slight improvements in the troposphere. These will be fully visible in the operational scores by June 2021.
The position error for forecasts of tropical cyclones increased compared to the previous year due to atmospheric variability, as indicated by forecasts based on the ERA5 reanalysis system, which show a very similar increase. High-resolution (HRES) tropical cyclone intensity errors (as measured by the error in central pressure) have reached their lowest value so far. However, the decrease relative to the previous year is also seen in ERA5.
For ocean waves in the extratropics, ECMWF leads other global wave forecasting systems in terms of significant wave height but ranks closer to average for peak period. In the tropics, ECMWF leads in terms of peak period.
Verification of ensemble forecasts of 2‑metre temperature anomalies in week two in the northern extratropics shows a statistically significant positive trend. There has also been a significant improvement in the headline score which monitors ENS probabilistic skill for weekly mean 2‑metre temperature in week three. In order to increase the signal-to-noise ratio, this score is based on the verification of re-forecasts. The initialization of the re-forecasts moved from ERA-Interim to ERA5 with the implementation of Cycle 46r1 in 2019, which means that the re-forecast verification now gives results that are closer to the actual real-time forecast skill.
Because of the lack of a strong El Niño or La Niña signal, a very strong positive phase of the Indian Ocean Dipole (IOD) peaking towards the end of 2019 became the main tropical driver of global long-range forecast skill. As a result, 2 m temperature anomaly patterns in boreal winter (DJF 2019–20) were reasonably well predicted over ocean areas, including the North Atlantic. In mid- and high-latitude regions of the American and Eurasian continents, however, forecast skill was lower. An extreme positive anomaly over Siberia (exceeding 1.5 standard deviations in some areas) was hinted at but not captured in magnitude, and the cold anomaly in Alaska and northern Canada was missed.
In spring 2020, both the IOD as well as temperature anomalies in the tropical Pacific returned to close-to-neutral values, leaving the 2020 boreal summer without two strong drivers on seasonal timescales. Forecast skill was accordingly low in many areas, including Europe, where a cold anomaly in the Mediterranean region and higher than normal temperatures in much of Scandinavia (forming the western extension of a strong Siberian warm anomaly) were missed in the forecast.
The complete set of annual verification results is available in ECMWF Technical Memorandum No. 880 on ‘Evaluation of ECMWF forecasts, including the 2020 upgrade’, downloadable from https://www.ecmwf.int/en/publications/technical-memoranda.
The following are other sources of information about verification and forecasting system changes.
- Verification pages: http://www.ecmwf.int/en/forecasts/charts
- Inter-comparison of global model forecast skill: http://apps.ecmwf.int/wmolcdnv/
- Ocean wave model intercomparison results: https://confluence.ecmwf.int/display/WLW/WMO+Lead+Centre+for+Wave+Forecast+Verification+LC-WFV
- A list of ‘Known IFS Forecasting Issues’: https://confluence.ecmwf.int/display/FCST/Known+IFS+forecasting+issues
- All IFS forecasting system cycle changes since 1985: http://www.ecmwf.int/en/forecasts/documentation-and-support/changes-ecmwf-model
Assessment of ECMWF’s Technical Advisory Committee, 15–16 October 2020
With regard to its overall view of the performance level of ECMWF’s operational forecasting system, the Committee:
- noted that ECMWF headline scores continue to show high and improving skill, especially in the light of the introduction of 46r1 and 47r1, and particularly for upper air, precipitation and 2 m temperature (away from higher latitudes);
- noted the lead over other centres has been maintained and acknowledged that scores for a number of elements were the highest ever;
- recognised that ECMWF maintains an overall lead compared to other centres in terms of ensemble spread and error and acknowledged that there remains under-dispersion in the summer;
- noted the highest ever medium-range verification scores for EFI wind and acknowledged the recovery of EFI precipitation scores following a recent drop in predictability as shown in ERA5;
- recognised that ECMWF has an overall lead in verification scores for extratropical ocean wave height but not for peak period;
- noted that the winter 2019/2020 SEAS5 forecast was good for many, but not all, regions and appreciated attribution studies looking into these errors; recognised the difficulties in producing good summertime seasonal forecasts;
- noted that ECMWF tropical cyclone track forecast errors were larger than in recent years but central pressure errors were lower, a signal also present in ERA5; appreciated improvement in the tropical cyclone maximum wind–central pressure relationship following 47r1 and welcomed further investigation into extratropical transition;
- noted the highest ever skill for the northern extratropics in week 2 extended-range output relative to persistence; acknowledged the lack of positive trend for weeks 3 and 4 whilst recognising ECMWF’s strategy to improve this;
- appreciated the introduction of new verification metrics and encouraged the development of further forecaster-relevant verification metrics;
- welcomed the move to ecCharts-2 and Dashboard-2 and appreciated the addition of new diagnostics as well as both the continuing support for this service and commitment to further improve its efficiency;
- welcomed the introduction of 47r1, including the extended range of web charts, improvements to convective diagnostics and new tropical cyclone diagnostics;
- appreciated the opportunity to join the Early Availability of ECMWF Data Pilot;
- welcomed proposed improvements at 47r2 and 48r1 to the ensemble vertical and horizontal resolution and changes to the extended range to run daily with 100 members;
- recognised an absence of any obviously detectable degradation in forecast performance in the face of loss of observations due to COVID‑19; appreciated efforts to exploit additional, new and novel observations to address possible shortcomings in observational networks;
- appreciated the continued very good support ECMWF provided to Member and Co-operating States over the last year, particularly in the face of COVID‑19 when training and events such as the annual UEF continued;
- appreciated the training, documentation and feedback processes provided by ECMWF and welcomed future training opportunities and webinars introducing new products and developments.