We continually monitor the accuracy of our operational forecasts. This evaluation provides essential feedback to both users and model developers on the quality of the forecasting system.
Evaluation of ECMWF forecasts, including the 2021 upgrade
Operational forecast evaluation
The overall performance of the operational medium and extended range forecasts is summarised using a set of headline scores which highlight different aspects of forecast skill.
These headline scores, together with additional verification of medium-range forecast performance are regularly updated on the ECMWF web site.
Evaluation is also carried out for the seasonal forecasts and results are available on the web.
We publish a detailed report on the performance of our forecasts every year as an ECMWF Technical Memorandum. This document presents recent verification statistics and evaluations of ECMWF forecasts (including weather, waves and severe weather events) along with information about changes to the data assimilation and forecasting system. The performance of the extended-range and seasonal forecasts is also included.
The 2020-2021 report is available at Evaluation of ECMWF forecasts, including the 2021 upgrade
Headline scores
In line with ECMWF’s Strategy 2016–2025, we have defined a set of two primary and six supplementary headline scores to evaluate long-term trends in forecast performance. The aim of this set of scores is to assess performance for various forecast lead times for surface weather parameters (such as precipitation, 2m temperature, and 10m wind gusts) as well as for the traditional upper-air fields. Four of the headline scores (two primary and two secondary) are expressed in terms of the lead time at which the score reaches a specific threshold value. The thresholds have been chosen to target the verification on the relevant forecast range for each measure of skill.
The two primary scores are:
- forecast lead-time at which the anomaly correlation of the HRES 500 hPa geopotential reaches 80% for the extra-tropical northern hemisphere; verification against analyses
- forecast lead-time at which the continuous ranked probability skill score (CRPSS) for ENS probabilistic forecasts of 850 hPa temperature reaches 25% for the extra-tropical northern hemisphere; verification against analyses
The supplementary scores are:
- forecast lead-time at which the SEEPS score for HRES forecasts of 24-hour total precipitation reaches 45% for the extra-tropics (northern and southern hemispheres); verification against station observations
- forecast lead-time at which the continuous ranked probability skill score (CRPSS) for ENS probabilistic forecasts of 24-hour total precipitation reaches 10% for the extra-tropics (northern and southern hemispheres); verification against station observations
- HRES tropical cyclone position error at forecast day 3
- Extreme Forecast Index (EFI) skill of 10m wind speed at forecast day 4, evaluated using the Relative Operating Characteristic (ROC); verification against station observations in Europe
- fraction of large ENS errors (CRPS>5K) in 2m temperature at forecast day 5 in the extra-tropics; verification against station observations
- discrete ranked probability skill score (RPSSD) for terciles of the mean 2 m temperature in the northern extra‐tropics in week 3 of the forecast (days 15‐21); the score is based on the evaluation of reforecasts against station observations
The primary and supplementary scores are all displayed on our web site.
Comparison with other centres
The World Meteorological Organisation (WMO) has defined procedures for producing and exchanging standard sets of verification scores. This allows us to compare the quality of our forecasts with those from other centres using a consistent evaluation framework.
The quality of our medium-range forecasts compared to those from other global centres can be seen on the web sites of the designated WMO Lead Centres:
- WMO Lead Centre for Deterministic Forecast Verification
- WMO Lead Centre for Verification of Ensemble Prediction System
- WMO Lead Centre for Wave Forecast Verification
Feedback from Member and Co-operating States
Each year ECMWF invites its Member and Co-operating States to provide feedback about their use of ECMWF forecasts. The national reports they provide include sections on verification. View the 'Application and verification of ECMWF products' reports from recent years in our Publications library.
ECMWF also publishes a summary of this (and other) feedback as an ECMWF Technical Memorandum. This latest report is available at Use and Verification of ECMWF Products in Member and Co-operating States (2021)