Use and Verification of ECMWF products

This report summarises Member and Co-operating State reports on use and verification of ECMWF forecast products, focussing on the period January 2022 to May 2024. This period included a key model upgrade, cycle 48r1 in June 2023, which ECMWF particularly wanted feedback on.

Many NMS (National Met Service) submissions were quite comprehensive, providing many verification charts and citing various case studies. There are also many new product requests, despite ECMWF having probably added far more products in the last year than ever before.

The “big features” of cycle 48r1, such as the ENS resolution upgrade, the increased frequency of sub-seasonal forecasts (formerly known as “extended range forecasts”) and the multi-layer snow scheme were all very well received. Verification evidence of performance improvements was provided – e.g. for wind speeds over Swiss mountains. Negative meteorological impacts were limited, although some mentioned technical challenges due to the higher volume of data.

In general ECMWF forecast products are liked, respected and widely used, from day 1 out to seasonal timescales, although most usage centres on the medium range – ~day 3 to day 10. Multiple visualisations are used operationally, including ECMWFs OpenCharts and ecCharts offerings.

Users in NMSs are generally very happy with IFS broadscale performance, and with weather parameter outputs. Several countries complimented ECMWF on cyclonic windstorm forecasts, despite some imperfections. Areas where improvements would be desirable, and where verification, survey results and case study data were presented in support, include 2m temperature extremes, winds over mountains, low cloud handling, snowmelt rates, lake and sea ice, and (most of all) convective precipitation and convective hazard prediction.

In general, and relative to the IFS, the skill of Limited Area Models (LAM) deterministic and ensemble systems run by NMSs has risen in recent years. For days 1 and 2 these systems have a performance edge over the IFS, as they should have given their higher resolution (~2.5km versus 9km), and they tend to be used in preference for those leads for weather guidance.

Again, NMSs reported many examples of post-processing and blending, of different types, imparting big improvements to model output, via bias correction and other adjustments. And a number of NMSs are delving into more complex post-processing approaches, related to the worldwide growth of artificial intelligence.

We asked for input on the new breed of data-driven forecast models run in real-time at ECMWF in experimental mode, such as the AIFS. Huge interest was expressed. Though outputs are not currently used in operational production and decision-making, quite a few NMSs mentioned that their forecasters occasionally monitor these outputs. Users want to build trust through better understanding, verification of extremes, seeing more outputs for more parameters and sharing experience.

Uptake of atmospheric composition outputs provided as part of the Copernicus Atmospheric Monitoring Service (CAMS) and Fire-related outputs has grown in recent years, with solar radiation and dust forecasts referenced. Meanwhile CAMS output post-processing, cited a few times, is a new activity.