ECMWF Newsletter #180

ECMWF moves to final stages of open data transition

Emma Pidduck
Umberto Modigliani
Victoria Bennett
Florian Pappenberger
Fabio Venuti
Xiaobo Yang
Ruth Coughlan
Maartje Kuilman
Ilaria Parodi

 

ECMWF is moving towards the final stages of its open data transition, with 2024 marking a significant year in achieving this goal. Recognising the economic and societal benefits demonstrated by existing open data initiatives and EU directives, the decision to move to open data was approved by the ECMWF Council in December 2020, with an ambitious target to achieve a fully open status by 2026.

During this transition, a tiered approach has been designed to provide balanced services for all users (see the first figure). Starting with data on a free and open tier, this is a subset of the full ECMWF Catalogue and is available with no charges under the CC-BY-4 open licence. This popular open-source licence allows the use of the data for commercial purposes. It permits redistribution, and the only requirement is that users must attribute the data owner and source. Due to the size of the ECMWF Real-time Catalogue, only a popular subset is freely available on the data portal. It is provided with limited support. Data is also published by popular third-party providers, such as Open-meteo.com, Amazon, Google and Microsoft. It is available with a one-hour delay after the operational dissemination. Also on this access tier are the Open Charts. These are available under the same CC-BY-4 licence, supported by tools such as Jupyter Notebooks to enable replication and adaptation as required.

Tiered access.
Tiered access. Access to ECMWF data and products is currently governed by a three-tier access structure, which will be reduced to the top two tiers by 2026.

In the next tier, open data is available with no data costs, but delivery costs may apply. The conditions of use for this data are consistent with the first tier, but the parameter options are wider, and the delivery time meets the operational schedule.

The final tier is the remaining restricted data at higher temporal and spatial resolutions. The conditions of use for the restricted dataset are governed by the ECMWF Standard Licence Agreement. Once the open data goal is fully achieved in 2026, the data accessible in this tier will move to the second level as ‘open data’ but will not necessarily be free (see the first figure). Users opting for the second and third tiers can pay for one of four service packs with a wide range of features, depending on their specific requirements.

Updates for 2024

In March 2024, the free and open data resolution was increased to 0.25 degrees (28 km), and the parameter subset was expanded to enable users to run limited-area models. In addition, data from the new Artificial Intelligence Forecasting System (AIFS) was added to the open data catalogue, allowing direct comparison with outputs from our Integrated Forecasting System (IFS).

From July 2024, ECMWF has made available all real-time data coarser than or equal to 0.4 degrees (~36 km) under the CC-BY-4 open licence. This means both extended-range ensemble forecast (ENS extended) and seasonal forecast datasets are now open and free of data cost at their native resolution, although they may be subject to service charges.

Additionally, data costs for the remaining restricted data have been reduced by up to 40%, enabling more users to take larger volumes of data to meet their requirements. The unit cost for data has been reduced by 40%, while the Maximum Charge cost has been decreased by 28%.

In addition to applying an open licence to much of the data, making data affordable while ensuring service quality has been part of the overall transition. Service charges, which comprise a volume charge and a service pack cost, have been reduced by an average of 60%, with the upper ceiling of volume limits increasing to 4 TiB per day.

Open data usage

Since releasing a much wider open dataset in March, ECMWF has seen a 30% increase in downloads retrieved via the ECMWF Data Portal, which serves over 400 TB per month and reached ~47 million retrievals in May 2024 (see the second figure). Distribution via third parties further enhances the popularity of these datasets.

Monthly data usage in 2024.
Monthly data usage in 2024. Since January, the data volume downloaded by users has increased to more than 400 TB per month, and the number of retrievals has gone up to about 47 million.

Future plans – 2025 and 2026

The goal is to achieve a fully open status in 2026 by removing the Information Cost (data cost) for all parameters in the ECMWF Real-time Catalogue and providing a subset of the catalogue at no cost to users. The ECMWF Council recently approved proposals to further reduce data costs and to further open up data availability in 2025 as part of the next steps to meet the final goal in 2026.

This includes an increase in the spatial resolution of the current free and open data available at 0.25 degrees to a resolution to match the new ERA6 dataset, which is planned to be 0.125 degrees (~14 km). Supporting this increase in resolution, ECMWF will further decrease the information cost for the remaining restricted data in the catalogue. The unit cost of the data will decrease by 33%, and the Maximum Charge fee will reduce by almost 50% in 2025. These changes will likely be introduced in the second half of 2025 and will be the final price decrease before removing the Information Cost in 2026.

Expanding support to WMO Members

As the open data offering expands, ECMWF will also remove barriers for World Meteorological Organization (WMO) Members by removing all data and service charges for full-resolution data in support of UN initiatives such as Early Warnings for All. Between now and 2027, the quality and accessibility of data will be improved for all WMO Members, first focusing on less developed nations in the WMO's Regional Association for Europe (WMO-RA6) and the countries supported by the WMO SOFF (Systematic Observations Financing Facility) initiative. SOFF aims to support some countries to generate and exchange GBON (Global Basic Observing Network) data, which are critical for improved weather forecasts and climate services.