As the complexity and resolution of global Earth system models (ESM) increase, so does the need for high-resolution data in space and time for the initialisation, verification and post-processing of models. As an example, convection-permitting models require very high-resolution initial conditions to provide a gain in forecast skill. A higher data density can be achieved by getting access to more weather stations. However, tapping into so-called unconventional or novel observations can complement conventional meteorological measurements to increase spatial and temporal resolutions. Here we describe ECMWF’s involvement in projects that investigate the use of such observations.
Multiple sources
Novel observations can be collected from devices that are designed to collect meteorological observations, such as personal weather stations. Parameters include temperature, air pressure, humidity and geographical information. They can also be time series of geophysical variables derived from different data sources, e.g. rainfall from telecommunication tower networks. Novel data sources also include the Internet of things (IoT) and crowdsourced information. There is a continuous growth in data from high-frequency, diverse, unmanaged sources. Harnessing these data has the potential to improve weather forecasts. However, it requires substantial development to combine the information from novel data with conventional observations. Challenges of these novel observations include the low reliability of data streams, uncertainties in measurement quality, and the heterogeneity of data types.
Three projects
ECMWF is involved in three European projects that have a focus on using citizen science and collecting novel observational data from private and public sensors, crowdsourced data and the IoT. The projects are I-CHANGE (Individual Change of Habits Needed for Green European transition – a Horizon 2020 project), TRIGGER (SoluTions foR mItiGatinG climate-induced hEalth thReats – a Horizon Europe project), and AD4GD (All Data for Green Deal). The main objectives of this work are: (1) the development of new infrastructure and the adaptation of existing infrastructure for data acquisition; (2) the adaptation and establishment of data storage infrastructure; and (3) the processing of observations, including a generic quality control framework.
A core task of these projects is developing and maintaining data hubs, which will need to perform quality control, encode observations in standardised forms, and store and index the observations. The data will follow standards including Open Geospatial Consortium (OGC), FAIR data, and European Data Governance standards (see the graph for an overview of the envisioned data infrastructure). The data processing includes data quality monitoring and flags, filtering, and encoding before the data are made available to users. The TRIGGER and I-CHANGE projects also share the idea of engaging citizens in the active collection of hydrometeorological, health and other data in real-life environments, so-called Living Labs (LLs). A number of these LLs are directly involved in setting up the envisioned data infrastructure to facilitate the assimilation and distribution of datasets to the general public. In TRIGGER, data containing personal health information will be randomised and anonymised already in the labs (i.e. clinics participating in TRIGGER) to protect patient confidentiality. The LLs will also be directly involved in the quality control and uncertainty assessment of the datasets before submitting them to the project’s data infrastructure. Project experts are tasked with the development of tools for increasing data usability and interoperability.
ECMWF’s role
ECMWF’s contribution to AD4GD is about connecting new observation sources to the EU’s Copernicus Atmosphere Monitoring Service (CAMS) run by ECMWF and supporting pilot studies that assess the value of socioeconomic and IoT data for estimating greenhouse gas concentrations. I-CHANGE aims to raise awareness of climate issues by making it easier to observe the environmental impacts of human activities, whereas TRIGGER looks at impacts on human health from weather and climate hazards. Both projects will collect environmental and socioeconomic data for further use within the project and for building applications for users. ECMWF’s role is to create data hubs.
These projects will pilot the handling of novel observations with the goal of exploring their potential use in operational models at ECMWF as well as in the EU initiative Destination Earth (DestinE), in which ECMWF participates. The end goal is a system that can merge novel and conventional observations into a high-resolution observational dataset with an estimated uncertainty and quality. There is still a long way to go before this becomes a reality, but these projects are a first step towards using novel observations at ECMWF.
Further information on the three projects can be found at: