The EUSTACE project: delivering global, daily information on surface air temperature

As an article on the outcomes of the EUSTACE Horizon 2020 project¹ is published, the project’s coordinator Nick Rayner and her project partners reflect on the experience of undertaking the ground-breaking challenge to create daily estimates of surface air temperature since 1850 across the globe for the first time by combining surface and satellite data using novel statistical techniques.

Daily variations in near-surface air temperature (the temperature about 2 metres above the surface of the ground) affect many aspects of our lives, including our health, food production and environment. It is important to understand these variations so that we can assess to what extent different parts of our lives are vulnerable to them and their extremes.  This is also a first step towards understanding how our lives might change in the future, enabling us to make effective decisions that ensure we continue to thrive as our climate changes. Near-surface air temperature is recorded using measurement stations in situ, and we don't have measurement stations everywhere. So, how do we get information about past daily variations in surface air temperature over the whole Earth? We can’t directly use measurements from satellites, because they record the temperature of surfaces - the ‘skin temperature’ - not the temperature of the air.

Figure 1. Schematic showing difference between satellite and near-surface air temperature (not to scale). Courtesy Janette Bessembinder.

We could use a “dynamical reanalysis”, which combines a state-of-the-art weather forecasting model with observations to recreate weather on any given day. Dynamical reanalyses are a great resource but, where surface observations are sparse, they rely on the effectiveness of the weather forecasting models. How confident can we be in their results there? It's difficult to say, because where observations are sparse, few are available to check them.

Wouldn't it be great if we could use satellites to estimate daily surface air temperature? Wouldn't it also be great if we could combine that information with data from weather stations to help us to produce a data set of daily air temperature back to 1850? Such a data set could help understanding of variations in daily temperature extremes, for example.

That is what we attempted to do in EUSTACE. EUSTACE arose from discussions within the EarthTemp Network between experts in measuring surface air temperature in situ (at weather stations, on scientific expeditions, on board ships and with drifting buoys) and "skin" temperature from satellite measurements of different surfaces (ice, ocean and land). Together they produced a plan to improve monitoring of surface temperature, and the EUSTACE project followed much of this plan.

The EUSTACE project produced the first global daily analysis using satellite and in situ measurements that is not a dynamical reanalysis. Many steps were needed along the way to get the data into the right shape to be able to use them in this way. A lot of the work involved quantifying how certain we could be of the air temperatures from the different data types we brought together.

Professor Chris Merchant of the University of Reading and Principal Investigator of the EarthTemp Network reflects: “A key strength of EUSTACE has been the make-up of the team: from statisticians, to people with statistical understanding, together with domain experts (including in talking to end users). Together we have quantified uncertainties in the temperature information and propagated them through our work, as well as achieving an understanding of how to use the  influence of the measurements in space and time in a balanced, tractable way.”

Darren Ghent, leader of the European Space Agency’s Climate Change Initiative (ESA CCI) Land Surface Temperature project and leader of one of the strands of work in EUSTACE agrees: “EUSTACE developed and applied a consistent approach to producing uncertainty estimates for satellite skin temperature data sets. This is something we have been able to take forwards in our work in ESA CCI.” He continued: “Validation of the EUSTACE uncertainty estimates both for skin and near-surface air temperature using  independent measurements, is an example of best practice in the field.”

Figure 2. Satellite temperature potentially available to projects like EUSTACE in one night (16/01/2006) from combining measurements from MODIS (over land) and AVHRR (over ocean and ice).

Professor Finn Lindgren - a statistician from the University of Edinburgh who originally developed the analysis approach used in EUSTACE - has quantified the leap we made: “This data set is 100,000 times bigger than anything we have attempted using these methods before.”

To make such a long climate data set useful we need consistent data. When instruments or station locations change, this can introduce jumps (or discontinuities) in the data that aren’t a result of a change in the climate. It’s important to try to remove these, a process referred to as homogenisation. Professor Stefan Brönnimann of the University of Bern has been working on this for many years. He reflects: “This is the first time that a global weather station data set of daily air temperatures has been provided with estimated locations of non-climatic discontinuities and their likelihood.”  Gerard van der Schrier, of the Netherlands Met Office (KNMI), adds: “This is also the first time that a pan-European station data set has been homogenised at the daily level to reduce the impact of non-climatic discontinuities.”

Figure 3. An observer taking a near-surface air temperature measurement in the 1960s. Courtesy KNMI.

Understanding what is going on at the poles, where there are few in situ measurements, but many satellite measurements is vital for preserving the fragile ecosystems that exist there. Jacob Hoyer from the Danish Meteorological Institute, and another EUSTACE work package leader, comments: “This is the first time we have calculated near-surface air temperature from satellite over sea ice and the first time it has been done in a systematic way over all ice caps with this many data.”

Combining terabytes of data needs efficient code and good infrastructure. Ag Stephens from the Science and Technology Facilities Council’s Centre for Environmental Data Analysis (CEDA) understands this very well: “EUSTACE created a state-of-the-art ‘big data’ processing pipeline. Providing a reproducible pipeline, with a number of processing stages, is challenging. From the start, EUSTACE has tried to keep track of what it is doing at each stage, tagging data to specific releases of the code.” All the EUSTACE products are accessible from CEDA.

EUSTACE was a proof of concept, and we have learned plenty during the ten years from the original EarthTemp Network concept through to the delivery of the EUSTACE global daily analysis.  It was not perfect and, in hindsight, there have inevitably been things we might have done differently.

The combination of weather station measurements with air temperature estimates from satellite data has boosted the amount of information available in many places. However, in locations where we have access to few daily in situ measurements, for example in Africa, we were unable to efficiently correct biases in our estimates of air temperature from satellites (underscoring the importance of data sharing and data recovery). Also, prior to 1895, we currently have access to few daily weather station measurements, so the analysis is less reliable then than data sets based on more plentiful monthly average data.  Good availability of in situ measurements is key to producing long-term, well-understood and well-validated climate data sets. More funding is needed for making in situ measurements in harder-to-reach places (and recovering data from paper archives). As a global community we need to share more of our measurements openly, so that we can work together to create the products and services that we all need to understand the world’s vulnerability to weather and climate change.

¹EUSTACE has received funding from the European Union's Horizon 2020 Programme for Research and Innovation, under Grant Agreement no. 640171.