U.S. Department of Energy Office of Biological and Environmental Research

BER Research Highlights

New Techniques for Filling Gaps in Instrument Records
Published: February 15, 2015
Posted: May 29, 2015

Best-estimate annual total cloud fraction from radar (black) and radar + lidar (red) observations, filled with information from the SOM technique. 95% confidence intervals are given by the grey and pink shading. [Graph courtesy Kennedy, Dong, and Xi. 2015. DOI: 10.1007/s00704-015-1384-3]

The Atmospheric Radiation Measurement (ARM) Climate Research Facility’s Southern Great Plains (SGP) site in Lamont, Oklahoma, is home to one of the longest records of actively sensed cloud information anywhere in the world. Despite the best efforts of facility staff, however, instruments occasionally fail or are taken down for maintenance, resulting in holes within the observational record.  These gaps lead to uncertainty in monthly statistics of observed variables such as cloud fraction that are often used to evaluate model simulations or diagnose trends in the observations.

Researchers funded by the Atmospheric System Research (ASR) program used a statistical technique called self-organizing maps (SOM) to reduce uncertainties in the instrument record. The analysis took advantage of the fact that cloud occurrence is partly controlled by the large-scale environment and that the long time series of ARM measurements allows robust classification into meteorological regimes. Testing a number of SOM configurations, the analysis showed that uncertainty in the monthly total cloud fraction record can be reduced significantly and that the largest gain is provided by SOMs that have a large number of classes and separate data by month.  Using the new technique, uncertainty in monthly total cloud fraction was reduced in half from previous values.

This proof-of-concept work opens the door to a number of other opportunities. The methodology is adaptable to other ARM sites. Further, the results suggest that a combination of ARM observations and reanalyses can provide a better historical record of cloud occurrence prior to the existence of actively sensed observations. Finally, this work can move beyond cloud fraction and the techniques can be applied to other variables such as records of specific cloud types.

Reference: Kennedy, A. D., X. Dong, and B. Xi. 2015. "Cloud Fraction at the ARM SGP Site: Reducing Uncertainty with Self-Organizing Maps," Theoretical and Applied Climatology, DOI: 10.1007/s00704-015-1384-3. (Reference link)

Contact: Sally McFarlane, SC-23.1, (301) 903-0943
Topic Areas:

  • Research Area: Atmospheric System Research
  • Facility: DOE ARM User Facility

Division: SC-23.1 Climate and Environmental Sciences Division, BER


BER supports basic research and scientific user facilities to advance DOE missions in energy and environment. More about BER

Recent Highlights

May 10, 2019
Quantifying Decision Uncertainty in Water Management via a Coupled Agent-Based Model
Considering risk perception can improve the representation of human decision-making processes in age [more...]

May 09, 2019
Projecting Global Urban Area Growth Through 2100 Based on Historical Time Series Data and Future Scenarios
Study provides country-specific urban area growth models and the first dataset on country-level urba [more...]

May 05, 2019
Calibrating Building Energy Demand Models to Refine Long-Term Energy Planning
A new, flexible calibration approach improved model accuracy in capturing year-to-year changes in bu [more...]

May 03, 2019
Calibration and Uncertainty Analysis of Demeter for Better Downscaling of Global Land Use and Land Cover Projections
Researchers improved the Demeter model’s performance by calibrating key parameters and establi [more...]

Apr 22, 2019
Representation of U.S. Warm Temperature Extremes in Global Climate Model Ensembles
Representation of warm temperature events varies considerably among global climate models, which has [more...]

List all highlights (possible long download time)