U.S. Department of Energy Office of Biological and Environmental Research

BER Research Highlights

Detecting Modeling Problems Early and Quickly
Published: February 03, 2017
Posted: January 26, 2018

Researchers develop a new method for testing the reproducibility of atmosphere model results.

The Science
Weather and climate models are large computer code structures that solve complex systems of mathematical equations. Researchers developed a new, objective, and computationally efficient method to determine whether simulations performed with these codes on new computers or built with new computer software are essentially similar when they don’t match exactly.

The Impact
Weather and climate models that provide important predictions for society are often developed and maintained by a large team of scientists working collaboratively. These teams require objective and efficient testing methods to assure the codes are producing expected behavior as they are being developed in ever-changing computing environments.

As computer codes are revised, or software and hardware environment are changed, there may be times when it is no longer possible to obtain numbers identical “digit for digit” to previous results. In these situations it is very important, and non-trivial, to distinguish whether these differences are just “noise” or discrepancies caused by unintended coding errors or computing-environment problems. Existing methods that evaluate these discrepancies using long-term statistics of model results are too computationally expensive to use for daily testing during phases of very active model development. A team of researchers led by scientists at Pacific Northwest National Laboratory developed a new method just as robust as existing methods, but hundreds of times cheaper. The new test identifies when simulations performed in a new model or computing environment are considered “changed beyond noise level” by recognizing when the numerical error calculated against a benchmark is found to be inconsistent with previously verified values. The team showed that the new method was effective when applied in the Community Atmosphere Model, and they expect that the underlying concept is generally applicable to atmosphere and geophysical models.

Contacts (BER PM)
Dorothy Koch
Earth System Modeling Program

(PNNL Contacts)
Hui Wan
Pacific Northwest National Laboratory

Phil Rasch
Pacific Northwest National Laboratory

This research was supported as part of the Accelerated Climate Modeling for Energy (ACME) program, funded by the US Department of Energy, Office of Science, Office of Biological and Environmental Research (BER). High-performance computing resources were provided by the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, supported by the Office of Science, and the National Center for Atmospheric Research (NCAR) Computational and Information Systems Laboratory.

Wan, H., K. Zhang, P. J. Rasch, B. Singh, X. Chen, and J. Edward. “A new and inexpensive non-bit-for-bit solution reproducibility test based on time step convergence (TSC1.0).” Geoscientific Model Development, 10, 537-552 (2017). [DOI: 10.5194/gmd-10-537-2017]
(Reference link)

Related Links
The design of the new test method was inspired by a previous study that evaluated numerical errors related to time evolution in the Community Atmosphere Model. PNNL Highlight: Tracking Down Time Missteps April 2015

Topic Areas:

  • Research Area: Earth and Environmental Systems Modeling

Division: SC-23.1 Climate and Environmental Sciences Division, BER


BER supports basic research and scientific user facilities to advance DOE missions in energy and environment. More about BER

Recent Highlights

May 10, 2019
Quantifying Decision Uncertainty in Water Management via a Coupled Agent-Based Model
Considering risk perception can improve the representation of human decision-making processes in age [more...]

May 09, 2019
Projecting Global Urban Area Growth Through 2100 Based on Historical Time Series Data and Future Scenarios
Study provides country-specific urban area growth models and the first dataset on country-level urba [more...]

May 05, 2019
Calibrating Building Energy Demand Models to Refine Long-Term Energy Planning
A new, flexible calibration approach improved model accuracy in capturing year-to-year changes in bu [more...]

May 03, 2019
Calibration and Uncertainty Analysis of Demeter for Better Downscaling of Global Land Use and Land Cover Projections
Researchers improved the Demeter model’s performance by calibrating key parameters and establi [more...]

Apr 22, 2019
Representation of U.S. Warm Temperature Extremes in Global Climate Model Ensembles
Representation of warm temperature events varies considerably among global climate models, which has [more...]

List all highlights (possible long download time)