BER launches Environmental System Science Program. Visit our new website under construction!

U.S. Department of Energy Office of Biological and Environmental Research

BER Research Highlights


Evaluation of Integrated Assessment Model Hindcast Experiments: a Case Study of the GCAM 3.0 Land Use Module
Published: November 29, 2017
Posted: January 23, 2018

Applying metrics in a case study, scientists found that no single evaluation measure likely exists for all variables in an integrated assessment model.

The Science
Integrated assessment modelers are increasingly conducting hindcast experiments—producing a model forecast for a time period in which observational data are available—across many scales of models. However, a community standard for evaluating integrated assessment models (IAMs) does not exist, making it more difficult to compare results of hindcast experiments from different models. Researchers at the U.S. Department of Energy’s Pacific Northwest National Laboratory presented different evaluation metrics for model output that provide information about varying aspects of model performance and, to aid interpretation, identified helpful performance benchmarks for several of these metrics. They then applied the metrics in a case study.

The Impact
Due to the structure of most IAMs, global aggregate metrics or otherwise highly aggregated skill scores commonly used to evaluate IAM hindcast experiments are likely to mask important deficiencies. Researchers identified an easy-to-manage suite of metrics to evaluate different features of model performance. This suite of metrics may be particularly useful in parameter estimation studies to help identify parameter values that best align with historical data.

Summary
Many types of performance statistics exist for IAMs and other models, but a large number of them operate on a pass-fail basis and offer little insight into why models fail. To make evaluating the large number of variable-region combinations in IAMs more feasible, researchers selected a set of measures that can be applied at different spatial scales (regional versus global). They also identified performance benchmarks for these measures (based on the statistics of the observational data set) that allow models to be evaluated in absolute versus relative terms. An ideal evaluation method for hindcast experiments in IAMs would feature absolute measures for evaluation of a single experiment for a single model. This method also would include relative measures to compare the results of multiple experiments for a single model or the same experiment repeated across multiple models. The performance benchmarks provide information about why a model might perform poorly on a given measure and therefore identify opportunities for improvement.

To demonstrate the use and types of results possible with the evaluation method, researchers applied the measures to results from a past hindcast experiment focused on land allocation in the Global Change Assessment Model (GCAM) version 3.0. Researchers found quantitative evidence that global aggregate metrics alone are insufficient for evaluating IAMs like GCAM that require global supply to equal global demand at each time period. These results indicate that no single evaluation measure likely exists for all variables in an IAM, and therefore sector-by-sector evaluation might be necessary.

Contacts (BER PM)
Bob Vallario
Integrated Assessment Research
Bob.Vallario@science.doe.gov

(PNNL Contact)
Katherine Calvin
Pacific Northwest National Laboratory - Joint Global Change Research Institute
katherine.calvin@pnnl.gov

Funding
This research was based on work supported by the U.S. Department of Energy Office of Science, Biological and Environmental Research as part of the Integrated Assessment Research program.

Publication
Snyder, A.C., R.P. Link, and K.V. Calvin. “Evaluation of Integrated Assessment Model Hindcast Experiments: A Case Study of the GCAM 3.0 Land Use Module.” Geoscientific Model Development 10, 4307-4319 (2017). [DOI: 10.5194/gmd-10-4307-2017]

Related Links
Reference link

Topic Areas:

  • Research Area: Multisector Dynamics (formerly Integrated Assessment)

Division: SC-33.1 Earth and Environmental Sciences Division, BER

 

BER supports basic research and scientific user facilities to advance DOE missions in energy and environment. More about BER

Recent Highlights

Jan 11, 2022
No Honor Among Copper Thieves
Findings provide a novel means to manipulate methanotrophs for a variety of environmental and in [more...]

Dec 06, 2021
New Genome Editing Tools Can Edit Within Microbial Communities
Two new technologies allow scientists to edit specific species and genes within complex laborato [more...]

Oct 27, 2021
Fungal Recyclers: Fungi Reuse Fire-Altered Organic Matter
Degrading pyrogenic (fire-affected) organic matter is an important ecosystem function of fungi i [more...]

Oct 19, 2021
Microbes Offer a Glimpse into the Future of Climate Change
Scientists identify key features in microbes that predict how warming affects carbon dioxide emi [more...]

Aug 25, 2021
Assessing the Production Cost and Carbon Footprint of a Promising Aviation Biofuel
Biomass-derived DMCO has the potential to serve as a low-carbon, high-performance jet fuel blend [more...]

List all highlights (possible long download time)