BER launches Environmental System Science Program. Visit our new website under construction!

U.S. Department of Energy Office of Biological and Environmental Research

BER Research Highlights


Evaluation of Integrated Assessment Model Hindcast Experiments: a Case Study of the GCAM 3.0 Land Use Module
Published: November 29, 2017
Posted: January 23, 2018

Applying metrics in a case study, scientists found that no single evaluation measure likely exists for all variables in an integrated assessment model.

The Science
Integrated assessment modelers are increasingly conducting hindcast experiments—producing a model forecast for a time period in which observational data are available—across many scales of models. However, a community standard for evaluating integrated assessment models (IAMs) does not exist, making it more difficult to compare results of hindcast experiments from different models. Researchers at the U.S. Department of Energy’s Pacific Northwest National Laboratory presented different evaluation metrics for model output that provide information about varying aspects of model performance and, to aid interpretation, identified helpful performance benchmarks for several of these metrics. They then applied the metrics in a case study.

The Impact
Due to the structure of most IAMs, global aggregate metrics or otherwise highly aggregated skill scores commonly used to evaluate IAM hindcast experiments are likely to mask important deficiencies. Researchers identified an easy-to-manage suite of metrics to evaluate different features of model performance. This suite of metrics may be particularly useful in parameter estimation studies to help identify parameter values that best align with historical data.

Summary
Many types of performance statistics exist for IAMs and other models, but a large number of them operate on a pass-fail basis and offer little insight into why models fail. To make evaluating the large number of variable-region combinations in IAMs more feasible, researchers selected a set of measures that can be applied at different spatial scales (regional versus global). They also identified performance benchmarks for these measures (based on the statistics of the observational data set) that allow models to be evaluated in absolute versus relative terms. An ideal evaluation method for hindcast experiments in IAMs would feature absolute measures for evaluation of a single experiment for a single model. This method also would include relative measures to compare the results of multiple experiments for a single model or the same experiment repeated across multiple models. The performance benchmarks provide information about why a model might perform poorly on a given measure and therefore identify opportunities for improvement.

To demonstrate the use and types of results possible with the evaluation method, researchers applied the measures to results from a past hindcast experiment focused on land allocation in the Global Change Assessment Model (GCAM) version 3.0. Researchers found quantitative evidence that global aggregate metrics alone are insufficient for evaluating IAMs like GCAM that require global supply to equal global demand at each time period. These results indicate that no single evaluation measure likely exists for all variables in an IAM, and therefore sector-by-sector evaluation might be necessary.

Contacts (BER PM)
Bob Vallario
Integrated Assessment Research
Bob.Vallario@science.doe.gov

(PNNL Contact)
Katherine Calvin
Pacific Northwest National Laboratory - Joint Global Change Research Institute
katherine.calvin@pnnl.gov

Funding
This research was based on work supported by the U.S. Department of Energy Office of Science, Biological and Environmental Research as part of the Integrated Assessment Research program.

Publication
Snyder, A.C., R.P. Link, and K.V. Calvin. “Evaluation of Integrated Assessment Model Hindcast Experiments: A Case Study of the GCAM 3.0 Land Use Module.” Geoscientific Model Development 10, 4307-4319 (2017). [DOI: 10.5194/gmd-10-4307-2017]

Related Links
Reference link

Topic Areas:

  • Research Area: Multisector Dynamics (formerly Integrated Assessment)

Division: SC-33.1 Earth and Environmental Sciences Division, BER

 

BER supports basic research and scientific user facilities to advance DOE missions in energy and environment. More about BER

Recent Highlights

Mar 23, 2021
Molecular Connections from Plants to Fungi to Ants
Lipids transfer energy and serve as an inter-kingdom communication tool in leaf-cutter ants&rsqu [more...]

Mar 19, 2021
Microbes Use Ancient Metabolism to Cycle Phosphorus
Microbial cycling of phosphorus through reduction-oxidation reactions is older and more widespre [more...]

Feb 22, 2021
Warming Soil Means Stronger Microbe Networks
Soil warming leads to more complex, larger, and more connected networks of microbes in those soi [more...]

Jan 27, 2021
Labeling the Thale Cress Metabolites
New data pipeline identifies metabolites following heavy isotope labeling.

Analysis [more...]

Aug 31, 2020
Novel Bacterial Clade Reveals Origin of Form I Rubisco
Objectives

  • All plant biomass is sourced from the carbon-fixing enzyme Rub [more...]

List all highlights (possible long download time)