U.S. Department of Energy Office of Biological and Environmental Research

BER Research Highlights


Evaluation of Integrated Assessment Model Hindcast Experiments: a Case Study of the GCAM 3.0 Land Use Module
Published: November 29, 2017
Posted: January 23, 2018

Applying metrics in a case study, scientists found that no single evaluation measure likely exists for all variables in an integrated assessment model.

The Science
Integrated assessment modelers are increasingly conducting hindcast experiments—producing a model forecast for a time period in which observational data are available—across many scales of models. However, a community standard for evaluating integrated assessment models (IAMs) does not exist, making it more difficult to compare results of hindcast experiments from different models. Researchers at the U.S. Department of Energy’s Pacific Northwest National Laboratory presented different evaluation metrics for model output that provide information about varying aspects of model performance and, to aid interpretation, identified helpful performance benchmarks for several of these metrics. They then applied the metrics in a case study.

The Impact
Due to the structure of most IAMs, global aggregate metrics or otherwise highly aggregated skill scores commonly used to evaluate IAM hindcast experiments are likely to mask important deficiencies. Researchers identified an easy-to-manage suite of metrics to evaluate different features of model performance. This suite of metrics may be particularly useful in parameter estimation studies to help identify parameter values that best align with historical data.

Summary
Many types of performance statistics exist for IAMs and other models, but a large number of them operate on a pass-fail basis and offer little insight into why models fail. To make evaluating the large number of variable-region combinations in IAMs more feasible, researchers selected a set of measures that can be applied at different spatial scales (regional versus global). They also identified performance benchmarks for these measures (based on the statistics of the observational data set) that allow models to be evaluated in absolute versus relative terms. An ideal evaluation method for hindcast experiments in IAMs would feature absolute measures for evaluation of a single experiment for a single model. This method also would include relative measures to compare the results of multiple experiments for a single model or the same experiment repeated across multiple models. The performance benchmarks provide information about why a model might perform poorly on a given measure and therefore identify opportunities for improvement.

To demonstrate the use and types of results possible with the evaluation method, researchers applied the measures to results from a past hindcast experiment focused on land allocation in the Global Change Assessment Model (GCAM) version 3.0. Researchers found quantitative evidence that global aggregate metrics alone are insufficient for evaluating IAMs like GCAM that require global supply to equal global demand at each time period. These results indicate that no single evaluation measure likely exists for all variables in an IAM, and therefore sector-by-sector evaluation might be necessary.

Contacts (BER PM)
Bob Vallario
Integrated Assessment Research
Bob.Vallario@science.doe.gov

(PNNL Contact)
Katherine Calvin
Pacific Northwest National Laboratory - Joint Global Change Research Institute
katherine.calvin@pnnl.gov

Funding
This research was based on work supported by the U.S. Department of Energy Office of Science, Biological and Environmental Research as part of the Integrated Assessment Research program.

Publication
Snyder, A.C., R.P. Link, and K.V. Calvin. “Evaluation of Integrated Assessment Model Hindcast Experiments: A Case Study of the GCAM 3.0 Land Use Module.” Geoscientific Model Development 10, 4307-4319 (2017). [DOI: 10.5194/gmd-10-4307-2017]

Related Links
Reference link

Topic Areas:

  • Research Area: Multisector Dynamics (formerly Integrated Assessment)

Division: SC-23.1 Climate and Environmental Sciences Division, BER

 

BER supports basic research and scientific user facilities to advance DOE missions in energy and environment. More about BER

Recent Highlights

Aug 24, 2019
New Approach for Studying How Microbes Influence Their Environment
A diverse group of scientists suggests a common framework and targeting of known microbial processes [more...]

Aug 08, 2019
Nutrient-Hungry Peatland Microbes Reduce Carbon Loss Under Warmer Conditions
Enzyme production in peatlands reduces carbon lost to respiration under future high temperatures. [more...]

Aug 05, 2019
Amazon Forest Response to CO2 Fertilization Dependent on Plant Phosphorus Acquisition
AmazonFACE Model Intercomparison. The Science Plant growth is dependent on the availabi [more...]

Jul 29, 2019
A Slippery Slope: Soil Carbon Destabilization
Carbon gain or loss depends on the balance between competing biological, chemical, and physical reac [more...]

Jul 15, 2019
Field Evaluation of Gas Analyzers for Measuring Ecosystem Fluxes
How gas analyzer type and correction method impact measured fluxes. The Science A side- [more...]

List all highlights (possible long download time)