Applying metrics in a case study, scientists found that no single evaluation measure likely exists for all variables in an integrated assessment model.
Integrated assessment modelers are increasingly conducting hindcast experiments—producing a model forecast for a time period in which observational data are available—across many scales of models. However, a community standard for evaluating integrated assessment models (IAMs) does not exist, making it more difficult to compare results of hindcast experiments from different models. Researchers at the U.S. Department of Energy’s Pacific Northwest National Laboratory presented different evaluation metrics for model output that provide information about varying aspects of model performance and, to aid interpretation, identified helpful performance benchmarks for several of these metrics. They then applied the metrics in a case study.
Due to the structure of most IAMs, global aggregate metrics or otherwise highly aggregated skill scores commonly used to evaluate IAM hindcast experiments are likely to mask important deficiencies. Researchers identified an easy-to-manage suite of metrics to evaluate different features of model performance. This suite of metrics may be particularly useful in parameter estimation studies to help identify parameter values that best align with historical data.
Many types of performance statistics exist for IAMs and other models, but a large number of them operate on a pass-fail basis and offer little insight into why models fail. To make evaluating the large number of variable-region combinations in IAMs more feasible, researchers selected a set of measures that can be applied at different spatial scales (regional versus global). They also identified performance benchmarks for these measures (based on the statistics of the observational data set) that allow models to be evaluated in absolute versus relative terms. An ideal evaluation method for hindcast experiments in IAMs would feature absolute measures for evaluation of a single experiment for a single model. This method also would include relative measures to compare the results of multiple experiments for a single model or the same experiment repeated across multiple models. The performance benchmarks provide information about why a model might perform poorly on a given measure and therefore identify opportunities for improvement.
To demonstrate the use and types of results possible with the evaluation method, researchers applied the measures to results from a past hindcast experiment focused on land allocation in the Global Change Assessment Model (GCAM) version 3.0. Researchers found quantitative evidence that global aggregate metrics alone are insufficient for evaluating IAMs like GCAM that require global supply to equal global demand at each time period. These results indicate that no single evaluation measure likely exists for all variables in an IAM, and therefore sector-by-sector evaluation might be necessary.
Contacts (BER PM)
Integrated Assessment Research
Pacific Northwest National Laboratory - Joint Global Change Research Institute
This research was based on work supported by the U.S. Department of Energy Office of Science, Biological and Environmental Research as part of the Integrated Assessment Research program.
Snyder, A.C., R.P. Link, and K.V. Calvin. “Evaluation of Integrated Assessment Model Hindcast Experiments: A Case Study of the GCAM 3.0 Land Use Module.” Geoscientific Model Development 10, 4307-4319 (2017). [DOI: 10.5194/gmd-10-4307-2017]
SC-23.1 Climate and Environmental Sciences Division, BER
New Approach for Studying How Microbes Influence Their Environment
A diverse group of scientists suggests a common framework and targeting of known microbial processes [more...]
Nutrient-Hungry Peatland Microbes Reduce Carbon Loss Under Warmer Conditions
Enzyme production in peatlands reduces carbon lost to respiration under future high temperatures. [more...]
Amazon Forest Response to CO2 Fertilization Dependent on Plant Phosphorus Acquisition
AmazonFACE Model Intercomparison. The Science Plant growth is dependent on the availabi [more...]
A Slippery Slope: Soil Carbon Destabilization
Carbon gain or loss depends on the balance between competing biological, chemical, and physical reac [more...]
Field Evaluation of Gas Analyzers for Measuring Ecosystem Fluxes
How gas analyzer type and correction method impact measured fluxes. The Science A side- [more...]
List all highlights (possible long download time)