Applying metrics in a case study, scientists found that no single evaluation measure likely exists for all variables in an integrated assessment model.
Integrated assessment modelers are increasingly conducting hindcast experiments—producing a model forecast for a time period in which observational data are available—across many scales of models. However, a community standard for evaluating integrated assessment models (IAMs) does not exist, making it more difficult to compare results of hindcast experiments from different models. Researchers at the U.S. Department of Energy’s Pacific Northwest National Laboratory presented different evaluation metrics for model output that provide information about varying aspects of model performance and, to aid interpretation, identified helpful performance benchmarks for several of these metrics. They then applied the metrics in a case study.
Due to the structure of most IAMs, global aggregate metrics or otherwise highly aggregated skill scores commonly used to evaluate IAM hindcast experiments are likely to mask important deficiencies. Researchers identified an easy-to-manage suite of metrics to evaluate different features of model performance. This suite of metrics may be particularly useful in parameter estimation studies to help identify parameter values that best align with historical data.
Many types of performance statistics exist for IAMs and other models, but a large number of them operate on a pass-fail basis and offer little insight into why models fail. To make evaluating the large number of variable-region combinations in IAMs more feasible, researchers selected a set of measures that can be applied at different spatial scales (regional versus global). They also identified performance benchmarks for these measures (based on the statistics of the observational data set) that allow models to be evaluated in absolute versus relative terms. An ideal evaluation method for hindcast experiments in IAMs would feature absolute measures for evaluation of a single experiment for a single model. This method also would include relative measures to compare the results of multiple experiments for a single model or the same experiment repeated across multiple models. The performance benchmarks provide information about why a model might perform poorly on a given measure and therefore identify opportunities for improvement.
To demonstrate the use and types of results possible with the evaluation method, researchers applied the measures to results from a past hindcast experiment focused on land allocation in the Global Change Assessment Model (GCAM) version 3.0. Researchers found quantitative evidence that global aggregate metrics alone are insufficient for evaluating IAMs like GCAM that require global supply to equal global demand at each time period. These results indicate that no single evaluation measure likely exists for all variables in an IAM, and therefore sector-by-sector evaluation might be necessary.
Contacts (BER PM)
Integrated Assessment Research
Pacific Northwest National Laboratory - Joint Global Change Research Institute
This research was based on work supported by the U.S. Department of Energy Office of Science, Biological and Environmental Research as part of the Integrated Assessment Research program.
Snyder, A.C., R.P. Link, and K.V. Calvin. “Evaluation of Integrated Assessment Model Hindcast Experiments: A Case Study of the GCAM 3.0 Land Use Module.” Geoscientific Model Development 10, 4307-4319 (2017). [DOI: 10.5194/gmd-10-4307-2017]
SC-23.1 Climate and Environmental Sciences Division, BER
Quantifying Decision Uncertainty in Water Management via a Coupled Agent-Based Model
Considering risk perception can improve the representation of human decision-making processes in age [more...]
Projecting Global Urban Area Growth Through 2100 Based on Historical Time Series Data and Future Scenarios
Study provides country-specific urban area growth models and the first dataset on country-level urba [more...]
Calibrating Building Energy Demand Models to Refine Long-Term Energy Planning
A new, flexible calibration approach improved model accuracy in capturing year-to-year changes in bu [more...]
Calibration and Uncertainty Analysis of Demeter for Better Downscaling of Global Land Use and Land Cover Projections
Researchers improved the Demeter model’s performance by calibrating key parameters and establi [more...]
Representation of U.S. Warm Temperature Extremes in Global Climate Model Ensembles
Representation of warm temperature events varies considerably among global climate models, which has [more...]
List all highlights (possible long download time)