Performance analysis of fully implicit Spectral Element Community Atmosphere Model (CAM-SE).
A Trilinos library based implicit solver that uses the Graphics Processing Unit (GPU) within the residual has been implemented within the spectral element CAM. The implicit solver provides accurate solutions for a range of problem types and scales very well on high performance computers, to more than 86,400 cores. The Newton-based implicit algorithms have been evaluated at this scale of complexity for atmosphere fluid flow applications for a wide variety of configurations.
The implicit solver is able to use large time step sizes such that no subcycling of tracers and physics is needed. Also, it shows equal performance to explicit time-stepping for strongly regionally refined configurations. There is potential for increased efficiency using the GPU, however more development of the interface between the solver library and the code is needed.
Several methods utilizing a Newton-Krylov nonlinear solver are evaluated for a range of configurations of the shallow-water dynamical core of the spectral element community atmosphere model to evaluate their computational performance. These configurations are designed to explore the attributes of each method under different but relevant model usage scenarios, including varied spectral order within an element, static regional refinement, and scaling to the largest problem sizes. The limitations and benefits to using explicit Runge-Kutta versus implicit multistep methods, with different parameters and settings, are discussed in light of the trade-offs with Message Passing Interface (MPI) communication and memory and their inherent efficiency bottlenecks. The recommendation for future work using the implicit solvers is conditional based on scale separation and the stiffness of the problem. For the regionally refined configurations, the implicit method has about the same efficiency as the explicit method, without considering efficiency gains from a preconditioner. Initial simulations with OpenACC directives to utilize a GPU when performing function evaluations show improvements locally, and that overall gains are possible with adjustments to data exchanges.
Contacts (BER PMs)
Earth System Modeling
Oak Ridge National Laboratory
Support from the U.S. Department of Energy Office of Science, Biological and Environmental Research, Earth System Modeling Program and the Advanced Scientific Computing Research Office, under the SciDAC program.
Evans, K. J., R. K. Archibald, D. J. Gardner, M. R. Norman, M. A. Taylor, C. S. Woodward, and P. H. Worley. "Performance Analysis of Fully Explicit and Fully Implicit Solvers within A Spectral Element Shallow-Water Atmosphere Model." The International Journal of High Performance Computing Applications (2017). [DOI: 10.1177/1094342017736373].
SC-23.1 Climate and Environmental Sciences Division, BER
BER supports basic research and scientific user facilities to advance DOE missions in energy and environment. More about BER
May 10, 2019
Quantifying Decision Uncertainty in Water Management via a Coupled Agent-Based Model
Considering risk perception can improve the representation of human decision-making processes in age [more...]
May 09, 2019
Projecting Global Urban Area Growth Through 2100 Based on Historical Time Series Data and Future Scenarios
Study provides country-specific urban area growth models and the first dataset on country-level urba [more...]
May 05, 2019
Calibrating Building Energy Demand Models to Refine Long-Term Energy Planning
A new, flexible calibration approach improved model accuracy in capturing year-to-year changes in bu [more...]
May 03, 2019
Calibration and Uncertainty Analysis of Demeter for Better Downscaling of Global Land Use and Land Cover Projections
Researchers improved the Demeter model’s performance by calibrating key parameters and establi [more...]
Apr 22, 2019
Representation of U.S. Warm Temperature Extremes in Global Climate Model Ensembles
Representation of warm temperature events varies considerably among global climate models, which has [more...]
List all highlights (possible long download time)