Yifeng Cui - US grants

Precise information about the structure of the solid Earth comes from seismograms recorded
at the surface of a highly heterogeneous lithosphere. Full-3D tomography based on adjoint and
scattering-integral methods can assimilate this information into three-dimensional models of elastic and
anelastic structure. These methods fully account for the physics of wave excitation, propagation, and
interaction by numerically solving the inhomogeneous equations of motion for a heterogeneous anelastic
solid. Full-3D tomography using adjoint and scattering-integral methods requires the execution of
complex computational procedures that challenge the most advanced high-performance computing
(HPC) systems. Current research is petascale; future research will require exascale capabilities.

This project will establish an interoperable set of community computational platforms?vertically
integrated systems of hardware, software, and wetware that will allow a significant community of
investigators to employ the techniques of full-3D tomography to refine Earth structures. Two tomographic
platforms will be built on highly scalable codes for solving the forward problem: the AWP-ODC 4th-order,
staggered-grid, finite-difference code, which has been widely used for regional earthquake simulation and
physics-based seismic hazard analysis, and the SPECFEM3D spectral element code, which is capable of modeling wave propagation through aspherical structures of essentially arbitrary complexity on scales ranging from local to global. A third platform, based on the Unified Community Velocity Model (UCVM) software developed by the Southern California Earthquake Center (SCEC), will provide a common framework for comparing and synthesizing Earth models and delivering model products to a wide community of geoscientists. These platforms will be deployed at the NCAR-Wyoming Supercomputing Center (NWSC).

Most of the traditional High-End Computing (HEC) applications and
current petascale applications are written using the Message Passing
Interface (MPI) programming model. Some of these applications are run
in MPI+OpenMP mode. However, it can be very difficult to use MPI or
MPI+OpenMP and maintain performance for applications which demonstrate
irregular and dynamic communication patterns. The Partitioned Global
Address Space (PGAS) programming model presents a flexible way for
these applications to express parallelism. Accelerators introduce
additional programming models: CUDA, OpenCL or OpenACC. Thus, the
emerging heterogeneous architectures require support for various
hybrid programming models: MPI+OpenMP, MPI+PGAS, and MPI+PGAS+OpenMP
with extended APIs for multiple levels of parallelism. Unfortunately,
there is no unified runtime which delivers the best performance and
scalability for all of these hybrid programming models for a range of
applications on current and next-generation HEC systems. This leads
to the following broad challenge: "Can a unified runtime for hybrid
programming model be designed which can provide benefits that are
greater than the sum of its parts?"

A synergistic and comprehensive research plan, involving computer
scientists from The Ohio State University (OSU) and Ohio Supercomputer
Center (OSC) and computational scientists from the Texas Advanced
Computing Center (TACC) and San Diego Supercomputer Center (SDSC),
University of California San Diego (UCSD), is proposed to address the
above broad challenge with innovative solutions. The investigators
will specifically address the following challenges: 1) What are the
requirements and limitations of using hybrid programming models for a
set of petascale applications? 2) What features and mechanisms are
needed in a unified runtime? 3) How can the unified runtime and
associated extension to programming model APIs be designed and
implemented? 4) How can candidate petascale applications be
redesigned to take advantage of proposed unified runtime? and 5) What
kind of benefits (in terms of performance, scalability and
productivity) can be achieved by the proposed approach? The research
will be driven by a set of applications from established NSF
computational science researchers running large scale simulations on
Ranger and other systems at OSC, SDSC and OSU. The proposed designs
will be integrated into the open-source MVAPICH2 library. The
established national-scale training and outreach programs at TACC,
SDSC and OSC will be used to disseminate the results of this research.

Earthquakes have major economic and societal consequences as can be seen from the aftermath of the recent large earthquakes in Japan, Chile, and New Zealand. This multidisciplinary project, which includes both geoscientists, computer scientists, and structural engineers, integrates high-level and middle-level scientific software elements developed by the Southern California Earthquake Center (SCEC) into a software environment for integrated seismic modeling that can be used for seismic hazard analysis. The framework includes integration of community velocity models, codes for dynamic and pseudo-dynamic rupture generation, deterministic and stochastic earthquake engines, and the applications necessary to employ forward simulations in two types of inverse problems: seismic source imaging and full 3D tomography. Modifications to already existing software packages slated to be significantly enhanced in the course of the workflow will allow simulations to be run on petascale machines and allow the better managing of scientific workflows. The work also focuses on software lifecycle issues such as model formation, verification, prediction, and validation and support the use of petascale computers by earthquake scientists. The goal of the project is to facilitate the incorporation of better theory and data into computationally intensive modeling of earthquake processes. Software will be designed to interface smoothly with OpenSHA, as well as OpenSEES, PEER, and NEES. Project partners will also develop and test two computational platforms, one that will have a user-friendly interface for calculating seismographs and the other will generate large suites of simulations for a layered earthquake hazard model. Models will be validated against datasets for 13 well-recorded historic California earthquakes of magnitude 6.0 or higher. The initial API will take advantage of the asynchronous IO features of Fortran 2003 with plans for adding C/C++ and Python interfaces. All codes developed will be open-source and publicly available and software distribution will be accompanied by sample input datasets and example forecast results. Broader impacts include the development of a new generation of time-dependent earthquake forecasts to produce ground-shake hazard maps, partnership with a federal agency and the private sector. It also includes a component of student and postdoctoral training and outreach to user communities. Undergraduate interns, many of whom have historically been from groups under-represented in STEM fields, will be trained in use of the software during an 8-week summer training course.

The principal investigators of this proposal are a collaboration of seismologists, computer scientists, and structural geologists who are developing computational platforms that can combine seismological and geological information into three dimensional (3D) representations of Earth structure. They derive the seismological information from earthquakes, controlled-source experiments, and observations of the ambient seismic field generated by ocean waves and other near-surface disturbances. They derive the geological information from an even wider variety of sources: field mapping, electromagnetic remote sensing, geotechnical measurements, drilling and well-logging, fault trenching, and laboratory measurements of material properties.

The systematic integration of such diverse datasets into unified structural representations (USRs) is a key problem of geoinformatics. The goal of this project is to develop for community use the complex cyberinfrastructure needed to manage the ?USR lifecycle?, including the effective application of high performance computing (HPC) to invert joint datasets in the iterative improvement of USRs. The lifecycle is initiated by the integration of several components into a starting model; the starting model is refined by full-3D tomographic inversion of seismic waveform data; the refined model is validated for its proposed applications and then disseminated to user communities. This new model becomes the main component of the starting model for the next lifecycle.

Earthquake simulations at the spatiotemporal scales required for probabilistic seismic hazard analysis (PSHA) present some of the toughest computational challenges in geoscience. PSHA is the scientific basis for many engineering and social applications: performance-based design, seismic retrofitting, resilience engineering, insurance-rate setting, disaster preparation and warning, emergency response, and public education. This project will extend deterministic earthquake simulations to seismic frequencies of 2 Hz and greater with the goal of reducing the epistemic uncertainties in physics-based PSHA. The research will address fundamental scientific problems that limit the scale range in current representations of source physics, anelasticity, and geologic heterogeneity. The research will improve the physical representations of earthquake processes and the deterministic codes for simulating earthquakes, which will benefit earthquake system science worldwide. The consequent decrease in mean exceedance probabilities, which could be up to an order of magnitude at high hazard levels, would have a broad impact on the prioritization and economic costs of risk-reduction strategies.

Previous research on Blue Waters has verified the scalability and computational readiness of the simulation codes. These codes will be used to advance physics-based PSHA through a coordinated program of numerical experimentation and large-scale simulation targeted at three primary objectives: (1) validation of high-frequency simulations against seismic recordings of historical earthquakes; (2) computation of high-frequency CyberShake hazard models for the Los Angeles region to support the development of high-resolution urban seismic hazard maps by the U. S. Geological Survey and the Southern California Earthquake Center (SCEC), and (3) high-frequency simulation of a M7.8 earthquake on the San Andreas fault to revise the 2008 Great California ShakeOut scenario and improve the risk analysis developed in detail for that scenario. The plan to accomplish this research has five computational milestones: (a) dynamic rupture simulations up to 8 Hz that include fault roughness and near-fault plasticity; (b) simulations of historic earthquakes up to 4 Hz for model validation; (c) simulations of the 1994 Northridge Earthquake up to 8 Hz for verification and validation; (d) extension of the 2008 ShakeOut scenario to 4 Hz; and (e) calculation of a complete CyberShake hazard model for the Los Angeles region up to 2 Hz.

The Software Environment for Integrated Seismic Modeling (SEISM) Project of the Southern California Earthquake Center (SCEC) will develop advanced earthquake simulation software capable of using high-performance computing to produce new information about earthquakes and the hazards they present. SCEC's SEISM project is developing an integrated, sustainable community software framework for earthquake system science to serve diverse communities of earthquake scientists and engineers, computer scientists, and at-risk stakeholders. The SEISM project is a collaboration among several diverse user communities with shared interests in reducing seismic risk and enhancing seismic resilience. SCEC SEISM researchers are addressing scientific problems that limit the accuracy and scale in current numerical representations of earthquake processes. SEISM computational improvements in seismic hazard calculations will benefit earthquake system science worldwide. The SCEC SEISM project will educate a diverse STEM workforce from the undergraduate to early-career level, and it will cross-train scientists and engineers in a challenging high-performance environment. As one application of SEISM, the researchers will develop new simulations for the Great California ShakeOut, which is engaging millions of people in earthquake preparedness exercises.

Earthquake simulations at the spatiotemporal scales required for probabilistic seismic hazard analysis present some of the toughest computational challenges in geoscience, requiring extreme-scale computing. The Southern California Earthquake Center is creating a Software Environment for Integrated Seismic Modeling (SEISM) that will provide the extreme-scale simulation capability needed to transform probabilistic seismic hazard analysis into a physics-based science. This project will advance SEISM through a user-driven research and development agenda that will push validated SEISM capabilities to higher seismic frequencies and towards extreme-scale computing. It will develop an integrated, sustainable community software framework for earthquake system science to serve diverse communities of earthquake scientists and engineers, computer scientists and at-risk stakeholders. A new SEISM-T framework will support both in-situ and post-hoc data processing to make efficient use of available heterogeneous architectures. The main goal of the project is to increase the 4D outer-scale/inner-scale ratio of simulations at constant time-to-solution by two orders of magnitude above current capabilities. The software development plan will use an agile process of test-driven development, continuous software integration, automated acceptance test suites for each application, frequent software releases, and attention to user feedback. The researchers will take advantage of the SCEC Implementation Interface to develop a dialog among user communities regarding the application of SEISM to the reduction of seismic risk and enhancement of seismic resilience. This research will address fundamental scientific problems that limit the accuracy and scale range in current numerical representations of earthquake processes, which will benefit earthquake system science worldwide. This project will educate a diverse STEM workforce from the undergraduate to early-career level, and it will cross-train scientists and engineers in a challenging high-performance environment. As one application of SEISM, the project team will develop new simulations for the Great California ShakeOut, which is engaging millions of people in earthquake preparedness exercises.

Earthquakes emerge from complex, multiscale interactions across time scales that range from milliseconds to millions of years within active faults systems that are incredibly difficult to observe. Large-scale physics-based earthquake simulations are essential scientific tools that can be used to better understand these hazardous natural phenomena. This project will develop physics-based codes for simulating earthquakes on Blue Waters and apply these simulation capabilities to improve existing hazard analysis methods. The very large scale computing and data management capabilities of the Blue Waters system will allow the project to develop and test earthquake models that capture physics in a more realistic manner, and to run simulations at finer resolutions and higher frequencies. The results will better quantify seismic hazards and their uncertainties.

This project will advance physics-based probabilistic seismic hazard analysis (PSHA) methods using numerical experimentation and large-scale simulations to increase the scale range in current representations of source physics, anelasticity, and geologic heterogeneity. Specifically, the research project will work towards seven computational objectives defined to improve our understanding of earthquake processes and advance physics-based PSHA: (1) Develop an empirically-calibrated physics-based earthquake forecast. (2) Develop a statistically sufficient, but reduced, rupture set representative of the new Unified California Earthquake Rupture Forecast. (3) Implement a new dynamic-rupture based kinematic source model to compute ground motions up to 8 cycles per second. (4) Evaluate the basin connectivity phenomenon observed in previous simulations to establish the importance of waveguide modeling at low seismic frequencies. (5) Investigate and characterize the influence of material and source models on the accuracy of ground motion simulations. (6) Validate and calibrate the rheological models used in simulations. (7) Test the physics-based hazard capabilities on a vulnerable embankment dam. The goal of this research is to improve the physical representations of earthquake processes and the deterministic codes for simulating earthquakes, which will improve PSHA practice in the United States and benefit earthquake system science worldwide.