Frank K. Wuerthwein - US grants

The Grid computing model connects computers that are scattered over a wide geographic area, allowing their computing power to be shared. Just as the World Wide Web enables access to information, computer grids enable access to distributed computing resources. These resources include detectors, computer cycles, computer storage, visualization tools, and more. Thus, grids can combine the resources of thousands of different computers that are not fully utilized, and assemble these to create a massively powerful resource, and, with GRID software, this resource can be accessible from a personal computer.

For scientists in international collaborations, grid computing provides the power that can enable effective collaborations whose members are widely dispersed geographically. Grids also can enable simulations that might take weeks on a single PC to run in hours on a grid. Further, the development of computing grids also develops new communities. Grids therefore encourage and require people from different countries and cultures to work together to solve problems.

Grid computing works because people participating in grids opt to share their computer power with others. This opens many questions, both social and technical. Who should be allowed to use each grid? Whose job should get priority in the queue to use grid power? What is the best way to protect user security? How will users pay for grid usage? Answering these questions requires all-new technical solutions, each of which must evolve as other grid and information technologies develop. Since grids involve countries and regions all over the world, these solutions must also suit different technical requirements, limitations and usage patterns.

The Large Hadron Collider (LHC), the accelerator facility discussed in this proposal, is a particle accelerator constructed as a collaboration between more than 50 countries. The world's largest machine, it accelerates particles to nearly the speed of light and then steers these particles into 600 million collisions every second. Data from these collisions is expected to change our basic understanding of antimatter, dark energy and more. The LHC will produce 15 million gigabytes of data a year: the storage capacity of around 20,000,000 CDs. Thousands of physicists all over the world want timely access to this data.

The LHC Computing Grid (LCG) combines the computing resources of more than 140 computing centers aiming to harness the power of 100,000 computers to process, analyze and store data produced from the LHC, making it equally available to all partners, regardless of their physical location.

Through this proposal, a new LCG computing model for the CMS experiment at the LHC will enable dynamic access to existing world-wide data caches and will provide the capabilities for applications on any laptop, server, or cluster, to access data seamlessly from wherever it is stored. Data access will no longer require the operation of large scale storage infrastructures local to the participating procesors..

This model will give the distributed physics groups automated access to "Any Data" at "Anytime" from "Anywhere", decreasing data access costs, and reducing application failure. When pre-staging of voluminous datasets into such local storage is not plausible - due to either a lack of local hardware capability or instantaneous demand - it will be replaced with on-demand access of only those data objects the analysis actually requires from any remote site where the data are available. This significantly reduces the overall I/O and storage space requirements outside the CMS computer systems. The total cost of ownership of computer centers at universities throughout the US is thus significantly reduced, as the dominant human as well as hardware costs related to provisioning and operating a storage infrastructure disappear. The project will enable smaller scale university clusters and physicists' desktop computers access to all types of CMS data and thus improve the scientific output of CMS scientists nationwide.

Further, this proposed infrastructure could be used for data-driven research in all fields, ranging from other natural sciences to social sciences and the humanities.

Research in data-intensive fields is increasingly multi-investigator and multi-institutional, depending on ever more rapid access to ultra-large heterogeneous and widely distributed datasets. The Pacific Research Platform (PRP) is a multi-institutional extensible deployment that establishes a science-driven high-capacity data-centric 'freeway system.' The PRP spans all 10 campuses of the University of California, as well as the major California private research universities, four supercomputer centers, and several universities outside California. Fifteen multi-campus data-intensive application teams act as drivers of the PRP, providing feedback to the technical design staff over the five years of the project. These application areas include particle physics, astronomy/astrophysics, earth sciences, biomedicine, and scalable multimedia, providing models for many other applications.

The PRP builds on prior NSF and Department of Energy (DOE) investments. The basic model adopted by the PRP is 'The Science DMZ,' being prototyped by the DOE ESnet. (A Science DMZ is defined as 'a portion of the network, built at or near the campus local network perimeter that is designed such that the equipment, configuration, and security policies are optimized for high-performance scientific applications rather than for general-purpose business systems'). In the last three years, NSF has funded over 100 U.S. campuses through Campus Cyberinfrastructure - Network Infrastructure and Engineering (CC-NIE) grants to aggressively upgrade their network capacity for greatly enhanced science data access, creating Science DMZs within each campus. The PRP partnership extends the NSF-funded campus Science DMZs to a regional model that allows high-speed data-intensive networking, facilitating researchers moving data between their laboratories and their collaborators' sites, supercomputer centers or data repositories, and enabling that data to traverse multiple heterogeneous networks without performance degradation over campus, regional, national, and international distances. The PRP's data sharing architecture, with end-to-end 10-40-100Gb/s connections, provides long-distance virtual co-location of data with computing resources, with enhanced security options.

Particle physics experiments at the Large Hadron Collider (LHC) at CERN seek to explore fundamental questions in modern physics, such as how particles attain mass, why gravity is weak, and the nature of dark matter. The large quantity of data produced at experimental facilities such as the LHC requires the development of complex pattern recognition algorithms and software techniques to achieve the scientific goals of these physics programs. This project will investigate new algorithms and techniques for data analysis using emerging computing processor architectures. These activities will enable the LHC experiments to take data more efficiently and improve the quality of the data that is recorded in order to extend the reach of the next generation of discoveries from planned hardware upgrades at the LHC over the next decade. The results of this research will significantly reduce the cost of computing for all LHC experiments. Software source code tools will be made available to the particle physics community. The investigators will host workshops to train post-doctoral fellows and graduate students from all areas of particle physics on how to use these advanced techniques. This training is valuable preparation for dealing with big data science in general.

This project will support research focused on novel compute architectures for parallelized and vectorized charged particle track reconstruction. This research will improve the reach of energy-frontier particle physics experiments, such as the ATLAS, CMS, LHCb and ALICE experiments at the LHC and any other fields where studying the passage of charged particles is of critical importance. The science targeted in this project includes studying the properties of the Higgs boson, probing dark matter by searching for supersymmetry, and exploring the unknown by looking for such proposed effects as large extra space-time dimensions.

Academic researchers need a simple data sharing architecture with end-to-end 10-to-100Gbps performance to enable virtual co-location of large amounts of data with computing. End-to-end is a difficult problem to solve in general because the networks between ends (campuses, data repositories, etc.) typically traverse multiple network management domains: campus, regional, and national. No one organization owns the responsibility for providing scientists with high-bandwidth disk-to-disk performance. Toward the National Research Platform (TNRP), addresses issues critical to scaling end-to-end data sharing. TNRP will instrument a large federation of heterogeneous "national-regional-state" networks (NRSNs) to greatly improve end-to-end network performance across the nation.

The goal of improving end-to-end network performance across the nation requires active participation of these distributed intermediate-level entities to reach out to their campuses. They are trusted conveners of their member institutions, contributing effectively to the "people networking" that is as necessary to the development of a full National Research Platform as is the stability, deployment, and performance of technology. TNRP's collaborating NRSNs structure leads to engagement of a large set of science applications, identified by the participating NRSNs and the Open Science Grid.

TNRP is highly instrumented to directly measure performance. Visualizations of disk-to-disk performance with passive and active network monitoring show intra- and inter-NSRN end-to-end performance. Internet2, critical for interconnecting regional networks, will provide an instrumented dedicated virtual network instance for the interconnection of TNRP's NRSNs. Cybersecurity is a continuing concern; evaluations of advanced containerized orchestration, hardware crypto engines, and novel IPv6 strategies are part of the TNRP plan.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

In 2015, the NSF-funded LIGO Observatory made the first-ever detection of gravitational waves, from the collision of two black holes, a discovery that was recognized by the 2017 Nobel Prize in Physics. In 2017, LIGO and its sister observatory Virgo in Italy made the first detection of gravitational waves from another extreme event in the Universe - the collision of two neutron stars. Gamma rays from the same neutron star collision were also simultaneously detected by NASA's Fermi space telescope. Meanwhile, the NSF-funded IceCube facility, located at the U.S. South Pole Station, has made the first detection of high-energy neutrinos from beyond our galaxy, giving us unobstructed views of other extreme objects in Universe such as supermassive black holes and supernova remnants. The revolutionary ability to observe gravitational waves, neutrinos, and optical and radio waves from the same celestial events has launched the era of "Multi-Messenger Astrophysics," an exciting new field supported by one of NSF's ten Big Ideas, "Windows on the Universe".

The success of Multi-Messenger Astrophysics depends on building new data infrastructure to seamlessly share, integrate, and analyze data from many large observing instruments. The investigators propose a cohesive, federated, national-scale research data infrastructure for large instruments, focused initially on LIGO and IceCube, to address the need to access, share, and combine science data, and make the entire data processing life cycle more robust. The novel working model of the project is a multi-institutional collaboration comprising the LIGO and IceCube observatories, Internet2, and platform integration experts. The investigators will conduct a fast-track two-year effort that draws heavily on prior and concurrent NSF investments in software, computing and data infrastructure, and international software developments including at CERN. Internet2 will establish data caches inside the national network backbone to optimize the LIGO data analysis. The goal is to achieve a data infrastructure platform that addresses the production needs of LIGO and IceCube while serving as an exemplar for the entire scope of Multi-messenger Astrophysics and beyond. In the process, the investigators are prototyping a redefinition of the role the academic internet plays in supporting science.

This project is supported by the Office of Advanced Cyberinfrastructure in the Directorate for Computer and Information Science and Engineering.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Exascale High Throughput Computing (HTC) demonstrates burst capability to advance multi-messenger astrophysics (MMA) with the IceCube detector. At peak, the equivalent of 1.2 Exaflops of 32 bit floating-point compute power is used. This is equivalent to approximately 3 times the scale of the #1 in the Top500 Supercomputer listing as of June 2019. In one hour, roughly 125 terabytes of input data is used to produce 250 terabytes of simulated data that is stored at the University of Wisconsin, Madison to be used to advance IceCube science. This data amounts to about 5% of the annual simulation data produced by the IceCube collaboration in 2018.

This demonstration tests and evaluates the ability of HTC-focused applications to effectively utilize availability bursts of Exascale-class resources to produce scientifically valuable output and explores the first 32-bit floating point Exaflop science application. The application concerns IceCube simulations of photon propagation through the ice at its South Pole detector. IceCube is the pre-eminent neutrino experiment for the detection of cosmic neutrinos, and thus an essential part of the MMA program listed among the NSF's 10 Big Ideas.

The simulation capacity of the IceCube collaboration is significantly enhanced by efficiently harnessing the power of short-notice Exascale processing capacity at leadership class High Performance Computing systems and commercial clouds. Investigating these capabilities is important to facilitate time-critical follow-up studies in MMA, as well as increasing the overall annual capacity in aggregate by exploiting opportunities for short bursts.

The demonstration is powered primarily by Amazon Web Services (AWS) and takes place in the Fall of 2019 during the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC19) in Denver, Colorado. It is a collaboration between the IceCube Maintenance & Operations program and a diverse set of Cyberinfrastructure projects, including the Pacific Research Platform, the Open Science Grid, and HTCondor. By further collaborating with Internet2 and AWS, the experimental project also explores more generally, large high-bandwidth data flows in and out of AWS. The outcomes of this project will thus have broad applicability across a wide range of domains sciences, and scales, ranging from small colleges to national and international scale facilities.

This project is supported by the Office of Advanced Cyberinfrastructure in the Directorate for Computer & Information Science & Engineering and the Division of Physics in the Directorate of Mathematical and Physical Sciences.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Advances in data-intensive science and engineering research, supported by ongoing developments in cyberinfrastructure, enable new insights and discoveries. Among these are progress in understanding fundamental processes and mechanisms from human public health to the health of the planet; predicting and responding to natural disasters; and promoting the increasing interconnectedness of science and engineering across many fields, including in astronomy, extreme-scale systems management, cell biology, high energy physics, social science, and satellite image analyses. Fundamentally new system architectures are required to accelerate such advances, including capabilities that integrate diverse computing and data resources, research and education networks, edge computing devices, and scientific instruments into highly usable and flexible distributed systems. Such systems provide both technological platforms for conducting research, and can catalyze distributed and multidisciplinary teams, which are developing new and transformative approaches to addressing disciplinary and multidisciplinary research problems.

Recent reports, informed through community visioning, including the NSF supported report “Transforming Science Through Cyberinfrastructure”, note that a cyberinfrastructure (CI) ecosystem designed to be open and scalable, and to grow with time may advance through in kind contributions of compute and data resources by the national science and education community. This CI ecosystem may be viewed, “more holistically as a spectrum of computational, data, software, networking, and security resources, tool and services, and computational and data skills and expertise that can be seamlessly integrated and used, and collectively enable new, transformative discoveries across S&E [science and education]”.

Aligned with this vision of a national scale CI ecosystem, the San Diego Supercomputer Center (SDSC) at the University of California, San Diego (UCSD), in association with partners at the University of Nebraska, Lincoln (UNL) and the Massachusetts Green High Performance Computing Center (MGHPCC), will deploy the “Prototype National Research Platform” (NRP). This novel, national scale, distributed testbed architecture includes: a high performance subsystem to be deployed at SDSC that integrates advanced processors to be available in association with extremely low latency national Research and Educational (R&E) networks operating at multiple 100Gbps speeds; additional highly optimized subsystems each constituting 288 Graphics Processing Units (GPUs) to be deployed at the University of Nebraska, Lincoln (UNL) and the Massachusetts Green High Performance Computing Center (MGHPCC), to be also interconnected to the R&E networks at 100Gbps speeds at each location; a minimum of additional 1 PB of high performance disk storage to be deployed at each of the three sites to establish a Content Delivery Network (CDN) providing prototype caliber access to data anywhere in the nation within a round trip time (RTT) of ~10ms to be available through a set of eight optimally positioned 50TB Non Volatile Memory (NVMe)-based network caches; and an innovative system software environment enabling both centralized management of the nationally distributed testbed system. Additionally, the system architecture will remain open to future growth through additional integration of capabilities to be achieved through a novel “bring your own resource” program.

The project is structured as a three-year testbed phase, followed by a two-year allocations phase. During the testbed phase, SDSC researchers, working closely with collaborators at UNL and MGHPCC, as well as with small numbers of research teams, will evaluate the NRP architecture and performance of constituent components. Semiannual workshops will bring teams together to share lessons learned, develop the knowledge and best practices to inform researchers, and explore the innovative architecture to accelerate S&E discoveries from ideas to publications. During the allocations phase, NRP will be available to researchers with projects deemed meritorious by an NSF-approved allocation process. Workshops continue through the allocations phase.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Machine learning (ML) is a rapidly expanding field. Computationally intensive workflows train neural nets and then use the results in smartphones, robots, drones, self-driving vehicles, and to run the Internet of Things. Access to graphics processing units (GPUs) is provided through CHASE-CI’s Nautilus, a highly distributed but centrally managed on-demand computer cluster designed for ML and Computational Media (CM). CHASE-CI provides over 20 campuses the scaffold for adding on-premises compute cycles and fast data handling and it offers researcher-focused support and training. Using CHASE-CI’s detailed measurements of performance, researchers learn to become experts in optimization of their computational resources.

CHASE-CI is a community-building effort that sustains a growing community of ML/CM researchers using a purpose-built continuously enhanced nationally distributed computing and data storage infrastructure. Researchers explore combinations of algorithms and architectures optimized with the help of graphed performance metrics. Researchers benefit from extensive shared workflows and open-source software. They use CHASE-CI’s on-line social media platform to receive and give support and share techniques. Community use of CHASE-CI informs computer architecture discussions about future national cyberinfrastructure research and instructional lab needs. CHASE-CI forms a national on-line community that is easy to join, designed for sharing code, data, and results.

The hardware, software, and socio-technical approaches developed by CHASE-CI have provided a roadmap for broader research uses and student training in ML/CM technologies. Researchers get expanded access to hundreds of GPUs for developing algorithms and software to train sensing devices and visualize results thus engaging the students who will soon join the essential workforce for the ongoing massive expansion of mobile platforms such as robots, drones, and self-driving cars. CHASE-CI impacts social diversity in computer science, broadening participation among Minority-Serving Institutions and underserved states. CHASE-CI thoroughly measures and monitors data access by applications over the regional and national R&E networks.

The repository for the project may be found at prp.ucsd.edu, to be maintained for the length of the project at a minimum. It is the anchor website containing pointers to all the research efforts that build upon the Pacific Research Platform. It contains code repositories, presentations, references like publications, presentations, and recorded lectures, and it maintains and archives an active social media channel. CHASE-CI is led by UC San Diego, partnering with investigators at UC Santa Cruz, The University of Nebraska-Lincoln, Florida Agricultural and Mechanical University, New York University, The University of Illinois at Chicago, and San Diego State University.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.