Dhabaleswar K. Panda - US grants

Panda This project studies wormhole routing in clustered multiprocessor systems. A clustered system raises new communication and synchronization issues in parallel systems by deviating from the traditional assumption of one processor per node. This project addresses these issues by: 1. developing communication algorithms for frequently used communication patterns on clustered systems; 2. developing synchronization mechanisms and algorithms to provide concurrent intercluster and intracluster synchronization; and 3. developing guidelines and architectural supports for interfacing processor clusters to k-ary n-cube wormhole-routed networks. Theoretical and probabilistic analysis together with simulation experiments are being used to develop and validate the results. This research promises significant impact on the development of future clustered scalable systems supporting various programming paradigms such as distributed-memory, cache- coherent shared memory, and distributed-shared memory.

Petascale applications are producing terabytes of data at a great rate. Storage systems in large-scale machines are significantly stressed as I/O rates are not growing as fast to cope with data production. A variety of HPC activities such as writing output and checkpoint data are all stymied by the I/O bandwidth bottleneck. Further to this, the post-processing and subsequent analysis/visualization of computational results is increasingly time consuming due to the widening gap between the storage/processing capacities of supercomputers and users' local clusters.

This research focuses on building a novel in-job dynamic data staging architecture and in bringing it to bear on the looming petascale I/O crisis. To this end, the following objectives are investigated: (i) the concerted use of node-local memory and emerging hardware such as Solid State Disks (SSDs), from a dedicated set of nodes, as a means to alleviate the I/O bandwidth bottleneck, (ii) the multiplexing of traditional user post-processing pipelines and secondary computations with asynchronous I/O on the staging ground to perform scalable I/O and data analytics, (iii) bypassing memory to access the staging area, and (iv) enabling QoS both in the staging ground and in the communication channel connecting it to compute client and persistent storage.

This study will have a wide-ranging impact on future provisioning of extreme-scale machines and will provide formative guidelines to this end. The result of this research will be a set of integrated techniques that can fundamentally change the current parallel I/O model and accelerate petascale I/O pipelines. Further, this research will help analyze the utility of SSDs in day-to-day supercomputing I/O and inform the wider HPC community of its viability.

The pace of scientific discovery has been rapid in recent years owing to cyberinfrastructures that enable researchers to: (a) remotely access distributed computing resources and big data sets, and (b) effectively collaborate with remote peers, at a global-scale. However, wide-adoption of these advances has been a challenge to researchers mainly due to limitations in traditional cyberinfrastructure equipment, policies and engineering practices at campuses.

This project addresses the adoption challenges for researchers within The Ohio State University (OSU), and for their collaborators at the state, national and international levels. The project team is integrating advanced technologies (e.g., 100Gbps connectivity, perfSONAR, OpenFlow, RoCE/iWARP) relevant to the use cases of diverse OSU researchers in a "Science DMZ" environment. A 100Gbps border router at OSU will be setup to connect to the state-funded OARnet-Internet2 peered 100Gbps network in support of the use cases. The project will investigate the tradeoffs to be balanced between researcher flow performance and campus security practices. Project activities also involve wide-area network experimentation to seamlessly integrate OSU's cyberinfrastructure and Science DMZ with a remote campus (at University of Missouri) cyberinfrastructure. This project will create and document the role of a "Performance Engineer on campus", who will be the primary "keeper" and "help-desk" of the Science DMZ equipment, and will function as a liaison with researchers and their collaborators to configure wide-area cyberinfrastructures. Best-practices are to be published, and open-source software applications will be developed for handling researcher application flows in production networks across diverse science/engineering disciplines and multiple university campuses.

The Message Passing Interface (MPI) is a very widely used parallel programming model on modern High-End Computing (HEC) systems. Many performance aspects of MPI libraries, such as latency, bandwidth, scalability, memory footprint, cache pollution, overlap of computation and communication etc. are highly dependent on system configuration and application requirements. Additionally, modern clusters are changing rapidly with the growth of multi-core processors and commodity networking technologies such as InfiniBand and 10GigE/iWARP. They are becoming diverse and heterogeneous with varying number of processor cores, processor speed, memory speed, multi-generation network adapters/switches, I/O interface technologies, and accelerators (GPGPUs), etc. Typically, any MPI library deals with the above kind of diversity in platforms and sensitivity of applications by employing various runtime parameters. These parameters are tuned during its release, or by
system administrators, or by end-users. These default parameters may or may not be optimal for all system configurations and applications.

The MPI library of a typical proprietary system goes through heavy performance tuning for a range of applications. Since commodity clusters provide greater flexibility in their configurations (processor, memory and network), it is very hard to achieve optimal tuning using released version of any MPI library, with its default settings. This leads to the following broad challenge: "Can a comprehensive performance tuning framework be designed for MPI library so that the next generation InfiniBand, 10GigE/iWARP and RoCE clusters and applications will be able to extract `bare-metal' performance and maximum scalability?" The investigators, involving computer
scientists from The Ohio State University (OSU) and Ohio Supercomputer Center (OSC) as well as computational scientists from the Texas Advanced Computing Center (TACC) and San Diego Supercomputer Center (SDSC), University of California San Diego (UCSD), will be addressing the above challenge with innovative solutions.

The investigators will specifically address the following challenges: 1) Can a set of static tools be designed to optimize performance of an MPI library during installation time? 2) Can a set of dynamic tools with low overhead be designed to optimize performance on a per-user and per-application basis during production runs? 3) How to incorporate the proposed performance tuning framework with the upcoming MPIT interface? 4) How to configure MPI libraries on a given system to deliver different optimizations to a set of driving applications? and 5) What kind of benefits (in terms of performance, scalability, memory efficiency and reduction in cache pollution) can be achieved by the proposed tuning framework? The research will be driven by a set of applications from established NSF computational science researchers running large scale simulations on the TACC Ranger and other systems at OSC, SDSC and OSU. The proposed designs will be integrated into the open-source MVAPICH2 library.

Most of the traditional High-End Computing (HEC) applications and
current petascale applications are written using the Message Passing
Interface (MPI) programming model. Some of these applications are run
in MPI+OpenMP mode. However, it can be very difficult to use MPI or
MPI+OpenMP and maintain performance for applications which demonstrate
irregular and dynamic communication patterns. The Partitioned Global
Address Space (PGAS) programming model presents a flexible way for
these applications to express parallelism. Accelerators introduce
additional programming models: CUDA, OpenCL or OpenACC. Thus, the
emerging heterogeneous architectures require support for various
hybrid programming models: MPI+OpenMP, MPI+PGAS, and MPI+PGAS+OpenMP
with extended APIs for multiple levels of parallelism. Unfortunately,
there is no unified runtime which delivers the best performance and
scalability for all of these hybrid programming models for a range of
applications on current and next-generation HEC systems. This leads
to the following broad challenge: "Can a unified runtime for hybrid
programming model be designed which can provide benefits that are
greater than the sum of its parts?"

A synergistic and comprehensive research plan, involving computer
scientists from The Ohio State University (OSU) and Ohio Supercomputer
Center (OSC) and computational scientists from the Texas Advanced
Computing Center (TACC) and San Diego Supercomputer Center (SDSC),
University of California San Diego (UCSD), is proposed to address the
above broad challenge with innovative solutions. The investigators
will specifically address the following challenges: 1) What are the
requirements and limitations of using hybrid programming models for a
set of petascale applications? 2) What features and mechanisms are
needed in a unified runtime? 3) How can the unified runtime and
associated extension to programming model APIs be designed and
implemented? 4) How can candidate petascale applications be
redesigned to take advantage of proposed unified runtime? and 5) What
kind of benefits (in terms of performance, scalability and
productivity) can be achieved by the proposed approach? The research
will be driven by a set of applications from established NSF
computational science researchers running large scale simulations on
Ranger and other systems at OSC, SDSC and OSU. The proposed designs
will be integrated into the open-source MVAPICH2 library. The
established national-scale training and outreach programs at TACC,
SDSC and OSC will be used to disseminate the results of this research.

The recently introduced Single Root I/O Virtualization (SR-IOV)
technique for InfiniBand and High Speed Ethernet provides native I/O
virtualization capabilities and enables us to provision the internal
PCI bus interface between multiple Virtual Machines (VMs). However,
achieving near native throughput for HPC applications that use both
point-to-point and collective operations on virtualized multi-core
systems with SR-IOV presents a new set of challenges for the designers
of high performance middleware, such as MPI. In order to solve this
problem, this project aims to address the following set of challenges:
1) How to redesign MPI communication library to achieve efficient
locality-aware communication and facilitate fair resource sharing on
modern virtualized high performance clusters, with SR-IOV? 2) Can
communication libraries be designed to deliver the best communication
performance across different VM subscription policies and network
communication modes? 3) What are the the challenges involved in
designing support for advanced features such as, live migration,
Quality of Service, and I/O storage virtualization? and 4) What kind
of benefits, in terms of performance and scalability, can be achieved
by the proposed approach for HPC applications? A synergistic and
comprehensive research plan is proposed to address the above
challenges for HPC Virtualization on clusters with SR-IOV and study
its impact for a set of HPC applications.

Managing and processing large volumes of data and gaining meaningful insights is a significant challenge facing the Big Data community. Thus, it is critical that data-intensive computing middleware (such as Hadoop, HBase and Spark) to process such data are diligently designed, with high performance and scalability, in order to meet the growing demands of such Big Data applications. While Hadoop, Spark and HBase are gaining popularity for processing Big Data applications, these middleware and the associated Big Data applications are not able to take advantage of the advanced features on modern High Performance Computing (HPC) systems widely deployed all over the world, including many of of the multi-Petaflop systems in the XSEDE environment. Modern HPC systems and the associated middleware (such as MPI and Parallel File systems) have been exploiting the advances in HPC technologies (multi/many-core architectures, RDMA-enabled networking, NVRAMs and SSDs) during the last decade. However, Big Data middleware (such as Hadoop, HBase and Spark) have not embraced such technologies. These disparities are taking HPC and Big Data processing into "divergent trajectories."

The proposed research, undertaken by a team of computer and application scientists from OSU and SDSC, aim to bring HPC and Big Data processing into a "convergent trajectory." The investigators will specifically address the following challenges: 1) designing novel communication and I/O runtime for Big Data processing while exploiting the features of modern multi-/many-core, networking and storage technologies; 2) redesigning Big Data middleware (such as Hadoop, HBase and Spark) to deliver performance and scalability on modern and next-generation HPC systems; and 3) demonstrating the benefits of the proposed approach for a set of driving Big Data applications on HPC system. The proposed work targets four major workloads and applications in the Big Data community (namely data analytics, query, interactive, and iterative) using the popular Big Data middleware (Hadoop, HBase and Spark). The proposed framework will be validated on a variety of Big Data benchmarks and applications. The proposed middleware and runtimes will be made publicly available to the community. The research enables curricular advancements via research in pedagogy for key courses in the new data analytics program at Ohio State and SDSC -- among the first of its kind nationwide.

A persistent problem facing academic cloud research is the lack of infrastructure and data to perform experimental research: large-scale hardware is needed to investigate the scalability of cloud infrastructure and applications, heterogeneous hardware is needed to investigate algorithmic and implementation tradeoffs, fully-configurable software environments are needed to investigate the performance of virtualization techniques and the differences between cloud software stacks, and data about how clouds are used is needed to evaluate virtual machine scheduling and data placement algorithms.

The Chameleon project will addresses these needs by providing a large-scale, fully configurable experimental testbed driven by the needs of the cloud research and education communities. The testbed, and the ecosystem associated with it, will enable researchers to explore a range of cloud research challenges, from large scale to small scale, including exploring low-level problems in hardware architecture, systems research, network configuration, and software design, or at higher levels of abstraction looking at cloud scheduling, cloud platforms, and cloud applications.

Chameleon will significantly enhance the ability of the computing research community to understand the behavior of Internet scale cloud systems, and to develop new software, ideas and algorithms for the cloud environment. As the tremendous shift to cloud as the primary means of providing computing infrastructure continues, a large-scale testbed tailored to researchers' needs is essential to the continued relevance of a large fraction of computing research.

The project is led by the University of Chicago and includes partners from the Texas Advanced Computing Center (TACC), Northwestern University, the Ohio State University, and the University of Texas at San Antonio, comprising a highly qualified and experienced team, with research leaders from the cloud and networking world blended with providers of production quality cyberinfrastructure. The team includes members from the NSF-supported FutureGrid project and from the GENI community, both forerunners of the NSFCloud solicitation under which this project is funded.

The Chameleon testbed, will be deployed at the University of Chicago (UC) and the Texas Advanced Computing Center (TACC) and will consist of 650 multi-core cloud nodes, 5PB of total disk space, and leverage 100 Gbps connection between the sites. While a large part of the testbed will consist of homogenous hardware to support large-scale experiments, a portion of it will support heterogeneous units allowing experimentation with high-memory, large-disk, low-power, GPU, and co-processor units. The project will also leverage existing FutureGrid hardware at UC and TACC in its first year to provide a transition period for the existing FutureGrid community of experimental users.

To support a broad range of experiments emphasizing a range of requirements ranging from a high degree of control to ease of use the project will support a graduated configuration system allowing full user configurability of the stack, from provisioning of bare metal and network interconnects to delivery of fully functioning cloud environments. In addition, to facilitate experiments, Chameleon will support a set of services designed to meet researchers needs, including support for experimental management, reproducibility, and repositories of trace and workload data of production cloud workloads.

To facilitate the latter, the project will form a set of partnerships with commercial as well as academic clouds, such as Rackspace and Open Science Data Cloud (OSDC). It will also partner with other testbeds, notably GENI and INRIA's Grid5000 testbed, and reach out to the user community to shape the policy an direction of the testbed.

The Chameleon project will bring a new dimension and scale of resources to the CS community who wish to educate their students about design, implementation, operation and applications of cloud computing, a critical skillset for future computing professionals. It will enhance the understanding and application of experimental methodology in computer science and generate new educational materials and resources, with the participation of, and for, Minority Serving Institution (MSI) students.

The project builds a comprehensive research infrastructure to meet the needs of a large research team at the Ohio State University allowing experimental research in a number of computer science areas including high performance computing, data analytics, storage, and virtualization. The project will lead to significant advances in many computer science areas, and its impact will be enhanced through active dissemination of software from the investigators. The project will contribute substantially to human resource development, education in computer science increasing diversity in related areas.

More specifically, researchers will acquire and deploy an infrastructure that includes three types of accelerators, conventional as well as energy-efficient nodes, large main memory including Solid State Drive (SSD) on a subset of nodes, and hardware for fine-grained power measurements. The requested resources will allow experimentation with a number of popular or emerging cluster configurations that address needs of a variety of compute-intensive and/or data-intensive workloads. Such an internally hosted and reconfigurable cluster will also allow power measurements, voltage margin reduction experiments, and failure detection and recovery studies in the presence of physical failures - none of which is typically feasible at national supercomputing and cloud infrastructures. The infrastructure is motivated by the multiple paradigm shifts that high performance computing is presently undergoing. They include increasing use of accelerators or coprocessors, increased criticality of energy and resilience beyond performance, synergy with data analytic applications, and popularity of massive main memory or SSD technologies. These trends provide opportunities and challenges for a variety of scientific, medical, and enterprise applications to be explored in this project.

Message-Passing Interface (MPI) continues to dominate the supercomputing landscape, being the primary parallel programming model of choice. A large variety of scientific applications in use today are based on MPI. On the current and next-generation High-End Computing (HEC) systems, it is essential to understand the interaction between time-critical applications and the underlying MPI implementations in order to better optimize them for both scalability and performance. Current users of HEC systems develop their applications with high-performance MPI implementations, but analyze and fine tune the behavior using standalone performance tools. Essentially, each software component views the other as a blackbox, with little sharing of information or access to capabilities that might be useful in optimization strategies. Lack of a standardized interface that allows interaction between the profiling tool and the MPI library has been a big impediment. The newly introduced MPI_T interface in the MPI-3 standard provides a simple mechanism that allows MPI implementers to expose variables representing configuration parameters or performance measurements from within the implementation for the benefit of tools, tuning frameworks, and other support libraries. However, few performance analysis and tuning tools take advantage of the MPI_T interface and none do so to dynamically optimize at execution time. This research and development effort aims to build a software infrastructure for MPI performance engineering using the new MPI_T interface.

With the adoption of MPI_T in the MPI standard, it is now possible to take positive steps to realize close interaction between and integration of MPI libraries and performance tools. This research, undertaken by a team of computer scientists from OSU and UO representing the open source MVAPICH and TAU projects, aims to create an open source integrated software infrastructure built on the MPI_T interface which defines the API for interaction and information interchange to enable fine grained performance optimizations for HPC applications. The challenges addressed by the project include: 1) enhancing existing support for MPI_T in MVAPICH to expose a richer set of performance and control variables; 2) redesigning TAU to take advantage of the new MPI_T variables exposed by MVAPICH; 3) extending and enhancing TAU and MVAPICH with the ability to generate recommendations and performance engineering reports; 4) proposing fundamental design changes to make MPI libraries like MVAPICH ``reconfigurable'' at runtime; and 5) adding support to MVAPICH and TAU for interactive performance engineering sessions. The framework will be validated on a variety of HPC benchmarks and applications. The integrated middleware and tools will be made publicly available to the community. The research will have a significant impact on enabling optimizations of HPC applications that have previously been difficult to provide. As a result, it will contribute to deriving "best practice" guidelines for running on next-generation Multi-Petaflop and Exascale systems. The research directions and their solutions will be used in the curriculum of the PIs to train undergraduate and graduate students.

Modern High-Performance Computing (HPC) systems are rapidly evolving with respect to processor, networking, and I/O technologies. In such a rapidly changing environment it is critical that the next-generation engineers and scientists get exposed to the modern architectural trends of HPC systems and learn how to use the features of these technologies to design HPC software stacks, learn about the process of open-source software developments and its sustainability. The annual MVAPICH User Group (MUG) meeting provides an open forum to exchange information on the design and usage of MVAPICH2 libraries, which are open-source, high-performance and scalable MPI libraries to take advantage of the RDMA technology. Travel funding from this project will enable a set of students (undergraduates and graduates) to attend the MUG meeting. Their participation will help them to enter the next-generation HPC workforce with increased expertise on software design, reuse, and sustainability. The project thus serves the national interest, as stated by NSF's mission: to promote the progress of science.

The MVAPICH project focuses on the design of high-performance MPI and PGAS runtimes for HPC systems. Over the years, this project has been able to incorporate new designs to leverage novel multi-/many-core platforms like Intel Xeon Phis, NVIDIA GPGPUs, Open POWER, and ARM architectures coupled with Remote Direct Memory Access (RDMA) enabled commodity networking technologies like InfiniBand, RoCE, Omni-Path, and 10/25/40/50/100GigE with iWARP. An annual MVAPICH User Group (MUG) meeting was created five years ago to provide an open forum to exchange information on MVAPICH2 libraries. The funding under this grant aims to achieve increased participation of undergraduate and graduate students working in the HPC area (systems and applications) in the annual MUG event. The requested student travel fund will help in attracting a set of students from a range of US institutions. The participation in an international event such as MUG will enable the students to get a global picture of the developments happening in the rapidly evolving HPC domain and open-source software design. The selection committee will stress on diversity to attract students from minority and under-represented groups.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

This award was partially supported by the CIF21 Software Reuse Venture whose goals are to support pathways towards sustainable software elements through their reuse, and to emphasize the critical role of reusable software elements in a sustainable software cyberinfrastructure to support computational and data-enabled science and engineering.

Parallel programming based on MPI (Message Passing Interface) is being used with increased frequency in academia, government (defense and non-defense uses), as well as emerging uses in scalable machine learning and big data analytics. The emergence of Dense Many-Core (DMC) architectures like Intel's Knights Landing (KNL) and accelerator/co-processor architectures like NVIDIA GPGPUs are enabling the design of systems with high compute density. This, coupled with the availability of Remote Direct Memory Access (RDMA)-enabled commodity networking technologies like InfiniBand, RoCE, and 10/40GigE with iWARP, is fueling the growth of multi-petaflop and ExaFlop systems. These DMC architectures have the following unique characteristics: deeper levels of hierarchical memory; revolutionary network interconnects; and heterogeneous compute power and data movement costs (with heterogeneity at chip-level and node-level).
For these emerging systems, a combination of MPI and other programming models, known as MPI+X (where X can be PGAS, Tasks, OpenMP, OpenACC, or CUDA), are being targeted. The current generation communication protocols and mechanisms for MPI+X programming models cannot efficiently support the emerging DMC architectures. This leads to the following broad challenges: 1) How can high-performance and scalable communication mechanisms for next generation DMC architectures be designed to support MPI+X (including Task-based) programming models? and 2) How can the current and next generation applications be designed/co-designed with the proposed communication mechanisms?

A synergistic and comprehensive research plan, involving computer scientists from The Ohio State University (OSU) and Ohio Supercomputer Center (OSC) and computational scientists from the Texas Advanced Computing Center (TACC), San Diego Supercomputer Center (SDSC) and University of California San Diego (UCSD), is proposed to address the above broad challenges with innovative solutions. The research will be driven by a set of applications from established NSF computational science researchers running large scale simulations on Stampede and Comet and other systems at OSC and OSU. The proposed designs will be integrated into the widely-used MVAPICH2 library and made available for public use. Multiple graduate and undergraduate students will be trained under this project as future scientists and engineers in HPC. The established national-scale training and outreach programs at TACC, SDSC and OSC will be used to disseminate the results of this research to XSEDE users. Tutorials will be organized at XSEDE, SC and other conferences to share the research results and experience with the community.

Novel neuroscience tools and techniques are necessary to enable insight into the building blocks of neural circuits, the interactions between these circuits that underpin the functions of the human brain, and modulation of these circuits that affect our behavior. To leverage rapid technological development in sensing, imaging, and data analysis new ground breaking advances in neuroscience are necessary to facilitate knowledge discovery using data science methods. To address this societal grand challenge, the project will foster new interdisciplinary collaborations across computing, biological, mathematical, and behavioral science disciplines together with partnerships in academia, industry, and government at multiple levels. The Big Data Neuroscience Spoke titled Midwest: Advanced Computational Neuroscience Network (ACNN) is strongly aligned with the national priority area of neuroscience and brings together a diverse set of committed regional partners to enable the Midwest region to realize the promise of Big Data for neuroscience. The ACNN Spoke will build broad consensus on the core requirements, infrastructure, and components needed to develop a new generation of sustainable interdisciplinary Neuroscience Big Data research. ACNN will leverage the strengths and resources in the Midwest region to increase innovation and collaboration for the understanding of the structure, physiology, and function of the human brain through partnerships and services in education, tools, and best practices.

The ACNN will design, pilot and support powerful neuroscientific computational resources for high-throughput, collaborative, and service-oriented data aggregation, processing and open-reproducible science. The ACNN Spoke framework will address three specific problems related to neuroscience Big Data: (1) data capture, organization, and management involving multiple centers and research groups, (2) quality assurance, preprocessing and analysis that incorporates contextual metadata, and (3) data communication to software and hardware computational resources that can scale with the volume, velocity, and variety of neuroscience datasets. The ACNN will build a sustainable ecosystem of neuroscience community partners in both academia and industry using existing technologies for collaboration and virtual meeting together with face-to-face group meetings. The planned activities of the ACNN Spoke will also allow the Midwest Big Data Hub to disseminate additional Big Data technologies resources to the neuroscience community, including access to supercomputing facilities, best practices, and platforms.

This award received co-funding from CISE Divisions of Advanced Cyberinfrastructure (ACI) and Information and Intelligent Systems (IIS).

As the computing, networking, heterogeneous hardware, and storage
technologies continue to evolve in High-End Computing (HEC) platforms,
it becomes increasingly essential and challenging to understand the
interactions between time-critical High-Performance Computing (HPC)
and Big Data applications, the software infrastructures upon which
they rely for achieving high-performing portable solutions, the
underlying communication fabric these high-performance middlewares
depend on and the schedulers that manage HPC clusters. Such
understanding will enable all involved parties (application
developers/users, system administrators, and middleware developers) to
maximize the efficiency and performance of the individual components
that comprise a modern HPC system and solve different grand challenge
problems. There is a clear need and unfortunate lack of a high-performance and
scalable tool that is capable of analyzing and correlating the
communication on the fabric with the behavior of HPC/Big Data
applications, underlying middleware and the job scheduler on existing
large HPC systems. The proposed synergistic and collaborative effort,
undertaken by a team of computer and computational scientists from OSU
and OSC, aims to create an integrated software infrastructure
for high-performance and scalable Fabric Analysis, Monitoring and
Introspection for HPC and Big Data. This tool will achieve the
following objectives: 1) be portable, easy to use and easy to
understand, 2) have high performance and scalable rendering and
storage techniques and, 3) be applicable to the different
communication fabrics and programming models that are likely to be
used on existing large HPC systems and emerging exascale systems. The
transformative impact of the proposed research and development effort
is to design a comprehensive analysis and performance monitoring tool
for applications of current and next generation multi
petascale/exascale systems to harness the maximum performance and
scalability.

The proposed research and the associated infrastructure will have a
significant impact on enabling optimizations of HPC and Big Data
applications that have previously been difficult to provide. These
potential outcomes will be demonstrated by using the proposed
framework to validate a variety of HPC and Big Data benchmarks and
applications under multiple scenarios. The integrated middleware and
tools will be made publicly available to the community through public
repositories and publications in the top forums, enabling other MPI
and Big Data stacks to adopt the designs. Research results will also
be disseminated to the collaborating organizations of the
investigators to impact their HPC software products and
applications. The proposed research directions and their solutions
will be used in the curriculum of the PIs to train undergraduate and
graduate students, including under-represented minorities and female
students. The technical challenges addressed by the proposal include: 1)
Scalable visualization of large and complex HEC networks so as to
provide a near instant rendering to end users, 2) A generalized data
gathering scheme which is easily portable to multiple communication
fabrics, novel compute architectures and high-performance middleware,
3) Enhanced data storage performance through optimized database
schemas and the use of memory-backed key value stores/databases, 4)
Support in MPI, PGAS, and Big Data libraries to enable the proposed
monitoring, analysis, and introspection framework, and 5) Enabling
deeper introspection of particular regions of application. The
research will also be driven by a set of HPC and Big Data
applications. The transformative impact of the proposed research and
development effort is to design a comprehensive analysis and
performance monitoring tool for applications of current and next
generation multi petascale/exascale systems to harness the maximum
performance and scalability.

The extremely high compute and communication capabilities offered by modern Graphics Processing Units (GPUs) and high-performance interconnects have led to the creation of High-Performance Computing (HPC) platforms with multiple GPUs and high-performance interconnects per node. Unfortunately, state-of-the-art production quality implementations of the popular Message Passing Interface (MPI) programming model do not have the appropriate support to deliver the best performance and scalability for applications on such dense GPU systems. These developments in High-End Computing (HEC) technologies and associated middleware issues lead to the following broad challenge: How can existing production quality MPI middleware be enhanced to take advantage of emerging networking technologies to deliver the best possible scale-up and scale-out for HPC and Deep Learning (DL) applications on emerging dense GPU systems? A synergistic and comprehensive research plan, involving computer scientists from The Ohio State University (OSU) and Ohio Supercomputer Center (OSC) and computational scientists from the Texas Advanced Computing Center (TACC), and San Diego Supercomputer Center (SDSC) and University of California San Diego (UCSD), is proposed to address the above broad challenges with innovative solutions. The proposed framework will be made available to collaborators and the broader scientific community to understand the impact of the proposed innovations on next-generation HPC and DL frameworks and applications in various science domains. Multiple graduate and undergraduate students will be trained under this project as future scientists and engineers in HPC. The proposed work will enable curriculum advancements via research in pedagogy for key courses in the new Data Science programs at OSU, SDSC and TACC. The established national-scale training and outreach programs at TACC, SDSC and OSC will be used to disseminate the results of this research to XSEDE users. Tutorials and workshops will be organized at PEARC, SC and other conferences to share the research results and experience with the community. The project is aligned with the National Strategic Computing Initiative (NSCI) to advance US leadership in HPC and the recent initiative of the US Government to maintain leadership in Artificial Intelligence (AI.)

The proposed innovations include: 1) Designing high-performance and scalable point-to-point, and collective communication operations that fully utilize multiple network adapters and advanced in-network computing features for GPU and CPU buffers within and across nodes; 2) Designing novel datatype processing and unified memory management to improve application performance; 3) Designing CUDA-aware I/O subsystem to accelerate MPI I/O and checkpoint-restart for HPC and DL applications; 4) Designing support for containerized environments to better enable easy deployment of proposed solutions on modern cloud environments; and 5) Carry out integrated development and evaluation to ensure proper integration of proposed designs with the driving applications. The proposed designs will be integrated into the widely-used MVAPICH2 library and made available. The project team members will work closely with internal and external collaborators to facilitate wide deployment and adoption of released software. The proposed solutions will be targeted to enable scale-up and scale-out of the driving science domains (molecular dynamics, lattice QCD, seismology, image classification, and fusion research) on emerging dense GPU platforms. The transformative impact of the proposed development effort is to achieve scalability, performance, and portability out of HPC and DL frameworks and applications to take advantage of emerging dense GPU platforms and hence, leading to significant advancements in science and engineering.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

In-network computing technologies, or the ability to offload significant portions of compute, communication, and I/O tasks to the network, have emerged as fundamental requirements to achieve extreme scale performance for end applications in the areas of High-Performance Computing (HPC) and Deep Learning (DL). Unfortunately, current generation communication middleware and applications cannot fully take advantage of these advances due to the lack of appropriate designs in the middleware-level. This leads to the following broad challenges: 1) Can middleware that are ?aware? of the computing capabilities of these emerging in-network computing technologies be designed in the most optimized manner possible for HPC and DL applications?, and 2) Can such a middleware be used to bene?t end applications in HPC and DL to achieve better performance and portability? A synergistic and comprehensive research plan is proposed to address the above broad challenges with innovative solutions. The proposed framework will be made available to collaborators and the broader scientific community to understand the impact of the proposed innovations on next-generation HPC and DL middleware and applications. Several graduate and undergraduate students will be trained under this project as future scientists and engineers in HPC. The proposed work will enable curriculum advancements via research in pedagogy for key courses at The Ohio State University. Tutorials and workshops will be organized at various conferences to share the research results and experience with the community. The project is aligned with the National Strategic Computing Initiative (NSCI) to advance US leadership in HPC and the recent initiative of the US Government to maintain leadership in Artificial Intelligence (AI.)

The proposed innovations include: 1) Designing scalable communication primitives (point-to-point and collectives) for using emerging switch and NIC based in-network computing features, 2) Exploiting in-network computing features to o?oad complex and user de?ned functions, 3) Designing high-performance I/O and storage subsystems using NVMe over Fabrics, 4) Designing enhanced in-network datatype processing schemes for MPI library, 5) Designing and optimizing in-network computing-based solutions for emerging cloud environment, and 6) Carrying out integrated development and evaluation of the proposed designs with a set of representative HPC and DL applications. The proposed designs will be integrated into the widely-used MVAPICH2 library and made available to the public. The project team members will work closely with collaborators to facilitate wide deployment and adoption of released software. The transformative impact of the proposed research is to achieve scalability, performance, and portability for HPC and DL frameworks/applications by leveraging emerging in-network computing technologies.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

The analysis of high-resolution images in both two and three-dimensions is becoming important for many scientific areas, such as in medicine, astronomy and engineering. Discoveries in these disciplines often require analyzing millions of images. The analysis of these images is complex and requires many steps on powerful computers. Some of these steps require looking through lots of images while some of these steps require deep analysis of each image. In many cases, these analyses have to be completed quickly, i.e. in "real-time", so that information and insights can be provided to humans as they do their work. These kinds of operations require powerful computers consisting of many different, heterogeneous but simple computing components. These components need to be configured and reconfigured so that they can efficiently work together to do these large-scale analyses. In addition, the software that controls these computers also has to be intelligently designed so that these analyses can be run on the right types of configurations. This project aims to acquire the necessary computing components and assemble such a powerful computer (named RADiCAL). Research done using RADiCAL will result in important scientific discoveries that will make us more prosperous, improve our health, and enable us to better understand the world and universe around us. Doing this research will also educate many students, including those from under-represented groups, who will become part of a highly-trained workforce capable of addressing our nation's needs long into the future.

The intellectual merit of RADiCAL is in the design a novel, high-performance, next-generation, heterogeneous, reconfigurable hardware and software stack to provide real-time interaction, analytics, machine/deep learning (ML/DL) and computing support for disciplines that involve massive observational and/or simulation data. RADiCAL will be built from commodity hardware, and designed for reconfiguration and observability. RADiCAL will enable a comprehensive research agenda on software that will facilitate rapid and flexible construction of analytics workflows and their scalable execution. Specific software research include: 1) a library with support for storage and retrieval of multi-resolution, multi-dimensional datasets, 2) scalable learning and inference modules, 3) data analytics middleware systems, and 4) context-sensitive human-in-the-loop ML models and libraries that encode domain expertise, coupling tightly with both lower level layers and the hardware components to facilitate scalable analysis and explainability. With the proposed hardware acquisition and software research, the transformative goal will be to facilitate decision-making and discovery in Computational Fluid Dynamics (CFD) and medicine (pathology). With respect to broader impacts, RADiCAL will provide a unique research, testing, and training infrastructure that will catalyze research in multiple disciplines as well as facilitate convergent research across disciplines. The advanced imaging applications and techniques for expert-assisted image analysis will be broadly applicable to other human-in-the-loop systems and have the potential to advance medicine and health. Projects that use RADiCAL will also provide unique test-beds for valuable empirical research on human-computer interaction and software engineering best practices. Well-established initiatives at The Ohio State University will facilitate the recruitment of graduate and undergraduate students from underrepresented groups for involvement in using the cyberinfrastructure. The heterogeneous and reconfigurable research instrument will be utilized to create sophisticated educational modules on how to co-design computational science experiments from the science goals to the underlying cyberinfrastructure. Tutorials and workshops will be organized at PEARC, Supercomputing and other conferences to share the research results and experience with the community.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

The ability to analyze and learn from large volumes of data is becoming important in many walks of human endeavor, including medicine, science, and engineering. Analysis workflows for high-resolution images (e.g. medical imaging, sky surveys), scientific simulations, as well as those for graph analytics and machine learning are typically time consuming because of the extreme scales of data involved. While the hardware elements of the modern data center are undergoing a rapid transformation to embrace the storage, processing, and analysis of needs of such applications - understanding of how the different layers of the systems stack interact with one another and contribute to end-to-end application performance is challenging. This planning project envisions the ACROPOLIS framework to address these challenges. ACROPOLIS will enable a comprehensive research agenda on systems software that will facilitate rapid and flexible construction of analytics workflows and their scalable execution. By facilitating the rapid prototyping of application drivers ACROPOLIS can also enable important scientific discoveries to potentially improve human health and better understand the world around us. The research enabled by ACROPOLIS will also educate many students, including those from under-represented groups, who will become part of a highly-trained workforce capable of addressing our nation's needs long into the future. With respect to broader impacts, ACROPOLIS will provide a unique research and training infrastructure that will catalyze research in multiple disciplines as well as facilitate convergent research across disciplines. Well-established initiatives at The Ohio State University, such as the Louis Stokes Alliances for Minority Participation (LSAMP) as well as new programs in Data Analytics, will facilitate the recruitment of graduate and undergraduate students for involvement in this research agenda. This project is aligned with two of NSF?s 10 Big Ideas: Harnessing the Data Revolution and Growing Convergence Research, as well as the American AI Initiative.

The project addresses five key research pillars: 1) Flexible abstractions for parallel computation and data representation, 2) Modeling data movement complexity at extreme scales, 3) Pattern-driven scalable communication and I/O systems, 4) Near-memory architectures for machine learning and analytics, and 5) Cross-layer observability and introspection. Specifically, the focus is on the design of an end-to-end framework inculcating a high-performance, next-generation, heterogeneous, reconfigurable hardware and software stack to facilitate real-time interaction, analytics, and machine learning for a range of scientific disciplines including Computational Pathology and Computational Fluid Dynamics and Emergency Response.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Although the world is witness to the tremendous successes of Artificial Intelligence (AI) technologies in some domains, many domains have yet to reap the benefits of AI due to the lack of easily usable AI infrastructure. The NSF AI Institute for Intelligent Cyberinfrastructure with Computational Learning in the Environment (ICICLE) will develop intelligent cyberinfrastructure with transparent and high-performance execution on diverse and heterogeneous environments. It will advance plug-and-play AI that is easy to use by scientists across a wide range of domains, promoting the democratization of AI. ICICLE brings together a multidisciplinary team of scientists and engineers, led by The Ohio State University in partnership with Case Western Reserve University, IC-FOODS, Indiana University, Iowa State University, Ohio Supercomputer Center, Rensselaer Polytechnic Institute, San Diego Supercomputer Center, Texas Advanced Computing Center, University of Utah, University of California-Davis, University of California-San Diego, University of Delaware, and University of Wisconsin-Madison. Initially, complex societal challenges in three use-inspired scientific domains will drive ICICLE’s research and workforce development agenda: Smart Foodsheds, Precision Agriculture, and Animal Ecology.

ICICLE’s research and development includes: (i) Empowering plug-and-play AI by advancing five foundational areas: knowledge graphs, model commons, adaptive AI, federated learning, and conversational AI. (ii) Providing a robust cyberinfrastructure capable of propelling AI-driven science (CI4AI), solving the challenges arising from heterogeneity in applications, software, and hardware, and disseminating the CI4AI innovations to use-inspired science domains. (iii) Creating new AI techniques for the adaptation/optimization of various CI components (AI4CI), enabling a virtuous cycle to advance both AI and CI. (iv) Developing novel techniques to address cross-cutting issues including privacy, accountability, and data integrity for CI and AI; and (v) Providing a geographically distributed and heterogeneous system consisting of software, data, and applications, orchestrated by a common application programming interface and execution middleware. ICICLE’s advanced and integrated edge, cloud, and high-performance computing hardware and software CI components simplify the use of AI, making it easier to address new areas of inquiry. In this way, ICICLE focuses on research in AI, innovation through AI, and accelerates the application of AI. ICICLE is building a diverse STEM workforce through innovative approaches to education, training, and broadening participation in computing that ensure sustained measurable outcomes and impact on a national scale, along the pipeline from middle/high school students to practitioners. As a nexus of collaboration, ICICLE promotes technology transfer to industry and other stakeholders, as well as data sharing and coordination across other National Science Foundation AI Institutes and Federal agencies. As a national resource for research, development, technology transfer, workforce development, and education, ICICLE is creating a widely usable, smarter, more robust and diverse, resilient, and effective CI4AI and AI4CI ecosystem.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Artificial Intelligence (AI) is used in many aspects of modern life such as language translation and image analysis. In addition to consumer and business applications, researchers are increasingly using AI techniques in their scientific processes. The growth in AI is heavily dependent on new Deep Learning (DL) and Machine Learning (ML) schemes. As datasets and DL and ML models become more complex the computing requirements for AI increase and researchers turn to high performance computing (HPC) facilities to meet these needs. This is leading to a critical need for a Cyberinfrastructure (CI) workforce that supports HPC systems with expertise in AI techniques and underlying technology. This project will pilot an AI bootcamp for CI professionals that is targeted based on the professional's job requirements. After attending the bootcamp CI professionals will be better equipped to provide computing and data services to AI research users. This in turn will broaden adoption and effective use of advanced CI by researchers in a wide range of disciplines and will have an impact on science and corresponding benefits to society from their successes. The training materials developed during this project will be openly shared with the CI community so that others can use and adapt the materials for similar training activities.

This project is novel in taking a holistic approach to addressing the AI expertise gap for CI professionals. The project will develop an AI Bootcamp for CI professionals with the overarching goal of increasing the confidence and effectiveness of their support of AI researchers. The project leverages the CI professionalization efforts of the Campus Research Computing Consortium (CaRCC) to organize the training outcomes based on four "facings" (Strategy/Policy facing, Researcher facing, Software/Data facing, and Systems facing). The project will identify learning outcomes for each CI facing and organize training tracks customized to specific roles. For this pilot the project is focused on developing a comprehensive training experience for Software/Data facing CI professionals. The AI Bootcamp will be offered virtually over twelve weeks. The instructional materials will be shared openly as notebooks, slide-decks and containers as appropriate so that they can be used for other training offerings. The project team is comprised of CI professionals, experienced in training CI users and providing CI operations, and Computer Science faculty members, experienced in offering courses in Data Analytics, AI and High Performance AI with active AI-based research programs. Drawing on extensive experience and materials in hands-on experiential learning for AI, the project team will create a comprehensive curriculum spanning foundational AI, software frameworks, and high performance computing for AI in a modularized virtual format to minimize barriers to access for the CI professional learner.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

The 2022 MVAPICH User group (MUG) Conference will be a venue for bringing together leading experts, including users, system administrators, researchers, engineers, and students, to discuss and share their knowledge on using the MVAPICH2 libraries to discuss and share their knowledge on using the MVAPICH2 libraries. The event will include a set of talks from researchers, users, and system administrators who are experts in the field. The event also includes contributed presentations (submitted through an open call for presentation and selected by the MVAPICH team) and presentations from the MVAPICH team on tuning and optimization strategies for various library components, troubleshooting guidelines, etc. The conference, to be held in Columbus, OH, from August 22-24, 2022, is organized by a leading group of experts in message passing (MPI) and networking technologies. The project will fund students to participate in the conference, to engage deeply with the MVAPICH research and user community. The project serves the national interest, as stated by NSF's mission, to promote the progress of science as it provides a forum to disseminate research efforts, connect researchers, and train the next generation of scholars.<br/><br/>The conference organizers are recruiting students wishing to participate in the program from a diverse set of institutions, and with the goal of a diverse set of participants. Prospective participants will have to submit an application, and submissions are evaluated by a committee for scientific merit and to promote diversity. The students will get several benefits from attending the conference: (1) exposure to the latest high performance computing (HPC) technologies (computing platforms, accelerators, and networks), their features, and their impact on designing HPC software environments; (2) in-depth exposure to the challenges and techniques in designing, developing and maintaining open-source software environments for HPC systems; (3) training on systematic tuning and optimization of HPC software stacks; and (4) interaction with national laboratory and industry professionals. The funding provided by NSF will have a significant impact on the careers of the future generation of researchers in HPC, networking, and message passing technologies, while encouraging diversity in the field.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.