Joseph Picone - US grants

This project is establishing a lasting collection of tools for converting speech to text and for speech understanding. The collection is based on an extensible C++ speech recognition system, for which both source code and object modules are available. Public access to this system is encouraged in three ways: by a web site, with community-wide design reviews, and with educational materials. The website includes the C++ software, together with a toolkit for evaluation of speech-recognition software and a set of Java applets for educational purposes. The design reviews are similiar to workshops, and cover topics such as software design and algorithm selection for new versions of the publicly available speech system. Finally, educational web sites and tutorials lower the barriers to performing research in speech-to-text systems.

This is the first year funding of a four-year continuing award. This project addresses issues relating to the construction of a system for answering questions about information contained in a collection of spoken documents. It focuses on the key scientific questions that arise in the integration of prosodic information, speech recognition and parsing in the retrieval of spoken documents, but will not involve implementation of a complete system. There are four key themes in the research: utilizing parsing in information retrieval; integrating prosodic information in parsing spoken language; incorporating uncertainty in parsing to handle speech recognition errors; and improvements to speech recognition of spontaneous speech. All components will share a probabilistic formulation, thereby affording a systematic framework for integrating the information they provide. A primary project goal is to better understand how information provided by one of these components might be effectively utilized to improve he performance of other components in the information retrieval task. Absent a corpus tailored to the information retrieval topics the PI and his team plan to study, progress will be evaluated using existing annotated text collections such as Switchboard and LDC's Broadcast News collections. The work will lead to advances in information extraction from telephone messages, conversations, university lectures, or from any text (such as encyclopedias), and should potentially serve as the basis for a sorely needed sophisticated web browser technology and data mining applications, which in turn would enable people who currently under-utilize computers to become full participants in the information revolution.

The most fundamental aspects of statistical modeling in speech recognition, linear Gaussian statistics, have been essentially unchanged in the past 20 years. Fundamental advancement is required if speech recognition is to become a pervasive technology in a myriad of applications requiring robustness to severe amounts of noise (e.g., cell phones and in-vehicle automotive applications). Nonlinear statistical models for speech were first proposed in the early 1980's when fractals and other such techniques promised great advances in compression. Since then progress has been slow but steady. Recent advances in various areas of speech processing, such as pitch determination and speech modeling, plus staggering advances in computational resources, suggest that these models are now viable for traditional problems such as speaker recognition, speaker verification, and speech recognition. Nonlinear dynamics provide a framework that supports parsimonious statistical models that may overcome many of the limitations of current hidden Markov model based techniques.

This research involves extending the traditional supervised-learning HMM paradigm to support a chaotic acoustic model that incorporates a nonlinear statistical model of observation vectors and then evaluating the impact of this model on text-independent speaker verification applications. The primary goal is to understand acoustic variation at the phonetic level in a more comprehensive and efficient manner. The proposed research could go far to enhance the potential practicality of nonlinear speech modeling. In addition, the computational tools and resources to be developed are expected to enhance the existing infrastructure for Internet accessible speech recognition, while promoting better understanding of speech in both research and education.

Despite significant progress in Neural Engineering in recent years, overall progress in the field does not appear to have been commensurate with the scope of investment. It has become evident that the field can benefit from a new paradigm in research and development that focuses on robust algorithm development. The purpose of this planning grant is to develop community support for the creation of the Neural Engineering Data Consortium (NEDC) at Temple University. The NEDC is focusing the attention of the research community on a progression of research questions as well as generating and curating massive data sets to be used in addressing those questions. Under this award, we are conducting an extensive series of workshops and site visits to build consensus amongst various research groups from academia and industry. Guided by this site research, we are developing an operational model for the NEDC that is simultaneously relevant to academic research, industry R&D, and government funding agencies. The NEDC is structured to significantly accelerate research progress and provide resources that promote the development of more robust technology. We are also broadening participation in neural engineering research by making data available to research groups who have significant signal processing expertise but who lack capacity for data generation. The impact of funded research is thus improved by enticing more participation in focused evaluation tasks. Finally, the NEDC is accelerating the transfer of technology to the healthcare marketplace, where it can directly enhance quality of life for a variety of patients.

PROJECT DESCRIPTION
In an age of ubiquitous mobile computing devices, students demand instant access to information. The project is creating an open-source, on-line laboratory in which students can engage in experiential learning of electrical engineering content in an on-demand setting. It is developing an open laboratory, which uses a virtual laboratory assistant (VLA) with voice input and output capabilities to provide personalized instruction for students participating in a self-paced undergraduate engineering laboratory. The intelligent agent is customizing the learning process by providing each student with personalized tutoring.

The components of the system include: pre-lab testing and instruction, engineering design exercises, short topic explanation videos, instrumentation instruction (including safety), and a corresponding post-lab test module. In the pre-lab, students are introduced to basic theory and simulation tools. If a student needs clarification on a topic, a short instructional video will provide guidance. These videos utilize a Khan Academy style format and explain many basic electrical engineering concepts. The lab instruction module assists students in executing an open-ended design problem using the VLA.

The project's pilot electrical engineering lab is studying the development of an extensible framework and evaluating the efficacy of open laboratory pedagogical approach. The effort is exploring: (1) to what extent is individual, self-paced learning possible, (2) what benefit comes with 24/7 accessibility, (3) how more timid students can receive personalized instruction in a non-threatening environment, and (4) how the level of engagement can be increased by allowing students to freely explore the subject matter. The evaluation activities involve both formative and summative efforts that will be supported by Temple University's Teaching and Learning Center. The instructional diagnosis and surveys will provide interested educators with data necessary to foster adoption of the project resources at other institutions.

BROADER SIGNIFICANCE
Temple University's open laboratory is being used as a recruiting tool during more than a dozen recruiting events per year, which are focused on high school and middle school students. These events are attended by hundreds of potential engineers from underrepresented groups. Through these events, the team is recruiting members of underrepresented groups to engineering. They are also integrating education and research through the open laboratories - in which undergraduate research students are performing simulations and prototyping. A successful development of the open laboratory environment is expected to more efficiently utilize lab space and student/faculty time. Moreover, the pilot is producing an open lab model that can be leveraged by other courses, along with a scalable architecture that can be used by other colleges and universities.

? DESCRIPTION (provided by applicant): Electronic medical records (EMRs) collected at every hospital in the country collectively contain a staggering wealth of biomedical knowledge. EMRs can include unstructured text, temporally constrained measurements (e.g., vital signs), multichannel signal data (e.g., EEGs), and image data (e.g., MRIs). This information could be transformative if properly harnessed. Information about patient medical problems, treatments, and clinical course is essential for conducting comparative effectiveness research. Uncovering clinical knowledge that enables comparative research is the primary goal of this proposal. We will focus on the automatic interpretation of clinical EEGs collected over 12 years at Temple University Hospital (over 25,000 sessions and 15,000 patients). Clinicians will be able to retrieve relevant EEG signals and EEG reports using standard queries (e.g. Young patients with focal cerebral dysfunction who were treated with Topamax). In Aim 1 we will automatically annotate EEG events that contribute to a diagnosis. We will develop automated techniques to discover and time-align the underlying EEG events using semi-supervised learning. In Aim 2 we will process the text from the EEG reports using state-of-the-art clinical language processing techniques. Clinical concepts, their type, polarity and modality shall be discovered automatically, as well as spatial and temporal information. In addition, we shall extract the medical concepts describing the clinical picture of patients from the EEG reports. In Aim 3, we will develop a patient cohort retrieval system that will operate on the clinical knowledge extracted in Aims 1 and 2. In addition we shall organize this knowledge in a unified representation: the Qualified Medical Knowledge Graph (QMKG), which will be built using BigData solutions through MapReduce. The QMKG will be able to be searched by biomedical researchers as well as practicing clinicians. The QMKG will also provide a characterization of the way in which events in an EEG are narrated by physicians and the validation of these across a BigData resource. The EMKG represents an important contribution to basic science. In Aim 4 we will validate the usefulness of the patient cohort identification system by collecting feedback from clinicians and medical students who will participate in a rigorous evaluation protocol. Inclusion and exclusion criteria for the queries shall be designed and experts will provide relevance judgments for the results. For each query, medical experts shall examine the top-ranked cohorts for common precision errors (false positives) and the bottom five ranked common recall errors (false negatives). User validation testing will be performed using live clinical data and the feedback wil enhance the quality of the cohort identification system. The existence of an annotated BigData archive of EEGs will greatly increase accessibility for non- experts in neuroscience, bioengineering and medical informatics who would like to study EEG data. The creation of this resource through the development of efficient automated data wrangling techniques will demonstrate that a much wider range of BigData bioengineering applications are now tractable.

Electroencephalograms (EEGs) are the most pervasive neural diagnostic tool; they require a highly trained neurologist to analyze them. Long-term EEG monitoring, used to diagnose rare events such as epileptic seizures, is difficult or impossible to scan manually without decision software support. Development of portable standalone diagnostic tools, which can address emerging markets such as contact sports, is highly difficult; and smaller or stand-alone medical practices often lack expertise to conduct diagnostics on-site and accordingly lose revenue. At present, innovation in commercial clinical decision support tools is minimal, whereas the global market for rapidly diagnosing brain-related injury and disease is growing. The proposed AutoEEGTM is a software tool that enhances productivity by auto-scanning EEG signals and flagging sections of the signal that need further review by a clinician. The proposed tool reduces the amount of data needing manual review by two orders of magnitude, offering substantial productivity gains in a clinical setting.

The proposed clinical decision support tool is based on proven, advanced, deep learning technology. It reduces time to diagnosis, reduces error and is sufficiently lightweight to run on portable standalone platforms. This technology is able to identify EEG events in the signal and subsequently to provide a report that summarizes its findings based on the event detected. The transcribed EEG signals can be viewed from any portable computing device. It also has the ability to learn from data, helping in future decision making, providing real-time feedback to aid in diagnosis, and, for patients undergoing long-term monitoring, creating an alert when abnormal signals are identified. This market-leading product will (1) Enable clinical neurologists employing a volume-based business mode to decrease the time spent analyzing an EEG and thereby increase billing; (2) Allow pharmas to assess changes quantitatively in neural activation during clinical trials; (3) Allow neurologists to order and bill for substantially more long-term monitoring tests based on this proven decision support tool; and (4) Add value to the commodity EEG headsets currently entering the market by providing meaningful, real-time signal analysis. This research project has two key components: (1) a detailed analysis of the market to understand various opportunities such as licensing to equipment manufacturers and off-line analysis for contract research organizations; and (2) usability design and engineering to understand the analytics and user interface issues that bring most value to potential users such as clinicians and primary care physicians. The outcomes of this research will be used to harden the technology and guide integration into existing EEG products.

? DESCRIPTION (provided by applicant): Electronic medical records (EMRs) collected at every hospital in the country collectively contain a staggering wealth of biomedical knowledge. EMRs can include unstructured text, temporally constrained measurements (e.g., vital signs), multichannel signal data (e.g., EEGs), and image data (e.g., MRIs). This information could be transformative if properly harnessed. Information about patient medical problems, treatments, and clinical course is essential for conducting comparative effectiveness research. Uncovering clinical knowledge that enables comparative research is the primary goal of this proposal. We will focus on the automatic interpretation of clinical EEGs collected over 12 years at Temple University Hospital (over 25,000 sessions and 15,000 patients). Clinicians will be able to retrieve relevant EEG signals and EEG reports using standard queries (e.g. Young patients with focal cerebral dysfunction who were treated with Topamax). In Aim 1 we will automatically annotate EEG events that contribute to a diagnosis. We will develop automated techniques to discover and time-align the underlying EEG events using semi-supervised learning. In Aim 2 we will process the text from the EEG reports using state-of-the-art clinical language processing techniques. Clinical concepts, their type, polarity and modality shall be discovered automatically, as well as spatial and temporal information. In addition, we shall extract the medical concepts describing the clinical picture of patients from the EEG reports. In Aim 3, we will develop a patient cohort retrieval system that will operate on the clinical knowledge extracted in Aims 1 and 2. In addition we shall organize this knowledge in a unified representation: the Qualified Medical Knowledge Graph (QMKG), which will be built using BigData solutions through MapReduce. The QMKG will be able to be searched by biomedical researchers as well as practicing clinicians. The QMKG will also provide a characterization of the way in which events in an EEG are narrated by physicians and the validation of these across a BigData resource. The EMKG represents an important contribution to basic science. In Aim 4 we will validate the usefulness of the patient cohort identification system by collecting feedback from clinicians and medical students who will participate in a rigorous evaluation protocol. Inclusion and exclusion criteria for the queries shall be designed and experts will provide relevance judgments for the results. For each query, medical experts shall examine the top-ranked cohorts for common precision errors (false positives) and the bottom five ranked common recall errors (false negatives). User validation testing will be performed using live clinical data and the feedback wil enhance the quality of the cohort identification system. The existence of an annotated BigData archive of EEGs will greatly increase accessibility for non- experts in neuroscience, bioengineering and medical informatics who would like to study EEG data. The creation of this resource through the development of efficient automated data wrangling techniques will demonstrate that a much wider range of BigData bioengineering applications are now tractable.

This project, developing a digital imaging system, aims to automatically characterize an enormous archive of digital images from a pathology lab. Based on open source programming language packages and using deep learning technologies, the effort entails developing software that will automatically annotate and classify these images. Thus the system consists of an annotated archive tool for high performance digital pathology involving digital images from pathology slides produced in clinical operations. This tool, presently unavailable, enables observation, annotation, and classification of images from tissues in pathology slides in order to create a very large data base that may be analyzed with algorithms that are designed to process and interpret the image data. By applying state of the art machine learning, the effort is expected to generate a sustainable facility to rapidly collect large amounts of data automatically. This facility enables deep learning systems to systematically address many operational challenges, such as ingestion of large, complex images.

Broader Impacts:
The instrumentation provides a useful technology capability. The work builds on the researchers' history of providing unencumbered resources for fields including human language technology and neuroscience. Several large, comprehensive databases of pathology slides will be released in an unencumbered manner; no comparable databases currently exist in terms of the quantity of data proposed. The urban setting of the project, as well as the diverse nature of the institution's client population, make it ideal for collecting this type of clinical data. A new generation of healthcare professionals will be trained using these resources to validate their knowledge in the longer term.

The broader impact/commercial potential of this PFI project is that it will lead to improved clinical outcomes for neurological patients in intensive care units (ICUs). Although the data acquired using continuous electroencephalography (EEG) in the ICU is inexpensive to record and a rich source of information for guiding clinical decision making, it is often not used because it takes too long to be analyzed manually. The proposed technology will be capable of evaluating EEGs in real-time in order to alert doctors when clinically relevant events such as seizures occur. This will improve patient outcomes by allowing doctors to intervene with medications in a timelier and more precise fashion. This work will also have the broader impact of improving science's understanding of the fundamentals of how machine learning can be applied specifically to neural signal processing, which is currently a poorly understood area.

The proposed project will enable and accelerate the commercialization of software technology that detects seizures and abnormal brain activity in Intensive Care Unit patients. This will be accomplished with three main tasks. In the first task, the existing seizure detection software, which currently works offline, will be converted to work in real-time with a target latency of 20 seconds to detect a seizure. This will be accomplished through intelligent memory handling and by developing a low-latency, highly optimized post-processing algorithm. The second task will strengthen the existing seizure detection code to operate at clinically acceptable levels of sensitivity and false alarm rates. This will be achieved by retraining our algorithms on a significantly more diverse and complex EEG database in order to expose the software to as many variations of seizure presentation as possible. In the third and final task, extensive software testing will be conducted in order to optimize the machine learning configuration that maximizes the gains achieved in Tasks 1 and 2.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

This project is concerned with research on very large digital images. Although image recognition tools are now commonplace for standard definition images, there is a lack of tools and techniques for analyzing very high resolution images (hundreds of millions of pixels per image). There are many important applications that would be well-served by the ability to automatically search such images, especially in cases where the properties of the search target aren't even fully known. Research of this nature is complicated by the fact that images of this size do not lend themselves well to the type of software tools that have been successful in standard-resolution image processing techniques such as handwriting or facial recognition. Data quantity alone is a major limitation; training a robust image recognition tool may require tens of thousands of images; at 100 million pixels per image, the sheer quantity of data becomes an issue for storing, sharing, and processing. New innovations will be required in machine learning, data provenance and warehousing, cloud computing, and image compression, all of which will serve the national interest.

The specific purpose of this project is to (a) build a community of stakeholders who have vested interests in innovations in high resolution image processing and (b) create a roadmap for future research. Specifically, the researchers will reach out to relevant groups across academia, industry, and not-for-profit consortia with the goal of building a robust community with diverse expectations. This community will engage in a series of workshops to define the goals and scope of a high definition image processing consortium. The workshops will seek to define data standards, image processing goals, standardized data sets, and best practices for sharing and computing images at the petabyte scale. These findings will be leveraged into a future project whose goal will be to actually build the tools and perform the research that has been road-mapped by this project.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Many crucial artificial intelligence applications in health sciences do not have sufficient practical data. Even when one collects incredible amounts of data, the data has significant gaps because of the unique circumstances (health conditions) of each patient. Training sizeable artificial intelligence systems on such data often results in an output that is either unacceptable for making decisive medical pronouncements or the systems do not perform any better than conventional methodologies. As such, the project's goal is to develop artificial intelligence, particularly machine learning algorithms that train rapidly, minimize errors, and do not require significant human expertise. The project's novelty is utilizing emerging quantum computing (QC) algorithms that offer the potential for rapid training of models and the ability to find better solutions quickly. In this project, QC will be applied to machine learning to demonstrate the efficacy of QC-based methods in two challenging applications: (a) seizure detection on encephalography signals and (b) automatic interpretation of digital pathology images. Positively impacting the two high-level applications will allow automated systems to approach domain expert (human) performance and increase the impact of this technology in the medical field, which will impact countless humans worldwide. Access to the highest levels of QC research will create career development opportunities, encouraging high schoolers to pursue computer and information science and engineering careers. <br/><br/>In this project, adiabatic quantum annealing (QA) will be used to solve two significant computational challenges: (a) finding a global minimum and (b) sampling from complex probability distributions. It will be demonstrated that training that utilizes QA-supported sampling can find better parameters than conventional parameter optimization approaches, and it also overcomes the deficiencies of current machine learning algorithms in challenging applications, such as seizure detection on encephalography signals and automatic interpretation of digital pathology images. Through these developments, it is also expected of this project to demonstrate that a wide range of configuration spaces (undirected probabilistic graphical models trained with a variety of application-relevant data) that have the property of "difficult to find local valleys in the probability distribution" to be easily sampled with QA. These findings will be applied to deep generative models for superior classification and pattern reconstruction accuracy. The ability of QA to reach difficult to sample regions of the configuration space will benefit many machine learning applications.<br/><br/>This project is jointly funded by Foundations of Emerging Technologies (FET) and the Established Program to Stimulate Competitive Research (EPSCoR).<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.