David Meir Blei - US grants

There is a growing need for (semi-)automated tools to analyze and organize large collections of electronic information. In response, there is a surge of research on machine learning of probabilistic topic models, which automatically discover the hidden thematic structure in a large collection of documents. Once made explicit, this hidden structure facilitates browsing, searching, organizing, and summarizing vast amounts of information.

This research program will significantly build on the current state-of-the-art in topic modeling.

1. We will develop topic modeling algorithms that discover trends in document streams. Modeling evolutionary and revolutionary change of topics over time will be an important new capability for corpora analysts, providing methods of forecasting and understanding the changing patterns in serial collections such as news feeds, scientific publications, or web blogs.

2. Many modern corpora, such as Wikipedia, contain important links between the documents. We will develop topic models of such interconnected collections that explicitly represent and generalize inter-document and/or inter-topic relationships. Such relationships may be hyper-links, scholarly citation, shared authorship, or statistical correlations. Capturing the patterns in these connections, and understanding their relationship to the texts, will have important implications for a great variety of scholarly, commercial, and personal 'recommender' systems.

3. Very often, analysts and other users approach a corpora with particular questions in mind. To facilitate focused, personalized exploration, we will develop supervised methods for discovering topic models that predict document-specific variables -- notably forms of relevance -- of online material such as scholarly papers, legal briefs, media sources, and product specifications.

This project addresses significant current limitations of topic modeling, and will provide practical new research and education tools for understanding and organizing modern repositories of information. We will make these tools available as open-source software to support and encourage their application to real-world problems, and we will fold the results of our research into ongoing education and outreach programs.

Finding and labeling semantic patterns in large, spatial data sets is one of the most important problems facing computer scientists today. Massive spatial data sets are being acquired in almost every scientific discipline, such as medicine, geology, biology, astrophysics, and others. Finding meaningful patterns in those data is often the bottleneck to scientific discovery. The proposed research is to develop a transformative machine learning methodology, where the process of discovering semantic patterns in large spatial data sets is interactive and semi-autonomous. With the proposed tools and algorithms, the user is provided with an interactive system that shows the most likely segmentations and labelings given the information provided so far, but allows the user to provide additional information as he/she sees fit. The user might adjust a segmentation, provide a label, or specify an expected pattern. The system will adapt in real time to each of these inputs, thus adjusting its predictions throughout the data.



The broad impact of the proposed plan will be enhanced through an integrated educational and outreach plan. Besides the published results of research results, the field will benefit from free distribution of research and education resources, including web pages, bibliographies, software, and data sets, including augmentations to WordNet. Further broad impacts include focused workshops and courses on shape analysis, machine learning, and visualization at both the university and professional levels. Finally, diversity enhancement programs will promote the opportunities for disadvantaged groups in research.

The PIs will develop new machine learning algorithms to explore how meaning is represented in the brain and how meaning representations shape human memory. Current neuroscientific theories of memory posit that forming a memory for a particular event involves associating the details of that event with the person's current mental context, i.e., everything else that she is thinking about at the time. When trying to remember the event, the person can access stored details by reinstating the mental context that was present when the memory was formed. This fits with the intuition that forgotten details (e.g., the location of misplaced house keys) can be retrieved by mentally "re-tracing steps", i.e., trying to reinstate the mindset that was present at the time of the original event. With these theories in mind, the goal of this work is to develop machine learning algorithms that make it possible to track, based on fMRI brain data and behavioral memory data, the process of "mentally re-tracing steps"---the proposed algorithms will be able to decode the state of a person's mental context as she forms memories and (later) as she searches for these memories.

The proposed work uses two fundamental ideas about memory and meaning: The first idea is that mental context is shaped by the meanings of recently encountered stimuli. The second idea is that semantic relationships between concepts in the brain mirror statistical relationships between words in naturally occurring language. The developed algorithms will bring together data from three sources---behavioral data from subjects performing memory recall tasks, fMRI neuroimaging data collected while subjects performed these tasks, and large collections of documents---to discover a latent meaning space that can simultaneously describe all three types of information. Each point in this space describes a mental context. Thus the core of the proposed work is to develop latent variable models and algorithms that can infer from data how the mental context moves through meaning space as a person stores and searches for memories.

The proposed work will lead to fundamental advances in machine learning (new algorithms for inferring hidden variables based on multiple, heterogeneous data types) and neuroscience (more refined theories of how memory search is accomplished in the brain). Furthermore, this work will catalyze the development of new technologies for diagnosing and remediating memory problems, by making it possible to track how the contextual reinstatement process is going awry in people experiencing memory retrieval failure.

Big data analytics is, fundamentally, the problem of bringing the massive amounts of data produced today down to human scale. In particular scientists, engineers, physicians, and many others in knowledge-intensive professions face data that is beyond human scale. This data is in the repositories that collect the data and the reports or results in their fields. This project will address the problem of bringing all this knowledge under control by using even more data, namely the individual and social patterns of how these repositories are accessed and used, and user-specific judgments (valuations) of the data. The proposed research will develop novel algorithms and an open-source infrastructure for improving discovery within and access to data repositories. These algorithms will aggregate and analyze the social analytic data, gathered from professional communities of data users, and will motivate them to participate by providing recommendations.

The transformative goal is to develop methods for organizing, and operationalizing the access and preference patterns of users of large repositories, and for integrating those valuations to accelerate discovery within the collections. Diverse human minds interacting with data collections, as they carry out their own research or operational activities, provide a powerful source of information about the value of the data itself. Those data items may be textual documents, numerical datasets, or other kinds of media content. The novel methods for representing, aggregating, organizing and valuating interactions between the users and the items can reveal structures within data collections, which were previously invisible to any individual. This discovery of interrelations within data, driven by the capture of human intelligence, will accelerate the processes of scientific discovery. Users who are permitted to valuate data, and who are motivated by receiving valuable recommendations in return, reveal more about their own interests. This makes it possible to discover relations among the data items and among the users themselves. The educational goals are to: (a) contribute to the education of specific graduate students supported by the project, and undergraduates via the REU mechanism; (b) generate new educational materials related to algorithmic innovations, and to research findings; and (c) improve access to and discovery within specific collections of materials. Research findings will be included in courses at all three collaborating universities.

Additional information about the project (including publication, software, data sets) will be made available through the project web site: http://arxiv_xs.rutgers.edu/.

In recent decades, scientific and technological fields have experienced "data moments" as researchers recognized the potential of drawing new types of inferences by applying techniques from computational statistics and machine learning to ever-growing datasets. At the same time, everyday life is increasingly saturated with products of data analysis: search engines, recommendation systems, autonomous vehicles, etc. These developments raise fundamental methodological questions, including how to collect and pre- pare data for analysis, and how to transform statistical inferences into effective action and new statistical inquiries. To address these questions, it is necessary to develop theoretical foundations for the practice of data science, and to provide practitioners with sound and practically relevant methodological training. The Columbia TRIPODS Institute pursues these goals through an integrated program of research in data science foundations, curriculum development, and center-building activities. The research program seeks to provide theoretical understanding of practical heuristics, develop modular and well-structured toolkits of computational primitives for data science, and to support the entirety of the data science cycle, from data collection and annotation, to the assessment of the analysis product.

The Institute pursues programs of research, education and center-building aimed at articulating theoretical foundations for data science. Its activities aim to have a major impact in shaping this emerging field. The research directions include understanding tractable classes of optimization problems, developing primitives that support efficient computation on data, and developing methodological foundations for interactive protocols in data science. These directions address challenging problems at the interface between theory and practice, the solutions of which require ideas spanning mathematics, statistics, and computing. The educational activities articulate model curricula in data science at the MS/professional and PhD levels, including interdisciplinary courses aimed at building a common language for a new generation of scientists and engineers. Center building activities are organized around cross-disciplinary themes and structured to encourage interaction across disciplines and to develop a common methodological community in Foundations of Data Science. These research and educational activities-including workshops, summer schools, distinguished lecture series, long-term visits, and outreach- help to further define and disseminate a common language for foundational research and education, and to increase diverse participation in data science. Located within the Center for Foundations of Data Science in the Data Science Institute at Columbia University, the Institute is at the center of the Northeast Big Data Hub. This position supports expansion of activities within Columbia, and also with other Big Data Hub members, TRIPODS Institutes, and research/industry organizations. Funds for the project come from CISE Computing and Communications Foundations, CISE Information Technology Research, MPS Division of Mathematical Sciences, and MPS Office of Multidisciplinary Activities.

A truly comprehensive theory of machine learning has the potential of informing science and engineering in the same profound way Maxwell’s equations did. It was the development of that theory by Maxwell that truly unleashed the potential of electricity, leading to radio, radars, computers, and the Internet. In an analogy, deep learning (DL) has found over the past decade many applications, so far without a comprehensive theory. An eventual theory of learning that explains why and how deep networks work and what their limitations are may thus enable the development of even more powerful learning approaches – especially if the goal of reconnecting DL to brain research bears fruit. In the long term, the ability to develop and build better intelligent machines will be essential to any technology-based economy. After all, even in its current – still highly imperfect –state, DL is impacting or about to impact just about every aspect of our society and life. The investigators also plan to complement their theoretical research with the educational goal of training a diverse population of young researchers from mathematics, computer science, statistics, electrical engineering, and computational neuroscience in the field of machine learning and of its theoretical underpinnings.

The investigators propose to join forces in a multi-pronged and collaborative assault on the profound mysteries of DL, informed by the sum of their experience, expertise, ideas, and insight. The research goals are threefold: to develop a sound foundational/mathematical understanding of DL; in doing so to advance the foundational understanding of learning more generally; and to advance the practice of DL by addressing its above-mentioned weaknesses. Of six foundational thrusts, the first two focus on the standard decomposition of the prediction error in approximation and sample (or estimation) error. Their goal is to extend classical results in approximation theory and theory of learnability to DL. These two are then supported by a research project that is specific to deep learning: analysis of the dynamics of gradient descent in training a network. The fourth theme is about robustness against adversaries and shifts, a powerful test for theories which is also important for practical deployment of learning systems. The fifth thrust is about developing the theory of control through DL, as well as exploring dynamical systems aspects of deep reinforcement learning. The final topic connects research on DL to its origins - and possibly its future: networks of neurons in the brain. The proposed research also promises to advance the foundations of learning theory. Success in this project will result in sharper mathematical techniques for machine learning and comprehensive foundations of machine learning robustness, broadly construed. It will also ultimately enable development of learning algorithms that transcend deep learning and guide the way towards creating more intelligent machines, and shed new light on our own intelligence.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Deep learning (DL) provides a powerful paradigm for modern machine learning (ML), with applications in a range of areas, such as natural language processing, computer vision and robotics. DL has proven powerful because it can capture complex relationships between input and output, and it enjoys efficient algorithms for analyzing massive datasets. But DL also has limitations. Currently, DL often provides ‘black box’ predictions. Black box indicates that the results do not involve clearly articulated assumptions or the degree of certainty in the results. To use ML in important applications, it is crucial to know from which assumptions the methods are based. Second, basic DL methods provide point predictions, but do not provide uncertainty about them. For ML to be safely deployed in critical decision-making systems, ML methods must provide calibrated measurement about the reliability of its predictions. Finally, all of these issues mean that DL does not provide easily interpretable predictions. Interpretability is important for understanding how ML makes mistakes, for deploying ML in high-stakes settings that require accountability, and when using ML predictions in the service of scientific understanding. This interdisciplinary project addresses these issues by using the rigorous methodology of probabilistic ML and applied Bayesian statistics to form interpretable DL models. Models that will be based on clearly stated assumptions and provide calibrated uncertainty about their predictions. This research aims to solve open problems in DL, provide mathematical clarity to some of its empirically proven ideas, and expand its reach to probabilistic modeling for broad applications in astronomy, language modeling for the computational social sciences, and electronic healthcare records.

The project will adapt modern ideas in DL for modern probabilistic models of complex datasets through research in two topics. The first topic develops the foundations of probabilistic deep learning, clarifying how deep neural network models draw from classical ideas like exponential families and generalized linear models, and expanding DL to Bayesian nonparametric models of infinite depth. The second topic develops empirical Bayes representation learning. Representation learning, a cornerstone of DL, is about finding low-dimensional descriptions of high-dimensional data. But from a statistical perspective, the problem is that many representations can accurately capture the distribution of the data. This project will explore how the powerful concept of empirical Bayes, a classical statistical idea that blends frequentist and Bayesian thinking, provides a natural framework for defining good representations. Through new theory, algorithms, and software, this project will significantly expand the capabilities of DL.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.