Kostas Daniilidis - US grants

Surround perception is crucial for an immersive sense of presence in communication and for efficient navigation and surveillance in robotics. To enable surround perception, new omnidirectional systems
provide a new impetus for us to rethink the way images are acquired and analyzed. This project will be focused on two fundamental issues: the intrinsic geometric properties of catadioptric omnidirectional sensors and the space-variant signal analysis of omnidirectional images with non-uniform resolution. The PI has proven in the preliminary work that every catadioptric system can be shown to be equivalent to a generalized stereographic projection from a virtual sphere to the real image plane. He intends to develop a unifying framework for all catadioptric systems, where conventional perspective cameras will be just a special case, and exhaustively study invariants of the projections on reflective urfaces as well as efficient representations for reconstruction and
motion estimation. Catadioptric designs need a new signal-theoretic treatment since
shift-invariant processing does not apply in the non-uniform resolution of omnidirectional images. This research will make use of integral transforms that map the 2D signal to
the spatial-frequency domain of a plane with virtually uniform resolution. The investigation will be extended from the simple edge detection to template matching and to the computation of optical flow in omnidirectional images.

This project, a collaborative with 03-24864 Papanikolopoulos at University of Minnesota, and 03-25017 Joel Burdick at California Institute of Technology, addresses research issues key to an important application of robot teams and information technology (emergency response in hazardous environments for various tasks). The research sets 6 goals:
Development of new algorithms that enable collaborative sensing.
Development of distributed localization/mapping methods that leverage capabilities of the heterogeneous robots.
In-depth study of communication issues with emphasis on transparent integration of ultra wideband communication methodologies.
Development of methods for team coordination and dynamic distribution of tasks to robots.
Creation of algorithms for the presentation of sensory information to users.
Experimental validation of the scalability of the aforementioned algorithms and techniques.
The PIs use the Scout and MegaScout robotic platforms designed at the University of Minnesota along with other testbeds at CalTech and U Penn to conduct the research. The project integrates the algorithms with first responder teams, emphasizing realistic scenarios; mentors students from underrepresented groups in order to retain them in CS/EE programs; conducts outreach activities through demonstrations at local schools and youth groups; conducts workshops that emphasize cross-disciplinary interaction; creates web resources; innovates classroom uses of multi-robot teams; and includes parts of the research in design projects for seniors. The project also includes international collaboration with groups at NTUA (Greece) and the University Louis Pasteur-Strasbourg I (France).

Today's archaeological excavations are slow and the cost for conservation can easily exceed the cost of excavation. This project is investigating and developing methods for the recovery of 3D underground structures from subsurface non-invasive measurements obtained with ground penetrating radar, magnetometry, and conductivity sensors. The results will not only provide hints for further excavation but also 3D models that can be studied as if they were already excavated. The three fundamental challenges investigated are the inverse problem of recovering the volumetric material distribution, the segmentation of the underground volumes, and the reconstruction of the surfaces that comprise interesting structures. In the recovery of the underground volume, high-fidelity geophysics models are introduced in their original partial differential equation form. Partial differential equations from multiple modalities are simultaneously solved to yield a material distribution volume. In segmentation, a graph spectral method for estimating graph cuts finds clusters of underground voxels with tight connections within partitions and loose connections between partitions. A method based on multi-scale graph cuts significantly accelerates the process while the grouping properties of the normalized cuts help in clustering together multiple fragments of the same material. In surface reconstruction, boundaries obtained from segmentation or from targeted material search are converted from unorganized voxel clouds to connected surfaces. A bottom-up approach is introduced that groups neighborhoods into facets whose amount of overlap guides the triangulation process. The archaeology PIs are providing prior knowledge on what structures are expected to be found which can lead the segmentation and the reconstruction steps.

The geoscience and archaeology PIs lead the effort of data acquisition at the Tiwanaku site in Bolivia. All original data as well as recovered 3D models will be made available to the public.

This project, developing an experimental testbed to work on different aspects of control and sensing for mobile networks of cameras and microphones, envisions a system of cameras, MACNet, moving in three dimensions enabling a Mobile Ad Hoc Camera Network. The development of this testbed provides the experimental infrastructure for the following interdisciplinary projects:
Monitoring, Evaluation, and Assessment for Geriatric Health Care,
Assistive Technology for People with Disabilities,
Digital Anthropology, and
Visual Servoing for Coordinating Aerial and Ground Robots.
MACNet cameras will be used to track patients and salient dynamic features for the 1st project. MACNet will simulate intelligent homes and museums with active cameras providing feedback to smart wheelchairs and providing information about target areas in the 2nd Project. MACNet will allow datasets of video to be acquired and analyzed for three-dimensional reconstruction and archiving in the 3rdProject; and for the last project, MACNet will provide dynamic platforms with cameras which will simulate aerial vehicles to track, localize, and coordinate with existing ground based autonomous vehicles with applications to surveillance and monitoring for homeland security. Finally, MACNet will be used as a testbed for research on distributed active vision, a theme that will bring together all researchers.

Broader Impact: The results will be directly applicable to a large class of problems in which communication, control, and sensing are coupled, with applications in smart homes, communities for assistive living, and surveillance and monitoring of homeland security. This cross-fertilization will contribute to train students with broader perspectives and potential new approaches to problem solving. The lab will be used by students and the institutions will leverage existing outreach programs in which both faculty and students participate, such as Robotics for Girls and PRIME.

The University of Pennsylvania has joined the multi-university Industry/University Cooperative Research Center for Safety, Security and Rescue Research located at the University of South Florida and the University of Minnesota. The I/UCRC will bring together industry, academe, and public sector users together to provide integrative robotics and artificial intelligence solutions in robotics for activities conducted by the police, FBI, FEMA, firefighting, transportation safety, and emergency response to mass casualty-related activities.

The need for safety, security, and rescue technologies has accelerated in the aftermath of 9/11 and a new research community is forming, as witnessed by the first IEEE Workshop on Safety, Security and Rescue Robotics. The Center is built upon the knowledge and expertise of multi-disciplinary researchers in computer science, engineering, industrial organization, psychology, public health, and marine sciences at member institutions.

There has been successful research on establishing metric representations of the environment required together with motion planning for any navigation task. Such metric maps require though excessive amounts of storage to memorize the robots' trajectories and all landmark positions. On the other hand, animals have excellent navigation capabilities based on visual sensing and simple path integration.

The technical approach can be summarized in the modeling of places and the map creation. An abstraction hierarchy is introduced for the visual modeling of places with the layers of feature landmarks, salient regions, and objects. A novel image similarity score will be used for tracking as well as loop closing and is robust to perceptual aliasing. Objects are learned from training sets of appearances of salient landmarks and in the highest abstraction level places are labeled depending on their object content and the constellation of objects in space. Topological maps are made of nodes labeled with place labels and associated with an action to neighboring nodes obtained from the relative pose between the two places. Learning of the maps will happen in the space of all possible topologies of place sets. A collaboration with biologists will try to cross-validate hypotheses based on visual inputs obtained from the animal's viewpoint.

Wayfinding is an essential capability for any person who wishes to have an independent life-style. It requires successful execution of several tasks including navigation and object and place recognition, all of which necessitate accurate assessment of the surrounding environment. For a visually-impaired person these tasks may be exceedingly difficult to accomplish and there are risks associated with failure in any of these. Guide dogs and white canes are widely used for the purpose of navigation and environment sensing, respectively. The former, however, has costly and often prohibitive training requirements, while the latter can only provide cues about obstacles in one?s surroundings. Human performance on visual information dependent tasks can be improved by sensing which provides information and environmental cues, such as position, orientation, local geometry, object description, via the use of appropriate sensors and sensor fusion algorithms. Most work on wayfinding aids has focused on outdoor environments and has led to the development of speech-enabled GPS-based navigation systems that provide information describing streets, addresses and points of interest. In contrast, the limited technology that is available for indoor navigation requires significant modification to the building infrastructure, whose high cost has prevented its wide use.

This proposal adopts a multi-faceted approach for solving the indoor navigation problem for people with limited vision. It leverages expertise from robotics, computer vision, and blind spatial cognition with behavioral studies on interface design to guide the discovery of information requirements and optimal delivery methods for an indoor navigation system. Designing perception and navigation algorithms, implemented on miniature-size commercially-available hardware, while explicitly considering the spatial cognition capabilities of the visually impaired, will lead to the development of indoor navigation systems that will assist blind people in their wayfinding tasks while facilitating cognitive-map development.

This award is funded under the American Recovery and Reinvestment Act of 2009 (Public Law 111-5).

This project provides infrastructure to the University of Pennsylvania to build from their existing work into the area of mobile manipulation.

The GRASP Lab is an internationally recognized robotics research group at the University of Pennsylvania. It hosts 14 faculty members and approximately 60 Ph.D. students from the primary departments of Computer and Information Science (CIS), Electrical and Systems Engineering (ESE), and Mechanical Engineering and Applied Mechanics (MEAM).

GRASP recently launched a multidisciplinary Masters program in Robotics and involves these students, as well as many Penn undergrads, in its research endeavors. The research conducted by the members of the GRASP laboratory includes such areas as vision, planning, control, multi-agent systems, locomotion, haptics, medical robotics, machine learning and modular robotics. This proposal requests funding for instruments that would enable us to broaden our research to include the important new area of mobile manipulation.

An increasing amount of research in robotics is being devoted to mobile manipulation because it holds great promise in assisting the elderly and the disabled at home, in helping workers with labor intensive tasks at factories, and in lessening the exposure of firemen, policemen and bomb squad members to dangers and hazards. As concluded by the NSF/NASA sponsored workshop on Autonomous Mobile Manipulation (AMM) in 2005, the technical challenges critically requiring the attention of researchers are:

? dexterous manipulation and physical interaction
? multi-sensor perception in unstructured environments
? control and safety near human beings
? technologies for human-robot interaction
? architectures that support fully integrated AMM systems

The researchers at GRASP fully support these recommendations and want to help lead the way in addressing them. Though GRASP conducts a great deal of research on similar challenges in related areas, this group has not worked on the unique and potentially transformative topic of mobile manipulation. The primary inhibitor to pursuing these challenges is that research in mobile manipulation critically depends on the availability of adequate experimental platforms. This group is requesting funding that would allow them to acquire such equipment.

In particular, this group is requesting funding for
(a) one human-scale mobile manipulator consisting of a Segway base, Barrett 7-DOF arm, and 3-fingered BarrettHand equipped with tactile sensors and a visual sensing suite; and
(b) four small Aldebaran Robotics NAO humanoid robots capable of locomotion and manipulation.

These instruments will allow the GRASP community to perform research in such areas as autonomous mobile manipulation, teleoperated mobile manipulation, navigation among people, bipedal locomotion, and coordination of multiple mobile manipulation platforms. Advances in this area are beneficial to society in a variety ways. For example, mobile manipulation is one of the most critical areas of research in household robotics, which aims at helping people (especially the elderly and the disabled) with their chores. Mobile manipulators will also see great use in office and industrial settings, where they can exibly automate a huge variety of manual tasks.

The project investigates computational methods for object detection, spatial scene construction, and natural language spatial descriptions derived from real-time visual images to describe prototypical indoor spaces (e.g., rooms, offices, etc.). The primary application of this research is to provide blind or visually impaired users with spatial information about their surroundings that may otherwise be difficult to obtain from non-visual sensing. Such knowledge will assist in development of accurate cognitive models of the environment and will support better informed execution of spatial behaviors in everyday tasks.

A second motivation for the work is to contribute to the improvement of spatial capacities for computers and robots. Computers and robots are similarly "blind" to images unless they have been provided some means to "see" and understand them. Currently, no robotic system is able to reliably perform high-level processing of spatial information on the basis of image sequences, e.g., to find an empty chair in a room, which not only means finding an empty chair in an image, but also localizing the chair in the room, and performing an action of reaching the chair. The guiding tenet of this research is that a better understanding of spatial knowledge acquisition from visual images and concepts of spatial awareness by humans can also be applied to reducing the ambiguity and uncertainty of information processing by autonomous systems.

A central contribution of this work is to make the spatial information content of visual images available to the visually impaired, a rapidly growing demographic of our aging society. In an example scenario a blind person and her guide dog are walking to her doctor's office, an office which she has not previously visited. At the office she needs information for performing some essential tasks such as finding the check-in counter, available seating, or the bathroom. No existing accessible navigation systems are able to describe the spatial parameters of an environment and help detect and localize objects in that space. Our work will provide the underlying research and elements to realize such a system.

This Integrative Graduate Education and Research Training (IGERT) award to the University of Pennsylvania supports the development of a new training paradigm for perception scientists and engineers, and is designed to provide them with a unique grasp of the computational and psychophysical underpinnings of the phenomena of perception. It will create a new role model of a well-rounded perceptual scientist with a firm grasp of both computational and experimental analytic skills. The existence of such a cadre of U.S. researchers will contribute to the country's global competitiveness in the growing machine perception and robotics industry.

Research and training activities are organized around five thematic areas related to complex scene perception: (1) Spatial perception and navigation; (2) Perception of material and terrain properties; (3) Neural responses to natural signals, saliency and attention; (4) Object Recognition in context and visual memory; and (5) Agile Perception. Interdisciplinary research will enable new insights into the astounding performance of human and animal perception as well as the design of new algorithms that will make robots perceive and act in complex scenes.

IGERT trainees will commit in advance of acceptance to a five-year graduate training program, comprising the following components: (1) Core disciplinary training; (2) one-year cross-disciplinary training in a chosen second discipline; (3) participation in two foundational and one integrational IGERT courses; (4) attendance of an interdisciplinary IGERT seminar; (5) co-advising throughout the 5 graduate years by an interdisciplinary faculty team ; and (6) completion of the Ph.D. dissertation.

IGERT is an NSF-wide program intended to meet the challenges of educating U.S. Ph.D. scientists and engineers with the interdisciplinary background, deep knowledge in a chosen discipline, and the technical, professional, and personal skills needed for the career demands of the future. The program is intended to catalyze a cultural change in graduate education by establishing innovative new models for graduate education and training in a fertile environment for collaborative research that transcends traditional disciplinary boundaries.

This Partnerships for Innovation (PFI) project, Autonomous Robotic Rotorcraft for Exploration, Surveillance, and Transportation (ARREST), from the University of Pennsylvania will create a partnership among small-business entrepreneurs in small rotor-craft and sensors and robotics researchers at the University of Pennsylvania developing autonomous robots. The partnership will enable the translation of fundamental, federally-sponsored, research results into products with societal benefits and commercial impact by implementing a loosely structured, application-focused, "play-like sandbox" environment among its partners. The intellectual merit of this project is derived from four projects aimed at developing products/ technologies enabled by the cross-fertilization of technologies for robotics and Unmanned Aircraft Vehicles (UAVs) and from the establishment of a sandbox to encourage exploration of potential applications and markets for technologies that are being, have been, or can be developed. New research contributions will include the development of small, autonomous micro aerial vehicles and the algorithms/software for sensing, perception, control and navigation without direct human command or intervention. Additionally, the partnership proposes innovation in doctoral education by adding a new dimension to student training to encourage "lateral and analogous thinking." It will consist of structured interactions with technology transfer specialists, inventors, and venture capitalists; collaborative research at the sites of the industrial partners; and new mechanisms for spurring innovation that includes an "inverted X-prize" competition called the "Y-prize," which focuses on the platform as the solution space (instead of as the problem space), thus fostering innovation by generating novel application ideas for using or adapting existing technological solutions and methodologies.

The broader impacts of the project are the new commercial applications and products that will be identified in collaboration with the knowledge-enhancement partners, and, in addition, the leveraging of already developed technologies to create these products. These products will be of direct use to end-users engaged in first response and also will provide incremental benefits to homeland security and to the military. The partnership will lead to the creation of a new robotics rotorcraft industry and help spawn a new generation of doctoral students, trained in the fundamentals with a depth that has come to be expected in top-notch doctoral programs, but also in the "lateral thinking" that is characteristic of agile, innovative small businesses. The partnership will establish a model for leveraging Department of Defense investment in basic research for dual-use applications and a framework for identifying and pursuing commercial applications that are not directly connected to security, first response, and defense applications.

Partners at the inception of the project include the Knowledge Enhancement Partnership (KEP) unit, consisting of the University of Pennsylvania (GRASP Robotics Laboratory); two small businesses: Dragon Pictures, Inc. (Essington, PA) and EmergentViews (San Francisco, CA); and a large private sector organization: Advanced Technologies Laboratory, Lockheed Martin Corporation. In addition, there are other core partners. These include public sector partners: the City of Philadelphia, and the Southeast Pennsylvania Ben Franklin Technology Partnership and their network of Angel and Venture Investors.

There are currently very few ways for the blind to navigate a new indoor space without the assistance of a fully-sighted person. The technology proposed by this project is designed to enable a visually-impaired individual to find their way through large indoor environments such as airports, train stations and shopping malls by recognizing semantic and salient visual features of the environment. There is no prior visit or mapping of the environment required, and there is no need to deploy or utilize any special infrastructure like WiFi access points or infrared beacons. Researchers plan to use publically available architectural lay-outs and information about the location of ships, tracks, gates and other visual cues. The platform is a cell-phone mounted on a necklace that provides turn-by-turn directions through an audio-voice command interface. This technology is designed to process video from the cell phone camera in real-time using text and logo detection, localization based on prior knowledge of the layout and integration of accelerometer and visual odometry.

The blind and visually-impaired population in the United States is large and expected to grow in the future. If successfully implemented, this technology could have broader reaching applications, including many location-based services such as aiding those with spatial learning difficulties or guiding users to a specific location. The project team has the expertise required to develop this technology at a relatively rapid rate and economical cost.

With advances in camera technologies, and as cloud storage, network bandwidth and protocols become available, visual media are becoming ubiquitous. Video recording became de facto universal means of instruction for a wide range of applications such as physical exercise, technology, assembly, or cooking. This project addresses the scientific and technological challenges of video shooting in terms of coverage and optimal views planning while leaving high level aspects including creativity to the video editing and post-production stages.

Camera placement and novel view selection challenges are modeled as optimization problems that minimize the uncertainty in the location of actors and objects, maximize coverage and effective appearance resolution, and optimize object detection for the sake of semantic annotation of the scene. New probabilistic models capture long range correlations when the trajectories of actors are only partially observable. Quality of potential novel views is modeled in terms of resolution that is optimized by maximizing the coverage of a 3D orientation histogram while an active view selection process for object detection minimizes a dynamic programming objective function capturing the loss due to classification error as well as the resources spent for each view.

The project advances active sensing and perception and provides the technology for further automation on video capturing. Such technology has broader impact on the production of education videos for online courses as well as in telepresence applications. Research results are integrated into robotics and digital media programs addressing K-12 students.

The proposed I/UCRC for Robots and Sensors for the Human Well-being (RoSeHuB) will focus on complementing a broad variety of off-the-self sensors with intelligent processing software that enables them to extract useful information about the operating environments in medicine and agriculture. RoSeHuB research will make heavy use of commercial cameras that can work in different parts of the electro-magnetic spectrum (i.e., visible, IR, Thermal, etc.), laser or radar sensors, etc. Sensors or sensor systems may exhibit different degrees of mobility. They may be embedded in robots or flying drones or they may be fixed with limited degrees of motion (PTZ cameras). In the areas of algorithms and learning methods the focus and the challenge is on creating methodologies that can balance real-time operation and computational power while providing high level semantic information either for planning, interaction or situational awareness for human operators. With respect to robots, efforts will focus on building systems with advanced mobility, manipulation, human-machine interaction, and coordination skills.

Robots and sensors can lead to more effective precision agriculture techniques that provide more food than current levels while they save water and prevent soil erosion. Similarly, robots play a critical and growing role in modern medicine, from training the next generation of doctors, dentists, and nurses, to comforting and protecting elderly patients in the early stages of dementia. The proposed Center will attract large companies to the pertinent domains and energize innovative startup companies, both through research and through the production of highly trained graduate students with advanced coursework in sensory-based robotic systems and hands-on exposure to multi-disciplinary, integrative systems. The Center will also fund a projects to pursue new pertinent high-risk initiatives, ensuring that technology continues to meet emerging societal needs. RoSeHuB faculty will aggressively recruit women and minority graduate and undergraduate students and host an annual summer camp for middle-schoolers from underrepresented groups.

This project wants to capitalize on a team of robots that move in order to better sense the environment and then perform basic manipulation tasks. The vision of the project is to integrate robots easily, provide human-team interfaces, and develop manipulation algorithms. The research will involve the development of perception strategies and manipulation schemes that will allow operation of the robot teams in real-world environments during search and rescue missions. In addition, this research will involve working in cluttered scenes where the lighting conditions may not be ideal. The project addresses: (i) Research on the novel problem of robotic perception and manipulation of target objects that interact with other objects as an integral part of the environment, which cannot be fully isolated in views and in physical arrangements before being manipulated; (ii) An appearance-based approach for recognition and pose estimation of 3D objects in cluttered scenes from a single view; (iii) Development of a measure of scene recognizability from each viewpoint to evaluate how accurately partially-occluded objects are recognized and how well their poses are estimated; (iv) Creation of solutions for disassembly analysis of 3D structures, extending our preliminary analysis of 2D structures; (v) Development of grasping based on the results of perception and with the aid of stability analysis of the arrangement of the objects and their interaction with the environment and with one another; and (vi) Experimental validation of the system in real-world settings, in close consultation with our industrial partners.

This project will allow the creation of manipulation capabilities along with perception schemes to facilitate the development of a multi-robot team for search and rescue missions. The impacts of the project include: (i) Expansion of the annual robot summer camp to include robot-team activities with the objective of attracting middle-schoolers from under-represented groups to computer science/electrical engineering; (ii) Integration of the activities with first responders through the UPenn and UNC Charlotte collaborations; (ii) Experimental validation in SAFL at UMN and the Disaster City in TX; (iii) Offering project themes to REU undergraduates or to the UROP program; (iv) Outreach programs that involve demonstrations to local K-12 institutions; and (v) Inclusion of the project theme to the regular curricula.

The project develops and tests novel compressive sensing and sensor locating techniques that are adaptable to a myriad of different mobile robot designs while operable on today's wireless communication infrastructures. Unique in-situ laboratory and field experiments provide tangible results to scientists and other stakeholders that can be leveraged to advance these systems into future real-world hazard management scenarios. The research team develops new technological approaches that results in mobilizing more intelligent, automated "eyes and ears on the ground." Outreach efforts include: (i) integration of the activities with practitioners; (ii) Seminars/webcasts to audiences like environmental engineers and first responders; (iii) Annual technology day camps to attract middle-schoolers from under-represented groups to engineering; (iv) Demonstrations to local K-12 institutions; (v) Inclusion of the project themes to the regular curricula; and (vi) International collaborations.

This project introduces Robotics 2.0; a framework that targets autonomous robots that are co-workers and co-protectors, adapting to and working with humans. The research team develops a Cyber-Control Network (CCN) to allow multiple fixed and mobile robotic environmental sensing and measurements to adapt quickly to the changing environment by dynamically linking sub-networks of actuation, sensing, and control together. The design of such CCN ControlWare, and compressive sensing architectures, could be adapted to other large-scale problems beyond disaster response, mitigation, and management, such as power grid monitoring and reconfiguration, or regional urban traffic operations to respond to traffic congestion and incidents. The robotic sensing platforms do not require a-priori knowledge of the hazardous and dynamically changing environments they are monitoring. The Robotics 2.0 framework allows to swiftly respond, to prepare, and to manage various types of disasters.

This project, developing a new instrument to enable an accurate quantitative analysis of the movement of animals and vocal expressions in real world scenes, aims to facilitate innovative research in the study of animal behavior and neuroscience in complex realistic environments. While much progress has been made investigating brain mechanisms of behavior, these have been limited primarily to studying individual subjects in relatively simple settings. For many social species, including humans, understanding neurobiological processes within the confines of these more complex environments is critical because their brains have evolved to perceive and evaluate signals within a social context. Indeed, today's advances in video capture hardware and storage and in algorithms in computer vision and network science make this facilitation with animals possible. Past work has relied on subjective and time-consuming observations from video streams, which suffer from imprecision, low dimensionality, and the limitations of the expert analyst's sensory discriminability. This instrument will not only automate the process of detecting behaviors but also provide an exact numeric characterization in time and space for each individual in the social group. While not explicitly part of the instrument, the quantitative description provided by our system will allow the ability to correlate social context with neural measurements, a task that may only be accomplished when sufficient spatiotemporal precision has been achieved.

The instrument enables research in the behavioral and neural sciences and development of novel algorithms in computer vision and network theory. In the behavioral sciences, the instrumentation allows the generation of network models of social behavior in small groups of animals or humans that can be used to ask questions that can range from how the dynamics of the networks influence sexual selection, reproductive success, and even health messaging to how vocal decision making in individuals gives rise to social dominance hierarchies. In the neural sciences, the precise spatio-temporal information the system would provide can be used to evaluate the neural bases of sensory processing and behavioral decision under precisely defined social contexts. Sensory responses to a given vocal stimulus, for example, can be evaluated by the context in which the animal heard the stimulus and both his and the sender's prior behavioral history in the group. In computer vision, we propose novel approaches for the calibration of multiple cameras "in the wild", the combination of appearance and geometry for the extraction of exact 3D pose and body parts from video, the learning of attentional focus among animals in a group, and the estimation of sound source and the classification of vocalizations. New approaches will be used on hierarchical discovery of behaviors in graphs, the incorporation of interactions beyond the pairwise level with simplicial complices, and a novel theory of graph dynamics for the temporal evolution of social behavior. The instrumentation benefits behavioral and neural scientists. Therefore, the code and algorithms developed will be open-source so that the scientific community can extend them based on the application. The proposed work also impacts computer vision and network science because the fundamental algorithms designed should advance the state of the art. For performance evaluation of other computer vision algorithms, established datasets will be employed.

For many decades, neuroscientists and evolutionary biologists have been interested in the mechanics and function of the songbird's "song system": the interconnected neural circuit that connects the higher-order auditory areas in the brain with the motor circuits in the brainstem that drive behavior. This work predominantly has focused on how the song system allows male songbirds to learn and produce song. The role of this circuit in female songbirds, which do not sing, has largely been ignored. Rather than acting as a circuit that generates vocal behavior, this work investigates the hypothesis that the "song system" in females serves to organize preferences for males' songs and guides their behavioral reactions to song in the form of a copulation solicitation display that ensures survival of the species. The project capitalize on the robustness, selectivity, and social malleability of the copulatory behavior in the brown-headed cowbird, to investigate how the song system transforms a sensory stimulus (the song) into a motor command that controls a postural response. The project also provides opportunities for undergraduate and graduate students to engage in interdisciplinary research, and it includes science education activities aimed at elementary school children as well as a comprehensive summer course in neuroscience for high school students.

The proposed work integrates disparate fields of science, including neuroscience, behavior, and engineering to provide unique insight into the evolution of neural circuits that control behavior. In the first aim, the investigators use a combination of classic pathway tracing techniques and recently developed transsynaptic tracer (vesicular stomatitis virus) to map the connectivity from the forebrain to the individual muscle groups that are activated during the production of a copulation solicitation display (CSD). In the second aim, the investigators record neural activity in forebrain song control nuclei HVC and RA during the production of CSD in female cowbirds to quantify the nature of the forebrain motor commands that control this highly selective sexual behavior. To evaluate the relationship between recorded neural activity patterns and the behavior, we will use a computer vision approach to quantify the copulatory behavior. In the final aim of the proposal, the investigators record neural responses to song in higher-order auditory forebrain areas (NCM, NIf, CM) within the context of CSD production. These experiments serve to test the hypothesis that these forebrain areas, which have known projections to song control nuclei, encode song valence and provide a direct link between song quality and the females' behavioral response. The neural and behavioral data will be made available at public internet site dedicated to Song Bird Science.

The objective of this award is to organize the annual Principal Investigators (PI) meeting for the National Robotics Initiative (NRI), which was launched in 2011. The PI meeting brings together the community of researchers, companies, and program officers who are actively engaged in the NRI to provide cross-project coordination in terms of common intellectual challenges, methods for education and training, best practices in terms of transition of results, and a centralized and lasting repository illustrating the research ideas explored and milestones achieved by the NRI projects. The meeting will be two days during late fall 2016 in the vicinity of Washington DC. The format will include short presentations by all the attending PIs, a poster session, keynote speeches, and panel discussions. Invitations to the meeting will include all PIs with active NRI grants, program managers with robotics-related programs, and members of the press.

Modern robots can be seen moving about a variety of terrains and environments, using wheels, legs, and other means, engaging in life-like hopping, jumping, walking, crawling, and running. They execute motions called gaits. An example of a gait is a horse trotting or galloping. Likewise, humans execute walking, running and skipping gaits. Essentially, for either a biological or mechanical systems, a gait is a locomotion pattern that involves large-amplitude body oscillations. Naturally, these motions cause impacts with terrain that jostle on-board perceptual systems and directly influence what the robots actually "see" as they move. For instance, the body motion of a bounding horse-like robot may result in significant occlusions and oscillations in on-board camera systems that confound motion estimation and perceptual feedback.

Focusing on complex mobility robots, this project seeks to better understand the coupling between locomotion and visual perception to improve perceptual feedback for closed-loop motion estimation. The work is organized around two key questions: 1) How should a robot look to move well? 2) How should a robot move to see well? To address the first challenge, the periodic structure of gait-based motions will be leveraged to improve perceptual filtering as the robot carries out fixed (pre-determined) motions. The second half of the project will derive perceptual objectives and a new perceptual gait design framework to guide how high degree-of-freedom, complex mobility robots should move (locomote). The goal is to optimize feedback for closed-loop motion implementation, on-line adaptation, and learning, which are currently difficult or impossible for many complex mobility robots.

Recent advances in artificial intelligence have led to significant progress in our ability to extract information from images and time sequences. Maintaining this rate of progress hinges upon attaining equally significant results in the processing of more complex signals such as those that are acquired by autonomous systems and networks of connected devices, or those that arise in the study of complex biological and social systems. This award establishes FINPenn, the Center for the Foundations of Information Processing at the University of Pennsylvania. The focus of the center is to establish fundamental theory to enable the study of data beyond time and images. The center's premise is that humans' rich intuitive understanding of space and time may not necessarily be applicable to the processing of complex signals. Therefore, matching the success in time and space necessitates the discovery and development of foundational principles to guide the design of generic artificial intelligence algorithms. FINPenn will support a class of scholar trainees along with a class of visiting postdocs and students to advance this agenda. The center will engage the community through the organization of workshops and lectures and will disseminate knowledge with onsite and online educational activities at the undergraduate and graduate level.

FINPenn builds on two observations: (i) To understand the foundations of data science it is necessary to succeed beyond Euclidean signals in time and space. This is true even to understand the foundations for Euclidean signal processing. (ii) Humans live in Euclidean time and space. To succeed in information processing beyond signals with Euclidean structure, operation from foundational principles is necessary because human intuition is of limited help. For instance, convolutional neural networks have found success in the processing of images and signals in time but they rely heavily on spatial and temporal intuition. To generalize their success to unconventional signal domains it is necessary to postulate fundamental principles and generalize from those principles. If the generalizations are successful they not only illuminate the new application domains but they also help establish the validity of the postulated principles for Euclidean spaces in the tradition of predictive science. The proposers further contend that the foundational principles of data sciences are to be found in the exploitation of structure and the associated invariances and symmetries that structure generates. The initial focus of the center is in advancing the theory of information processing in signals whose structure is defined by a group, a graph, or a topology. These three types of signals generate three foundational research directions which build on the particular strengths of the University of Pennsylvania on network sciences, robotics, and autonomous systems which are areas in which these types of signals appear often.

This project is part of the National Science Foundation's Harnessing the Data Revolution (HDR) Big Idea activity.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Consider two future autonomous system use-cases: (i) a bomb defusing rover sent into an unfamiliar, GPS and communication denied environment (e.g., a cave or mine), tasked with the objective of locating and defusing an improvised explosive device, and (ii) an autonomous racing drone competing in a future autonomous incarnation of the Drone Racing League. Both systems will make decisions based on inputs from a combination of simple, single output sensing devices, such as inertial measurement units, and complex, high dimensional output sensing modalities, such as cameras and LiDAR. This shift from relying only on simple, single output sensing devices to systems that incorporate rich, complex perceptual sensing modalities requires rethinking the design of safety-critical autonomous systems, especially given the inextricable role that machine and deep learning play in the design of modern perceptual sensors. These two motivating examples raise an even more fundamental question however: given the vastly different dynamics, environments, objectives, and safety/risk constraints, should these two systems have perceptual sensors with different properties? Indeed, due to the extremely safety critical nature of the bomb defusing task, an emphasis on robustness, risk aversion, and safety seems necessary. Conversely, the designer of the drone racer may be willing to sacrifice robustness to maximize responsiveness and lower lap-time. This extreme diversity in requirements highlights the need for a principled approach to navigate tradeoffs in this complex design space, which is what this proposal seeks to develop. Existing approaches to designing perception/action pipelines are either modular, which often ignore uncertainty and limit interaction between components, or monolithic and end-to-end, which are difficult to interpret, troubleshoot, and have high sample-complexity.

This project proposes an alternative approach and rethinks the scientific foundations of using machine learning and computer vision to process rich high-dimensional perceptual data for use in safety-critical cyber-physical control applications. Thrusts will develop integration between perception, planning and control that allow for their co-design and co-optimization. Using novel robust learning methods for perceptual representations and predictive models that characterize tradeoffs between robustness (e.g., to lighting & weather changes, rotations) and performance (e.g., responsiveness, discriminativeness), jointly learned perception maps and uncertainty profiles will be abstracted as ``noisy virtual sensors? for use in uncertainty aware perception-based planning & control algorithms with stability, performance, and safety guarantees. These insights will be integrated into novel perception-based model predictive control algorithms, which allow for planning, stability, and safety guarantees through a unifying optimization-based framework acting on rich perceptual data. Experimental validation of the benefits of these methods will be conducted at Penn using photorealistic simulations and physical camera equipped quadcopters, and be used to demonstrate perception-based planning and control algorithms at the extremes of speed/safety tradeoffs. On the educational front, the research outcomes of this proposal will be used to develop a sequence of courses on safe autonomy, safe perception, and learning and control at the University of Pennsylvania. Longer term, the goal of this project is to create a new community of researchers that focus on robust learning for perception-based control. Towards this goal, departmental efforts will be leveraged to increase and diversify the PhD students working on this project.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Advances in technology, mathematics, computing and engineering are making it possible to quantify behaviors within complex naturalistic environments and to relate them to underlying neural mechanisms. For social animals, which have evolved to perceive and evaluate signals within a community context, the ability to link neural function with the precise social environment is especially important and challenging. Little is currently known about how the brain integrates complex social information and how such information might be encoded. This stems in part from the experimental challenge of measuring and assessing the variables that determine a social context and then linking the state of a social network to precise neural events. This project has assembled an interdisciplinary team of engineers, neurobiologists and computational scientists to create a platform to record and evaluate brain dynamics in individual animals navigating a complex social environment. In addition to the challenge and opportunity of using sophisticated engineering and computational approaches to study how brains encode social information, this work will generate a complex dataset that will offer unique opportunities for developing novel mathematical methods to quantify and visualize social networks that can be applied to other disciplines.

In this study, a "smart aviary" is equipped with arrays of cameras and microphones to create a fully automated system for tracking moment-to-moment behavioral events for each individual songbird within a social group. The songbirds are of a highly gregarious species (brown-headed cowbird, Molothrus ater) that uses vocal communications to form and maintain a complex social system. As a variable, social context needs to be mathematically constructed over multiple timescales from the sequence of all behavioral events. This entails the development of new mathematical approaches and statistical models for quantifying social network state so that individual neural events can be linked back to the precise social contexts. The project will develop new machine learning approaches for automated capture of social interactions over months-long time periods. In addition, an articulated mesh model enables visual signals to be captured in unprecedented detail, allowing the quantification of shape-mediated social signaling.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

This project aims to advance the state of the art in robotic mapping and navigation by enabling spatial understanding using semantic maps and spatial reasoning for following language instructions given only visual inputs. Current performance in those tasks is low because of the inability to ground semantic entities and instructions spatially. Instead of grounding semantics to images, spatial understanding and navigation can be achieved if a system uses maps as an intermediate representation, as also indicated by behavioral and neural findings in spatial cognition. Building a map of an unseen space without exhaustive exploration can be learned, and this process can be facilitated by cross-modal language-vision attentional mechanisms. The project will integrate research with education and outreach underrepresented groups in Philadelphia neighborhoods as a target broadening the participation.<br/><br/>This research is centered around understanding how vision and language interact to create better spatial representations like maps and facilitate navigation. The project will approach the vision-language from three angles. (i) How robots can learn to predict a map when entering an unseen environment using active learning. (ii) How navigation instructions can be encoded into spatial configuration schemata and navigational concepts that can be better aligned to maps and paths than raw language embeddings, and (iii) how navigational language representations can facilitate the creation of maps in unseen environments, and how one can follow instructions by using maps and language to create paths to follow.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Robots will perform better at everyday activities when they can quickly combine their sensory data into a model of their environment, just like how humans instinctively use all their senses and knowledge to accomplish daily tasks. Robots, however, must be programmed to create these models that humans do intuitively, effortlessly, and robustly. This robotics project explores a novel algorithmic approach that combines visual and tactile sensory data with a knowledge of physics and a capability to learn that makes robot planning and reasoning more effective, efficient, and adaptable. The project includes the development and testing of research prototypes, preparation of new curriculum, and outreach to high school students and teachers and to the general public.<br/><br/>This project introduces a new data representation, called a Visual Tactile Neural Field (VTNF), that allows robots to combine data from visual and tactile sensors to create a unified model of an object. The VTNF is designed to be used in a closed-loop manner, where a robot may use data from its physical interactions with an object to create or improve a model and may use its current understanding of a model to inform how best to interact with a physical object. Towards this end, the investigators create the mathematical techniques, computational tools, and robot hardware necessary to generate a VTNF model. The investigators also develop techniques to quantify the uncertainty about an object and use this uncertainty to learn search policies that allow robots to generate accurate models as quickly as possible. The VTNF, which allows for the easy addition of new properties about an object, provides a flexible representational foundation for other researchers and practitioners to use to enable robots to learn faster by having a more detailed understanding of both the surrounding environment and their interactions with it.<br/><br/>This project is supported by the cross-directorate Foundational Research program in Robotics and the National Robotics Initiative, jointly managed and funded by the Directorates for Engineering (ENG) and Computer and Information Science and Engineering (CISE).<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.