Kostas Daniilidis - US grants

This project, a collaborative with 03-24864 Papanikolopoulos at University of Minnesota, and 03-25017 Joel Burdick at California Institute of Technology, addresses research issues key to an important application of robot teams and information technology (emergency response in hazardous environments for various tasks). The research sets 6 goals:
Development of new algorithms that enable collaborative sensing.
Development of distributed localization/mapping methods that leverage capabilities of the heterogeneous robots.
In-depth study of communication issues with emphasis on transparent integration of ultra wideband communication methodologies.
Development of methods for team coordination and dynamic distribution of tasks to robots.
Creation of algorithms for the presentation of sensory information to users.
Experimental validation of the scalability of the aforementioned algorithms and techniques.
The PIs use the Scout and MegaScout robotic platforms designed at the University of Minnesota along with other testbeds at CalTech and U Penn to conduct the research. The project integrates the algorithms with first responder teams, emphasizing realistic scenarios; mentors students from underrepresented groups in order to retain them in CS/EE programs; conducts outreach activities through demonstrations at local schools and youth groups; conducts workshops that emphasize cross-disciplinary interaction; creates web resources; innovates classroom uses of multi-robot teams; and includes parts of the research in design projects for seniors. The project also includes international collaboration with groups at NTUA (Greece) and the University Louis Pasteur-Strasbourg I (France).

Today's archaeological excavations are slow and the cost for conservation can easily exceed the cost of excavation. This project is investigating and developing methods for the recovery of 3D underground structures from subsurface non-invasive measurements obtained with ground penetrating radar, magnetometry, and conductivity sensors. The results will not only provide hints for further excavation but also 3D models that can be studied as if they were already excavated. The three fundamental challenges investigated are the inverse problem of recovering the volumetric material distribution, the segmentation of the underground volumes, and the reconstruction of the surfaces that comprise interesting structures. In the recovery of the underground volume, high-fidelity geophysics models are introduced in their original partial differential equation form. Partial differential equations from multiple modalities are simultaneously solved to yield a material distribution volume. In segmentation, a graph spectral method for estimating graph cuts finds clusters of underground voxels with tight connections within partitions and loose connections between partitions. A method based on multi-scale graph cuts significantly accelerates the process while the grouping properties of the normalized cuts help in clustering together multiple fragments of the same material. In surface reconstruction, boundaries obtained from segmentation or from targeted material search are converted from unorganized voxel clouds to connected surfaces. A bottom-up approach is introduced that groups neighborhoods into facets whose amount of overlap guides the triangulation process. The archaeology PIs are providing prior knowledge on what structures are expected to be found which can lead the segmentation and the reconstruction steps.

The geoscience and archaeology PIs lead the effort of data acquisition at the Tiwanaku site in Bolivia. All original data as well as recovered 3D models will be made available to the public.

The University of Pennsylvania has joined the multi-university Industry/University Cooperative Research Center for Safety, Security and Rescue Research located at the University of South Florida and the University of Minnesota. The I/UCRC will bring together industry, academe, and public sector users together to provide integrative robotics and artificial intelligence solutions in robotics for activities conducted by the police, FBI, FEMA, firefighting, transportation safety, and emergency response to mass casualty-related activities.

The need for safety, security, and rescue technologies has accelerated in the aftermath of 9/11 and a new research community is forming, as witnessed by the first IEEE Workshop on Safety, Security and Rescue Robotics. The Center is built upon the knowledge and expertise of multi-disciplinary researchers in computer science, engineering, industrial organization, psychology, public health, and marine sciences at member institutions.

This award is funded under the American Recovery and Reinvestment Act of 2009 (Public Law 111-5).

This project provides infrastructure to the University of Pennsylvania to build from their existing work into the area of mobile manipulation.

The GRASP Lab is an internationally recognized robotics research group at the University of Pennsylvania. It hosts 14 faculty members and approximately 60 Ph.D. students from the primary departments of Computer and Information Science (CIS), Electrical and Systems Engineering (ESE), and Mechanical Engineering and Applied Mechanics (MEAM).

GRASP recently launched a multidisciplinary Masters program in Robotics and involves these students, as well as many Penn undergrads, in its research endeavors. The research conducted by the members of the GRASP laboratory includes such areas as vision, planning, control, multi-agent systems, locomotion, haptics, medical robotics, machine learning and modular robotics. This proposal requests funding for instruments that would enable us to broaden our research to include the important new area of mobile manipulation.

An increasing amount of research in robotics is being devoted to mobile manipulation because it holds great promise in assisting the elderly and the disabled at home, in helping workers with labor intensive tasks at factories, and in lessening the exposure of firemen, policemen and bomb squad members to dangers and hazards. As concluded by the NSF/NASA sponsored workshop on Autonomous Mobile Manipulation (AMM) in 2005, the technical challenges critically requiring the attention of researchers are:

? dexterous manipulation and physical interaction
? multi-sensor perception in unstructured environments
? control and safety near human beings
? technologies for human-robot interaction
? architectures that support fully integrated AMM systems

The researchers at GRASP fully support these recommendations and want to help lead the way in addressing them. Though GRASP conducts a great deal of research on similar challenges in related areas, this group has not worked on the unique and potentially transformative topic of mobile manipulation. The primary inhibitor to pursuing these challenges is that research in mobile manipulation critically depends on the availability of adequate experimental platforms. This group is requesting funding that would allow them to acquire such equipment.

In particular, this group is requesting funding for
(a) one human-scale mobile manipulator consisting of a Segway base, Barrett 7-DOF arm, and 3-fingered BarrettHand equipped with tactile sensors and a visual sensing suite; and
(b) four small Aldebaran Robotics NAO humanoid robots capable of locomotion and manipulation.

These instruments will allow the GRASP community to perform research in such areas as autonomous mobile manipulation, teleoperated mobile manipulation, navigation among people, bipedal locomotion, and coordination of multiple mobile manipulation platforms. Advances in this area are beneficial to society in a variety ways. For example, mobile manipulation is one of the most critical areas of research in household robotics, which aims at helping people (especially the elderly and the disabled) with their chores. Mobile manipulators will also see great use in office and industrial settings, where they can exibly automate a huge variety of manual tasks.

This Integrative Graduate Education and Research Training (IGERT) award to the University of Pennsylvania supports the development of a new training paradigm for perception scientists and engineers, and is designed to provide them with a unique grasp of the computational and psychophysical underpinnings of the phenomena of perception. It will create a new role model of a well-rounded perceptual scientist with a firm grasp of both computational and experimental analytic skills. The existence of such a cadre of U.S. researchers will contribute to the country's global competitiveness in the growing machine perception and robotics industry.

Research and training activities are organized around five thematic areas related to complex scene perception: (1) Spatial perception and navigation; (2) Perception of material and terrain properties; (3) Neural responses to natural signals, saliency and attention; (4) Object Recognition in context and visual memory; and (5) Agile Perception. Interdisciplinary research will enable new insights into the astounding performance of human and animal perception as well as the design of new algorithms that will make robots perceive and act in complex scenes.

IGERT trainees will commit in advance of acceptance to a five-year graduate training program, comprising the following components: (1) Core disciplinary training; (2) one-year cross-disciplinary training in a chosen second discipline; (3) participation in two foundational and one integrational IGERT courses; (4) attendance of an interdisciplinary IGERT seminar; (5) co-advising throughout the 5 graduate years by an interdisciplinary faculty team ; and (6) completion of the Ph.D. dissertation.

IGERT is an NSF-wide program intended to meet the challenges of educating U.S. Ph.D. scientists and engineers with the interdisciplinary background, deep knowledge in a chosen discipline, and the technical, professional, and personal skills needed for the career demands of the future. The program is intended to catalyze a cultural change in graduate education by establishing innovative new models for graduate education and training in a fertile environment for collaborative research that transcends traditional disciplinary boundaries.

There are currently very few ways for the blind to navigate a new indoor space without the assistance of a fully-sighted person. The technology proposed by this project is designed to enable a visually-impaired individual to find their way through large indoor environments such as airports, train stations and shopping malls by recognizing semantic and salient visual features of the environment. There is no prior visit or mapping of the environment required, and there is no need to deploy or utilize any special infrastructure like WiFi access points or infrared beacons. Researchers plan to use publically available architectural lay-outs and information about the location of ships, tracks, gates and other visual cues. The platform is a cell-phone mounted on a necklace that provides turn-by-turn directions through an audio-voice command interface. This technology is designed to process video from the cell phone camera in real-time using text and logo detection, localization based on prior knowledge of the layout and integration of accelerometer and visual odometry.

The blind and visually-impaired population in the United States is large and expected to grow in the future. If successfully implemented, this technology could have broader reaching applications, including many location-based services such as aiding those with spatial learning difficulties or guiding users to a specific location. The project team has the expertise required to develop this technology at a relatively rapid rate and economical cost.

This project wants to capitalize on a team of robots that move in order to better sense the environment and then perform basic manipulation tasks. The vision of the project is to integrate robots easily, provide human-team interfaces, and develop manipulation algorithms. The research will involve the development of perception strategies and manipulation schemes that will allow operation of the robot teams in real-world environments during search and rescue missions. In addition, this research will involve working in cluttered scenes where the lighting conditions may not be ideal. The project addresses: (i) Research on the novel problem of robotic perception and manipulation of target objects that interact with other objects as an integral part of the environment, which cannot be fully isolated in views and in physical arrangements before being manipulated; (ii) An appearance-based approach for recognition and pose estimation of 3D objects in cluttered scenes from a single view; (iii) Development of a measure of scene recognizability from each viewpoint to evaluate how accurately partially-occluded objects are recognized and how well their poses are estimated; (iv) Creation of solutions for disassembly analysis of 3D structures, extending our preliminary analysis of 2D structures; (v) Development of grasping based on the results of perception and with the aid of stability analysis of the arrangement of the objects and their interaction with the environment and with one another; and (vi) Experimental validation of the system in real-world settings, in close consultation with our industrial partners.

This project will allow the creation of manipulation capabilities along with perception schemes to facilitate the development of a multi-robot team for search and rescue missions. The impacts of the project include: (i) Expansion of the annual robot summer camp to include robot-team activities with the objective of attracting middle-schoolers from under-represented groups to computer science/electrical engineering; (ii) Integration of the activities with first responders through the UPenn and UNC Charlotte collaborations; (ii) Experimental validation in SAFL at UMN and the Disaster City in TX; (iii) Offering project themes to REU undergraduates or to the UROP program; (iv) Outreach programs that involve demonstrations to local K-12 institutions; and (v) Inclusion of the project theme to the regular curricula.

The proposed I/UCRC for Robots and Sensors for the Human Well-being (RoSeHuB) will focus on complementing a broad variety of off-the-self sensors with intelligent processing software that enables them to extract useful information about the operating environments in medicine and agriculture. RoSeHuB research will make heavy use of commercial cameras that can work in different parts of the electro-magnetic spectrum (i.e., visible, IR, Thermal, etc.), laser or radar sensors, etc. Sensors or sensor systems may exhibit different degrees of mobility. They may be embedded in robots or flying drones or they may be fixed with limited degrees of motion (PTZ cameras). In the areas of algorithms and learning methods the focus and the challenge is on creating methodologies that can balance real-time operation and computational power while providing high level semantic information either for planning, interaction or situational awareness for human operators. With respect to robots, efforts will focus on building systems with advanced mobility, manipulation, human-machine interaction, and coordination skills.

Robots and sensors can lead to more effective precision agriculture techniques that provide more food than current levels while they save water and prevent soil erosion. Similarly, robots play a critical and growing role in modern medicine, from training the next generation of doctors, dentists, and nurses, to comforting and protecting elderly patients in the early stages of dementia. The proposed Center will attract large companies to the pertinent domains and energize innovative startup companies, both through research and through the production of highly trained graduate students with advanced coursework in sensory-based robotic systems and hands-on exposure to multi-disciplinary, integrative systems. The Center will also fund a projects to pursue new pertinent high-risk initiatives, ensuring that technology continues to meet emerging societal needs. RoSeHuB faculty will aggressively recruit women and minority graduate and undergraduate students and host an annual summer camp for middle-schoolers from underrepresented groups.

This project, developing a new instrument to enable an accurate quantitative analysis of the movement of animals and vocal expressions in real world scenes, aims to facilitate innovative research in the study of animal behavior and neuroscience in complex realistic environments. While much progress has been made investigating brain mechanisms of behavior, these have been limited primarily to studying individual subjects in relatively simple settings. For many social species, including humans, understanding neurobiological processes within the confines of these more complex environments is critical because their brains have evolved to perceive and evaluate signals within a social context. Indeed, today's advances in video capture hardware and storage and in algorithms in computer vision and network science make this facilitation with animals possible. Past work has relied on subjective and time-consuming observations from video streams, which suffer from imprecision, low dimensionality, and the limitations of the expert analyst's sensory discriminability. This instrument will not only automate the process of detecting behaviors but also provide an exact numeric characterization in time and space for each individual in the social group. While not explicitly part of the instrument, the quantitative description provided by our system will allow the ability to correlate social context with neural measurements, a task that may only be accomplished when sufficient spatiotemporal precision has been achieved.

The instrument enables research in the behavioral and neural sciences and development of novel algorithms in computer vision and network theory. In the behavioral sciences, the instrumentation allows the generation of network models of social behavior in small groups of animals or humans that can be used to ask questions that can range from how the dynamics of the networks influence sexual selection, reproductive success, and even health messaging to how vocal decision making in individuals gives rise to social dominance hierarchies. In the neural sciences, the precise spatio-temporal information the system would provide can be used to evaluate the neural bases of sensory processing and behavioral decision under precisely defined social contexts. Sensory responses to a given vocal stimulus, for example, can be evaluated by the context in which the animal heard the stimulus and both his and the sender's prior behavioral history in the group. In computer vision, we propose novel approaches for the calibration of multiple cameras "in the wild", the combination of appearance and geometry for the extraction of exact 3D pose and body parts from video, the learning of attentional focus among animals in a group, and the estimation of sound source and the classification of vocalizations. New approaches will be used on hierarchical discovery of behaviors in graphs, the incorporation of interactions beyond the pairwise level with simplicial complices, and a novel theory of graph dynamics for the temporal evolution of social behavior. The instrumentation benefits behavioral and neural scientists. Therefore, the code and algorithms developed will be open-source so that the scientific community can extend them based on the application. The proposed work also impacts computer vision and network science because the fundamental algorithms designed should advance the state of the art. For performance evaluation of other computer vision algorithms, established datasets will be employed.

Modern robots can be seen moving about a variety of terrains and environments, using wheels, legs, and other means, engaging in life-like hopping, jumping, walking, crawling, and running. They execute motions called gaits. An example of a gait is a horse trotting or galloping. Likewise, humans execute walking, running and skipping gaits. Essentially, for either a biological or mechanical systems, a gait is a locomotion pattern that involves large-amplitude body oscillations. Naturally, these motions cause impacts with terrain that jostle on-board perceptual systems and directly influence what the robots actually "see" as they move. For instance, the body motion of a bounding horse-like robot may result in significant occlusions and oscillations in on-board camera systems that confound motion estimation and perceptual feedback.

Focusing on complex mobility robots, this project seeks to better understand the coupling between locomotion and visual perception to improve perceptual feedback for closed-loop motion estimation. The work is organized around two key questions: 1) How should a robot look to move well? 2) How should a robot move to see well? To address the first challenge, the periodic structure of gait-based motions will be leveraged to improve perceptual filtering as the robot carries out fixed (pre-determined) motions. The second half of the project will derive perceptual objectives and a new perceptual gait design framework to guide how high degree-of-freedom, complex mobility robots should move (locomote). The goal is to optimize feedback for closed-loop motion implementation, on-line adaptation, and learning, which are currently difficult or impossible for many complex mobility robots.

Consider two future autonomous system use-cases: (i) a bomb defusing rover sent into an unfamiliar, GPS and communication denied environment (e.g., a cave or mine), tasked with the objective of locating and defusing an improvised explosive device, and (ii) an autonomous racing drone competing in a future autonomous incarnation of the Drone Racing League. Both systems will make decisions based on inputs from a combination of simple, single output sensing devices, such as inertial measurement units, and complex, high dimensional output sensing modalities, such as cameras and LiDAR. This shift from relying only on simple, single output sensing devices to systems that incorporate rich, complex perceptual sensing modalities requires rethinking the design of safety-critical autonomous systems, especially given the inextricable role that machine and deep learning play in the design of modern perceptual sensors. These two motivating examples raise an even more fundamental question however: given the vastly different dynamics, environments, objectives, and safety/risk constraints, should these two systems have perceptual sensors with different properties? Indeed, due to the extremely safety critical nature of the bomb defusing task, an emphasis on robustness, risk aversion, and safety seems necessary. Conversely, the designer of the drone racer may be willing to sacrifice robustness to maximize responsiveness and lower lap-time. This extreme diversity in requirements highlights the need for a principled approach to navigate tradeoffs in this complex design space, which is what this proposal seeks to develop. Existing approaches to designing perception/action pipelines are either modular, which often ignore uncertainty and limit interaction between components, or monolithic and end-to-end, which are difficult to interpret, troubleshoot, and have high sample-complexity.

This project proposes an alternative approach and rethinks the scientific foundations of using machine learning and computer vision to process rich high-dimensional perceptual data for use in safety-critical cyber-physical control applications. Thrusts will develop integration between perception, planning and control that allow for their co-design and co-optimization. Using novel robust learning methods for perceptual representations and predictive models that characterize tradeoffs between robustness (e.g., to lighting & weather changes, rotations) and performance (e.g., responsiveness, discriminativeness), jointly learned perception maps and uncertainty profiles will be abstracted as ``noisy virtual sensors? for use in uncertainty aware perception-based planning & control algorithms with stability, performance, and safety guarantees. These insights will be integrated into novel perception-based model predictive control algorithms, which allow for planning, stability, and safety guarantees through a unifying optimization-based framework acting on rich perceptual data. Experimental validation of the benefits of these methods will be conducted at Penn using photorealistic simulations and physical camera equipped quadcopters, and be used to demonstrate perception-based planning and control algorithms at the extremes of speed/safety tradeoffs. On the educational front, the research outcomes of this proposal will be used to develop a sequence of courses on safe autonomy, safe perception, and learning and control at the University of Pennsylvania. Longer term, the goal of this project is to create a new community of researchers that focus on robust learning for perception-based control. Towards this goal, departmental efforts will be leveraged to increase and diversify the PhD students working on this project.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Robots will perform better at everyday activities when they can quickly combine their sensory data into a model of their environment, just like how humans instinctively use all their senses and knowledge to accomplish daily tasks. Robots, however, must be programmed to create these models that humans do intuitively, effortlessly, and robustly. This robotics project explores a novel algorithmic approach that combines visual and tactile sensory data with a knowledge of physics and a capability to learn that makes robot planning and reasoning more effective, efficient, and adaptable. The project includes the development and testing of research prototypes, preparation of new curriculum, and outreach to high school students and teachers and to the general public.<br/><br/>This project introduces a new data representation, called a Visual Tactile Neural Field (VTNF), that allows robots to combine data from visual and tactile sensors to create a unified model of an object. The VTNF is designed to be used in a closed-loop manner, where a robot may use data from its physical interactions with an object to create or improve a model and may use its current understanding of a model to inform how best to interact with a physical object. Towards this end, the investigators create the mathematical techniques, computational tools, and robot hardware necessary to generate a VTNF model. The investigators also develop techniques to quantify the uncertainty about an object and use this uncertainty to learn search policies that allow robots to generate accurate models as quickly as possible. The VTNF, which allows for the easy addition of new properties about an object, provides a flexible representational foundation for other researchers and practitioners to use to enable robots to learn faster by having a more detailed understanding of both the surrounding environment and their interactions with it.<br/><br/>This project is supported by the cross-directorate Foundational Research program in Robotics and the National Robotics Initiative, jointly managed and funded by the Directorates for Engineering (ENG) and Computer and Information Science and Engineering (CISE).<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.