Nilanjan Sarkar - US grants

9701614 Yuh Our primary research objective is to investigate and develop intelligent control strategies for underwater robotic vehicles (URVs) with manipulator workpackages. The motion of the manipulator, which is attached to the vehicle's main body, affects the motion of the vehicle, whose dynamics are also subject to parameter uncertainties, changes in payload and environment, and nonlinear behavior. It is necessary to develop an intelligent control system for such vehicles to provide: automatic compensation for the errors of the vehicle motion, due to manipulator motion and underwater currents. coordinated control of both vehicle and manipulator using deliberate vehicle motion; such motion will help task performance and add the degrees of freedom of the vehicle to those of the manipulator workpackage; learning and adaptation capabilities to parameter uncertainties and changes in the environment; and close coupling of control knowledge transfer between the high-level and low-level subsystems of the overall URV control system. The study: extends a theoretical modeling of the underwater robotic vehicle and manipulator, including dynamic interaction between links and vehicle main body, kinematic redundancy, contact stability of the system in contact with the environment, and fixtureless manipulation; develops a theoretical framework for intelligent control strategies based on fuzzy neural network (FNN) control using fuzzy clustering techniques and on-line reinforcement learning technique; and includes experimental demonstration of the proposed approach on the University of Hawaii's underwater robotic vehicle, ODIN (Omni-Directional Intelligent Navigator). ***

The project will investigate ways of making robots more user-friendly via an innovative approach whereby the robot will be able to recognized the affecting state of the interacting human and modify it's (robot's) own actions to make the human feel comfortable to work with the robot. Wearable biofeedback sensors will be used to measure a variety of physiological indices to infer the underlying affective human states.

Proposal #: CNS 08-21640
PI(s): Bodenheimer, Robert E.
Adams, Julie A.; McNamara, Timothy P.; Rieser, John J.; Sarkar, Nilanjan
Institution: Vanderbilt University
Nashville, TN 37235-7749
Title: MRI/Acq.: Instruments for Interaction, Learning, and Perception in Virtual Environments
Project Proposed:
This project, acquiring a high-fidelity instrument designed to facilitate and assess perception, interaction, and learning in immersive environments, pursues an ambitious research agenda dealing with people, their interactions with virtual environments, and the design factors underlying successful environments. The work aims to build a program to develop a better understanding of the cognitive capabilities of humans in immersive virtual environments, to inform the design process of such environments and to understand how humans reason about space. The instrument will be shared among diverse and interdisciplinary groups collaborating in the area of virtual environments, including Computer Science and Engineering (graphics, animation, artificial intelligence, human factors, robotic, etc.) and the Psychological Science (cognitive psychology, child development, rehabilitation engineering, brain sciences, etc.) The component parts of the instrument (comprising optical motion capture equipment, a head-mounted display with binocular eye-tracking, and high-performance wireless data gloves) allow the measurement, tracking, rendering, and animation of subjects in virtual environments (from their overall position, to their posture, to the actions of their hands and fingers) coupled with the measurement of their gaze. The project ranges from low-level research in how people experience virtual environments to user evaluations involving high-level interface and simulation design. Children with autism will also be studied.
Broader Impacts: This project improves the quality of learning in virtual environments, reducing the time and cost of authoring and overcoming likely impediments to their widespread use. The instrument enables courses in robotics currently infeasible with real robots and provides experience for students. The work builds a scientific program to develop a better understanding of the cognitive capabilities of humans in immersive virtual environments and may be applied to understanding the development of children?s abilities to reason about space and to coordinate perceptual-motor skills as they develop. Moreover, it may help to treat autism spectral disorder.

PI: Sarkar N. & Warren, Z.
Proposal Number: 0967170
This interdisciplinary project proposes fundamental, transformative technology improvements to assist children with Autism Spectrum Disorder (ASD). The objective is to develop a novel affect and attention sensitive virtual reality -based assistive technology for ASD intervention. Social communication and social information processing are thought to represent core domains of impairment in children with ASD. The proposed research will develop technologies capable of targeting individual deficit by flexibly and adaptively responding to subtle affective and attentive changes in individuals with ASD during social paradigms and create VR-based ASD intervention technology that can present itself as a realistic and powerful intervention platform.
The individual, familial, and societal impact associated with ASD is enormous. Therefore, an important direction for research on ASD is the identification and development of technology that can make application of effective intensive treatment more accessible and cost effective. To address this need, this project will combine VR-based technology with affective computing using physiological signals and attention inference through eye gaze measurement to develop a new paradigm for autism intervention that will appreciably transform the ability to understand and tailor interventions to the specific vulnerabilities of children with ASD. The project will develop an assistive technology that will result in an affect and attention sensitive system for detecting, adaptively responding to, and optimizing levels of social interaction for children with ASD during VR interactions. The system is scalable and adaptable to a broad range of intervention strategies, providing flexibility in design and implementation.

Intellectual Merit: The proposed research advances the design and authoring of adaptive virtual environments for use with a challenged population of children. It has the potential to significantly contribute new computational methods for affective computing, particularly affective computing mediated by physiologic signal processing. Primarily, the project will develop new framework to design virtual environments for social interaction that intelligently combine both affective and attentive information into adaptive and controllable response systems. The proposed activity also represents a new technology that will fundamentally advance the engineering knowledge of human-computer interactions as well as the understanding of the mechanisms that underlie the presumed core social impairments, and associated interventions, with ASD.

Broader Impacts: Emerging research suggests as many as 1 in 150 children are diagnosed with an autism spectrum disorder and ASD related care costs the nation over $35 billion annually. The low-cost and highly deployable technologies developed in this proposal could have significant impact on this population. They may also create a completely new technological intervention methodology for children with ASD. This research may further a technology that can enable all core components of effective intervention at only a fraction of the cost of typical intervention programs, while at the same time increasing the ability of the intervention provider to systematically control and promote intervention related skills targeting individual deficit. The educational activities will train and mentor undergraduate and graduate students in the proposed research, and develop one module in a one-credit seminar course targeted to engineering freshman class. The outreach activities will include offering research opportunities to high school students, especially among groups currently underrepresented in STEM (science, technology, engineering, and mathematics) fields, and providing high school teachers with research experience during summer in the design and research evaluation of advanced technologies for autism intervention. Outreach objectives will also involve usability testing to elicit feedback from behavioral professionals and children with ASD. This project represents an opportunity to leverage a unique collaboration that brings cutting-edge engineering, psychology, and clinical knowledge to the development of a pragmatic and efficacious assistive intervention technology addressing core symptoms of ASD.

DESCRIPTION (provided by applicant): It is estimated that 1 in 110 children in the United States are affected by Autism Spectrum Disorders (ASD). The identification and effective treatment of ASD is often characterized as a public health emergency incurring a $35-90 billion annual cost. Traditional interventions designed to address higher level social and adaptive impairments have been demonstrated to be minimally effective for school-aged children and adolescents with ASD and are thought to be a result of a failure of traditional methodologies to systematically match intervention strategies to specific skill deficits within and across naturalistic settings in appropriately intensive dosages. In this grant proposal, in an attempt to address the above concerns, a research plan is developed to apply intelligent adaptive technology as a Virtual Reality (VR) based intervention modality for treatment of a) core social deficits and b) applied adaptive behavioral skills for children and adolescents with ASD. The technology will be applied with capacities for controllable levels of difficulty that can be adapted with the VR environment into both social and adaptive behavioral intervention scenarios in real-time based on predictions of behavioral engagement of participants while engaged with the tasks. Behavioral engagement is operationalized by affective and attentive states and as such, the intervention technology will be sensitive to how variations in an individual child's engagement predict task performance. Rule-governed adaptation will be incorporated into VR interactions and examined explicitly in terms of how such modifications improve performance. Simply, this technology will be applied to automatically adjust task characteristics based on an individual's unique profile in hopes that task performance within core domains of impairment will be greatly bolstered by application of individually modulated and scaffolded reinforcement modalities. The specific aims of this research are: 1) to refine the intelligent adaptive response technology that the authors have already developed in their pilot work, and design both adaptive behavior and social tasks in a VR environment;2) to develop physiology-based individualized affective models and attention inference mechanisms from eye gaze information, and to design a rule-based supervisor to allow adaptive reinforcement strategies sensitive to behavioral engagement;and 3) to apply and examine the efficacy of prediction models and the adaptive technology in children with ASD relative to improving specific learning related to VR-based and realistic social and adaptive tasks. The success of such an adaptive response technology in ASD intervention will pave the way for systematic exploration and application of adaptive intervention paradigms aimed at matching individual deficit with targeted intervention. PUBLIC HEALTH RELEVANCE: The project will apply and examine the efficacy of a novel adaptive virtual reality (VR) technology as a potential intervention tool for children and adolescents with Autism Spectrum Disorders (ASD). The proposed technology is designed as an 'intelligent'system that automatically adjusts intervention tasks based on physiological data, information about where the child is looking (e.g., eye tracking), and how well the child is performing, in order to enhance performance on tasks that have real-world application. It is believed that the successful application of this new technology has the potential to usher in a new era of personalized, targeted computer-based ASD intervention capable of addressing the core deficits of the disorder in a manner that is more efficacious and accessible.

This proposal requests travel funds from NSF to assist 50 US students to participate in ICRA 2012, which will be held in St. Paul, MN, May 14-18, 2012. The purpose of this Group Travel Grant Proposal is to make it possible for US students and postdocs to attend the conference, present their work, and forge connections with colleagues from around the world.

This year marks the 50th anniversary of robotics. In addition to the regular conference, ICRA 2012 will feature interactive presentations, robot demonstrations, thematic plenary sessions on design, bio-robotics, and intelligent transportation, and special-topic symposia celebrating the conference theme of "Innovation for Tomorrow's Needs." As part of this award, students will also have the opportunity to participate in special events and learn more about the field of robotics. Travel funding will be in the form of partial airfare reimbursement and full reimbursement for student registration to approximately 50 U.S. students who plan to present at least one paper at the Conference.

Broader Impacts: Most major international robotics conferences were outside of North America this past year, so demand should be very high from American students and professors to attend this major robotics event. The benefit to student learning and mentorship by encouraging attendance of students by direct monetary means brings large dividends in student confidence, knowledge and expertise.

PI: Sarkar, Nilanjan and Warren, Zachary
Proposal Number: 1264462

Project Summary: A novel and transformative robotic intervention technology, called ARIA
(Adaptive Robot-mediated Intervention Architecture), with the potential to accelerate social communication skill development for young children with autism spectrum disorders (ASD) is proposed in this research. ARIA will fluidly integrate a humanoid robot, multiple spatially distributed network of cameras, an array of display monitors, as well as a complex but efficient computational face, gaze and gesture detection methodology in order to create a highly flexible and adaptive intelligent environment to potentially advance early joint attention and imitation related skills for young children with ASD. Application of this system will be examined across two user studies with well-defined samples of young children with ASD to provide specific answers and direction to important questions of generalization and potential impact of robotic intervention.

Intellectual Merit: The proposed research advances the design and development of intelligent adaptive robotic platforms to offer a potentially transformative intervention application for young children with ASD. The specific technological innovation proposed here has the potential to significantly contribute to new non-invasive and closed-loop human-robot interaction learning paradigms with potential broad extension to individuals with a vast array of neurodevelopmental conditions and limiting sensory vulnerabilities across the lifespan. From the perspective of the science and technology of robotics, the project will contribute towards the design and development of smart environments for learning, intelligent system architecture for adaptive robotics as well as affective computing and control of dynamic human-robot interaction. In particular, it has the potential to significantly contribute towards developing novel efficient applications of computational methods for affective computing, particularly affective computing mediated by non-invasive gaze and attention processing. It will also contribute towards closed loop gesture-based human-robot interaction by developing new methodologies for gesture recognition and adaptive response from the robot. The project will develop a framework and tools to design adaptive environments for enhanced robotic and embodied social interaction that intelligently and fluidly integrates real-time behavioral indices of attentive and gesture information into flexible and controllable response systems. In short, the proposed activity represents a system has the potential to fundamentally advance the engineering knowledge of intelligent human-robotic interaction. This paradigm may also potently impact our understanding of the science of ASD intervention itself. The embedded user studies will test the potential efficacy of robotic intervention on the earliest core symptoms of ASD.

Broader Impacts: With the most recent Centers for Disease Control and Prevention (CDC) prevalence estimates for children with ASD at 1 in 88, effective early identification and treatment is often characterized as a public health emergency. The costs of ASD are thought to be enormous across the lifespan, with recent individual incremental lifetime cost projections exceeding $3.2 million and national cost over $35 billion annually. The proposed research explicitly focuses on realizing robotic intervention technologies with potential for improving early ASD related impairments and could have significant beneficial impact on this population. This research may further a technology that can enable all core components of effective intervention at only a fraction of the cost of typical intervention programs, while at the same time increasing the ability of the intervention provider to systematically control and promote intervention related skills targeting individual deficit. The educational activities will train and mentor undergraduate and graduate students in the proposed research, and bring research into classroom through several courses. The outreach activities will include offering research opportunities to high school students, especially among groups currently underrepresented in STEM (science, technology, engineering, and mathematics) fields, and providing high school teachers with research experience during summer. The project offers a strong community connection through formal dissemination to ASD family, clinical, and scientific communities.

DESCRIPTION (provided by applicant): With the most recent Centers for Disease Control and Prevention (CDC) prevalence estimates for children with ASD at 1 in 88, effective early identification and treatment is often characterized as a public health emergency. Given the present limits of intervention science and the enormous costs of the disorder across the lifespan, there is an urgent need for more efficacious treatments that can enhance intervention outcomes. A growing number of studies have investigated the application of advanced interactive technologies to ASD intervention, including computer technology, virtual reality (VR) environments, and more recently, robotic systems. The primary goal of the proposed research is the design and preliminary testing of a robust robotic intervention platform and environment specifically designed to accelerate improvements in early areas of core ASD deficit. In particular, it focuses on developing a robotic intervention system (ARIA: Adaptive Robot-mediated Intervention Architecture) capable of seamlessly integrating real-time, non-invasive detection of gaze with an intelligent environment endowed with the ability to autonomously alter system function based on child performance to impact early joint attention skills, thought to be fundamental social communication building blocks central to etiology and treatment of ASD. While it is unlikely that technological advances will take the place of traditional intervention paradigms, we hypothesize that adaptive robotic systems may hold great value as potential accelerant technologies that enhance learning intervention modalities in potent ways, particularly for children who show powerful differences in their contingent responses to non-biological actions and events relative to interactions with social partners at very early ages. In the proposed research, we will investigate the realistic potential of robotic technology for young children with ASD via explicit design and tests of such a system to improve performance within the domain of early joint attention skills. Thus, the specific aims and milestones of the R21 phase of our project focus on: 1) Achieving integration of noncontact gaze technology into the system, 2) realizing autonomous closed-loop operation of our system architecture, and 3) demonstrating adequate initial user functioning while interacting with the autonomous system. Subsequent to attaining these milestones and demonstrating initial system capacity, the R33 phase of our project will assess the performance of the robot-mediated architecture during a pilot intervention experiment. This trial will specifically evaluate system capacities as related t very young children with ASD as well as feasibility data related to recruitment and retention in the protocol, user tolerance, and patient satisfaction. This pilot trial will also assess within system improvement in joint attention skills, as well as a methodology for assessing potential generalization of such skills in a larger trial.

? DESCRIPTION (provided by applicant): Social impairments are core features of schizophrenia that lead to poor outcome. Social skills and competence improve quality of life and protect against stress-related exacerbation of symptoms, while supporting resilience, interpersonal interactions, and social affiliation. To improve outcome, we must remediate social deficits. Existing psychosocial interventions are moderately effective but the effort-intensive nature (high burden), low adherence, and weak transfer of skills to everyday life present significant hurdles toward recovery. Thus, there is a dire need to develop effective, engaging and low-burden social interventions for people with schizophrenia that will result in better compliance rates and functional outcome. We will test the effectiveness of a novel adaptive virtual reality (VR) intervention in improving targeted social cognitive function (social attention as indexed by eye scanning patterns) in individuals with schizophrenia. VR technology offers a flexible alternative to conventional therapies, with several advantages, including a simplified and low-stress social interaction environment with targeted opportunities to simulate, exercise and reinforce basic elements of social skills in a very wide range of realistic scenarios, and to repea exposure to naturalistic situations from multiple angles. Our desktop VR `game' is designed as an `intelligent' system that adaptively adjusts the difficulty of social training tasks based on participant's physiological, eye tracking and performance data in real time. Such dynamic feedback-based, `closed-loop' VR supports and enhances training because it adjusts and personalizes the learning environment in real time for each participant so that he/she always learns at an optimal arousal and attentional state. Furthermore, the VR environment can potentially simulate any social scenarios, which allows the participants to exercise social skills n a wide variety of situations. Such simulation exercises can help generalize learned skills to everyday life. The R21 phase will aim to implement the social intervention VR task, test its efficacy on improving social attention (target) in schizophrenia, and determine an optimal `dose'. We hypothesize that the VR training will engage social attention and improving social attention will lead to better social outcome. The R33 phase will test the adaptive social VR game against an active control condition in a pilot randomized controlled trial to evaluate the relative efficac of the social VR on enhancing social attention and associated neural circuitry. We will also examine social outcome. If this initial work is successful, our long-term goals are to develop VR social skills training modules that are personalized, accessible, and portable so that social remediation can become an integral part of one's daily life.

PROJECT SUMMARY / ABSTRACT The aging population with its concomitant medical conditions, physical and cognitive impairments, at a time of strained health care resources, establishes the urgent need to design and implement technologies that enhance or maintain physical and cognitive function and improve quality of life among older adults. Robotic technologies in the form of socially assistive robots (SAR), with capabilities for autonomously detecting and meaningfully responding to older adults' engagement and behavior, have the potential for addressing physical, cognitive, and/or social conditions. The field of SAR is still in early development. We propose to introduce SAR for not only one person - one robot interaction, but also to utilize the robot to foster interactions between two people. The specific aims of this two-year pilot study are: 1) design and validation of an autonomous robotic architecture for older adults with and without cognitive impairment or apathy and 2) assess the feasibility, acceptability and tolerance of the robot-mediated intervention. To accomplish Aim 1, we will refine our current robotic architecture ARIA (Adaptive Robot-Mediated Intervention Architecture) to create several new engaging cognitive, physical and social tasks suitable for older adults and extend the ARIA's capability to include more than one person in the robot-mediated activities to foster human-to-human interaction. We will conduct laboratory experiments involving 30 older adults (10 without cognitive impairment, 10 with mild cognitive impairment, and 10 with mild to moderate dementia) in which we will test the individual capabilities as well as integrated functioning of the robotic system. In Aim 2, we will conduct two pilot feasibility studies at an independent living, assisted living and dementia care senior center. In Pilot Study 1, we will enroll 12 adults, 4 with no cognitive impairment, 4 with mild cognitive impairment, and 4 with mild to moderate dementia, to participate three times weekly for 4 weeks in a series of physical, cognitive and social activities. Pilot Study 2 will include 6 pairs of older adults (2 with no cognitive impairment, 2 pairs with MCI, and 2 pairs with dementia) to attend three sessions a week for 4 weeks. This will allow us to expand the robotic capabilities to monitor more than one person and to encourage social engagement between the older adults. All study procedures will remain the same as in Pilot Study 1. Aims 1 and 2 experiments will be videotaped. Older adults' reactions to the robotic interactions will be gathered by survey, and observation using observer-rated tools. In addition to informing future clinical trials of SAR effectiveness, the information from the proposed study will contribute to the long-term goals supporting the development of robotic strategies to enhance physical function, cognition and socialization of older adults in the community setting.

PROJECT SUMMARY Autism Spectrum Disorder (ASD) is a common, costly, heterogeneous neurodevelopmental disorder comprised of impairments related to social communication and interaction as well as restricted interests and repetitive behaviors. Social communication deficits are core to ASD with challenges in this domain often translating into serious lifespan impairment for most individuals. While certain behavioral and pharmacological interventions have some benefits to many children with ASD, such interventions often require significant time and effort- intensive burdens for implementation, suffer from low-adherence and limited availability in community settings, and ultimately have demonstrated weak transfer of skills to real world settings of meaning. While advances in technological capacity have the potential to yield intervention platforms of meaning, current technological systems have focused on simple discrete social skill development or required confederate or remote operation, limits which have impeded progress toward successful mainstream clinical use and benefit. In order to address these issues we propose to develop and test the feasibility, tolerability, and potential for clinical benefit of a novel technological paradigm for meaningful social communication that combines two innovative technologies: (1) collaborative virtual environments (CVE) and (2) artificially intelligent (AI) agent-based interactions to create a virtual intelligent system (VIS) for ASD intervention. We will design a VIS in which children with ASD will participate in a series of engaging game-like interactive tasks where they collaborate with peers and with artificially intelligent partners (AIP). We hypothesize that the proposed paradigm will preserve all advantages of a traditional CVE for meaningful interaction, but will allow unrestricted verbal communication with real partners and eliminate the need for manual coding by providing consistent unbiased measurements of meaningful aspects of social interaction. The proposed system will autonomously index both traditional markers of performance as well as social communicative behaviors contributing to task performance and attempt to modify tasks to meaningfully alter subsequent interactions. It is hypothesized that this measurement and intervention strategy, deployed across engaging, dynamic interactions highly relevant to real-world social interaction, will readily yield quantifiable metrics of important components of social communication that are potentially sensitive to change and responsive to intervention strategies. The specific aims of this project are: 1) Development of a virtual intelligent system and embedded artificially intelligent partners capable of dynamic meaningful interaction during collaborative game activities, and 2) Validation of feasibility and tolerability of the system with children and adolescents with ASD. The proposed technology is simple, cost-effective and is easy to translate for mainstream use.

The landscape of jobs and work is changing rapidly, driven by the development of new technologies. Intelligent, automated machines and services are a growing part of jobs and the workplace. New technologies are enabling new forms of learning, skills assessments, and job training. The potential benefits of these technologies include increased productivity and satisfaction, and more job opportunities. The workshop supported by this award aims to harness these innovations to enhance the science, technology, engineering, and mathematics (STEM) job opportunities and workforce engagement of individuals with autism spectrum disorder (ASD), and related developmental disabilities. The workshop will promote the convergence of psychology, data science, computer science, engineering, learning science, special education, organizational behavior, and business to define key challenges and research imperatives at the nexus of humans, technology, and work. This convergence workshop will employ deep integration of knowledge, theories, methods, and data from multiple fields to form new and expanded frameworks for addressing scientific and societal challenges and opportunities. The results of the workshop will include the identification and sharing of new research directions and tools to enhance STEM workforce engagement of individuals with ASD and related developmental disabilities. This convergence workshop addresses the future of work at the human-technology frontier.

The workshop will explore tools and approaches to enhance retention, engagement, and productivity in STEM jobs, and specifically to harness unique capabilities and accommodate for individual needs of individuals with ASD. The workshop will develop a convergence research agenda around four topics, including 1) human-technology partnerships to support success in K-12 STEM education, 2) tools for characterizing individual capabilities and affinities and mapping these to STEM workforce needs, 3) artificial-intelligence and visual-cognition tools for human interaction with data, and 4) technologies to accommodate for unique needs and capabilites in the workplace. These topics will integrate previously disparate disciplines and research approaches, with speakers encompassing a wide range of subject matter expertise; from engineers and technologists who are developing human-technology interfaces and devices, to psychologists who are harnessing human-technology partnerships to better understand unique human capabilities for STEM, to computer scientists who are studying and developing novel data-visualization approaches patterned on autistic visual thinking, to organizational scientists developing innovative employment models for the creation of STEM sector employment spaces and technologies that leverage and support autistic individuals in the workforce. The conclusions and recommendations from the workshop will be disseminated via a white paper, and will be used to design a research agenda to help leverage human-technology advances to maximize workforce opportunities and productivity.

Neurodiversity is an emerging concept through which certain neurological differences - Autism, Attention Deficit Hyperactivity Disorder, Dyslexia, and others - are considered a natural part of human neurocognitive variation, associated not only with impairments but also with unique strengths. Indeed, many neurodiverse people have capabilities that are in high demand across many sectors, yet their potential remains vastly underutilized. This National Science Foundation Research Traineeship (NRT) award to Vanderbilt University will address this potential by training graduate students in a new interdisciplinary field of Neurodiversity Inspired Science and Engineering (NISE), which links human-technology frontiers (HTF) research and education across STEM disciplines through a cohesive focus on autism. The project anticipates providing a unique and comprehensive training opportunity for one hundred fifty (150) MS and PhD students, including forty-five (45) funded trainees, from computer science, mechanical engineering, data science, psychology, organizational science, and neuroscience. Students will engage in research that has as its goals: (i) understanding the unique capabilities associated with autism and learning to match these capabilities to 21st-century workforce needs, (ii) prototyping assistive technologies to enable employment and workplace success, and (iii) exploring organizational practices that help leverage the talents of autistic individuals and enhance organizational innovation.

The NISE NRT project seeks to train a new type of engineer and scientist, one who can devise innovations that support workforce engagement of individuals with autism and/or that are inspired by autistic capabilities. Building on the strengths of Vanderbilt's new Frist Center for Autism & Innovation, this NRT project will engage trainees in the development, deployment, and commercialization of HTF approaches and devices, providing broadly applicable skills in artificial intelligence, data science, robotics, virtual reality, and inclusive design. Collaboration with practitioners in clinical psychology, special education, and business, will ensure relevance of trainees' projects to the clinical, educational, and/or commercial domains. Trainees will participate in the Vanderbilt NSF I-Corps program as well as invention disclosure and patents. Research projects will also impact K-12 students and teachers in the communities where NRT trainees conduct their work. A central part of the program's plan to recruit, mentor, and advance women, underrepresented minorities, and persons with disabilities is the Fisk-Vanderbilt Masters-to-PhD Bridge Program, a national exemplar in STEM graduate diversity. Trainees will undertake, in addition to their regular graduate program requirements, a common core of three new NISE courses, summer school, workshops, and internships, culminating in a graduate certificate in NISE.

The NSF Research Traineeship (NRT) Program is designed to encourage the development and implementation of bold, new potentially transformative models for STEM graduate education training. The program is dedicated to effective training of STEM graduate students in high priority interdisciplinary or convergent research areas through comprehensive traineeship models that are innovative, evidence-based, and aligned with changing workforce and research needs.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

The NSF Convergence Accelerator supports team-based, multidisciplinary efforts that address challenges of national importance and show potential for deliverables in the near future.

The broader impact/potential benefit of this Convergence Accelerator Phase I project is to dramatically increase the engagement of individuals with autism spectrum disorders (ASD) in the workforce. While approximately two-thirds of 2.5 million adults with ASD in the US have average intelligence, more than 50% of them remain unemployed or underemployed. Many individuals with ASD have unique capabilities that are in high demand across many job sectors; optimizing workforce engagement for these individuals holds the potential to transform great societal cost into great societal value. This project utilizes convergent expertise in Artificial Intelligence (AI), virtual reality, robotics, together with expertise in neuroscience, and behavioral and organizational psychology, to develop intelligent tools and systems to facilitate employment of individuals with ASD that have high potential for rapid commercialization and deployment. Specifically, the proposed research will develop intelligent training systems for interviews and other job relevant social interaction skills for individuals with ASD, and skill assessment tools for employers to enhance recruitment and retention. The entire project is based on the foundational idea that many people with ASD have the potential to participate in the workforce in ways that contribute to society while also sustaining personal success and well-being.

This Convergence Accelerator Phase I project presents a comprehensive research plan to create new AI tools, systems, and predictive models, inclusive of employer and stakeholder input, to connect people with ASD to employers via embedded, technologically based, research-informed supports for individuals and organizations alike. For people with ASD, inherent challenges related to social initiation, engagement, and communication impede their adaptive independence, including finding and keeping jobs. This issue has become a top priority of the National Institutes of Health Interagency Autism Coordinating Committee. The project involves six convergent, mutually reinforcing research components: (1) a pipeline to employment for people with ASD; (2) an affect-sensitive, closed-loop virtual reality interview training platform to assess and intervene on skill deficits while also gathering aggregate data relevant to employer training; (3) opportunities for home assessment and practice outside of traditional educational settings through the use of AI-agent mediated collaborative virtual environments and (4) closed-loop interactive socially assistive robots; (5) novel computer vision and wearable computing tools for assessment of real-world generalization of skills learned within VR and robotic systems; and (6) customizable, innovative assessment tools using data-driven visual AI to identify strengths, talents, and job-relevant skills.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Project Summary/Abstract A large proportion of older adults residing in long term care (LTC) settings, such as nursing homes and assisted living facilities, suffer from Mild Cognitive Impairment (MCI), Alzheimer's disease (AD) and AD-related dementias (ADRD). Apathy is common in persons with AD and ADRD with prevalence rates up to 72%. It is associated with further cognitive decline, functional deficits, reduced quality of life, social isolation, and increased mortality. Apathy imposes significant burden on LTC staff and negatively impact quality of care, staff satisfaction and turnover. Since few pharmacologic options exist, a major strategy is to foster older adults' engagement in social, physical and cognitive activities, primarily those that are multimodal in nature. However, these interventions often require significant personnel time and resources, a major concern given the current nursing shortage and high turnover among LTC nursing personnel. The Centers of Medicare and Medicaid Services mandates LTC facilities to provide meaningful engaging activities for residents, which can be resource intensive and are difficult for many US LTC settings. In order to partially mitigate some of these issues, intervention based on an intelligent socially assistive robot (SAR) based architecture, called ARIA (Adaptive Robot-mediated Intervention Architecture), developed under a R21 grant, that can adaptively and dynamically interact with older adults with MCI, AD and ADRD who reside in LTC settings, is proposed in this grant application. This interdisciplinary proposal is directly aligned with the NIA goals of understanding and developing effective interventions using smart technology to reduce the burden of age-related diseases and address the special caregiver needs of those caring for persons with dementia (PWD). This multi-phase, multi-site, mixed methods clinical trial will systematically examine responsiveness and engagement among persons with MCI or dementia to two types of SARs (humanoid and animal), its effect on cognitive, physical and social function as well as the impact of SARs on informal and formal caregivers with a goal towards future scalability and sustainability. The specific aims of the proposed research are: Aim 1: To improve our novel social robotic interaction architecture through additional software development to a) make it more versatile in combining multimodal quantitative data capturing engagement, b) more robust such that non-experts can operate it and create new tasks by concatenating task primitives, and c) expand tasks to address varying degrees of cognitive and physical impairments of older adults (Months 1-18). Aim 2: To compare the effect of usual care (UC) group to UC+ARIA group on reducing apathy among older adults with mild cognitive impairment (MCI), mild dementia, or moderate dementia (Months 19-48). Aim 3: To identify barriers and facilitators to SAR implementation across sites to address future scalability and sustainability (Months 18-42). This study will contribute to the development of improved intelligent technology as an effective approach to engage older PWD with the long term goal of enhancing function and quality of life.

The NSF Convergence Accelerator supports use-inspired, team-based, multidisciplinary efforts that address challenges of national importance and will produce deliverables of value to society in the near future.

Neurodiversity is an emerging concept through which certain neurological differences?Autism, Attention Deficit Hyperactivity Disorder, Dyslexia, and others?are considered a natural part of human neurocognitive variation, associated not only with impairments but also with unique strengths. Indeed, many neurodiverse people have capabilities that are in high demand across many sectors. Yet, while some 70,000 Americans with autism enter adulthood every year, currently 85% of them will be unemployed or underemployed relative to their skill levels, representing a cost to the United States of $175 billion annually. Thus, optimizing workforce engagement for individuals with autism holds the potential to transform great cost into great value. This National Science Foundation Convergence Accelerator (C-Accel) award to Vanderbilt University will address this grand challenge by bringing together cutting-edge Artificial Intelligence (AI) innovations with transdisciplinary expertise?spanning engineering and computer science to organizational psychology, clinical translation, and implementation science?to create a suite of commercially viable technologies that integrate AI within virtual environments, robotic systems, human-human interactions, and novel assessment tools. These technologies will be created using input from stakeholders, including employers of individuals with autism, companies that develop technological products to help employment, state vocational and rehabilitation services that provide job training, and advocacy groups that provide guidance regarding community needs. The technologies will be transitioned to practice through deployment with private- and public-sector partners, together with analysis using implementation science to ensure long-term sustainability and the broadest impact.

This C-Accel Phase II program will advance the scientific and technological methodologies of the projects initiated in Phase I that are designed to create a pipeline to employment for people with autism. Specifically, the suite of tools to be developed include: (1) Visual and Cognitive AI Tools to Assess Autistic Talent; (2) Virtual Reality (VR)-based Simulator for Improving Job-Interview Skills; (3) Collaborative Virtual Environments with Embedded Intelligent Agent for Social Interaction Assessment and Support; (4) Social Robotic System to Assess and Train Tolerance to Interruption; and (5) Computer Vision Tools to Measure and Improve Non-verbal Communication. Across these projects, we will make fundamental scientific and technological advancements in: (i) data-driven visual AI for innovative assessment tools to identify strengths, talents, and job-relevant skills, as well as employer-identified work needs; (ii) novel VR-based platform for job interview training that utilizes real-time closed-loop multimodal affective computing for stress and attention recognition; (iii) a collaborative virtual environment that create new skill estimation algorithms and a peer-based learning paradigm mediated by an AI agent; (iv) a home-based skill assessment and training systems using socially assistive robotics; and (v) novel computer-vision and deep learning methods and algorithms to assess real-world generalization of nonverbal social communication. The project?s intellectual property plan includes advancing each of these technologies from prototype to minimum viable product (MVP) stage and into commercial use through licensing agreements within the two-year project period. Through Vanderbilt University?s Frist Center for Autism & Innovation, graduate students and neurodiverse interns will participate in all aspects of the C-Accel research and development efforts.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Adults with neurodevelopmental disabilities (e.g., autism spectrum disorder; ASD) and/or motor impairments (e.g., multiple sclerosis; MS) have the lowest rates of any type of employment in the US (unemployment rate of 63-68%). The potential economic benefits of improving employment outcomes for these individuals are enormous. For example, the current economic opportunity cost to the US economy of unemployed individuals with ASD is estimated at $90 billion per year, to say nothing of the immeasurable human costs; for individuals with MS, the estimated loss to the economy is approximately $25 billion per year. This National Science Foundation Engineering Research Center (ERC) Planning Grant award to Vanderbilt University will address this potential by creating the capacity for a future NSF ERC for the Employment of Persons with Disabilities through Inclusion Engineering (EDIE). EDIE would seek to create engineered systems to lower barriers to employment for neurodiverse and motor-impaired individuals, such that they can more readily maintain or pursue meaningful employment. To be fully ready for ERC scale, we need to more fully develop a number of critical elements through this planning grant, specifically: (1) An ERC work plan guided by a 5-year timeline for deliverables that emphasizes (a) stakeholder engagement and assessment of user needs, (b) pilot testing of technologies with and through partner organizations, (c) development of technologies to minimum viable product (MVP) stage, (d) testing through large-scale deployment, and (e) sustainability through commercialization of the technologies. (2) Commercialization strategies (e.g., start-ups and licensing agreements) that would create a self-sustained innovation ecosystem. (3) A plan to gradually expand the Center’s scope to include a wider range of both technologies and disabilities, consistent with the mission of creating a more inclusive workforce.

The proposed vision of the planned EDIE ERC is to create intelligent, adaptive technologies that enable employment and workplace success for individuals with neurodevelopmental and/or motor disabilities, increasing human productivity and economic impact, and leading to a more fully inclusive workforce and society. Guiding these Engineered Systems would be the enabling idea of Inclusion Engineering, an emerging research paradigm representing the convergence of engineering with implementation science, labor and economic policy, and commercial innovation, toward enabling full societal inclusion of all individuals. Multiple technological innovations will need to be pursued in concert toward a truly systems-level solution for full inclusion of motor-impaired and neurodiverse individuals for the future of work. Thus, we envision that surrounding and supporting EDIE’s core research thrusts will be an innovation ecosystem that includes commercialization efforts coupled to key stakeholder needs and feedback, a culture of diversity and inclusion with regards both to Center personnel and end-users, and a multi-pronged effort toward developing a future engineering workforce that is trained, equipped, and inspired to advance the future of Inclusion Engineering. The preliminary convergent research thrusts that we have identified are: (1) Workplace disability inclusion and employment nondiscrimination policy to ensure these individual technologies and the resulting engineering systems are based on Participatory Action Research (PAR) to have real-world uptake and societal impact; (2) Social Human-Machine Interaction (sHMI) to support neurodiverse individuals; and (3) Physical Human-Machine Interaction (pHMI) to support motor-impaired individuals. The second and third thrusts map directly into engineered systems, while the first guides them and also addresses issues that are not directly addressable via technological solutions.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Children with Intellectual and Developmental Disabilities (IDD) are at increased risk of showing “problem behavior” that place them at risk of getting hurt, removed from the classroom, or hospitalized. Approximately 1 in 6 children and adolescents in the United States are diagnosed with IDD and half of them experience some form of problem behavior. Therapists trained in Applied Behavior Analysis, or ABA, can help determine why problem behavior happens and how to prevent it. These therapists watch children, try to evoke problem behaviors by changing a child’s environment, then try things that might change behavior, and see if the behavioral data changes. Because problem behavior can be triggered during this process, this strategy sometimes put them or their patient at risk. It also takes a lot of time. Wearable technology and advanced computational strategies could help increase the safety and helpfulness of strategies to prevent problem behavior. Specifically, small sensors worn in clothing or on the wrist could provide data about a child’s body, or “physiological responses,” like heart rate or sweat. Machine learning can then be used to determine what combination of body signals imply a problem behavior is about to happen. This project has two stages. In the first stage, the team will design new sensors that detect biological signals such as sweating, motion, and heart rate. The team will then measure how well these sensors work. This includes asking people with IDD what they think about the sensors. Based on that input, the team will change the sensors and then use them in a larger study. The goal is to test whether the system can predict problem behavior, how well it works when used in the real-world with real therapists, and what users think about the system. Results of this study will help researchers and practitioners understand if this kind of wearable technology is helpful and acceptable as part of supporting people with problem behavior and IDD.


This project proposes to integrate transdisciplinary expertise in cutting-edge wearable sensing, affective computing, machine learning, and behavioral and clinical science to enhance and transform existing models of behavioral intervention for problem behaviors in children and adolescents with IDD. Problem behaviors, including self-injury, aggression, property destruction, and wandering not only can cause serious injury or death, but also interfere with the ability to participate in school, home, and other community settings. In the context of problem behavior and IDD, this project will fundamentally advance the scientific and the technological methodologies of multimodal wearable sensing-based design of predictive machine learning models. The two research thrusts are: (1) Design of multimodal sensor framework; and 2) Real-time precursor prediction. Across these thrusts, the project will make fundamental scientific and technological advancements in: (i) A low-power, open-access, user-centric wearable sensor framework that can sense physiological responses and gestures to be used for affective computing; and (ii) A set of novel, clinically grounded, semi-supervised machine learning models to predict problem behavior that can be used by behavioral interventionists in real-time. An important novelty of this research that separates it from existing work in the field is that the team proposes to address the detection of problem behavior through its precursors, rather than the behaviors themselves, with the goal of increasing the safety and efficiency of sessions. These scientific and technological advancements will be created within a state-of-the-art clinical and behavioral science framework. The proposed work will foster interdisciplinary research in engineering and health sciences. The team proposes a number of outreach and educational activities that will have broader impact in STEM education: i) involve individuals with ASD directly in the research through the Frist Center for Autism & Innovation’s Neurodiversity Corps; ii) provide interdisciplinary training opportunities for early stage clinical scientists; iii) provide research opportunity to high school, undergraduate, and graduate students; iv) provide research opportunity to high school teachers; v) bring research into the classroom; and vi) disseminate the research through seminars, presentation, and publication. Emphasis will be placed on recruiting candidates from minority and underrepresented groups to improve diversity in STEM.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

In 2022, approximately 70,000 young adults with autism will leave high school and face a litany of disheartening statistics regarding independent living, community participation, and employment. Adults with autism rate employment as their top concern. A major impediment for autistic individuals to access work opportunities specifically is lack of independence with transportation; fewer than a third of driving-age individuals with autism are licensed to drive while many more are capable with the right training. This project offers a community-driven planning process to explore how a prototype of an advanced artificial-intelligence-based driving-training (AIDT) system, specifically designed for neurodiverse individuals, that together with a curriculum built on a cognitive behavioral intervention for driving, can offer an effective solution for communities to help capable autistic individual become licensed drivers, who are confident in their driving ability and able to more fully participate in community life.<br/><br/>This Stage-1 project seeks to plan for a full pilot deployment with multiple types of civic partners and support providers – including community-based training centers and clinics, schools, and others – toward an effective, low-cost, commercializable, driving-instruction platform, with a value proposition that offers increased independence and expanded career options for individuals with autism. An AIDT prototype has been tested and demonstrated in Nashville, Tennessee, and San Diego, California. The research in this Planning Grant will involve a large group of civic entities in a community-engaged, mixed methods quantitative and qualitative data collection process to identify context determinants that are important to address within the AIDT system and in constructing a training protocol in preparation for Stage-2 pilot deployment across vocational rehabilitation, clinical services and education sites. The project expects to learn how to make the AIDT system adaptable to use within multiple employment contexts and multiple employment outcomes of relevance to stakeholder communities, how to make the AIDT outputs responsive to community service providers’ needs for maximal useability and minimal training demands, and how to finance its long-term sustainment and growth. <br/><br/>This project is in response to the Civic Innovation Challenge program — Track B. Bridging the gap between essential resources and services & community needs — and is a collaboration between NSF, the Department of Homeland Security, and the Department of Energy.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

This project seeks to reduce loneliness in older adults who reside in long term care (LTC) communities through new augmented reality (AR) technology. Loneliness is a serious condition that is related to increases in heart disease, depression, suicide, mental and physical decline, and reduced quality of life and death. Two out of five older adults in the U.S. report being lonely. Even more alarming, three out of four LTC older adults experience loneliness. The COVID-19 pandemic, with its accompanying safety protocols, has intensified loneliness across the LTCs. The project will discover how augmented reality can reduce loneliness in LTC older adults by linking them with family members who reside elsewhere. This project will allow older adults and family members to see each other’s 3-dimensional realistic images, eat meals together, and interact with one another in various activities, such as playing cards. Investigators of this project are experts in engineering, computer science, gerontology, nursing, medicine and social health science. Working with older adults and family members in the design and testing of the AR technology, the team will compare AR to 2D interactive communication technologies, such as Zoom or Facetime. Initial understanding of the feasibility and acceptability of this enhanced AR technology among older adults, families and LTC staff will guide future studies targeting loneliness, ultimately improving quality of life for older adults. The community focus for this project will be older adults residing in LTC communities in Middle Tennessee with the potential to scaling the solution across the nation.<br/><br/>The project will fundamentally advance the scientific and the technological methodologies of collaborative Augmented Reality to enhance social presence and thus social connectedness, to create realistic and socially appropriate interactions. It will make several fundamental contributions in both technology and social science during the course of this research: 1) create a novel multi-objective optimization based framework that minimizes positional errors of the hand of the avatar while preserving its nonverbal behavior with respect to the human it represents; such an ability will allow shared activities (e.g., drinking tea together) with appropriate social nonverbal behavior (e.g., gaze and postures), a critical component of communication; 2) create a new methodology of a user’s motions onto its avatar to generate naturalistic, socially appropriate motion that respects dissimilarities between the user’s and its avatar’s environments (e.g., differences in room geometries) through novel motion mapping and optimization that ensures natural walking patterns; 3) develop a greater understanding of the feasibility, acceptability and social presence in the use of varying collaborative AR activities and environments for older adults with different levels of cognitive impairment and their family members; 4) develop a greater understanding of the impact of collaborative AR on loneliness based on level of cognitive impairment; 5) gain a greater understanding of the logistics and deployment of this technology in LTCs and family homes to inform scalability; and 6) create activity design guidelines for reduction of loneliness in older adults. The research will be conducted through participatory design using key stakeholders (e.g., older adults, activity directors, LTC management) and evaluated using a two-arm experimental design comparing collaborative AR to current state-of-the-art 2D interactive communication technologies.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.