Francis Quek - US grants

EIA-9802090
Defanti, Thomas A.

University of Illinois

CISE Research Infrastructure: CAVERN: The CAVE Research

UIC's research goal is to move Tele-Immersion and Data Mining from the laboratory to the Next Generation Internet. Scientists are just now beginning to get access to high-speed networks to retrieve information from remote datasets (whether large disk farms, scientific instrumentation, or satellites), analyze that information using remote supercomputers, and use virtual reality (VR) to collaborate with distant colleagues. UIC has promoted VR and data mining in select communities of users, and now wishes to create the teams, tools, hardware, system software, and human-interface models to enable national-scale, multi-site collaborations to facilitate solutions to National Challenge and Grand Challenge problems.

Planned CAVE Research Network (CAVERN) activities are twofold: enabling technology development and application driver support. This proposal lays out several application disciplines- - computational science and engineering, education and training, and every citizen interfaces (ECI) with emphasis on rehabilitation for the disabled - - to drive the development of the tools and techniques for tele-immersion and data mining. Enabling technologies for tele-immersion and data mining include CAVERNsoft, ECI interfaces, data mining tools, video, and performance optimization.

Behaviorally Situated Avatars for Tutoring: Beyond the Talking Head and Animated Icon
Abstract

One-on-one tutoring is a critical component of teaching and learning. This project investigates technology to deploy an anywhere-tutor anywhere-student system that permits a tutor to provide instruction via an animated avatar. Exploiting psycholinguistic information, the system translates time-situated pointing and communicative gestures performed on a LCD tablet in conjunction with speech into a stream of behaviorally correct, spatially situated 3D gestures performed by the avatar tutor. A camera tracks the student, facilitating socially aware behavior by the avatar. The system exploits the student's ability to use body motion cues in the embodied avatar to direct her attention to appropriate locations of projected graphics that serve as artifacts of instruction. The technology will potentially facilitate a tutoring program in which students may connect to a pool of tutors for help with a variety of topics. The approach essentially implements tutoring 'telephone handsets' at both ends of the interaction. Tutors may be distributed across multiple sites at their convenience and take calls as they are needed. Widespread access to such tutoring will facilitate the vision of universally available educational opportunities. The project will investigate the necessary artificial intelligence and interaction technologies and evaluate the efficacy of this tutoring methodology.

When we speak, our embodied behavior of gesture, gaze, posture, and facial expression are brought into the service of the communicative process. This research is grounded in psycholinguistic theories of multimodal human communication. One path from multimodal behavior to language is bridged by the underlying mental imagery. This spatial imagery, for a speaker, relates not to the elements of syntax, but to the units of thought that drive the expression (vocal utterance and visible display). Gestures reveal the focal points of the accompanying utterances, and relate to the meaning of the newsworthy elements of the unfolding discourse. This is the underlying premise of the current project, in which the PI will focus on math discourse and education for blind students.

Mathematical reasoning is rich in spatial imagery that is revealed in gesture. Furthermore, gesture has the capacity to create images of the math concepts that serve as "objects of contemplation." When a graphic/illustration is available for math instruction, the discourse stream is typically situated with gestures of spatial reference into the graphic. Research with individuals who are blind suggests that they have remarkable capacity for visual imagery, memory, and conceptualization. However, students who are blind tend to lag behind sighted students in mathematics education. The PI posits that a significant impediment to math instruction for students who are blind lies in the lack of visual access to the embodiment of the instructor. He notes that we have, in such students, a population that is able to access graphical content (through tactile image technology), but is not visually aware of the embodied behavior of the interlocutor. He proposes to remedy this problem by affording blind students a sense of embodiment awareness, with the help of augmentative communication approaches that employ tactile devices to provide elements of embodiment awareness. To this end, the PI will perform a series of "perception and action" experiments to assess the efficacy of such devices, followed by a second series of experiments with blind and sighted students in mathematics instruction, to be captured on video, and with pre- and post- tests, to assess the quality and quantity of imagistic content both conveyed by the teacher and by the student in the course of the instruction, and to determine the correlation, if any, between these and the formation of math conceptualization.

Broader Impacts: This project should have significant direct impact on inclusive mathematics instruction at all grade levels for visually impaired students; provision of a sense of embodiment awareness to students who are blind has not hitherto been studied, and has the potential for empowering such students. Furthermore, understanding the channels for embodiment awareness will affect the design of future distance learning systems, and will afford insights on how best to provide embodiment cues to students in Internet-based instruction.

This project, exploring the concept and human-computer interaction (HCI) implications of the embodied mind, builds a comprehensive environment for HCI that includes vision/graphics/large displays, sound capture, textile, and haptics. The infrastructure aims at servicing 14 projects in five areas of research:

-Embodiment Awareness and Aware Spaces
-Expansive High-Resolution Visual Displays
-Situated Interfaces in Varying Attention Settings
-Augmentation of Interactive Spaces with Aware Devices and Interfaces
-Embodied Human Communicative Behavior

The research elements are combined in three integrative exemplar application domains:

-Interaction around geographical information in collaborative situation rooms,
-Aware classrooms, and
-Embodiment awareness technologies to assist Aging in place.

Broader Impact: Specific applications, such as aware classroom and aware homes for aging convey technological benefits that in turn might benefit society. Furthermore, the collaboration with minority and undergraduate-serving institutions (Norfolk State and Hollins U), the integration of the project goals with teaching, and the general knowledge dissemination, form a worthwhile outreach program.

The PI believes research in human-computer interaction is on the cusp of solidifying a paradigm of embodied cognition that emphasizes the role of bodily reality in the creation of meaning, which constitutes the essence of our thought and experience. When we speak, our heads, eyes, bodies, arms, hands, and face are brought into the service of communication. There is substantial evidence that communication does not exist for us separately from its embodied roots. The cognitive and perceptual resources of attention (both visual and conceptual), working memory, long-term memory, temporal sequencing, language processes, touch, internal physiological experience, and proprioception must be considered together to understand the range of meanings a behavior has to the actor and receiver in a situation. The PI recently submitted a proposal entitled "CRI: Interfaces for the Embodied Mind" by means of which he was able to secure funds for the acquisition of equipment necessary to exploring ideas such as those outlined above, but not for the exciting applications he envisages in collaboration with partner institutions. The supplemental funding provided by this SGER will enable the PI to explore the implications of embodied interaction technology to communication in the domain of the liberal arts, specifically through participant and travel funding to support collaborative projects: (a) with Prof. Cassandra Newby-Alexander and others in Norfolk State University's Department of History who are investigating the Underground Railroad network in the four large urban slave centers of Norfolk, Portsmouth, Richmond, and Petersburg; and (b) with Prof. Pinckney Benedict of Hollins University's English Department on a pilot project in which two or more creative writing students develop an embodied adaptation of James Joyce's classic short story "Araby" which is rich in concrete environmental cues and written in a dreamlike narrative style that lends itself gracefully to nonlinear storytelling. Each project will involve two faculty and two student members from the partner organization, along with students and faculty from Virginia Tech.

Broader Impacts: Norfolk State University is an HBCU, and Hollins is a women's university. This project will afford an opportunity to these partner institutions for their students and faculty to participate in the broader federally-funded research enterprise, and will at the same time expose students at Virginia Tech to students with diverse backgrounds and perspectives that will improve their ability to engage realistic HCI problems. To these ends the PI will develop a collaborative research outreach model based on cross-institutional faculty teams. He will define research projects that span multiple institutions and provide a scaffold for outreach partner institution students to work with their professors during the summers on the Virginia Tech campus. Partner institution students will continue their research upon returning to their home institution, thereby transferring their experiences to home faculty and students while keeping their connection with the faculty and students they befriended over the summer. Electronic communications and periodic cross-visits by the faculty team will keep the research vital. An annual workshop at Virginia Tech will expose partner institution students and faculty to the broader research and stimulate greater synergy.

Many proposed technological solutions to future disasters involve the use of wireless cellular technology. However, these same networks become quickly saturated soon after a disaster and remain saturated for critical periods. This project will capture data on cell phone usage (and other communication modes) in terms of when, with whom (what relationship with subject), and for how long the calls take place, and whether the calls are incoming or outgoing. Such data should help in the modeling of communication network behavior in such situations and in understanding how such technologies may be utilized or extended in future emergencies in any community. The goal of this exploratory research is to capture time sensitive communications and other ephemeral data regarding the tragic events surrounding the Virginia Tech (VT) shootings on April 16, 2007. These data will be available for further analyses by multiple interested groups in order to plan for future emergency response and critical peer-to-peer communication.

Intellectual Merit: The intellectual merit of this exploratory research is two fold: 1) it contributes to the body of knowledge regarding emergency response and disaster management via mobile communication networks; and 2) it provides a model of communication behavior among ordinary citizens during the first 24 hours of a crisis. The data to be collected can be used to visualize communication patterns and emergency response networking, can reveal support networks for communication and information, and can lead to policy analysis and debate regarding mobile communication, emergency preparedness and response.

Broader Impact: The results from this study have the potential to affect the general public and emergency response groups in communities across the US. By understanding the citizen usage and network impact of mobile communication during a crisis in a geographic area, network overload can be mitigated while critical interpersonal communication can be accommodated. This research will also consider problems with the user interface for typing under duress on a small keyboard; and cell phone use by citizens with low computing or reading literacy (such as, the elderly, lower socioeconomic groups, and non-native English speakers) who may not be completely comfortable in English or in cell phone text usage.

This research project will investigate the effects social organization, learning technologies, and their interrelationships, have on the geometric reasoning of PreK-3 students. Social organization is operationalized here as the strategic grouping of students into triads to engage in collaborative problem solving. Learning technologies are operationalized here as physical and virtual manipulative types (tangrams, pentominoes, geoboards) together with supporting software designed to facilitate students in accessing their mathematical knowledge. Based on findings from preliminary work in this area, it is proposed that learning technologies allow for fluid exchanges between peers, and transitions between physical and virtual manipulatives, significantly enhancing geometric reasoning. The current project presents a robust research and design framework that combines iterative cycles of field studies in the classroom with development and experimentation in the lab. The goal is to design, develop and prototype a tabletop computer interface that allows for unprecedented exchanges among student peers and transitions between manipulative types. The tabletop interface will serve first as the apparatus that facilitates experimental manipulations, later being used, through iterative refinement, as the learning technology platform by which students gain access to mathematical knowledge. The advancement proposed by the current work is that it places all interaction on the horizontal tabletop. This is critical in that it removes confounding effects of different seating and screen orientations from pilot study findings. Manipulations of geometric artifacts reside along two dimensions of research questions: the effects of social interaction and of learning technology on geometric reasoning. Hence, the proposed studies will allow investigation of different transitions between fully functional and fully operational thinking, functional and qualitative operational thinking, and qualitative and quantitative parametric thinking with the same interaction morphologies.

The proposed research has the potential to make significant contributions to the following intellectual domains: 1) child development -understanding children's development of mathematical reasoning in terms of learning trajectories 2) mathematics education - understanding teachers' use of advanced learning technologies and role in classroom instruction, and children's learning of mathematics using advanced learning technologies; 3) learning sciences - refining techniques for the analysis, design, and development of advanced learning technologies in use; 4) human-computer interaction -design of interfaces that aid learning to include gestural interaction, tangible manipulation, and horizontal displays; and 5) psycholinguistics - grounding of concepts developed in other domains in the classroom environment. Results from the research will also have much to say about the participation of girls in the mathematics classroom, a goal being to encourage rich discourse of all students. In addition, developed technologies may have implications for the education of special needs children as instructional interfaces can be modified and reconfigured to meet the diverse needs of this underrepresented group. A significant outcome of collaboration among these disciplines will be cutting-edge research opportunities for doctoral students and groundwork for the establishment of new curricula and, possibly, degree programs.

The purpose of this exploratory project is to test a proof-of-concept that reflects an emerging and potentially transformative mobile system to enhance recognition and way-finding function for users with severe visual impairments. The PI's Near and Far Environmental Awareness System (NaFEAS) will run on a mobile phone platform in concert with RFID tags and sensors which serve as inputs and outputs to a knowledge database that can be updated by the user. NaFEAS will recognize tagged objects and provide audio and other feedback to users within a designated envelope around the sagittal and coronal body planes. In addition, NaFEAS will facilitate way-finding using tagged objects in the near and far environments. The focus here is on the development of a low-fidelity prototype using participatory design; the theoretical basis is embodied cognition, which emphasizes the full use of senses, gestures, and space to understand objects and the environment. The PI will take special pains to minimize the cognitive biases occurring when sighted designers or researchers retrofit visually-dominant interaction patterns to develop accessible devices. The prototype will be evaluated by target group members using an experimental design with self-report, behavioral, and physiological variables.

Broader Impacts: Besides resulting in a prototype product that will be of great value to the target population, the findings from the experiments will advance our knowledge of how individuals with severe visual impairments successfully interact with next generation systems, including mobile, wearable, and ubiquitous systems. This knowledge will be generalizable to technologies beyond the test bed the PI will be using.

Modern direct manipulation and visualization systems have made key strides in bringing powerful data transformations and algorithms to the analyst's desktop. But to further promote the vision of powerful visual analytics, wherein automated algorithms and visual representations complement each other to yield new insight, we must continually increase the expressiveness with which analysts interact with data. This project focuses on the task of storytelling, that is to say the stringing together of seemingly unconnected pieces of data into a coherent thread or argument. To support storytelling, which requires both human judgment and algorithmic assistance, the PIs will first develop a new theory of relational redescriptions that provides a uniform way to describe data and to compose data transformation algorithms across a multitude of domains. Using this theory, the PIs will be able to define stories formally as compositions of relational redescriptions. They will develop scalable and steerable algorithms for storytelling that will respond to dynamic user input, such as preferences and constraints, and they will contextualize their use in interactive visualizations that harness the power of spatial layout. Finally, they will investigate how analysts engage in sense-making using the new storytelling algorithms and visualizations, in the hope of finding answers to questions such as: How do analysts achieve insight and advance their conceptualization of patterns derived from datasets? Project outcomes will include the formal conceptualization of storytelling as well as the compositional approach to building complex chains of inference.

Broader Impacts: This research will make it easier for analysts to interactively explore connections in large-scale heterogeneous datasets. The PIs will work with the FODAVA-lead team at Georgia Tech and PNNL's NVAC to investigate applications of relational redescriptions and storytelling to domains of interest to NSF and DHS, and will develop in consultation with real users across these groups a layered software framework for storytelling (both analysis and visualization) capabilities; the framework will be released into the public domain under the GNU GPL/Lesser GNU GPL license, and APIs will be provided that allow analysts to tailor it to suit their needs. Although this project will focus on cyber-analytics scenarios such as those motivated by the VAST 2009 challenge, project outcomes will generalize across other domains such as bioinformatics, systems biology, electronic commerce, and social networks. The unified notion of redescriptions will help integrate multiple data sources (numeric, symbolic, textual, and categorical), and situate them on a common footing for visual analytics; it will also enable visual analysts from different application domains to use a common vocabulary while interacting with one other.

This research explores the concept of grounded imagination where creativity is grounded in knowledge and informed by experience. How might one nurture and cultivate such grounded imagination in K-12 students? To address this question, this project will develop and deploy a storytelling system for authoring and interacting with hyper-dramas. It incorporates the theory advanced by Lev Vygotsky, that creativity involves a process of combinatory imagination by which an individual creates new things for herself from elements of prior experience encoded in everyday concepts and new culturally transmitted information. This situates creativity within two developmental streams: intellectual development (acquiring grounding in the form of knowledge and experience), and the process of flexible re-combination. The aspect of social-cultural engagement suggests that creativity is a discourse with the larger culture and society. Drawing from this theoretical foundation, this project explores whether the creativity trough (decline in creativity from the 4th grade through middle school) may occur because social-cultural awareness precedes intellectual development so that the student judges herself an inadequate contributor. Facilitating hyper-drama authoring will allow students to draw on their experience in hyper-media and dramatic presentation, and thus nurture the combinatory creativity process.

In this research, students will construct non-linear hyper-narratives in which each path through the narrative tree represents an individual narrative arc. Each node represents an occurrence along the narrative timeline, called a story fragment, that may happen in one or more places. The occurrence in each place takes the form of an animated dramatic scene. Storytelling, then, becomes a process of authoring the story along with the hyper-narrative, and the content and goings on within each scene (including the characters, activities, animation, scenes, and dialogue). This project will test the system and address two scientific questions by deploying different versions of the authoring system in classes at the elementary, middle, and high school levels within the Roanoke County School District. The complexity of stories constructed and the kinds of content will be graduated to match the learning goals within each level. This research will follow cohorts of these students over a period of two years as they transition between elementary, middle, and high school. Two key questions are: 1. How does a situated process of storytelling employing a hyperdrama model support creative imagination? 2. How does one engage a generation of students in a media-saturated world to become creative participants, rather than passive consumers of digital media?

The immediate research aim, to enhance and nurture creative processes in K-12 students, has significant implications for society. A more innovative workforce is critical to the success both of the nation and of the individual worker. Furthermore, creativity is necessary for all learning, including mathematics and science. By addressing the fundamental process of creativity, this project can positively impact learning of all domains including STEM. More directly, the explicit construction of hyper-narratives, and the assembly of resources (graphics, animation, characters, etc.) for each hyper-drama node make explicit the algorithmic and flow-of-control mechanisms necessary for mathematics, science, and engineering. Second, the will develop interest in scientific research among elementary, middle, and high school students. Third, the software and intellectual product of this research will be disseminated broadly through publications and open source mechanisms.

The goal of this project is to foster new research collaborations across different disciplines by developing a novel massive interaction game that represents a unique genre wherein teams of users strategically direct armies of virtual 3D "terra cotta" soldiers by employing physical drum beats as control signals to determine the warriors' behavior. It is intended that the game will engage and enthrall both user teams and a live audience with the special demands of cooperative and competitive game play in a compelling and creative scenario. Successful completion of the research will require intensive collaboration across multiple fields of computational science, art, and music. The culmination of the project will be a social event centrally located on the Virginia Tech campus, in which the finished game is publically introduced to the community at large.

To achieve these goals, the PI team will engage interactive game design approaches to draft and enrich the overall game experience. They will adopt commoditized Graphics Processing Units (GPUs) to meet the computational challenges of real-time simulation, animation, and rendering of massive number of 3D characters. Novel numerical methods will be developed to enhance the efficiency of global path finding for massive crowd simulation. The individual combat behavior of the virtual soldiers will be automatically regulated by an artificially intelligent component, which is supported by GPU accelerated agent-based simulation. In order to provide visualization capacity for a massive crowd in an enormous environment, the investigators will develop technologies to enable level-of-detail rendering and animation. For massive interaction support, signal processing and computer vision approaches will be used to recognize the drumbeats and to interpret the audience participation behavior (which will be tracked with the aid of microphones embedded among the members of audience and cameras aimed at them). Synthetic music generation approaches will be utilized to produce real-time accompaniment to the drums.

Broader Impacts: The outcome outlined above will be the result of broad and intensive collaboration between engineering and the arts. Computer engineers will design and develop the overall game engine architecture, graphics rendering technologies, numerical solutions for character path planning, and approaches to recognize drum beats. Graphics artists will design 3D geometry models and animations for "terra cotta" soldiers, and will work closely with engineers to build a smooth "pipeline" to import digital content into the game software system. Researchers in modern music will produce accompaniment to the cadence of the drums. The team-based approach adopted in this project will expose students to cross-disciplinary research, so that they come away with an understanding and appreciation of the need to function cooperatively to achieve rewarding and exciting results. Teaming of undergraduate students with graduate students will, it is hoped, lead some of the undergraduates to choose research related career paths while also encouraging the graduate students to learn how to better mentor others. The PI team hopes that the mass media event at the culmination of the project will imbue the broader public (and especially K-12 students) with the excitement of computing and the arts.

This project explores a multimodal corpus for vision-based meeting analysis. The research team is working on: (1) extracting the data from tapes and organizing them into multimedia databases; (2) developing a database visualization and analysis tool to support model development; and (3) developing an agent-based algorithm to extract hand and head tracking information so that higher level models may be built onto the data.

The project provides datasets that are organized into a usable corpus with many unique properties, such as the ground truth at the psycholinguistic/psycho-social level of the social roles status, purpose of each meeting, and at the video level in the form of motion tracking data collected co-temporally with the video, for developing and testing new algorithms. The developed tools improve the access to the multimedia database of multi-view group human behavior. The agent-based approach provides a novel way in video annotation. The developed tools and algorithms from this project can be applied to many other applications. For example, the tools may be applied to analyze classroom behavior and in learning scenarios. The project provides research opportunities for undergraduate and graduate students including women and individuals from underrepresented populations. The project outreaches to the user communities through publications, presentations, web presence, and broader collaborative interactions.

The PI's goal in this research is to develop tools on a state-of-the-art platform (the Apple iPad) that will afford access to textual information for individuals who are blind or who suffer from severe visual impairments (IBSVI). The PI's approach is to use an embossed screen overlay to provide spatial and tactile correlates to text read aloud, and to engage the spatial cognition and memory resources of the target population for navigating through a document and annotating it if/as desired. The PI argues that from the invention of print media forward, information has been formulated and optimized for consumption by beings (people) with a dominant visual capability. This visuo-spatial bias is not well-understood or studied in the context of information access by and delivery to the IBSVI community; most technological information aids funnel information to them as sequential aural streams, obviating the use of broader spatial cognitive resources. In this project the PI will develop a Spatial Touch Audio Annotator and Reader (STAAR) testbed to explore a multimodal alternative that enables the user to fuse spatial layout and informational content through touch location on a slate-type device and audio rendering of text to speech, respectively. STAAR will enable self-paced reading using a tactile overlay pattern on an iPad surface, which will be designed to provide tactile landmarks to help the user navigate the "page." STAAR will render the text chunk touched audibly. The use of touch gestures to enable contextualized highlighting and note-taking will also be investigated. The PI will study how the target population may employ spatial strategies and exploration to re-find and re-access information both in the act of reading and for recall after some time interval.

Broader Impacts: A new generation of slate-type devices exemplified by the Apple iPad threatens to widen the accessibility gap between the IBSVI community and the majority of the population. By supporting the use of spatial cognitive and memory resource for both reading and contextualized annotation, the PI hopes to ameliorate this endemic barrier to participation. Project outcomes will contribute to our understanding of the role of space in information design and representation, and the spatial cognition and memory resources needed for uptake of such information. And they will contribute to the domain of mobile computing, for the IBSVI community in particular but ultimately for the population in general, through the STAAR system, which will be designed and implemented from the ground up as a portable device on a state-of-art interaction form-factor. The multimodal fusion of haptics and speech in STAAR will have implications for the designs of such things as navigational aids and service delivery systems for the non-sighted as well as the sighted. The project will in addition provide unique cross-disciplinary educational and learning opportunities for undergraduate and graduate students. All software developed in this project will be placed in open source.

As new personal devices of various form factors (from smart phones to laptop computers) proliferate, we believe that their potential to support human learning, activity, and information access can be fully realized only when these devices function together with a broader technology environment and with each other. We envision the day when such technology envelops society, and in concert empower individuals to function more effectively both individually and as members of society. In essence the projected infra-structure will allow us to investigate this future.

In a sense, technologically savvy users already construct ecologies for themselves in an ad hoc manner. We ensure that hand-held devices we carry will synchronize with our larger devices like laptops and desktops. We ensure data exchange among the various applications we use, and we create personal and group workflows among our computational environments. We keep appropriate adaptors so that our portable devices have access to the projection screens or group displays in our work environment. We learn the interaction that is needed on each platform to ensure cross-platform interactivity. Finally, we organize our work and life around these interoperations ? synching our iPods and iPads periodically, selecting applications on each platform to support the way we work and our lifestyle as a whole. The challenge of this proposal is to remove the ad hoc nature of the construction of such one-of-a-kind ecologies.

Our proposed infrastructure comprises two related subgroups. First, we propose a Rapid Prototyping Laboratory (RPL) where faculty and students can develop new devices to populate the ecologies. The RPL will support electronic and single-board-computer designs, printed-circuit board prototyping, and 3D object construction. Second, we will construct the larger environment to be populated by these devices. This environment includes a variety of large to personal display technologies and a set of tracking technologies to enable devices to situate themselves in space.

We have assembled two sets of research projects that benefit from this infrastructure. First we inves-tigate how our envisioned ecologies may be realized. We will study how this myriad of technologies may interoperate to give the user a set of consistent expectations of interaction across devices and display environments. We shall also investigate HCI methodologies for humans to function within the 3D world populated by devices and displays. Our second set of projects investigates various uses of our envi-sioned ecologies. Extending the ecology analogy, each use scenario furnishes a habitat that contextual-izes the ecologies within it. We advance a set of research areas: education and learning; universal access; insight formation; smart homes; usable security; mediated social interaction; and electronic-textiles to furnish such contexts where technology-interoperation and user-interaction may be explored.

An important aspect of this project is to encourage a kind of computational thinking among our re-searchers and students that span the digital and physical environment. As such our proposed infrastruc-ture will facilitate a broad interdisciplinary research. We will be using the infrastructure in the classes we teach bringing graduate students, undergraduates, and individuals from underrepresented populations into our research programs.

Building on the highly energized national Maker's Movement involving government offices, schools, and businesses now sweeping the country, this project will explore ways to engage more children in the effort and become motivated and interested in STEM. The project will target 200 children at grades 3-5 to involve them in the "Maker's mindset" from an early age. The goal is for children to develop strong self-identities in STEM areas through their involvement in arts, crafts, and narrative storytelling.

For three years, Latino and African American children will participate in a STEM-inspired intervention based on principles of the Making Movement from a constructivist perspective in a project-based learning environment. Students will learn basic concepts in electricity, circuitry, Ohm's Law, 3D printing, electronic load balancing, LEDs, resistances, transistors, and diodes as well as age-appropriate knowledge and skills in geometry that is intended to foster self-identity with a Maker mindset. Research shows that engaging children early on in these types of activities builds strong affiliations with the larger STEM community, creates a sense of belonging in that community, and helps prepare children to easily assimilate within the rapidly changing technological world. To ensure that the Making activities are not an end unto themselves, teachers will participate in ongoing professional development activities to learn how to integrate Making activities into the existing science curriculum for the longer term. This iteratively designed study stems from prior NSF support and is structured around four threads aimed at: (a) conceptualizing the Maker's movement in terms of early childhood development; (b) instilling a Maker's mindset in children; (c) influencing children's identity about STEM; and (d) benefiting society through contributions to the future STEM workforce. Research questions structured around these threads will guide the project team's work with 100 children in five cohorts over three years who will participate in the Maker's intervention to observe the effects on children's self-identity with STEM. The project team will use a mixed-methods design to allow comparisons across three grades each year. A control group of 100 children at the same grades will be used for comparison purposes. The expected outcome is that over time, children will begin to see themselves as active partakers of and contributors to STEM fields and what STEM careers might offer.

This project will expand the methodologies and applications for computer-based haptics, by investigating how remote tactile communication, functioning in partnership with other contextual channels like speech, may facilitate supportive discourse for at-risk individuals. The research enhances infrastructure for research and education by building a testbed to study a heretofore unexplored mode of computer-assisted perception between people. It will advance science while promoting teaching, training, and learning by providing unique cross-disciplinary educational and learning opportunities for undergraduate and graduate students. Potentially, this new technology will facilitate greater partnership and emotional interaction among people, over long distances. This will help people understand the meaning of the other person's communications, maintain valued relationships, and enhance each other's well-being. It could usher in a new dimension in telecommunications that has significant economic potential.

A hardware and software testbed will be developed to study this sub-space of human-computer interaction, including tactile modalities and parameters (e.g., compression, pattern, and onset dynamics) and the timing between a tactile stimulus and the accompanying contextualizing channel. Input and output devices will be developed through iterative design, and evaluated by means of: (1) usability and characterization studies; (2) in-laboratory affective communication studies including a one-way controlled study to assess affect conveyance and a dyadic communication study for the range of communication dynamics supported; (3) in vivo extended communication studies. One connection to general communication theory is the hypothesis that tactile interaction conveys affect immediately without interposing any explicitly coded symbolic message. More generally, this research will take an interdisciplinary approach that draws from and contributes to each participating discipline. Broad dissemination will enhance scientific and technological understanding through publication of results and insights in the journals and conferences of the various fields represented in the project. All software developed will be shared in open source form.

As part of an overall strategy to enhance learning within maker contexts in formal and informal environments, the Innovative Technology Experiences for Students and Teachers (ITEST) and Advancing Informal STEM Learning (AISL) programs partnered to support innovative models in Making poised to catalyze new approaches in STEM learning and innovation. Employing a novel design and development approach, this Early Concept Grant for Exploratory Research (EAGER) will test the feasibility of integrating Making concepts with real world micro-manufacturing engineering principles within the context of intense, multi-year team apprenticeship experiences for high school students. The apprenticeship model is particularly novel, as current Making research and experiences predominately take place in afterschool and summer programs for up to 25 youth. The proposed apprenticeships will require a two year commitment by a small cohort of Texas high school students, which will provide an opportunity to examine the feasibility and impact of the effort longitudinally. The cohort will learn to think critically, solve problems, and work together as a Making Production Team (MPT) in a customized makerspace in their high school, constructing engineering-based science kits for implementation in a local elementary school. Not only will the students enhance their content knowledge while developing design and development skills but the students will also receive stipends which will address two very practical needs for the targeted high need population - employment and workforce development. Few, if any, efforts currently serve the targeted population through the contextualization of Making within a supply chain management and micro-manufacturing framework that extends the Making experience by integrating the student designed products into elementary classrooms. As such, this project will contribute to essentially unexplored areas of Making research and development.

Six high school students from high poverty, underserved Texas communities along the Texas-Mexico border (colonias) will be selected for the Making Production Team (MPT). In Years 1 and 2, the students will meet regularly during the academic school year and over the summer with Texas A & M University undergraduates, graduate students, and the project team to learn key aspects of Making and manufacturing (i.e., ideation, prototyping, design, acquisition, personnel, and production) through hands-on making activities and direct instruction. Concurrently, a research study will be conducted to explore: (a) the actualization of the model in an underserved community, (b) the effectiveness of problem-based learning to train students in the model, and (c) STEM knowledge and self-concept. Data will be collected from multiple sources. An adapted version of the Academic Self-Description Questionnaire will be administered to the students to assess their STEM technical knowledge and skills as well as their self-concept in relation to STEM domains. Remote and in person interviews will be conducted with the students to track the evolution of the primary dependent variables, STEM learning and self-concept, over time. Program facilitators and partners will be interviewed to examine the feasibility of the making experience within the given context and for the targeted students. Finally, the students' diary reflections, products, and video recordings of their work sessions will also be examined. Time-series quantitative tests and in-depth qualitative methods will be used to analyze the data.

This cyberlearning project explores a novel approach, called Digitally-Augmented Enactment (DAE), to support learning expressive writing by children at the critical grade 4-5 developmental period, with a focus on English Language Learners (ELLs). Expressive writing is core to the learning of all school subjects. An alarmingly high percentage of students enter middle school with low proficiency in writing, which hinders further learning in high school and college. Children from these underserved populations are particularly at risk of withdrawing from writing during the grade 4-5 stage in school when it matters most. The key reason for children's difficulties is that writing requires not only mastery of writing mechanics (e.g., grammar, sentence structure), but also the possession of ideas to convey, and a mastery of the process of translating ideas into expression. Accordingly, the DAE approach bifurcates the writing process by harnessing the power of pretend play to bring forth children's imagination, enabling them to focus on ideas, without at the same time having to be concerned with the technical details of writing. DAE combines low-cost motion-tracking technologies, such as Kinects, with animation technologies. This technological environment allows children to enact with props and produces animated cartoons as the 'outline' for their essays which serves as a scaffold for their writing. The new technological approach to expressive writing is explored in a charter school and a public elementary school, both of which serve a high percentage of ELLs, and the majority of the students are on reduced lunch programs. The project will help children from diverse backgrounds to become literate individuals in a modern society. The project also provides unique cross-disciplinary educational and learning opportunities for undergraduate and graduate students to serve predominantly underserved communities.

With the intensification of self-evaluation and the gradual engagement in symbolic thinking during this critical developmental period, children are at risk of developing negative perceptions of writing if no support is provided. The DAE approach intends to help children to move from ideas to written expression through a process of enacted imagination. The idea is for children to materialize their imagination through embodied enactment, and to transform this into an animated cartoon. The child is able to view and interact with this congealed imagination to engage their technical writing skills. This gives children license to engage their capacity for vivid pretend play to give ideas form and fuel the writing process. Hence, DAE facilitates a "minds-on" bridge between ideas and the formal writing activity through enactment and feedback. The DAE approach is embodied in a representative system that combines motion-tracking, sketch-recognition and animation technologies. The system will scaffold children's externalization, organization and conversion of ideas into writing-ready imagination. An iterative development, test, and evaluation strategy through lab studies will be used to ensure usability in Years 1 and 2, and classroom studies will be conducted in Year 3 to test for effectiveness and potential of the new technological genre. One grade 4 or 5 class from each of two schools (a charter school and a public elementary school) will engage in two writing assignments using the DAE approach, and two comparable assignments with conventional methods (e.g., graphic organizers). Both schools serve a high percentage of ELLs, who will participate in the studies. Video-based psycholinguistics and gesture analysis, interviews, questionnaires, and writing assessments will be used to understand how DAE may bridge the children's ideas to formal writing. This project will expand core understanding of how embodied processes participate in structuring and articulating ideas and imagination for writing.

Intelligent, interactive, and highly networked machines --with which people increasingly share their autonomy and agency--are a growing part of the landscape, particularly in regard to work. As automation today moves from the factory floor to knowledge and service occupations, insight and action are needed to reap the benefits in increased productivity and increased job opportunities, and to mitigate social costs. The workshop supported by this award will promote the convergence of multiple fields of inquiry from Computer Science, Human-Computer Interaction, Electrical and Mechanical Engineering, Design, Economics, the Social Sciences, Industrial Engineering and Education to define key challenges and research imperatives of the nexus of humans, technology, and work. Convergence is the deep integration of knowledge, theories, methods, and data from multiple fields to form new and expanded frameworks for addressing scientific and societal challenges and opportunities. This convergence workshop addresses the future of work at the human-technology frontier.

The specific focus of this workshop is to foster a discussion about the re-conceptualization of wealth creation in economically distressed areas by enabling the production of high quality manufactured goods at scales of hundreds and thousands in many distributed locations, but in an effective and sustainable manner. This approach, called "Micro-Manufacture," is grounded in technological advances that facilitate customized production of artifacts (e.g., 3D printing, the Internet; mobile devices, etc.). The workshop aims at initiating a discussion about the feasibility of the micro-manufacturing model. The workshop will also explore the interplay between Maker-based innovation and the engineering processes of Micro-Manufacture. To initiate the Micro-Manufacturing discussion, the perspectives of multiple science, technology, educational and societal interests need to be included. In particular, it is expected that research questions emanating from the workshop will require the deep integration of more than one discipline, from supply chain modeling and product design, to economics, sociology and education, to name only a few. The expected outcomes of the workshop will include the formation of a community of researchers with ideas that cross-cut fields to address Micro-Manufacture; an understanding of how the multiple related research areas interact and the research that is needed to enable Micro-Manufacture; and a publication in the form of a collected volume to inform the research community and broader society of the need, possibility, and challenges relating to Micro-Manufacture. The broader impacts accrue mainly from its potential to address society's pressing need for new configurations of work that may bring employment to a broader range of workers.

As workplaces change, people will need to develop stronger and more flexible skills around technology. Schools are one place students might learn these skills, but in the case of electronics, 3D design, and similar Making technologies, many students lack access to technically-oriented teachers and mentors. To help address this gap, this project will develop technologies, processes, and training to develop programs where STEM university students provide distance mentorship for rural students around technology. New technologies are needed because of the hands-on nature of Making activities, which makes remote mentoring through screen sharing or videoconferencing difficult. Thus, the project will develop technology to enable mentors to provide instruction that includes gesture, eye-gaze, and speech interaction through distance telepresence robotics and augmented reality. New processes and training are needed as well, to help the university mentors be effective teachers; this will be done through a MentorCorps program that provides education on classroom pedagogy and technology learning. The technologies and training programs will be developed and validated with high school students in three rural school districts, who will participate in a career and technology education (CTE) course. The course will provide knowledge in 3D design, fabrication, electronics, computer programming, and basic manufacturing; the high school students will use these skills in turn to prepare science learning kits for elementary schools in their districts. In addition to advancing the science of remote technology education, the project will have a number of broader impacts. The chosen school districts in the Colonias region at the Texas-Mexico border serve students who are underrepresented in STEM, and the work will improve their STEM education and work preparedness. The university mentors, meanwhile, will gain service learning opportunities that develop educational, interpersonal, and empathy skills beneficial to future professional lives.

This project studies how embodied communication involving speech and gestures may be mediated through mobile telepresence robots and augmented reality to support hands-on distance mentoring. At the heart of this project is what a situated distance mentoring experience really means. The project is informed by the psycholinguistics of embodied communication where meaning is expressed through gesture, gaze, and speech as an indivisible whole. Four design-implement-test-deploy-evaluate cycles are planned to learn whether and how powerful multimodal language may be mediated over distance through the designs to support explanation and mentoring. In addition to in-lab workshops to test design versions, the project will deploy and evaluate the finalized design of each cycle in CTE classes in three rural high schools. College science and technology students will pilot the augmented telepresence robots to teach these classes that combine hands-on Maker-based skills and knowledge with engineering principles of manufacturing in three rural high schools to produce learning kits for their local elementary school. The scientific contributions of the project are fourfold: 1. It studies the nexus of embodied communication of space, gaze, and gesture with technical explanation, using resources like artifacts for demonstration and illustration, diagrams and drawing to support distance mentoring for hands-on technology learning. 2. It studies how this multimodal communicative compound may be engaged distally through a mobile telepresence robot extended to provide gaze and gaze awareness and gestures in augmented reality that blends the project space of the student and the gestural space of the mentor. 3. It combines the science and technology research with the real-world educational mission of rural technology education, thus bringing context to the research. 4.It contributes across the fields of psycholinguistics, computer science/HCI, and education. To psycholinguistics: the technical implementation enables the exploration of the embodied language behavior on a platform that can be manipulated to provide new insight. To computer science/HCI: the project investigates embodied interaction techniques that enable powerful multimodal communication strategies via remote platforms. To education: the project investigates means to deliver remote mentoring for hands-on learning that typically requires physical co-presence, and advances Maker-based learning to the engineering practices of manufacturing.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Reading, doing, and sharing mathematics are cognitively intensive tasks that require significant amounts of learning and preparation. These skills form an indispensable foundation for participation in all STEM fields and most careers. So the lack of mathematical literacy severely disadvantages individuals who are blind or who have a severe visual impairment (IBSVI). By the time they reach middle school, IBSVI are typically 1-3 years behind their seeing counterparts in mathematics, and the gap continues to widen in subsequent years. The key challenge to accessing mathematical expressions is that they are laid out spatially, and this spatial notation needs to be accessed interactively. This project will develop and study solutions to support reading, doing, and sharing of mathematics on modern tablet and handheld devices for IBSVI by balancing across multiple modes of low-vision access, interactive sound, tactile and haptic support, spatial landmarking, audio reading, interaction methodologies and the structure of the mathematical expression. This will open the door to personalized solutions that honor the ranges of visual abilities (e.g., degree of tunnel vision, low-light capacities, tactile abilities) of IBSVI. The research will investigate such multimodal reading, doing and sharing of mathematics expressions starting in later elementary and middle school.

The research begins with the proposition that information representation is designed for the sighted. The non-linearity and hyper-dimensionality of mathematical expressions are encoded spatially and are intended for visuo-spatial access and exploration. Furthermore, reading and doing of mathematics is cognitively intensive, requiring years of learning to build literacy. The project will address this chicken-and-egg problem where literacy is needed to access mathematical material, and mathematical material is needed to build literacy. This problem is compounded by the range of visual and tactile abilities of IBSVI, who have a range of central vision acuity, light and color sensitivity, degree of tunnel and tactile ability. The project will develop a 6-element model comprising coarse visual cues, tactile-landmark overlay grid configuration, audio/sonification feedback and reading, interaction methodologies, dynamic haptic feedback; and structural organization of the mathematical content. The model will be balanced for the range of mathematical access needs and IBSVI visual/tactile abilities, and will curate a solution space for supporting reading, doing, and sharing of mathematics. Reading, doing, and sharing will be treated in tandem for students at the upper elementary and middle school levels at the Texas School of the Blind and Visually Impaired (TSBVI), where teachers will work with project researchers in an iterative design, implement, and test process to address the authentic mathematics needs of their students. Students will have sustained access to the research products so that researchers can gain insight into how IBSVI learn 'and adapt to technologies, avoiding the 'one-shot-syndrome' that often prevents research approaches from moving beyond insights from first use scenarios.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

From an early age, humans learn to see, touch, hold, manipulate, and alter their physical surroundings using both of their hands and tools. A sculptor holds clay in one hand and precisely carves out material to create an intricate work of art. A mechanic uses hands and tools to reach the most inaccessible nooks of a car?s engine to make repairs. All who work their hands, threading needles, carving soap sculptures, or wiring electrical wall sockets know the extraordinary dexterity and precision they possess when working in the space within the near reach of both hands. The project investigates a new class of augmented and virtual reality (AR/VR) environments that allow designers to be more creative in this workspace. Existing AR/VR systems only allow users to interact at an arm?s length making it impossible to leverage the innate human ability to use hands and tools for precise actions close to the body. As a result, the original promise of AR and VR in facilitating creative thinking and problem-solving is still unrealized. Towards this, a workspace is designed in this project to enable designers to create virtual designs in three-dimensional (3D) space by mirroring the intricate, complex, and precise actions that humans can perform in daily life. The research team will develop AR/VR technology and approaches that will enable users to directly ?sketch? 3D digital objects, precisely control those objects, and assemble them to create physically feasible designs. The research team will integrate the new technology and tools from this project to promote geometric reasoning in K-12 students, undergraduate, and graduate curriculum. The project has the potential to make a long-term contribution to the future of creative work and may positively impact U.S. competency in design, manufacturing, and education by enabling small businesses to innovate effectively and economically.

In order to enable the creation of a new class of AR and VR interfaces for 3D design, this research investigates precise spatial manipulation of virtual objects to facilitate 3D digital design. The research team will systematically explore new spatial interactions by leveraging the perceptual-motor sweet spot, i.e., a portion of frontal space in the immediate vicinity of the human body where highly precise interactions are possible through the confluence of visuo-tactile perception, proprioception, and fine bi-manual motor control. Through a series of iterative design-prototype-evaluate cycles, the research team will (1) study the visuo-motor and biomechanical aspects of tangible bi-manual interactions in the workspace for spatial object manipulation; (2) develop digital workflows for 3D design and modeling of virtual mechanical artifacts based on the those interactions; and (3) expand the digital workflows to enable physics-based design powered by haptics. First and foremost, the project will contribute to the understanding of how inherent human dexterity can be leveraged to enable precise spatial interactions. Second, the project will reveal how precise spatial interactions promote design creativity and enable creators to design functional artifacts. Finally, the research will provide tested methods for operationalizing the perceptual-motor sweet-spot through new interactive methods that integrate user intent, design feasibility, and physical realization of functional products.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

The mathematics that are used to describe spatial transformations can be very difficult for undergraduate students. While moving something in the physical world may be easy to understand, describing the same operation with math in the digital world can be daunting. This project will develop new technology using Augmented Reality (AR) and Artificial Intelligence (AI) to improve the teaching and learning of these difficult concepts in STEM disciplines as well as creative endeavors. The project will test the use of new AR/AI technology to enhance students’ learning of the mathematics behind spatial transformations. This will improve the use of AR/AI-powered precise motion tracking of objects that can collect high resolution in-situ motion and scene data to enhance learning analytics. The project will identify the AR capabilities that can help students conceive, connect, and compare mathematical representations of motion to overcome the well-documented difficulties students face when learning spatial transformations. Strengthening this skill will support their continued development across many STEM disciplines.

An AR/AI-powered innovative learning environment will be developed and evaluated for its effects on teaching and learning major rotation and orientation representations in the Euclidean space. Different levels of abstraction, including Axis-Angle, Euler Angles, Matrices, and Quaternions will be tested. In workshops participants will play with and transform 3D-printed geometry models to evaluate the effectiveness of the AR/AI technology. The project will assess student learning outcomes by comparing math skills in pre- and post-workshop tests compared to students working through the same tasks without the use of the AR/AI technology. The project will contribute to advancing knowledge across disciplines of spatial and mathematical pedagogies, by exploring: a) the role of novel AR interaction that allows the interplay between physical and virtual manipulatives to engage students in embodied learning; b) the capabilities of AR to make difficult invisible concepts visible for supporting an intuitive and formal understanding of spatial reasoning and mathematical formulation; c) the features of AR that help students see relationships between spatial manipulations and mathematical operations; and d) the potential impact of individual differences in spatial transformations when using AR-assisted mathematical learning.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Since the early in in the COVID-19 pandemic, telemedicine has seen tremendous growth in the U.S, providing people in under-resourced areas with broader access to healthcare through remote telemedicine visits with health professionals and experts to whom they may not otherwise have access. However, these visits are currently restricted to video teleconference-style exchanges and lack the capacity for physical examinations, which contribute significantly to diagnoses and the identification of health concerns that may not otherwise surface during a medical visit. This project investigates the possibility of supporting physical examination as a paired practice between a caregiver local to the patient and a remote physician, using touch-based and augmented-reality technologies. Caregivers will perform examinations while wearing touch-sensitive gloves, guided by remote physicians using a physical examination cockpit that lets them see and feel the patient using the outputs from the caregiver’s gloves and a video feed. The technology and practice will potentially transform the work of physicians and local caregivers, opening the door to more effective and more accessible telemedicine visits.<br/><br/>The foregoing discussion uncovers a number of questions that the project is designed to address. First, tactile interpretation is active, meaning that one has to typically be in control of the sensing process to be able to interpret the sensory output. The proposal will explore conditions under which one can meaningfully interpret passively received tactile information, such as what the remote physician would feel when a local (to the patient) caregiver is now in control of the sensing motions. One method to be tested for returning agency to the physician is the psychological phenomenon known as the ‘rubber hand’ or body transfer illusion, where one psychologically associates an appropriately-placed rubber hand as one’s own, and an impact on the rubber hand is viscerally felt by the subject. Even if the rubber hand illusion does not fully activate, joint expertise between the local touch explorer and the distant interpreter may facilitate the needed tactile understanding, and this understanding may be enhanced through practice. The project team will explore configurations (e.g., relative positions of the distal toucher’s hands to the subject’s hidden hands, degree of and type of movement/tactile exploration, visual/augmented reality presentation) that allow body transfer illusions or joint expertise to enable interpretation. The team will also conduct a series of video-based grounded theory explorations of physicians performing physical examinations to gain a better understanding of the process and to categorize specific actions that may serve as the basis of communication between the local caregiver and the distal physician, and to inform the technology developments necessary to enable this collaborative examination. Finally, the project team, comprising physicians and nurses, mechanical engineers, human-computer interaction researchers, and perceptual psychologists, will collaborate on the research described, engaging in team development exercises to gain a stronger shared understanding of the joint physical examination process and support future research and development of these practices and technologies.<br/><br/>This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.