Sidney D'Mello - US grants

This research will explore interactions between cognition and emotion during the learning of scientific methods in the context of a computer tutoring environment. The primary focus will be on the relations between impasses, cognitive disequilibrium, and the affective-cognitive state of confusion. Confusion correlates with learning gains because it is diagnostic of cognitive disequilibrium, a state that occurs when learners face obstacles to goals, contradictions, incongruities, anomalies, conflicts, and system breakdowns. Cognitive equilibrium is normally restored after thought, reflection, problem solving and other effortful cognitive activities. Therefore, pedagogical tactics that challenge, perplex, and productively confuse learners are stimulating alternatives to the typical information delivery systems in education that promote shallow knowledge in the comfort zone of the learner, but rarely deep comprehension. This research will develop tutorial interventions that induce, track, and regulate confusion and cognitive disequilibrium in the minds of learners, as well as the cognitive and emotional mechanisms that restore cognitive equilibrium. The research has three specific objectives: (1) To promote deep learning by developing tutorial interventions that experimentally induce impasses, cognitive disequilibrium, and the resulting confusion; (2) to integrate sensing devices and signal processing algorithms that detect and track the associated confusion; and (3) to develop affect-sensitive pedagogical strategies to help learners regulate their confusion. The three objectives will be accomplished by augmenting an existing Intelligent Tutoring System (ARIES, Acquiring Research Investigative and Evaluative Skills) with technologies that automate assessment of emotion and cognition, as well as an intelligent handling of emotions. State-of-the-art sensing devices detect relevant emotions during learning (confusion, frustration, boredom, flow/engagement, delight, surprise) on the basis of the dialogue history, facial expressions, and body posture. The ARIES system promotes scientific inquiry skills by presenting case studies that exhibit flawed scientific methods and that require learners to offer thoughtful critiques on the scientific merits of the studies. The critiques encourage (a) the general cognitive processes of drawing inferences, constructing causal models, identifying problems, and asking diagnostic questions and (b) skills that directly target scientific reasoning, such as stating hypotheses, identifying dependent and independent variables, isolating potential confounds in designs, interpreting trends in data, and determining whether data support predictions. Students interact with ARIES through conversational trialogues in natural language with two animated agents: a tutor agent and a peer agent. Cognitive disequilibrium is created when the agents produce messages with contradictions, conflicts, and clashes with what the student knows. Correct information eventually emerges in the trialogue, which restores cognitive equilibrium.

The broader significance and importance of the project is to advance science education, intelligent learning environments, and human-computer interfaces. It is widely acknowledged that the level of science understanding among students and adults in the United States needs improvement and does not compare favorably with several other nations. The proposed research will help fill this gap by developing technological interventions to fortify citizens and aspiring scientists with the skills needed for critical thinking, complex reasoning, and problem solving in science. The project will develop intelligent learning environments targeted for deeper learning, which is needed for a technologically sophisticated workforce, and for a motivating learning experience, which is expected in recent generations of students. The project will develop advanced sensing devices for detecting emotions and cognition, a contribution that should impact the fields of human-computer interaction, cognitive science, and the learning sciences.

The ability to concentrate on tasks is critical to learning. This project will develop attention-aware cyberlearning as a new genre of learning technologies that automatically detect and respond to students' attentional states. In particular, this project will implement technology that will detect mind wandering (MW) which is when attention shifts from task-related thoughts to task-unrelated thoughts. MW has been studied in the context of complex comprehension tasks and it has been found that a high degree of MW leads to inferior performance. However MW has not been studied in the context of learning with technology and technology solutions have not been proposed to reduce MW. This project addresses MW in the context of learning with technology. The detection of MW is through the use of inexpensive eye-tracking devices. The devices will be integrated with software to detect MW while students are engaged in learning high school biology through an interactive system called Guru. Once MW is detected, software strategies will be used to enable the students return to the learning task. The primary research will be in the development and testing of MW detection algorithms and in the development and testing of strategies to reduce MW.

In more detail, the attention-aware Guru will include an integrated eye tracker, an automated gaze-based MW detector, and intervention strategies to improve learning by mitigating the costs of MW. The research will be conducted in 9th grade biology classrooms in Northern Indiana, where the core technological components will be formatively studied, iteratively refined, and summatively evaluated. Generalizable insights will be identified at every stage of the project in order to promote transferability of the findings to future attention-aware technologies, thereby helping students learn to their fullest potential. In summary, the proposed attention-aware Guru technology will be used to advance fundamental research focused on uncovering: (1) the incidence of MW during learning with technology, (2) relationships between MW and learning, (3) patterns of eye-gaze that are diagnostic of MW, (4) automated intervention strategies to reorient attention and reduce the detrimental effects of MW, and (5) generalizable insights to catalyze future implementations of attention-aware cyberlearning.

Collaborative problem solving (CPS) is an essential skill in our increasingly connected and globalized world. Yet, there is a paucity of knowledge on how to define, measure, and develop this skill, especially in the context of STEM learning. A team of investigators from Notre Dame University, Florida State University, and Arizona State University will seek to discover how interpersonal interactions arise and influence CPS processes and outcomes in digital STEM learning environments. The research will focus on groups of high school and college students collaborating virtually within a STEM educational game called Physics Playground. The hypothesis is that CPS effectiveness can be improved by providing real-time automated feedback on the ongoing collaboration. The investigators will collect an array of data, ranging from individual physiological measures to learning outcome measures to group communication patterns. Their goal is to improve the design of future CPS learning environments, making them more enjoyable, engaging, and effective. The project is funded by the EHR Core Research (ECR) program, which supports work that advances the fundamental research literature on STEM learning.

The research will involve dyads and triads collaborating virtually within a learning environment called Physics Playground, a STEM educational game. Data will be collected in both the lab and classroom, using a diverse sample of high school and college students from three sites across the U.S. The team will integrate data from a range of sources including modeling of small group problem solving and collaborative learning and from the modeling of low-level data from eye tracking, facial feature tracking, psychophysiology, and linguistic/paralinguistic speech analysis. Additionally, the team will study the moderating effects of task constraints and group composition on CPS processes and outcomes. A further goal is to model dynamic CPS processes using nonlinear time series analyses and multimodal deep recurrent neural networks. The computational models will be integrated into the learning environment to test the hypothesis that CPS outcomes can be improved by providing automated feedback on unfolding collaborative processes. Thus, by blending basic experimental research, multisensor-multimodal analysis, computational modeling, and dynamic computerized intervention, the researchers will attempt to make foundational theoretical, methodological, and technological advances.

This project will use multiple sources of middle school classroom data to give feedback and assessment information to teachers so that their teaching ability is enhanced. The data includes anonymized student performance data (grades and standardized test results) and anonymized existing audio recordings of classroom discussions between students and teachers. The audio data will be used to analyze the student-teacher discussions for effectiveness of the student-teacher discussions in student learning. As the effectiveness measures are developed, feedback for instructional improvement will be provided to the teachers in a design cycle for continuous improvement. The technological innovations are in the analysis of the student-teacher discussions, in natural language understanding of student-teacher discussions, and in machine learning to classify effective from non-effective student-teacher discussions.

This project will advance cyber-enabled, teacher analytics as a new genre of technology that provides automated feedback on teacher performance with the goal of improving teaching effectiveness and student achievement. The exemplary implementation will autonomously analyze audio from real-world English and language arts classes for indicators of effective discourse to enable a new paradigm of datadriven reflective practice. The project emphasizes six theoretical dimensions of discourse linked to student achievement growth: goal clarity, disciplinary concepts, and strategy use for teacher-led discourse, and challenge, connection, and elaborated feedback for transactional discourse. The innovation aims to help teachers develop expertise on these dimensions and will be developed and tested in 9th grade classrooms in Western Pennsylvania. The team will first generate initial insights on how teacher discourse predicts student achievement via a re-analysis of large volumes (128 hours) of existing
classroom audio. Next, they will design and iteratively refine hardware/software interfaces for efficient,flexible, scalable audio data collection by teachers. The data will be used to computationally model dimensions of effective discourse by combining linguistic, discursive, acoustic, and contextual analysis of audio with supervised and semi-supervised deep recurrent neural networks. The model-based estimates will be incorporated into an interactive analytic/visualization platform to promote data-driven reflective practice. After refinement via design studies, the impact of the innovation on instructional improvement and student literacy outcomes will be evaluated in a randomized control trial. Finally, generalizable insights will be identified at every stage of the project to promote transferability to future cyber-enabled, teacher-analytics technologies.

The project, based at the University of Colorado, will advance fundamental knowledge on learning from complex STEM texts. This is a critical skill for success in an increasingly information-driven world and workforce, but it is also an area where students consistently struggle. Scores on standardized assessments are stubbornly stagnant, troublesome achievement gaps remain, and the U.S. continues to lag behind its international peers. This is especially relevant to the reading of complex STEM content. The main idea of the project is that learning from text is fundamentally about the coordination of vision (the eye) with thought (the mind) in a manner constrained by the content (the text), the context (the task), and the individual (the learner). Thus, there is a need to understand how these factors interact in order to advance basic knowledge and to inform effective interventions. The team will study how brain and behavioral signals can be used to understand the reading process and associated learning outcomes in a manner that is sensitive to individual differences. Anticipated near-term broader impacts include the interdisciplinary training of students, especially those from groups underrepresented in STEM fields, community outreach, and promoting scientific reasoning skills relevant to multiple STEM areas. In the longer term, the results of the project will inform the design of future interventions that aim to make learning from text more efficient, engaging, and effective. The project is funded by the EHR Core Research (ECR) program, which supports work that advances the fundamental research literature on STEM learning.

The intellectual merit of the project is to gain fundamental knowledge on how the eye and brain dynamically coordinate to construct mental representations during complex learning from STEM texts. The project will address this goal by collecting eye movements, reading behaviors, noninvasive brain signals, and learning outcomes from high-school and college students in controlled lab settings and in school environments. It is expected that students will improve critical thinking and scientific reasoning skills by engaging with multiple texts on scientific research methods. The project will use behavioral (eye tracking and reading times) and non-invasive brain measures (high-density functional near-infrared spectroscopy and electroencephalography) to identify neurobehavioral markers of core processes involved in reading comprehension (e.g., focused attention vs. mind wandering, inference generation and elaboration) and the extent to which these measures can predict learning outcomes (comprehension, knowledge, and transfer) assessed immediately after learning and a week later. The project will also leverage advances in deep machine learning to build computational neurobehavioral models of the process and outcome variables in a manner that is sensitive to textual difficulty and to individual differences (e.g., prior knowledge, interest). Thus, the project will advance fundamental knowledge by integrating three research areas: theories of low-level reading processes and associated models of eye movements, models of higher-order STEM text comprehension including multiple document comprehension, and emerging research on the cognitive neuroscience of reading including co-registration with eye movements.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

A central challenge of science, technology, engineering, and mathematics (STEM) learning is how to promote deep conceptual learning via rich socio-collaborative learning experiences. To meet this challenge, the Institute for Student-AI Teaming will reframe the role of AI (Artificial Intelligence) in education, moving towards a future where AI is viewed as a social, collaborative partner that helps students work and learn more effectively, engagingly, and equitably, while helping educators focus on what they do best: inspiring and teaching students. The Institute will develop, deploy, and study AI Partners that interact naturally with students and teachers through speech, gesture, gaze, and facial expression in real-world classrooms and remote learning settings. The AI partners will be designed in close collaboration with educators with the aim of supporting students to develop STEM competencies, disciplinary practices, and 21st century skills of collaborative problem solving and critical thinking. These AI Partners will observe, participate in, and support small groups of students to engage in deep and sustained learning conversations, while assisting teachers in orchestrating effective learning experiences at the individual, small group, and whole-class levels. The focal content domain will be AI education, thus contributing to growing a diverse workforce of future AI researchers and practitioners. The long-term impact of the Institute is to help realize the grand challenge of ?Education for All?, by leading the nation towards a future where all students ? especially those whose identities are underrepresented in STEM ? routinely participate in rich and rewarding AI-enabled collaborative learning experiences that scale across a large number of classrooms, resulting in deeper student engagement and persistence in STEM, more inclusive classroom cultures, and significant improvements in learning outcomes. The Institute will support workforce development though sustained engagement of diverse youth and national dissemination of AI-enabled curricula. Realizing this impact will require the Institute to develop foundational advances in AI technology, consistent with the National AI Research and Development plan which calls for significant advances in human-AI collaboration.

The Institute for Student-AI Teaming brings together a geographically distributed team of researchers from nine Universities (University of Colorado Boulder; Colorado State University; University of California, Santa Cruz; University of California, Berkeley; Brandeis University; Worcester Polytechnic Institute; Georgia Institute of Technology; University of Illinois at Urbana Champagne; University of Wisconsin-Madison) with partners from academia, K-12 school districts, and industry. The Institute will adopt responsible innovation and polycultural approaches for developing ethical AI technologies by integrating foundational and use-inspired AI research across more than 12 interdisciplinary research areas - spanning the computing, learning, cognitive and affective sciences. This research program will be conducted in close partnership with K12 educators, gender and racially diverse students, parents, and other community stakeholders to ensure that the resulting technologies address community needs and reflect community values. Efforts will grow a diverse workforce of future AI researchers and practitioners by engaging 5,000 middle/high school students in innovative AI education through AI-enabled pedagogies. The Institute?s research program will yield fundamental advances in multimodal processing, natural language understanding, affective computing, and knowledge representation to develop AI models that can autonomously monitor the unfolding learning discourse at multiple levels ? understanding the content, the conversational dynamics, gestures, and social signals? and learn to generate appropriate dialog moves to be effective partners in the learning conversations. It will advance the new science of student-AI teaming in the form of theoretical frameworks, innovative interaction paradigms that consider the cognitive, affective, and social dimensions of the interaction, and novel human-AI architectures for orchestrating effective student and teacher interactions with AI learning partners. The Institute will advance the science of inclusive co-design by developing methodologies to empower diverse stakeholders to envision, co-create, and critique AI technologies for their schools and communities. These advances will be achieved through a highly interdisciplinary research methodology blending algorithm development, user-centered design, neurophysiological modeling, design-based research, curriculum co-design, and classroom field trials. Research products, including curricula and software, will be openly and broadly disseminated, including sharing of data after establishing consent, data security, and privacy-protection. To promote collaboration and knowledge transfer, a dedicated Community Hub will provide services to integrate participants and partner organizations and will coordinate opening up the Institute to the broader community. The Institute will be evaluated along metrics of effectiveness, equity, efficiency, generalizability, and impact, both internally and via an external advisory board. Overall, the Institute will serve as a national nexus point for empowering diverse stakeholders to engage in responsible co-design of student-AI collaborative technologies.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

The ability to effectively work as a team is essential to meet the demands of the modern world and workforce. However, the COVID-19 crisis has drastically changed how teams collaborate, including periods of extended remote work, mixed remote and in-person teams, blurred home and work boundaries, elevated stress and anxiety, and extreme uncertainty about the future. The swift onset of the crisis required individuals, teams, and organizations to abruptly adapt to rapidly changing circumstances with little to no preparation. The proposed research will investigate disruptions to teamwork and how teams adapt during the COVID-19 crisis and in the ensuing recovery period. The project will investigate 30 real-world teams over a three-month period while in the midst of the crisis and for an additional one-month follow-up as events unfold. The goal is to understand how teams respond to changing contexts, how teams support each other, how conflict is managed, and how teams develop, adapt, and sustain the rhythms of teamwork during COVID-19 and in the ensuing recovery period. This foundational research will be essential to help organizations establish team structures and collaborative processes that enable them to more successfully address disruptions in the current and in future crises. The project will provide unique opportunities for interdisciplinary training of students in computer science and psychology, will broaden participation by recruiting diverse students, and will share a rich and unique dataset with the broad scientific community.

The specific aims are to: (1) understand states, processes, and behaviors (e.g., team cohesion, communication patterns, collective stress) of teams during the COVID-19 crisis with a focus on factors associated with team performance; (2) investigate how individuals and teams experience and adapt to major COVID-related life events, such as school closings, enforcing of social distancing, budget cuts, illnesses, and so on; and (3) identify patterns in team states and behaviors over time, detect disruptions to these patterns, and study how new patterns emerge during the crisis and in the ensuing period of recovery. The project will use wearable sensors to track heart rate, sleep, physical activity, and relative location (home or away), communication tools (e.g., team calendars, email metadata), ecological momentary assessments (EMAs), validated survey instruments, and semi-structured interviews to investigate team states, team processes, team behaviors, and team performances in context and over time. The findings will contribute basic knowledge on teaming under the unique context of COVID-19, what factors are associated with team performance, and whether changes teams adopt are temporary or permanent.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.