Daniel Yurovsky - US grants

DESCRIPTION (provided by applicant): The infant's world is filled with objects; objects with unknown names. But learning the names for these objects - mapping auditory word forms onto objects in visual scenes - requires young learners to contend with significant uncertainty. When the infant's caretaker produces a word, the child must determine to which object, if any, it refers The solution is straightforward for adults and for older children: they consult the social cues produced by the speaker (e.g. pointing, gaze) to determine the target object. However, the way that infants solve this referential uncertainty problem, and the way that they learn to use social cues to reference, remains the topic of significant debate. The proposed project will produce insight into both of these questions through three integrated lines of research. Because competing theories about the origins and development of the use of social cues have used very different experimental methodology, their results have proven to be difficult to integrate. Thus, Specific Aim 1 is to measure infants' use of cues to reference, in a single paradigm, over a broad range of early development. Experiments 1 and 2 use eye-tracking methods to measure how infants learn words from different cues to reference (salience, eye-gaze, pointing, and action), both alone and together, from 9-months to 24-months. This data will let us determine both the ages at which infants begin to use these cues, and also the trajectories they follow to become mature users of social cues. Second, we will use head-mounted cameras to record natural naming events to children, collecting data about the events from which infants could learn about social cues to reference. These data will allow us to determine the extent to which trajectories of cue use can be explained by their presence and predictive validity in children's language input. In conjunction with this data, we will use a set of computational models to address Specific Aim 2: to understand how children learn to use social cues to learn language from social partners. These three integrated approaches will advance our understanding of the basic learning mechanisms used by young children in order to learn language. Together, they will also inform the construction of tools for assessment and diagnosis of language learning in young children. Further, modeling the normative mechanisms by which children learn to use social information for language learning can point to potential interventions for children with difficulties in this aspect of language acquisition. This is likely to be of particular relevance t children with autism spectrum disorder.

As natural language systems become ubiquitous (e.g. phone trees, chatbots, and smart homes) they must learn to adapt to users by modeling them each as individuals with different abilities, knowledge, and tastes. Theory of mind is the human ability to reason about the hidden mental states of others, but is a complex phenomenon that does not emerge in children until late in their development compared to other more basic communicative skills. The questions of interest to this EAGER project are: (1) what makes this skill hard for children to learn, (2) what can computers learn from how children are taught, and (3) in what ways can machine learning models provide insight into human development. This project sits at this intersection of machine learning, developmental psychology, and pedagogy.

This project includes formal models of information sharing and teaching grounded in shared referential games. Agents and children are tasked with asking an instructor to efficiently distinguish similar objects -- a task which requires understanding common ground and identifying distinguishing features. While the learner will often make ambiguous statements, the teacher will provide corrections and instruction to guide the learning process. This formulation allows for variation along several dimensions of relevance to successful communication: working memory, visual and lexical complexity, and specificity of instruction. Experiments with children will provide benchmarks against which computational agents can be compared, and experiments with agents will allow us to decompose the contribution of each of these factors to the difficulty of developing a theory of mind.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.