Shari Speer - US grants

9818341
Brown, Frank M.
Agah, Arvin
University of Kansas

Ambiguity Resolution for Intelligent Systems Using a Cognitive Robot

This research instrumentation contributes to the acquisition, by the Departments of Electrical Engineering and Computer Science, Psychology, and Speech Language and Hearing, of a Cognitive Robot (base system, arm and hand manipulator, binocular color vision, laser ranger, sound reception and interpretation, and sound generation) enabling research projects in:

- Ambiguity Resolution in Spoken Language Understanding,
- Ambiguity Resolution in Context Driven Active Robot Vision,
- Ambiguity Resolution within the Memory System, and
- Ambiguity Resolution in Robot Reasoning and Actuation.

The Cognitive Robot, with visual, speech, movement, and hand/arm manipulation capabilities, will enable research plausible interactions among the processes of ambiguity resolution in multiple cognitive subsystems including subsystems for speech and natural language understanding, scene interpretation, the retrieval of appropriate memory context, and robot planning and action execution. The research methodology is to integrate together computational models of these various ambiguity resolution processes of these subsystems into the Cognitive Robot and to test them in an integrated fashion by observing the behavior of the Cognitive Robot. The significance of this research lies in the fact that ambiguity in one subsystem can often only be resolved by the ambiguity resolution capabilities of the other subsystems. Thus, the study of ambiguity resolution in each of the four research projects using the Cognitive Robot will be enhanced by its integration with the ambiguity resolution capabilities of the others. Since the human capability for effortlessly resolving ambiguity seems to be a fundamental aspect of intelligence, an explication of the interactions among the ambiguity resolution capabilities of these subsystems may be highly significant.

This research is designed to demonstrate that naive speakers in a natural conversational situation reliably use prosody (loosely, the 'melody in speech') to structure the information they provide to listeners, and that listeners in turn make use of this prosodic structure to facilitate accurate recovery of the speaker's intended meaning. The broad objective of the research is to develop a psycholinguistic model of sentence comprehension in which prosody plays a fundamental role, often determining the syntactic and semantic interpretation assigned to a sentence. Participants in these experiments will play a cooperative board game that elicits semi-spontaneous productions of syntactically ambiguous sentences. The game involves two speakers, each with a game board the other cannot see, who must use to a limited set of scripted sentences to determine and communicate joint moves. Key utterances contain syntactic ambiguities such as the ambiguous prepositional phrase attachment in "Move the square with the triangle," (meaning either 'push the square with the triangle' or 'move the house-shaped piece.') To evaluate the production of prosody, players' speech will be recorded digitally and submitted to phonetic and phonological analysis. To evaluate the use of prosody in comprehension, the timing and fixation locations of players eyemovements will be recorded as they listen to and comprehend the conversation and implement the game moves. The experiments will use head-mounted eye tracking, a relatively new technology recently shown to be sensitive to prosodic and syntactic factors in experiments with isolated spoken sentences that refer to objects in the listeners' visual scene. The apparatus is ideal because it does not restrict movement, does not introduce an additional experimental task, and does not interrupt speaking or listening.

There are two novel aspects to this research. The first is that the game task focuses the participants' attention on the act of communication rather than on the process of reading, and can provide a more accurate picture of the use prosody in natural conversation between naive speakers. Recent work in language production has argued that speakers use prosody to disambiguate syntax only when they are aware of the ambiguity and are given explicit instructions, implying that although listeners can use prosody, they may have few opportunities to do so. However, speakers in these studies read sentences from notecards. Because the pragmatic goals of a reader differ dramatically from those of speakers in typical conversation, such tasks may misrepresent the nature and extent of prosodic disambiguation. The second is that the combination of the game task with head-mounted eyetracking technology will develop a much-needed novel method that will provide a temporally-precise, on-line measure of comprehension that can be used while comprehension is occurring as part of semi-spontaneous spoken discourse. This will allow Speer to investigate the issues concerning the relative influence of prosodic, syntactic, and situationally-determined discourse variables over the time course of sentence comprehension, which form the core of current work in modeling human language processing.

This POWRE grant will allow Speer to purchase a head-mounted system, providing her the opportunity to utilize this new methodology to which she currently has no access, and to conduct exploratory work necessary to begin a new line of inquiry. The availability of such methodology will greatly enhance her ability to provide empirical data relevant to important theoretical questions at the most basic level of her research program and thus will be highly important to the advancement of her career.

When people engage each other in conversation, a substantial part of their message is conveyed by prosody, or 'the melody of speech.' Prosody includes variation in the rate at which we speak, the volume of our voices, and the rhythm and tune of our phrases. For example, prosody can change the string of speech sounds that make up the two English words 'triceratops oil' into three quite different words 'Try, Sarah, topsoil,' or even four words 'Trice, air, atop, soil.' With National Science Foundation support, Dr. Shari Speer and Dr. Kiwako Ito will explore the way speakers and listeners use prosody to communicate in spontaneous dialogues in English and Japanese. The cross-linguistic comparison addresses the question of whether prosody has the same kind of communicative function in languages that are melodically very different. What aspects of the melodies are most important for language understanding, and are these common across languages? Do speakers pronounce certain melodies because they will be helpful for listeners, or do they focus primarily on their own interpretation of a message? These questions are motivated by a more basic one: What is universal about prosody in human conversation, and how does it reflect the structure of cognitive function during language use?

Language plays an important role in the way individuals express their social identities. In addition, listeners are sensitive to variation in pronunciation and employ their knowledge of language cues in order to recognize familiar voices and identify unknown talkers as members of particular social groups within society. This research project explores listeners' abilities to tune their speech perception systems according to their expectations of talkers' social identities. The socially marked pronunciation variant investigated in this study is the process of liaison in spoken French. Liaison consonants occur in word final position, and while they are never pronounced when the following word begins with a consonant, they can be pronounced when the following word begins with a vowel. A series of experiments will test whether French listeners internalize the variation in liaison use according to talker age (i.e., older talkers produce more liaison) and talker social class (i.e., talkers of higher social classes produce more liaison), and test whether French listeners use this information in speech perception. Participants will listen to short phrases which either do or do not contain liaison consonants. Analyses will include a calculation of the effect which the presence or absence of the liaison consonant has on word recognition of the following vowel initial word. The phrases will consist of spontaneous speech produced by various French speakers. Before listening to a talker's phrases, participants will be presented with information about the talker's age, social class, or both. Although all participants will listen to identical recordings, they will receive different information about the talkers' social identities. For example, some participants will be told that a given talker is a member of a higher social class, while other participants will be told that the same talker is a member of a lower social class. The expected outcome is that the presence or absence of liaison consonants will affect word recognition of the following vowel initial word according to the talker information the participants received. In other words, participants will tune their speech perception systems according to their expectations of the talkers' social identities.

The intellectual merit of this research lies primarily in its contribution to speech perception research. By testing the effects of social expectations on word recognition, this study effectively bridges the fields of psycholinguistics and sociolinguistics. The broader impacts of this project are many, as we live in a world in which we regularly come into contact with speakers of different accents, dialects, and languages, and in which we are challenged daily by social stereotypes. We can all benefit from learning about how socially marked pronunciation variants affect our perception. Participants in this study will be impacted directly through the debriefing process, while others will benefit from the dissemination of the research results. For instance, results from this study will inform automatic speech recognition research, aid French language instructors teaching about the liaison process, and inspire training programs which teach individuals about the impacts of language stereotypes on speech perception.

[unreadable] DESCRIPTION (provided by applicant): Spoken dialogue is 1 of the most basic forms of human language, ubiquitous across cultures. It differs in form from written language and read speech, with different vocabulary usage, different syntax, and an expanded use of visually available context. Most importantly, speakers and hearers of dialogue must rely on intonation, or the 'melody in speech.' When we talk, we manipulate the pitch, rate, phrasing, and volume of our speech. Patterns of intonation across a conversation can indicate complex discourse information not available from the words alone, such as what is already known by both speakers, what is newly introduced to 1 or both of them, which information they are finished discussing and what is still to be talked about. Although intonational structuring of discourse information is reported for numerous languages, a theory of the general cognitive mechanism underlying the universal use of intonation has yet to be established. The cross-linguistic research proposed here is crucial for the development of a general theory of intonation use in human language processing. The focus on analyses of unscripted conversational speech provides the most accurate information available about basic human language performance. Studying spontaneous speech has been considered an intractable problem, because it is hard to predict the specific words and sentence structures a speaker will use. We have piloted novel methodology that allows collection of multiple tokens of like utterances from the same speaker in varying intonational conditions. To understand how listeners respond in conversation, we use head-mounted eye-movement monitoring, an immediate, implicit measure of comprehension that allows the listener to speak and move while looking at the objects described by a conversational partner. The comparisons of English and Japanese, 2 languages that differ substantially in their syntax and intonation, test whether intonation is used differently in a language that provides melodic cues more or less reliably, and in different physical forms. Understanding how consistently intonation marks the information status of words and whether intonational cues facilitate listeners' comprehension of messages is important, not only for theories of language processing and development, but also for accurate speech identification and generation systems in artificial intelligence, and for the development of effective diagnoses and therapies for aphasic patients and others with communication loss. [unreadable] [unreadable] [unreadable]

When two people engage in dialogue, a substantial part of their message is conveyed by its intonation. Even when interlocutors are not visually co-present for conversation, listeners have access to a great deal of information beyond the spoken strings of consonants and vowels, including variation in rhythm, tune, tempo, loudness, tenseness and tone of voice. Intonation provides an organizational structure for speech, and can convey simultaneously a speaker's attitude, utterance purpose, and the relative importance of particular words or phrases. Intonation can mark the difference between immediately relevant vs. background information, express contrast, contradiction, and correction; and indicate the intended syntax of ambiguous utterances. Basic research into the relationship between intonation and speakers/hearers' intentions about syntax and information structure addresses whether, when, and how speakers use intonational information to signal linguistic and paralinguistic meaning as well as whether, how and when listeners use this information to recover meaning.

With National Science Foundation support, Dr. Shari Speer and Dr. Kiwako Ito will explore the way speakers and listeners use intonation to communicate information structure in spontaneous dialogues in English and Japanese. The cross-linguistic comparison addresses the question of whether prosody has the same kind of communicative function in languages that are syntactically and melodically very different. What aspects of intonation are most important for language understanding, and are these common across languages? Do speakers pronounce certain tunes because they will be helpful for listeners, or do they focus primarily on their own interpretation of a message? These questions are motivated by a more basic one: What is universal about intonation in human conversation, and how does it reflect the structure of cognitive function during language use?

Understanding how humans employ pragmatic principles to facilitate communication is an important element in understanding human interaction. Research in experimental pragmatics has shown marked recent theoretical progress. Current work successfully models not only contextual influences from linguistic structure and visually available context, but also factors such as the interlocutors involved, the world knowledge available to them as individuals and as they interact, and their mutually sustained beliefs and intentions. Recent progress in the field can be linked to researchers' ability to leverage novel methods, such as mining multi-interlocutor text interactions in social media, web-crawling language produced and understood via electronic means such as texting, internet blogs and other web-based platforms, and collecting data via Mechanical Turk. These approaches have expanded the available testable hypotheses as well as the diversity and representativeness of the language data under study. Pragmatics theorists have also been particularly open to incorporating insights from experimentally-generated results into developing theories of language interpretation. The 27th Annual CUNY Human Sentence Processing Conference, to be held in March 2014 at The Ohio State University in Columbus, Ohio, will include a Special Session on Experimental Pragmatics that aims to take stock of recent progress in this burgeoning sub-field and help to set the agenda for future research in this domain.

The core of the special session is a series of invited talks by six prominent researchers, diverse with respect to gender and academic seniority, with relevant expertise. Together, the speakers represent a wide range of theoretical and methodological approaches (corpus linguistics as well as experimental studies; language acquisition as well as adult processing; speech perception as well as the processing of quantifiers and anaphora). The special session promises to provide a fuller understanding of the operation of pragmatic influences and their interaction with other aspects of linguistic competence and performance that we believe will lead to a more realistic and representative picture of how the human mind represents and processes language.