| This research proposal is concerned with a functional approach to verbal and visual prosody in spoken conversations. The problem to be addressed in the project is about the combined use of specific auditive cues (such as intonation, tempo, voice quality and pausing) and specific visual cues (such as facial expressions and specific body gestures) for marking different dialogue phenomena. First, we will explore how audiovisual prosody can be exploited to highlight the information status of words. Then, we will investigate how it can be used to signal whether or not the process of information exchange in a dialogue is going well. Next, we will explore how it can support the turn-taking mechanism in spontaneous interactions. Finally, we will see to what extent audiovisual prosody may reflect speakers emotions and attitudes. The results of these different substudies will be integrated in one coherent, functional model of audiovisual prosody. All the questions will be tackled from the point of view of both the speaker and the listener, and from a crosslinguistic perspective. Insight into functional aspects of audiovisual prosody is relevant from both a theoretical and applied perspective. First, it is remarkable to observe that this important communicative device is still largely unexplored. Knowledge about how audiovisual prosody works may yield new insights into how people mark important words, deixis, turn-taking, discourse structure, etc. and more general into how languages can differ in the way they signal linguistic and paralinguistic phenomena. Second, there is an increasing interest in computer interfaces that rely on what is termed embodied conversational agents , i.e., specific software components that appear to users as animated characters. To make these agents believable and communicative , it is important to know in full detail how specific auditive and visual parameters contribute to speech communication. (Keywords: audiovisual prosody, dialogue analysis, speech technology, embodied conversational agents). |
