| Language is a key component of human communication. In understanding and producing language different forms of information have to be retrieved from memory and integrated into well-formed utterances. These sources of information include knowledge about word forms, word meanings, word class (e.g., nouns, verbs, etc), as well as principles of combining individual words in longer utterances, prosody, etc. In recent decades a substantial amount of knowledge has been accumulated on the cognitive organization of the language system, and, to a lesser degree, on the underlying neural organization (for a recent overview, see Brown & Hagoort, 1999). However, in ordinary conversation the speaker and listener have to integrate the speech signal with information from the visual modality. A striking example of this integration are the gestures that accompany speech. The cross-modal integration of the auditory speech signal and the gestures is an interesting example of how the brain solves the binding problem across modalities. (a) The aim of this project is to investigate the multimodal integration of auditory and visual information during the simultaneous comprehension of speech and gesture. Detailed information about the interaction between speech and gesture comprehension not only reveals central aspects of human communication, but will also further our understanding of information binding in the brain across input modalities. (b) The project will address questions related to the time course of the integration of speech and gestures with the help of electrophysiological recordings (ERPs). In addition, fMRI will be used to determine the brain areas that are crucial for the multimodal integration of speech and gestures. The central questions are: (i) What is the nature and time course of speech and gesture integration when both speech and gesture are in the focus of attention of the listener? (ii) Do the characteristics of integration change in a more naturalistic setting when gestures are not necessarily in the focus of attention? (iii) Do gestures have a special status among visuo-spatial information when it comes to cross-modal integration with speech? (iv) What are the crucial brain areas for the integration of speech and gesture? (c) This proposal is highly innovative, since it will be the first investigation of the neural organization of the integration between speech and gesture. (d) The proposal is relevant for the NWO COGNITION programme since it fits its themes "from neuron to cognition" and "communication and socially situated cognition." The project is cross-disciplinary involving the areas of linguistics, psychology, and cognitive neuroscience. It involves respected research institutions, namely the Max Planck Institute for Psycholinguistics and the F.C. Donders Centre for Cognitive Neuroimaging. |
