Hostname: page-component-586b7cd67f-rdxmf Total loading time: 0 Render date: 2024-11-23T15:32:57.389Z Has data issue: false hasContentIssue false
  • English
  • Français

Gesturing integrates top-down and bottom-up information: Joint effects of speakers' expectations and addressees' feedback

Published online by Cambridge University Press:  11 March 2014

Anna K. Kuhlen ,
Alexia Galati  and
Susan E. Brennano
Anna K. Kuhlen*
Affiliation:
Humboldt University of Berlin, Germany
Alexia Galati
Affiliation:
University of Cyprus, Cyprus
Susan E. Brennano*
Affiliation:
Stony Brook University, USA
*
Correspondence addresses: Anna Kuhlen, Berlin School of Mind and Brain, Humboldt University of Berlin, 10117 Berlin, Germany. E-mail: [email protected]
Correspondence addresses: Anna Kuhlen, Berlin School of Mind and Brain, Humboldt University of Berlin, 10117 Berlin, Germany. E-mail: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Speakers adapt their speech based on both prior expectations and incoming cues about their addressees' informational needs (Kuhlen and Brennan 2010). Here, we investigate whether top-down information, such as speakers' expectations about addressees' attentiveness, and bottom-up cues, such as addressees' feedback during conversation, also influence speakers' gestures. In 39 dyads, addressees were either attentive when speakers told a joke or else distracted by a second task, while speakers expected addressees to be either attentive or distracted. Independently of adjustments in speech, both speakers' expectations and addressees' feedback shaped quantitative and qualitative aspects of gesturing. Speakers gestured more frequently when their prior expectations matched addressees' actual behavior. Moreover, speakers with attentive addressees gestured more in the periphery of gesture space when they expected addressees to be attentive. These systematic adjustments in gesturing suggest that speakers flexibly adapt to their addressees by integrating bottom-up cues available during the interaction in light of attributions made from top-down expectations. That these sources of information lead to adjustments patterning similarly in speech and gesture informs theoretical frameworks of how different modalities are deployed and coordinated in dialogue.

Keywords

speech-accompanying gesturescommunicationaudience designspontaneous spoken dialoguefeedback cuesdistracted addressees
Type
Research Article
Information
Language and Cognition , Volume 4 , Issue 1 , March 2012 , pp. 17 - 41
Copyright
Copyright © UK Cognitive Linguistics Association 2012

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Alibali, M. W., Heath, D. C. & Myers, H. J.. 2001. Effects of visibility between speaker and listener on gesture production: Some gestures are meant to be seen. Journal of Memory and Language 44. 160–188.CrossRefGoogle Scholar
Alibali, M. W., Kita, S. & Young, A. J.. 2000. Gesture and the process of speech production: We think, therefore we gesture. Language and Cognitive Processes 15. 593–613.Google Scholar
Argyle, M. & Cook, M.. 1976. Gaze and mutual gaze. Cambridge: Cambridge University Press.Google Scholar
Bavelas, J. B., Chovil, N., Lawrie, D. A. & Wade, A.. 1992. Interactive gestures. Discourse Processes 15. 469–489.Google Scholar
Bavelas, J., Coates, L. & Johnson, T.. 2000. Listeners as co-narrators. Journal of Personality and Social Psychology 79. 941–952.CrossRefGoogle ScholarPubMed
Bavelas, J. B., Gerwing, J., Sutton, C. & Prevost, D.. 2008. Gesturing on the telephone: Independent effects of dialogue and visibility. Journal of Memory and Language 58. 495–520.Google Scholar
Brennan, S. E. 1991. Conversation with and through computers. User Modeling and User-Adapted Interaction 1. 67–86.Google Scholar
Brennan, S. E. 2005. How conversation is shaped by visual and spoken evidence. In Trueswell, J. & Tanenhaus, M. (eds.), Approaches to studying world-situated language use: Bridging the language-as-product and language-action traditions, 95–129. Cambridge, MA: MIT Press.Google Scholar
Brennan, S. E., Chen, Z., Dickinson, C. A., Neider, M. B. & Zelinsky, G. J.. 2008. Coordinating cognition: The costs and benefits of shared gaze during collaborative search. Cognition 106. 1465–1477.CrossRefGoogle ScholarPubMed
Brennan, S. E., Galati, A. & Kuhlen, A. K.. 2010. Two minds, one dialog: Coordinating speaking and understanding. In Ross, B. (ed.), The psychology of learning and motivation (Vol. 51), 301–344. Burlington: Academic Press.Google Scholar
Brugman, H. & Russell, A.. 2004. Annotating multimedia/multi-modal resources with ELAN. Paper presented at the LREC 2004, Fourth International Conference on Language Resources and Evaluation, Lisbon, Portugal.Google Scholar
Church, R. B. & Goldin-Meadow, S.. 1986. The mismatch between gesture and speech as an index of transitional knowledge. Cognition 23. 43–71.Google Scholar
Clark, H. H. & Brennan, S. E.. 1991. Grounding in communication. In Resnick, L. B., Levine, J. & Teasley, S. D. (eds.), Perspectives on socially shared cognition, 127–149. Washington, DC: APA.CrossRefGoogle Scholar
Clark, H. H. & Wilkes-Gibbs, D.. 1986. Referring as a collaborative process. Cognition 22. 1–39.Google Scholar
Cohen, L. 1968. Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin 70. 213–220.Google Scholar
de Jaegher, H., de Paolo, E. & Gallagher, S.. 2010. Can social interaction constitute social cognition? Trends in Cognitive Sciences 14. 441–447.CrossRefGoogle ScholarPubMed
de Ruiter, J. P. A. 2000. The production of gesture and speech. In McNeill, D. (ed.), Language and gesture: Window into thought and action, 284–311. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
Gerwing, J. & Bavelas, J.. 2004. Linguistic influences on gesture's form. Gesture 4. 157–195.Google Scholar
Goodwin, C. 1979. The interactive construction of a sentence in natural conversation. In Psathos, G. (ed.), Everyday language. Studies in ethnomethodology, 97–121. New York: Irvington.Google Scholar
Gullberg, M. & Holmqvist, K.. 1999. Keeping an eye on gestures: Visual perception of gestures in face-to-face communication. Pragmatics and Cognition 7. 35–63.Google Scholar
Gullberg, M. & Kita, S.. 2009. Attention to speech-accompanying gestures: Eye movements and information uptake. Journal of Nonverbal Behavior 33. 251–277.CrossRefGoogle ScholarPubMed
Hanna, J. E. & Brennan, S. E.. 2007. Speakers' eye gaze disambiguates referring expressions early during face-to-face conversation. Journal of Memory and Language 57. 596–615.Google Scholar
Holler, J. & Stevens, R.. 2007. The effect of common ground on how speakers use gesture and speech to represent size information. Journal of Language and Social Psychology 26. 4–27.Google Scholar
Holler, J. & Wilkin, K.. 2009. Communicating common ground: How mutually shared knowledge influences speech and gesture in a narrative task. Language and Cognitive Processes 24. 267–289.Google Scholar
Jacobs, N. & Garnham, A.. 2007. The role of conversational hand gestures in a narrative task. Journal of Memory and Language 56. 291–303.Google Scholar
Kendon, A. 1967. Some functions of gaze-direction in social interaction. Acta Psychologica 26. 22–63.Google Scholar
Kraljic, T., Brennan, S. E. & Samuel, A. G.. 2008. Accommodating variation: Dialects, idiolects, and speech processing. Cognition 107. 54–81.Google Scholar
Kraljic, T., Samuel, A. G. & Brennan, S. E.. 2008. First impressions and last resorts: How listeners adjust to speaker variability. Psychological Science 19. 332–338.Google Scholar
Krauss, R. M., Chen, Y. & Gottesman, R.. 2000. Lexical gestures and lexical access: A process model. In McNeill, D. (ed.), Language and Gesture, 261–283. Cambridge: Cambridge University Press.Google Scholar
Kraut, R. E., Lewis, S. H. & Swezey, L. W.. 1982. Listener responsiveness and the coordination of conversation. Journal of Personality and Social Psychology 43. 718–731.CrossRefGoogle Scholar
Kuhlen, A. K. & Brennan, S. E.. 2010. Anticipating distracted addressees: How speakers' expectations and addressees' feedback influence storytelling. Discourse Processes 47. 567–587.Google Scholar
Landis, J. R. & Koch, G. G.. 1977. The measurement of observer agreement for categorical data. Biometrics 33. 159–174.Google Scholar
Leys, C. & Schumann, S.. 2010. A nonparametric method to analyze interactions: The adjusted rank transform test. Journal of Experimental Social Psychology 46. 684–688.CrossRefGoogle Scholar
Liszkowski, U., Albrecht, K., Carpenter, M., M. & Tomasello, M.. 2008. Infants' visual and auditory communication when a partner is or is not visually attending. Infant Behavior and Development 31. 157–167.Google Scholar
McNeill, D. 1992. Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press.Google Scholar
McNeill, D. 2000. Catchments and context: non-modular factors in speech and gesture production. In McNeill, D. (ed.), Language and gesture, 312–328. Cambridge: Cambridge University Press.Google Scholar
McNeill, D. & Duncan, S.. 2000. Growth Points in thinking-for-speaking. In McNeill, D. (ed.), Language and gesture, 141–161. Cambridge: Cambridge University Press.Google Scholar
Özyürek, A. 2000. The influence of addressee location on spatial language and representational gestures of direction. In McNeill, D. (ed.), Language and gesture, 64–83. Cambridge: Cambridge University Press.Google Scholar
Özyürek, A. 2002. Do speakers design their cospeech gestures for their addressees? The effects of addressee location on representational gestures. Journal of Memory and Language 46. 688–704.Google Scholar
Richardson, D. C., Dale, R. & Tomlinson, J. M.. 2009. Conversation, gaze coordination, and beliefs about visual context. Cognitive Science 33. 1468–1482.Google Scholar
Russell, A. W. & Schober, M. F.. 1999. How beliefs about a partner's goal affect referring in goal-discrepant conversations. Discourse Processes 27. 1–33.CrossRefGoogle Scholar
Seyfeddinipur, M. 2006. Disfluency: Interrupting speech and gesture. (MPI Series in Psycholinguistics, 39). Nijmegen, NL: Max Planck Institute of Psycholinguistics.Google Scholar
Shockley, K., Richardson, D. C. & Dale, R.. 2009. Conversation and coordinative structures. Topics in Cognitive Science 1. 305–319.Google Scholar
Teufel, C., Alexis, D. M., Todd, H., Lawrance-Owen, A. J., Clayton, N. S. & Davis, G.. 2009. Social cognition modulates the sensory coding of observed gaze direction. Current Biology 19. 1274–1277.Google Scholar
Teufel, C., Fletcher, P. C. & Davis, G.. 2010. Seeing other minds: Attributed mental states influence perception. Trends in Cognitive Science 14. 376–382.Google Scholar
Wilkes-Gibbs, D. 1986. Individual goals and collaborative actions: Conversations as collective behavior. Stanford, CA: Stanford University dissertation.Google Scholar
Yngve, V. H. 1970. On getting a word in edgewise. Papers from the 6th Regional Meeting of the Chicago Linguistic Society, 567–578. Chicago, IL: Chicago Linguistic Institute.Google Scholar