Hostname: page-component-6d856f89d9-xkcpr Total loading time: 0 Render date: 2024-07-16T06:08:00.642Z Has data issue: false hasContentIssue false
  • English
  • Français

Knowing ‘who she is’ based on ‘where she is’: The effect of co-speech gesture on pronoun comprehension

Published online by Cambridge University Press:  11 March 2014

Whitney Goodrich Smith  and
Carla L. Hudson Kam
Whitney Goodrich Smith*
Affiliation:
University of Maryland
Carla L. Hudson Kam
Affiliation:
University of British Columbia
*
Correspondence addresses: Whitney Goodrich Smith, University of Maryland, College Park, Department of Hearing and Speech Sciences, Room 0100 Lefrak Hall, College Park, MD 20742, USA. E-mail: [email protected].
Get access Rights & Permissions [Opens in a new window]

Abstract

We examine whether pronoun interpretation is affected by naturalistic co-speech gesture. Participants in three conditions watched narrations containing ambiguous pronouns. In one condition the narrator produced gestures consistent with order-of-mention; in another, they conflicted with order-of-mention; and in the third, she did not gesture. Results showed that when the gestures conflicted with order-of-mention participants were much less likely to interpret the pronoun as referring to the first-mentioned character. In a second experiment we ruled out the possibility that participants were simply picking up on differences within the speech itself. These results extend previous work on gesture and language processing by showing that the information in gesture can influence the way people interpret words which by their nature are ambiguous, and that this influence is similar to that of well-known speech internal cues.

Keywords

gesturelanguage comprehensionpronoun resolutionreferent processing
Type
Research Article
Information
Language and Cognition , Volume 4 , Issue 2 , June 2012 , pp. 75 - 98
Copyright
Copyright © UK Cognitive Linguistics Association 2012

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Akmajian, A. & Jackendoff, R.. 1970. Coreferentiality and stress. Linguistic Inquiry 1. 124–126.Google Scholar
Alibali, M., Heath, D. & Meyers, H.. 2001. Effects of visibility between speaker and listener on gesture production: Some gestures are meant to be seen. Journal of Memory and Language 44. 169–188.CrossRefGoogle Scholar
Argyle, M. & Cook, M.. 1976. Gaze and mutual gaze. Cambridge: Cambridge University Press.Google Scholar
Arnold, J. E. 1998. Reference form and discourse patterns. Stanford, CA: Stanford University dissertation.Google Scholar
Arnold, J. E. 2001. The effect of thematic roles on pronoun use and frequency of reference continuation. Discourse Processes 31. 137–162.Google Scholar
Arnold, J. E. 2008. Reference production: Production-internal and Addressee-oriented Processes. Language and Cognitive Processes 23. 495–527.Google Scholar
Arnold, J. E. 2010. How speakers refer: the role of accessibility. Language and Linguistic Compass 4. 187–203.CrossRefGoogle Scholar
Arnold, J. E., Altmann, R., Fagnano, M. & Tanenhaus, M. K.. 2004. The Old and thee, uh, new. Psychological Science 9. 578–582.Google Scholar
Arnold, J. E., Brown-Schmidt, S. & Trueswell, J.. 2007. Children's use of gender and order-of-mention during pronoun comprehension. Language and Cognitive Processes 22. 527–565.Google Scholar
Arnold, J. E., Brown-Schmidt, S., Trueswell, J. & Fagnano, M.. 2004. Children's use of gender and order-of-mention during pronoun comprehension. In Trueswell, J. & Tanenhaus, M. (eds.), Approaches to studying world-situated language use. Bridging the language-as-product and language-as-action traditions, 261–282. Cambridge, MA: MIT Press.Google Scholar
Arnold, J. E., Eisenband, J., Brown-Schmidt, S. & Trueswell, J.. 2000. The rapid use of gender information: Evidence of the time course of pronoun resolution from eye tracking. Cognition 76. B13–B26.Google Scholar
Arnold, J. E., Hudson Kam, C. & Tanenhaus, M. K.. 2007. If you say thee uh- you're describing something hard: The on-line attribution of disfluency during reference comprehension. Journal of Experimental Psychology: Learning, Memory and Cognition 33. 914–930.Google Scholar
Bavelas, J., Chovil, N.Lawrie, D. & Wade, A.. 1992. Interactive gestures. Discourse Processes 15. 469–489.Google Scholar
Beattie, G. W. & Shovelton, H. K.. 1999. Do iconic hand gestures really contribute anything to the semantic information conveyed by speech? An experimental investigation. Semiotica 123. 1–2.Google Scholar
Brennan, S. E.& Clark, H. H.. 1996. Conceptual pacts and lexical choice in conversation. Journal of Experimental Psychology: Learning, Memory, and Cognition 22. 1482–1493.Google ScholarPubMed
Butterworth, B. & Hadar, U.. 1989. Gesture, speech, and computational stages: A reply to McNeill. Psychological Review 96. 168–174.Google Scholar
Campana, E., Silverman, L., Tanenhaus, M. K., Bennetto, L. & Packard, S.. 2005. Real-time integration of gesture and speech during reference resolution. Proceedings of the 27th Annual Meeting of the Cognitive Science Society. 378–383.Google Scholar
Cassell, J., McNeill, D. & McCullough, K. E.. 1999. Speech-gesture mismatches: Evidence for one underlying representation of linguistic and non-linguistic information. Pragmatics and Cognition 7. 1–33.Google Scholar
Chambers, C. G.& Smyth, R.. 1998. Structural parallelism and discourse coherence: A test of centering theory. Journal of Memory and Language 39. 593–608.Google Scholar
Clark, E. V. 2009. First language acquisition. Cambridge: Cambridge University Press.Google Scholar
Crawley, R. A., Stevenson, R. J. & Kleinman, D.. 1990. The use of heuristic strategies in the interpretation of pronouns. Journal of Psycholinguistic Research 194. 245–265.Google Scholar
Cutler, A., Dahan, D. & van Donsellar, W.. 1997. Prosody in the comprehension of spoken language: A literature review. Language and Speech 40. 141–201.Google Scholar
Dahan, D., Tanenhaus, M. K. & Chambers, C. G.. 2002. Accent and reference resolution in spoken-language comprehension. Journal of Memory and Language 47. 292–314.Google Scholar
Foraker, S. 2011. Gesture and discourse: How we use our hands to introduce and refer back. In Stam, G., Ishino, M. & Ashley, R. (eds.), Integrating gestures, 279–292. Amsterdam: John Benjamins.CrossRefGoogle Scholar
Foraker, S. & Goldin-Meadow, S.. 2007. Gesture and discourse: How we use our hands to refer back. Poster presented at 20th Annual CUNY Conference on Human Sentence Processing, San Diego, CA. 03.Google Scholar
Gernsbacher, M. A.& Hargreaves, D. J.. 1988. Accessing sentence participants: The advantage of first mention. Journal of Memory and Language 27. 699–717.Google Scholar
Goldin-Meadow, S. 2003. The resilience of language: What gesture creation in deaf children can tell us about how all children learn language. New York: Psychology Press.Google Scholar
Goodrich, W. & Hudson Kam, C. L.. 2009. Co-speech gesture as input in verb learning. Developmental Science 12. 81–87.Google Scholar
Goodwin, C. 1986. Gestures as a resource for the organization of mutual orientation. Semiotica 62. 29–49.CrossRefGoogle Scholar
Gordon, P. C., Grosz, B. J. & Gilliom, L. A.. 1993. Pronouns, names and the centering of attention in discourse. Cognitive Science 17. 311–347.Google Scholar
Grosz, B. J., Joshi, A. K. & Weinstein, S.. 1995. Centering: A framework for modeling the local coherence of discourse. Computational Linguistics 21. 203–226.Google Scholar
Gullberg, M. & Holmqvist, K.. 2006. What speakers do and what addressees look at: Visual attention to gestures in human interaction live and on video. Pragmatics and Cognition 14. 53–82.Google Scholar
Hadar, U. & Krauss, R.. 1999. Iconic gestures: the grammatical categories of lexical affiliates. Journal of Neurolinguistics 12. 1–12.Google Scholar
Järvikivi, J., van Gompel, R. P., Hyönä, J. & Bertram, R.. 2005. Ambiguous pronoun resolution: Contrasting the first-mention and subject-preference accounts. Psychological Science 16. 260–264.Google Scholar
Kehler, A. 2002. Coherence, reference, and the theory of grammar. Stanford, CA: CSLI Publications.Google Scholar
Kelly, S. D., Kravitz, C. & Hopkins, M.. 2004. Neural correlates of bimodal speech and gesture comprehension. Brain and Language 89. 253–60.Google Scholar
Kelly, S. D., Özyürek, A. & Maris, E.. 2010. Two sides of the same coin: Speech and gesture mutually interact to enhance comprehension. Psychological Science 21. 260–267.Google Scholar
Kendon, A. 1980. Gesticulation and speech: Two aspects of the process of utterance. In Key, M. R. (ed.), Relationship of verbal and nonverbal communication, 207–228. Berlin: Mouton de Gruyter.Google Scholar
Kendon, A. 1981. Geography of gesture. Semiotica 37. 129–163.Google Scholar
Kendon, A. 1990. Conducting interaction. Cambridge: Cambridge University Press.Google Scholar
Kendon, A. 1994. Do gestures communicate? A review. Research on Language and Social Interaction 27. 175–200.Google Scholar
Kimbara, I. 2007. Indexing locations in gesture: Recalled stimulus image and interspeaker coordination as factors influence gesture form. In Duncan, S., Cassel, J. & Levy, E. (eds.), Gesture and the dynamic dimension of language, 213–220. Amsterdam: John Benjamins.Google Scholar
Kousta, S. T. 2008. A constraint-based approach to pronoun resolution. University of Cambridge Research Centre for English and Applied Linguistics Working Papers 12. 33–69.Google Scholar
Krauss, R. M. 1998. Why do we gesture when we speak? Current Directions in Psychological Science 7. 54–60.Google Scholar
Krauss, R. M., Chen, Y. & Gottesman, R.. 2000. Lexical gestures and lexical access: A process model. In McNeill, D. (ed.), Language and gesture, 261–283. Cambridge: Cambridge University Press.Google Scholar
Krauss, R. M., Morrel-Samuels, P. & Colasante, C.. 1991. Do conversational hand gestures communicate? Journal of Personality and Social Psychology 61. 743–754.Google Scholar
MacWhinney, B. 1977. Starting points. Language 53. 152–168.Google Scholar
MacWhinney, B., Bates, E. & Kliegl, R.. 1984. Cue validity and sentence interpretation in English, German, and Italian. Journal of Verbal Learning and Verbal Behavior 23. 127–150.Google Scholar
Maratsos, M. P. 1973. The effects of stress on the understanding of pronominal coreference in children. Journal of Psycholinguistic Research 2. 1–8.Google Scholar
McGurk, H. & MacDonald, J.. 1976. Hearing lips and seeing voices. Nature 264. 746–748.Google Scholar
McNeill, D. 1992. Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press.Google Scholar
McNeill, D. 2000. (ed.). Language and gesture: Window into thought and action. Cambridge: Cambridge University Press.Google Scholar
Munhall, K. G., Jones, J. A.Callan, D. E., Kuratate, T. & Vatikiotis-Bateson, E.. 2004. Visual prosody and speech intelligibility: Head movement improves auditory speech perception. Psychological Science 15. 133–137.Google Scholar
Padden, C. 1988. Interaction of morphology and syntax in American Sign Language. NY: Garland.Google Scholar
Palmer, S. 1999. Vision science: Photons to phenomenology. Cambridge, MA: MIT Press.Google Scholar
Rogers, W. T. 1978. The contribution of kinesic illustrators toward the comprehension of verbal behavior within utterances. Human Communication Research 5. 54–62.Google Scholar
Rutter, D. R. 1984. Looking and seeing: The role of visual communication in social interaction. Chichester: Wileys.Google Scholar
Singer, M. A.& Goldin-Meadow, S.. 2005. Children learn when their teacher's gestures and speech differ. Psychological Science 16. 85–89.Google Scholar
So, W. C., Coppola, M.Licciardello, V. & Goldin-Meadow, S.. 2005. The seeds of spatial grammar in the manual modality. Cognitive Science 29. 1029–1043.Google Scholar
So, W. C., Kita, S. & Goldin-Meadow, S.. 2009. Using the hands to identify who does what to whom: Gesture and speech go hand-in-hand. Cognitive Science 33. 115–125.CrossRefGoogle Scholar
Streeck, J. 1993. Gesture as communication I: Its coordination with gaze and speech. Communication Monographs 60. 275–299.Google Scholar
Tfouni, L. & Klatzky, R.. 1983. A discourse analysis of deixis: Pragmatic cognitive, and semantic factors in the comprehension of ‘this’, ‘that’, ‘here’, and ‘there’. Journal of Child Language 10. 123–133.CrossRefGoogle ScholarPubMed
Thompson, L. A.& Massaro, D. W.. 1994. Children's integration of speech and pointing gestures in comprehension. Journal of Experimental Child Psychology 57. 327–354.CrossRefGoogle ScholarPubMed
Willems, R. M., Özyürek, A. & Hagoort, P.. 2007. When language meets action: The neural integration of gesture and speech. Cerebral Cortex 17. 2322–2333.Google Scholar
Wu, Y. C.& Coulson, S.. 2007. How iconic gestures enhance communication: An ERP study. Brain and Language 101. 234–245.Google Scholar