Hostname: page-component-848d4c4894-xm8r8 Total loading time: 0 Render date: 2024-07-04T18:23:52.309Z Has data issue: false hasContentIssue false
  • English
  • Français

Visual speech cues speed processing and reduce effort for children listening in quiet and noise

Published online by Cambridge University Press:  30 July 2020

Rebecca Holt Open the ORCID record for Rebecca Holt [Opens in a new window] ,
Laurence Bruggeman  and
Katherine Demuth
Rebecca Holt*
Affiliation:
Department of Linguistics, Macquarie University
Laurence Bruggeman
Affiliation:
Department of Linguistics, Macquarie University The MARCS Institute for Brain, Behaviour & Development and ARC Centre of Excellence for the Dynamics of Language, Western Sydney University
Katherine Demuth
Affiliation:
Department of Linguistics, Macquarie University
*
*Corresponding author. Email: [email protected]
Get access Rights & Permissions [Opens in a new window]

Abstract

Processing speech can be slow and effortful for children, especially in adverse listening conditions, such as the classroom. This can have detrimental effects on children’s academic achievement. We therefore asked whether primary school children’s speech processing could be made faster and less effortful via the presentation of visual speech cues (speaker’s facial movements), and whether any audio-visual benefit would be modulated by the presence of noise or by characteristics of individual participants. A phoneme monitoring task with concurrent pupillometry was used to measure 7- to 11-year-old children’s speech processing speed and effort, with and without visual cues, in both quiet and noise. Results demonstrated that visual cues to speech can facilitate children’s speech processing, but that these benefits may also be subject to variability according to children’s motivation. Children showed faster processing and reduced effort when visual cues were available, regardless of listening condition. However, examination of individual variability revealed that the reduction in effort was driven by the children who performed better on a measure of phoneme isolation (used to quantify how difficult they found the phoneme monitoring task).

Keywords

audio-visual speechchildrenmotivationprocessing effortprocessing speed
Type
Original Article
Information
Applied Psycholinguistics , Volume 41 , Issue 4 , July 2020 , pp. 933 - 961
Check for updates
Copyright
© The Author(s), 2020. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Alhanbali, S., Dawes, P., Millman, R. E., & Munro, K. J. (2019). Measures of listening effort are multidimensional. Ear and Hearing, 40, 1084–1097.10.1097/AUD.0000000000000697CrossRefGoogle ScholarPubMed
Ayasse, N. D., Lash, A., & Wingfield, A. (2017). Effort not speed characterizes comprehension of spoken sentences by older adults with mild hearing impairment. Frontiers in Aging Neuroscience, 8, 329.10.3389/fnagi.2016.00329CrossRefGoogle Scholar
Baayen, R. H., & Milin, P. (2010). Analyzing reaction times. International Journal of Psychological Research, 3, 12–28.10.21500/20112084.807CrossRefGoogle Scholar
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68, 255–278.10.1016/j.jml.2012.11.001CrossRefGoogle ScholarPubMed
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 69, 1–33.Google Scholar
Beatty, J., & Lucero-Wagoner, B. (2000). The pupillary system. In Cacioppo, J. T., Tassinary, L. G., & Berntson, G. G. (Eds.), Handbook of psychophysiology (2nd ed.). Cambridge: Cambridge University Press.Google Scholar
Boersma, P., & Weenink, D. (2018). Praat: doing phonetics by computer [Computer software]. Retrieved from http://www.praat.org Google Scholar
Campbell, R. (2008). The processing of audio-visual speech: Empirical and neural bases. Philosophical Transactions of the Royal Society B, 363, 1001–1010.10.1098/rstb.2007.2155CrossRefGoogle ScholarPubMed
Choi, S., Lotto, A., Lewis, D., Hoover, B., & Stelmachowicz, P. (2008). Attentional modulation of word recognition by children in a dual-task paradigm. Journal of Speech, Language, and Hearing Research, 51, 1042–1054.10.1044/1092-4388(2008/076)CrossRefGoogle Scholar
Connine, C. M., & Titone, D. (1996). Phoneme monitoring. Language and Cognitive Processes, 11, 635–645.Google Scholar
Corretge, R. (2012). Praat vocal toolkit [Computer software]. Retrieved from http://www.praatvocaltoolkit.com Google Scholar
Crandell, C. C., & Smaldino, J. J. (2000). Classroom acoustics for children with normal hearing and with hearing impairment. Language, Speech, and Hearing Services in Schools, 31, 362–370.10.1044/0161-1461.3104.362CrossRefGoogle ScholarPubMed
Cutler, A., & Norris, D. (1979). Monitoring sentence comprehension. In Garrett, M. F., Cooper, W. E., & Walker, E. C. T. (Eds.), Sentence processing: Psycholinguistic studies presented to Merrill Garrett. Hillsdale, NJ: Erlbaum.Google Scholar
Cvejic, E., Kim, J., & Davis, C. (2010). Prosody off the top of the head: Prosodic contrasts can be discriminated by head motion. Speech Communication, 52, 555–564.10.1016/j.specom.2010.02.006CrossRefGoogle Scholar
Davis, C., & Kim, J. (2004). Audio-visual interactions with intact clarly audible speech. Quarterly Journal of Experimental Psychology, 57A, 1103–1121.10.1080/02724980343000701CrossRefGoogle Scholar
Dijkstra, T., Roelofs, A., & Fieuws, S. (1995). Orthographic effects on phoneme monitoring. Canadian Journal of Experimental Psychology, 49, 264–271.10.1037/1196-1961.49.2.264CrossRefGoogle ScholarPubMed
Engelhardt, P. E., Ferreira, F., & Patsenko, E. G. (2010). Pupillometry reveals processing load during spoken language comprehension. Quarterly Journal of Experimental Psychology, 63, 639–645.10.1080/17470210903469864CrossRefGoogle ScholarPubMed
Erdener, D., & Burnham, D. (2013). The relationship between auditory-visual speech perception and language-specific speech perception at the onset of reading instruction in English-speaking children. Journal of Experimental Child Psychology, 116, 120–138.10.1016/j.jecp.2013.03.003CrossRefGoogle ScholarPubMed
Fort, M., Spinelli, E., Savariaux, C., & Kandel, S. (2010). The word superiority effect in audiovisual speech perception. Speech Communication, 52, 525–532.10.1016/j.specom.2010.02.005CrossRefGoogle Scholar
Fort, M., Spinelli, E., Savariaux, C., & Kandel, S. (2012). Audiovisual vowel monitoring and the word superiority effect in children. International Journal of Behvioral Development, 36, 457–467.10.1177/0165025412447752CrossRefGoogle Scholar
Fraser, S., Gagné, J.-P., Alepins, M., & Dubois, P. (2010). Evaluating the effort expended to understand speech in noise using a dual-task paradigm: The effects of providing visual speech cues. Journal of Speech, Language and Hearing Research, 53, 18–33.10.1044/1092-4388(2009/08-0140)CrossRefGoogle ScholarPubMed
Gathercole, S. E., & Pickering, S. J. (2000). Assessment of working memory in six- and seven-year-old children. Journal of Educational Psychology, 92, 377–390.10.1037/0022-0663.92.2.377CrossRefGoogle Scholar
Gosselin, P. A., & Gagné, J.-P. (2011). Older adults expend more listening effort than young adults recognizing audiovisual speech in noise. International Journal of Audiology, 50, 786–792.10.3109/14992027.2011.599870CrossRefGoogle ScholarPubMed
Hicks, C. B., & Tharpe, A. M. (2002). Listening effort and fatigue in school-age children with and without hearing loss. Journal of Speech, Language, and Hearing Research, 45, 573–584.10.1044/1092-4388(2002/046)CrossRefGoogle Scholar
Holt, C. M., Demuth, K., & Yuen, I. (2016). The use of prosodic cues in sentence processing by prelingually deaf users of cochlear implants. Ear and Hearing, 37, e256–e262.10.1097/AUD.0000000000000253CrossRefGoogle ScholarPubMed
Howard, C. S., Munro, K. J., & Plack, C. J. (2010). Listening effort at signal-to-noise ratios that are typical of the school classroom. International Journal of Audiology, 49, 928–932.10.3109/14992027.2010.520036CrossRefGoogle ScholarPubMed
Hsu, B. C.-L., Vanpoucke, F., & van Wieringen, A. (2017). Listening effort through depth of processing in school-age children. Ear and Hearing, 38, 568–576.10.1097/AUD.0000000000000436CrossRefGoogle ScholarPubMed
Hurtado, N., Marchman, V. A., & Fernald, A. (2008). Does input influence uptake? Links between maternal talk, processing speed and vocabulary size in Spanish-learning children. Developmental Science, 11, F31–F39.10.1111/j.1467-7687.2008.00768.xCrossRefGoogle ScholarPubMed
Jerger, S., Damian, M. F., Karl, C., & Abdi, H. (2018). Developmental shifts in detection and attention for auditory, visual, and audiovisual speech. Journal of Speech, Language and Hearing Research, 61, 3095–3112.10.1044/2018_JSLHR-H-17-0343CrossRefGoogle ScholarPubMed
Jerger, S., Damian, M. F., Spence, M. J., Tye-Murray, N., & Abdi, H. (2009). Developmental shifts in children’s sensitivity to visual speech: A new multimodal picture word task. Journal of Experimental Child Psychology, 102, 40–59.10.1016/j.jecp.2008.08.002CrossRefGoogle ScholarPubMed
Jerger, S., Damian, M. F., Tye-Murray, N., & Abdi, H. (2014). Children use visual speech to compensate for non-intact auditory speech. Journal of Experiemental Child Psychology, 126, 295–312.10.1016/j.jecp.2014.05.003CrossRefGoogle ScholarPubMed
Jerger, S., Damian, M. F., Tye-Murray, N., & Abdi, H. (2017). Children perceive speech onsets by ear and eye. Journal of Child Language, 44, 185–215.10.1017/S030500091500077XCrossRefGoogle Scholar
Jesse, A., & Janse, E. (2012). Audiovisual benefit for recognition of speech presented with single-talker noise in older listeners. Language and Cognitive Processes, 27, 1167–1191.10.1080/01690965.2011.620335CrossRefGoogle Scholar
Kahneman, D., & Beatty, J. (1966). Pupil diameter and load on memory. Science, 154, 1583–1585.10.1126/science.154.3756.1583CrossRefGoogle ScholarPubMed
Kim, J., Cvejic, E., & Davis, C. (2014). Tracking eyebrows and head gestures associated with spoken prosody. Speech Communication, 57, 317–330.10.1016/j.specom.2013.06.003CrossRefGoogle Scholar
Knowland, V. C. P., Evans, S., Snell, C., & Rosen, S. (2016). Visual speech perception in children with language learning impairments. Journal of Speech, Language, and Heraing Research, 59, 1–14.10.1044/2015_JSLHR-S-14-0269CrossRefGoogle ScholarPubMed
Lalonde, K., & Holt, R. F. (2015). Preschoolers benefit from visually salient speech cues. Journal of Speech, Language, and Hearing Research, 58, 135–150.10.1044/2014_JSLHR-H-13-0343CrossRefGoogle ScholarPubMed
Lalonde, K., & Holt, R. F. (2016). Audiovisual speech perception development at varying levels of perceptual processing. Journal of the Acoustical Society of America, 139, 1713–1723.10.1121/1.4945590CrossRefGoogle ScholarPubMed
Lalonde, K., & Werner, L. A. (2019). Infants and adults use visual cues to improve detection and discrimination of speech in noise. Journal of Speech, Language, and Hearing Research, Advance online publication.10.1044/2019_JSLHR-H-19-0106CrossRefGoogle ScholarPubMed
Leonard, L. B., Ellis Weismer, S., Miller, C. A., Francis, D. J., Tomblin, J. B., & Kail, R. V. (2007). Speed of processing, working memory, and language impairment in children. Journal of Speech, Language, and Hearing Research, 50, 408–428.10.1044/1092-4388(2007/029)CrossRefGoogle ScholarPubMed
Lo, S., & Andrews, S. (2015). To transform or not to transform: Using generalized linear mixed models to analyse reaction time data. Frontiers in Psychology, 6, 1171.CrossRefGoogle ScholarPubMed
Luciana, M., Conklin, H. M., Hooper, C. J., & Yarger, R. S. (2005). The development of nonverbal working memory and executive control processes in adolescents. Child Development, 76, 697–712.10.1111/j.1467-8624.2005.00872.xCrossRefGoogle ScholarPubMed
Maidment, D. W., Kang, H. J., Stewart, H. J., & Amitay, S. (2015). Audiovisual integration in children listening to spectrally degraded speech. Journal of Speech, Language and Hearing Research, 58, 61–68.10.1044/2014_JSLHR-S-14-0044CrossRefGoogle ScholarPubMed
McGarrigle, R., Dawes, P., Stewart, A. J., Kuchinsky, S. E., & Munro, K. J. (2017). Measuring listening-related effort and fatigue in school-aged children using pupillometry. Journal of Experimental Child Psychology, 161, 95–112.CrossRefGoogle ScholarPubMed
McGarrigle, R., Gustafson, S. J., Hornsby, B. W. Y., & Bess, F. H. (2019). Behavioral measures of listening effort in school-age children: Examining the effects of signal-to-noise ratio, hearing loss, and amplification. Ear and Hearing, 40, 381–392.10.1097/AUD.0000000000000623CrossRefGoogle ScholarPubMed
McGarrigle, R., Munro, K. J., Dawes, P., Stewart, A. J., Moore, D. R., Barry, J. G., & Amitay, S. (2014). Listening effort and fatigue: What exactly are we measuring? A British Society of Audiology Cognition in Hearing Special Interest Group “white paper.” International Journal of Audiology, 53, 433–440.10.3109/14992027.2014.890296CrossRefGoogle Scholar
McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746–748.CrossRefGoogle ScholarPubMed
Mealings, K. T., Buchholz, J. M., Demuth, K., & Dillon, H. (2015). Investigating the acoustics of a sample of open plan and enclosed Kindergarten classrooms in Australia. Applied Acoustics, 100, 95–105.CrossRefGoogle Scholar
Mealings, K. T., Demuth, K., Buchholz, J. M., & Dillon, H. (2015). The effect of different open plan and enclosed classroom acoustic conditions on speech percpetion in Kindergarten children. Journal of the Acoustical Society of America, 138, 2458–2469.10.1121/1.4931903CrossRefGoogle Scholar
Nelson, P. B., & Soli, S. (2000). Acoustical barriers to learning: Children at risk in every classroom. Language, Speech, and Hearing Services in Schools, 31, 356–361.CrossRefGoogle ScholarPubMed
Peter, M. S., Durrant, S., Jessop, A., Bidgood, A., Pine, J. M., & Rowland, C. F. (2019). Does speed of processing or vocabulary size predict later language growth in toddlers? Cognitive Psychology, 115, 101238.10.1016/j.cogpsych.2019.101238CrossRefGoogle ScholarPubMed
Pichora-Fuller, M. K., Kramer, S. E., Eckert, M. A., Edwards, B., Hornsby, B. W. Y., Humes, L. E., … Wingfield, A. (2016). Hearing impairment and cognitive energy: the Framework for Understanding Effortful Listening (FUEL). Ear and Hearing, 37, 5S–27S.10.1097/AUD.0000000000000312CrossRefGoogle Scholar
Picou, E. M., Ricketts, T. A., & Hornsby, B. W. Y. (2011). Visual cues and listening effort: Individual variability. Journal of Speech, Language and Hearing Research, 54, 1416–1430.10.1044/1092-4388(2011/10-0154)CrossRefGoogle ScholarPubMed
Picou, E. M., Ricketts, T. A., & Hornsby, B. W. Y. (2013). How hearing aids, background noise, and visual cues influence objective listening effort. Ear and Hearing, 34, e52–e64.CrossRefGoogle ScholarPubMed
R Core Team. (2018). R: A language and environment for statistical computing [Computer software]. Vienna, Austria: R Foundation for Statistical Computing.Google Scholar
Reisberg, D., McLean, J., Goldfield, A. (1987). Easy to hear but hard to understand: A lip-reading advantage with intact auditory stimuli. In Dodd, B., & Campbell, R. (Eds), Hearing by eye: the psychology of lip-reading. Hillsdale, NJ: Erlbaum.Google Scholar
Remez, R. E. (2005). Three puzzles of multimodal speech perception. In Vatikiotis-Bateson, E., Bailly, G., & Perrier, P. (Eds.), Audiovisual speech. Cambridge, MA: MIT Press.Google Scholar
Reynolds, C. R. (1997). Forward and backward memory span should not be combined for clinical analysis. Archives of Clinical Neuropsychology, 12, 29–40.10.1093/arclin/12.1.29CrossRefGoogle Scholar
Ross, L. A., Molholm, S., Blanco, D., Gomez-Ramirez, M., Saint-Amour, D., & Foxe, J. J. (2011). The development of multisensory speech perception continues into the late childhood years. European Journal of Neuroscience, 33, 2329–2337.10.1111/j.1460-9568.2011.07685.xCrossRefGoogle ScholarPubMed
Ross, L. A., Saint-Amour, D., Leavitt, V. M., Javitt, D. C., & Foxe, J. J. (2007). Do you see what I am saying? Exploring visual enhancement of speech comprehension in noisy environments. Cerebral Cortex, 17, 1147–1153.10.1093/cercor/bhl024CrossRefGoogle Scholar
Sasisekaran, J., & Weber-Fox, C. (2012). Cross-sectional study of phoneme and rhyme monitoring abilities in children between 7 and 13 years. Applied Psycholinguistics, 33, 253–279.CrossRefGoogle ScholarPubMed
Schmidtke, J. (2014). Second language experience modulates word retrieval effort in bilinguals: Evidence from pupillometry. Frontiers in Psychology, 5, 137.CrossRefGoogle ScholarPubMed
Schmidtke, J. (2018). Pupillometry in linguistic research: an introduction and review for second language researchers. Studies in Second Language Acquisition, 40, 529–549.10.1017/S0272263117000195CrossRefGoogle Scholar
Strand, J. F., Brown, V. A., Merchant, M. B., Brown, H. E., & Smith, J. (2018). Measuring listening effort: Convergent validity, sensitivity, and links with cognitive and personality measures. Journal of Speech, Language, & Hearing Research, 61, 1463–1486.10.1044/2018_JSLHR-H-17-0257CrossRefGoogle ScholarPubMed
Strelnikov, K., Rouger, J., Laglayre, S., Fraysse, B., Deguine, O., & Barone, P. (2009). Improvement in speech-reading ability by auditory training: Evidence from gender differences in normally hearing, deaf and cochlear implanted subjects. Neuropsychologia, 47, 972–979.10.1016/j.neuropsychologia.2008.10.017CrossRefGoogle ScholarPubMed
Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America, 26, 212–215.CrossRefGoogle Scholar
Tamási, K., McKean, C., Gafos, A., Fritzsche, T., & Höhle, B. (2017). Pupillometry registers toddlers’ sensitivity to degrees of mispronunciation. Journal of Experimental Child Psychology, 153, 140–148.10.1016/j.jecp.2016.07.014CrossRefGoogle ScholarPubMed
Treiman, R., Salasoo, A., Slowiaczek, L. M., & Pisoni, D. B. (1982). Effects of syllable structure on adults’ phoneme monitoring performance. In Pisoni, D. B. (Ed.), Research on speech perception progress report no. 8. Bloomington, IN: Indiana University Speech Research Laboratory.Google Scholar
Van Heuven, W. J. B., Mandera, P., Keuleers, E., & Brysbaert, M. (2014). SUBTLEX-UK: A new and improved word frequency database for British English. Quarterly Journal of Experimental Psychology, 67, 1176–1190.10.1080/17470218.2013.850521CrossRefGoogle ScholarPubMed
Vatikiotis-Bateson, E., & Kuratate, T. (2012). Overview of audiovisual speech processing. Acoustical Science and Technology, 33, 135–141.10.1250/ast.33.135CrossRefGoogle Scholar
Wagner, A. E., Toffanin, P., & Başkent, D. (2016). The timing and effort of lexical access in natural and degraded speech. Frontiers in Psychology, 7, 398.CrossRefGoogle ScholarPubMed
Wagner, R. K., Torgesen, J. K., Rashotte, C. A., & Pearson, N. A. (2013). CTOPP-2: Comprehensive Test of Phonological Processing–Second edition. Austin, TX: Pro-Ed.Google Scholar
Winn, M. B., Edwards, J. R., & Litovsky, R. Y. (2015). The impact of auditory spectral resolution on listening effort revealed by pupil dilation. Ear and Hearing, 36, e153–e165.10.1097/AUD.0000000000000145CrossRefGoogle ScholarPubMed
Zeileis, A., Grothendieck, G., Ryan, J. A., Ulrich, J. M., & Andrews, F. (2018). S3 infrastructure for regular and irregular time series (Z’s ordered observations) [Computer software]. Retrieved from http://zoo.R-Forge.R-project.org Google Scholar
Zekveld, A. A., Heslenfeld, D. J., Johnsrude, I. S., Versfeld, N., & Kramer, S. E. (2014). The eye as a window to the listening brain: Neural correlates of pupil size as a measure of cognitive listening load. Neuroimage, 101, 76–86.10.1016/j.neuroimage.2014.06.069CrossRefGoogle ScholarPubMed