Hostname: page-component-745bb68f8f-grxwn Total loading time: 0 Render date: 2025-02-11T19:53:39.852Z Has data issue: false hasContentIssue false
  • English
  • Français

YOUNG LEARNERS’ PROCESSING OF MULTIMODAL INPUT AND ITS IMPACT ON READING COMPREHENSION

AN EYE-TRACKING STUDY

Published online by Cambridge University Press:  15 April 2020

Ana Pellicer-Sánchez Open the ORCID record for Ana Pellicer-Sánchez [Opens in a new window] ,
Elsa Tragant Open the ORCID record for Elsa Tragant [Opens in a new window] ,
Kathy Conklin Open the ORCID record for Kathy Conklin [Opens in a new window] ,
Michael Rodgers Open the ORCID record for Michael Rodgers [Opens in a new window] ,
Raquel Serrano Open the ORCID record for Raquel Serrano [Opens in a new window]  and
Ángels Llanes Open the ORCID record for Ángels Llanes [Opens in a new window]
Ana Pellicer-Sánchez*
Affiliation:
UCL Institute of Education
Elsa Tragant
Affiliation:
University of Barcelona
Kathy Conklin
Affiliation:
University of Nottingham
Michael Rodgers
Affiliation:
Carleton University
Raquel Serrano
Affiliation:
University of Barcelona
Ángels Llanes
Affiliation:
University of Lleida
*
*Correspondence concerning this article should be addressed to Ana Pellicer-Sánchez, UCL Institute of Education, 20 Bedford Way, London WC1H 0AL, United Kingdom. E-mail: [email protected].
Get access Rights & Permissions [Opens in a new window]

Abstract

Theories of multimedia learning suggest that learners can form better referential connections when verbal and visual materials are presented simultaneously. Furthermore, the addition of auditory input in reading-while-listening conditions benefits performance on a variety of linguistic tasks. However, little research has been conducted on the processing of multimedia input (written text and images) with and without accompanying audio. Eye movements were recorded during young L2 learners’ (N = 30) processing of a multimedia story text in reading-only and reading-while-listening conditions to investigate looking patterns and their relationship with comprehension using a multiple-choice comprehension test. Analysis of the eye-movement data showed that the presence of audio in reading-while-listening conditions allowed learners to look at the image more often. Processing time on text was related to lower levels of comprehension, whereas processing time on images was positively related to comprehension.

Type
Research Article
Information
Studies in Second Language Acquisition , Volume 42 , Special Issue 3: Language Learning from Multimodal Input , July 2020 , pp. 577 - 598
Check for updates
Copyright
© The Author(s) 2020. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

This manuscript is an output from an ELT Research Award funded by the British Council to promote innovation in English language teaching research. The views expressed are not necessarily those of the British Council. We would like to thank Fabio Parente and Ling Shuping (University of Nottingham), and Radha Chandy (University of Barcelona) for their help with data collection and analysis.

References

REFERENCES

Alemdag, E., & Cagiltay, K. (2018). A systematic review of eye tracking research on multimedia learning. Computers & Education, 125, 413–428.CrossRefGoogle Scholar
Arnau, J., Bendayan, R., Blanca, M. J., & Bono, R. (2013). The effect of skewness and kurtosis on the robustness of linear mixed models. Behavior Research Methods, 45, 873–879.CrossRefGoogle ScholarPubMed
Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67, 1–48. https://doi.org/10.18637/jss.v067.i01.CrossRefGoogle Scholar
Bisson, M.-J., Van Heuven, W., Conklin, K., & Tunney, R. (2014). Processing of native and foreign language subtitles in films: An eye tracking study. Applied Psycholinguistics, 35, 399–418. https://doi.org/10.1017/S0142716412000434.CrossRefGoogle Scholar
Bisson, M.-J., Van Heuven, W., Conklin, K., & Tunney, R. (2015). The role of verbal and pictorial information in multi-modal incidental acquisition of foreign language vocabulary. Quarterly Journal of Experimental Psychology, 68, 306–326. https://doi.org/10.1080/17470218.2014.979211.CrossRefGoogle Scholar
Brown, R., Waring, R., & Donkaewbua, S. (2008). Incidental vocabulary acquisition from reading, reading-while-listening, and listening to stories. Reading in a Foreign Language, 20, 136–163.Google Scholar
Buck, G. (2001). Assessing listening. Cambridge University Press.CrossRefGoogle Scholar
Chang, C.-S. (2009). Gains to L2 listeners from reading while listening vs. listening only in comprehending short stories. System, 37, 652–663. https://doi.org/10.1016/j.system.2009.09.009.CrossRefGoogle Scholar
Chang, C.-S. (2011). The effect of reading while listening to audiobooks: Listening fluency and vocabulary gain. Asian Journal of English Language Teaching, 21, 43–64.Google Scholar
Chang, C.-S., & Millett, S. (2014). The effect of extensive listening on developing l2 listening fluency: Some hard evidence. ELT Journal, 68, 31–40. https://doi.org/10.1093/elt/cct052.CrossRefGoogle Scholar
Chang, C.-S., & Millett, S. (2015). Improving reading rates and comprehension through audio-assisted extensive reading for beginner learners. System, 52, 91–102. https://doi.org/10.1016/j.system.2015.05.003.CrossRefGoogle Scholar
Chang, Y., & Choi, S. (2014). Effects of seductive details evidenced by gaze duration. Neurobiology of Learning and Memory, 109, 131–138.CrossRefGoogle ScholarPubMed
Cobb, T. (n.d.). Lextutor. Vocabprofile [computer program]. https://www.lextutor.ca/cgi-bin/range/texts/index.plGoogle Scholar
Conklin, K., Pellicer-Sánchez, A., & Carrol, G. (2018). Eye-tracking: A guide for applied linguistics research. Cambridge University Press.CrossRefGoogle Scholar
Cowan, N. (1995). Attention and memory: An integrated framework. Oxford psychology series, No. 26. Oxford University Press.Google Scholar
Cullen, A. C., & Frey, H. C. (1999). Probabilistic techniques in exposure assessment: A handbook for dealing with variability and uncertainty in models and inputs. Plenum Press.Google Scholar
Delignette-Muller, M. L., & Dutang, C. (2015). fitdistrplus: An R Package for Fitting Distributions. Journal of Statistical Software, 64, 1–34.CrossRefGoogle Scholar
Diao, Y., & Sweller, J. (2007). Redundancy in foreign language reading comprehension instruction: Concurrent written and spoken presentations. Learning and Instruction, 17, 78–88.CrossRefGoogle Scholar
d’Ydewalle, G., & de Bruycker, W. (2007). Eye movements of children and adults while reading television subtitles. European Psychologist, 12, 196–205. https://doi.org/10.1027/1016-9040.12.3.196CrossRefGoogle Scholar
Eitel, A. (2016). How repeated studying and testing affects multimedia learning: Evidence for adaptation to task demands. Learning and Instruction, 41, 70–84. https://doi.org/10.1016/j.learninstruc.2015.10.003.CrossRefGoogle Scholar
Elley, W. B., & Mangubhai, F. (1983). The impact of reading on second language learning. Reading Research Quarterly, 19, 53–67. https://doi.org/10.2307/747337.CrossRefGoogle Scholar
Godfroid, A., Ahn, J., Choi, I., Ballard, L., Cui, Y., Johnston, S., Lee, S., Sarkar, A., & Yoon, H. (2018). Incidental vocabulary learning in a natural reading context: An eye-tracking study. Bilingualism: Language and Cognition, 21, 563–584. https://doi.org/10.1017/S1366728917000219.CrossRefGoogle Scholar
Griffiths, R. (1990). Speech rate and nonnative speaker comprehension: A preliminary study in the time-benefit analysis. Language Learning, 40, 311–336.CrossRefGoogle Scholar
Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. Routledge.Google Scholar
Hill, D. (2013). Graded readers. ELT Journal, 67, 85–125.CrossRefGoogle Scholar
Hu, M., & Nation, I. S. P. (2000). Unknown vocabulary density and reading comprehension. Reading in a Foreign Language, 13, 403–430.Google Scholar
Jamet, E., & Le Bohec, O. (2007). The effect of redundant text in multimedia instruction. Contemporary Educational Psychology, 32, 588–598.CrossRefGoogle Scholar
Johnson, C. I., & Mayer, R. E. (2012). An eye movement analysis of the spatial contiguity effect in multimedia learning. Journal of Experimental Psychology: Applied, 18, 178–191. https://doi.org/10.1037/a0026923.Google ScholarPubMed
Kalyuga, S., Chandler, P., & Sweller, J. (1999). Managing split-attention and redundancy in multimedia instruction. Applied Cognitive Psychology, 13, 351–371.3.0.CO;2-6>CrossRefGoogle Scholar
Kalyuga, S. & Sweller, J. (2014). The redundancy principle in multimedia learning. In Mayer, R. E. (Ed.), The Cambridge handbook of multimedia learning (2nd ed.) (pp. 247–262). Cambridge University Press.CrossRefGoogle Scholar
Koller, M. (2016). robustlmm: An R package for robust estimation of linear mixed-effects models. Journal of Statistical Software, 75, 1–24. https://doi.org/10.18637/jss.v075.i06.CrossRefGoogle Scholar
Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2017). lmerTest package: Tests in linear mixed effects models. Journal of Statistical Software, 82, 1–26. https://doi.org/10.18637/jss.v082.i13.CrossRefGoogle Scholar
Liceralde, V. R. T., & Gordon, P. C. (2018). Consequences of power transforms as a statistical solution in linear mixed-effects models of chronometric data. osf.io/ygc7sCrossRefGoogle Scholar
Lightbown, P. M. (1992). Can they do it themselves? A comprehension-based ESL course for young children. In Courchene, R., John, J. St, Therien, C., & Glidden, J. I. (Eds.), Comprehension-based second language teaching (pp. 353–370). University of Ottawa Press.Google Scholar
Lightbown, P., Halter, R., White, J., & Horst, M. (2002). Comprehension-based learning: The limits of “Do It Yourself.” The Canadian Modern Language Review, 58, 427–464. https://doi.org/10.3138/cmlr.58.3.427.CrossRefGoogle Scholar
Lo, S., & Andrews, S. (2015). To transform or not to transform: Using generalized linear mixed models to analyse reaction time data. Frontiers in Psychology, 6, 1171. https://doi.org/10.3389/fpsyg.2015.01171.CrossRefGoogle ScholarPubMed
Mason, L., Pluchino, P., Tornatora, M. C., & Ariasi, N. (2013). An eye-tracking study of learning science text with concrete and abstract illustrations. Journal of Experimental Education, 81, 356–384. https://doi.org/10.1080/00220973.2012.727885.CrossRefGoogle Scholar
Mason, L., Tornatora, M. C., & Pluchino, P. (2015). Integrative processing of verbal and graphical information during re-reading predicts learning from illustrated text: An eye movement study. Reading and Writing, 28, 851–872. https://doi.org/10.1007/s11145-015-9552-5.CrossRefGoogle Scholar
Massaro, D. W. (2012). Multimodal learning. In Seel, N. M. (Ed.) Encyclopedia of the sciences of learning (pp. 2375–2378). Springer.CrossRefGoogle Scholar
Mayer, R. E. (2001). Multimedia learning. Cambridge University Press.CrossRefGoogle Scholar
Mayer, R. E. (2009). Multimedia learning (2nd ed.). Cambridge University Press.CrossRefGoogle Scholar
Mayer, R. E. (2014). Introduction to multimedia learning. In Mayer, R. E. (Ed.), The Cambridge handbook of multimedia learning (2nd ed.) (pp. 1–24). Cambridge University Press.CrossRefGoogle Scholar
Mayer, R. E. (2017). Using multimedia for e-learning. Journal of Computer Assisted Learning, 36, 1–21. https://doi.org/10.1111/jcal.12197.Google Scholar
Mayer, R., Heiser, J., & Lonn, S. (2001). Cognitive constraints on multimedia learning: When presenting more material results in less understanding. Journal of Educational Psychology, 93, 187–198.CrossRefGoogle Scholar
Mayer, R. E., & Moreno, R. (1998). A split-attention effect in multimedia learning: Evidence for dual processing systems in working memory. Journal of Educational Psychology, 90, 312–320.CrossRefGoogle Scholar
McCulloch, C. E., & Neuhaus, J. M. (2011). Misspecifying the shape of a random effects distribution: Why getting it wrong may not matter. Statistical Science, 26, 388–402. https://doi.org/10.1214/11-STS361.CrossRefGoogle Scholar
Meara, P. M., & Milton, J. L. (2003). X_Lex: The swansea vocabulary levels test. Express Publishing.Google Scholar
Montero Perez, M., Peters, E., Clarebout, G., & Desmet, P. (2014). Effects of captioning on video comprehension and incidental vocabulary learning. Language Learning and Technology, 18, 118–141.Google Scholar
Montero Perez, M., Peters, E., & Desmet, P. (2015). Enhancing vocabulary learning through captioned video: An eye-tracking study. The Modern Language Journal, 99, 308–328. https://doi.org/10.1111/modl.12215.CrossRefGoogle Scholar
Moreno, R., & Mayer, R. E. (1999). Cognitive principles of multimedia learning: The role of modality and contiguity. Journal of Educational Psychology, 91, 358–368.CrossRefGoogle Scholar
Moussa-Inaty, J., Ayres, P., & Sweller, P. (2012). Cognitive load and the impact of spoken English on learning English as a foreign language. Applied Cognitive Psychology, 63, 391–402.CrossRefGoogle Scholar
Muñoz, C. (2017). The role of age and proficiency in subtitle reading. An eye-tracking study. System, 67, 77–86. https://doi.org/10.1016/j.system.2017.04.015.Google Scholar
Niegeman, H., & Heidig, S. (2012). Multimedia learning. In Seel, N. M. (Eds.) Encyclopedia of the sciences of learning (pp. 2372–2375). Springer.CrossRefGoogle Scholar
Omaggio, A. C. (1979). Pictures and second language comprehension: Do they help? Foreign Language Annals, 12, 107–116. https://doi.org/10.1111/j.1944-9720.1979.tb00153.x.CrossRefGoogle Scholar
Paivio, A. (1986). Mental representations: A dual coding approach. Oxford University Press.Google Scholar
Paivio, A. (2006). Mind and its evolution: A dual coding approach. Lawrence Erlbaum.Google Scholar
Pellicer-Sánchez, A. (2016). Incidental L2 vocabulary acquisition from and while reading: An eye-tracking study. Studies in Second Language Acquisition, 38, 97–130. https://doi.org/10.1017/S0272263115000224.CrossRefGoogle Scholar
Pellicer-Sánchez, A., & Conklin, K. (2020). Eye-tracking as a data collection method. In Rose, H. & McKinley, J. (Eds.), The Routledge handbook of research methods in applied linguistics . Routledge.Google Scholar
Peters, E. (2019). The effect of imagery and on‐screen text on foreign language vocabulary learning from audiovisual input. TESOL Quarterly, 53, 1008–1032. https://doi.org/10.1002/tesq.531.CrossRefGoogle Scholar
Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372–422.CrossRefGoogle ScholarPubMed
Rayner, K. (2009). Eye movements and attention in reading, scene perception, and visual search. The Quarterly Journal of Experimental Psychology, 62, 1457–1506.CrossRefGoogle ScholarPubMed
Core Team, R. (2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/.Google Scholar
Sawilowsky, S. S. (2009). New effect size rules of thumb. Journal of Modern Applied Statistical Methods, 8, 597–599. https://doi.org/10.22237/jmasm/1257035100.CrossRefGoogle Scholar
Schmidt-Weigand, F., Kohnert, A., & Glowalla, U. (2010). A closer look at split visual attention in system-and self-paced instruction in multimedia learning. Learning and Instruction, 20, 100–110. https://doi.org/10.1016/j.learninstruc.2009.02.011.CrossRefGoogle Scholar
Schnotz, W. (2014). An integrated model of text- and picture comprehension. In Mayer, R. E. (Ed.), The Cambridge handbook of multimedia learning (pp. 72–103). Cambridge University Press.CrossRefGoogle Scholar
Serrano, R., & Pellicer-Sánchez, A. (2019). Young L2 learners’ online processing of information in a graded reader during reading-only and reading-while-listening conditions: A study of eye movements. Applied Linguistics Review. Advance online publication. https://doi.org/10.1515/applirev-2018-0102.CrossRefGoogle Scholar
Sung, E., & Mayer, R. E. (2013). Online multimedia learning with mobile devices and desktop computers: An experimental test of Clark’s methods-not-media hypothesis. Computers in Human Behavior, 29, 639–647. https://doi.org/10.1016/j.chb.2012.10.022.CrossRefGoogle Scholar
Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12, 257–285. https://doi.org/10.1207/s15516709cog1202_4.CrossRefGoogle Scholar
Tragant, E., & Pellicer-Sánchez, A. (2019). Young learners’ engagement with multimodal exposure: An eye-tracking study. System, 80, 212–223. https://doi.org/10.1016/j.system.2018.12.002.Google Scholar
Tragant, E., & Vallbona, A. (2018). Reading while listening to learn: Young EFL learners’ perceptions. ELT Journal, 72, 395–404. https://doi.org/10.1093/elt/ccy009.CrossRefGoogle Scholar
Tragant, E., Muñoz, C., & Spada, N. (2016). Maximizing young learners’ input: An intervention program. The Canadian Modern Language Review, 72, 234–257.CrossRefGoogle Scholar
Trofimovich, P., Lightbown, P. M., Halter, R., & Song, H. (2009). Comprehension based practice: The development of L2 pronunciation in a listening and reading program. Studies in Second Language Acquisition, 31, 609–639.CrossRefGoogle Scholar
Warren, P., Boers, F., Grimshaw, G., & Siyanova-Chanturia, A. (2018). The effect of gloss type on learners’ intake of new words during reading. Studies in Second Language Acquisition, 40, 883–906. https://doi.org/10.1017/S0272263118000177.CrossRefGoogle Scholar
https://doi.org/10.3138/cmlr.1204.1CrossRefGoogle Scholar
Webb, S., & Chang, A. (2015). Second language vocabulary learning through extensive reading with audio support: How do frequency and distribution of occurrence affect learning? Language Teaching Research, 19, 667–686.CrossRefGoogle Scholar
Webb, S., & Chang, A. C.-S. (2017). How does mode of input affect the incidental learning of multiword combinations? Paper presented at the AAAL Annual. Conference, Portland, OR.Google Scholar
Webb, S., Newton, J., & Chang, A. (2013). Incidental learning of collocation. Language Learning, 63, 91–120.CrossRefGoogle Scholar
Whitford, W., & Joanisse, M. F. (2018). Do eye movements reveal differences between monolingual and bilingual children’s first-language and second-language reading? A focus on word frequency effects. Journal of Experimental Child Psychology, 173, 318–337.CrossRefGoogle Scholar
Wilde, O. (2012). The Canterville Ghost. In Retold by Cadwallader, J. ELI Graded: Readers. ELI Publishing.Google Scholar
Winke, P., Gass, S., & Sydorenko, T. (2013). Factors influencing the use of captions by foreign language learners: An eye-tracking study. The Modern Language Journal, 97, 254–275. https://doi.org/10.1111/j.1540-4781.2013.01432.x.CrossRefGoogle Scholar
Wright, A. (2010). Pictures for language learning. Cambridge University Press.Google Scholar
Yang, F., Chang, C., Chien, W., Chien, Y., & Tseng, T. (2013). Tracking learners’ visual attention during a multimedia presentation in a real classroom. Computers and Education, 62, 208–220. https://doi.org/10.1016/j.compedu.2012.10.009.CrossRefGoogle Scholar
Supplementary material: File

Pellicer-Sánchez et al. supplementary material

Pellicer-Sánchez et al. supplementary material

Download Pellicer-Sánchez et al. supplementary material(File)
File 15.4 KB