Abstract

This chapter presents an integrated model of text and picture comprehension that takes into account that learners can use multiple sensory modalities. The model encompasses reading comprehension, listening comprehension, visual picture comprehension, and sound comprehension (i.e., auditory picture comprehension). The model's cognitive architecture consists of sensory registers, working memory, and long-term memory. It furthermore includes a cognitive level and a perceptual level. The cognitive level is characterized by two representational channels: a verbal channel and a pictorial channel. The perceptual level is characterized by multiple sensory channels. After presenting the model, the chapter derives predictions, which can be empirically tested. It reports research findings that can be explained by the model, and it derives practical suggestions for instructional design. Finally, the chapter discusses limitations of the model and points out directions for further research.

Introduction

The term multimedia means different things on different levels. On the level of technology, it means the use of multiple delivery media such as computers, screens, and loudspeakers. On the level of presentation formats, it means the use of different forms of representation such as texts and pictures. On the level of sensory modalities, it means the use of multiple senses such as the eye and the ear. The level of technology is of course very important in practice, but it is not of high interest from a psychological point of view, because comprehension is not fundamentally different when a text passage is delivered either by a computer screen or by a printed book, or if a picture is presented by a poster or by a slide.