Text comprehension and picture comprehension can be synthesized into a common conceptual framework which differentiates between external and internal descriptive and depictive representations. Combining this framework with the human cognitive architecture including sensory registers, working memory, and long-term memory leads to an integrated model of text and picture comprehension. The model consists of a descriptive branch and a depictive branch of processing. It includes multiple sensory modalities. Due to a flexible combination of sensory modalities and representational formats, the model covers listening comprehension, reading comprehension, visual picture comprehension, and sound comprehension. The model considers text comprehension and picture comprehension to be different routes for constructing mental models and propositional representations with the help of prior knowledge. It allows us to explain the effects of coherence, text modality, split attention, text–picture contiguity, redundancy, sequencing, and the effects of different types of visualization.

Sign systems help to create descriptive and depictive representations. Descriptive representations operate on symbols. They are based on conceptual analyses identifying objects or events as well as attributes and interrelations. Attributes and relations are ascribed by predications to entities according to syntactic rules resulting in so-called propositions (“idea units”). These propositions can be integrated into coherent semantic networks. Propositional representations are considered as mental structures which can be externalized in the form of spoken or written texts. Despite their informational incompleteness, descriptions have high representational power. Depictive representations are based on inherent commonalities between a representing object and the represented subject matter. The inherent commonalities can be based on similarity or analogy. These representations are complete with regard to a certain class of information. Due to their completeness and consistency and because information can be read off directly, depictive representations have high computational efficiency.