In human social behaviour, interactants build communicative sequences, move by move. These moves are never semiotically simple. Their composite nature is widely varied in kind: a word combined with other words, a string of words combined with an intonation contour, a diagram combined with a caption, an icon combined with another icon, a spoken utterance combined with a hand gesture. By what means does an interpreter take multiple signs and draw them together into unified, meaningful packages? This book explores the question with special reference to one of our most familiar types of move, the speech-with-gesture composite.

This introductory chapter sketches a view of how it is that interpreters may derive meaning from composite utterances. This view of meaning has emerged from the empirical studies in Chapters 2–7, but is intended to have more general application to the analysis of any kind of communicative move, regardless of whether it involves speech, gesture, both, or neither.

Meaning does not begin with language

In a person's vast array of communicative tools, language is surely unrivalled in its expressive richness, speed, productivity, and ease. But the interpretation of linguistic signs is driven by broader principles, principles of rational cognition in social life, principles which underlie other processes of human judgement, from house-buying to gambling to passing people on a crowded street.