Present day Human–Computer Interaction (HCI) revolves around typing at a keyboard, moving and pointing with a mouse, selecting from menus and searching through manuals. These interfaces are far from ideal, especially when dealing with graphical input and trying to visualise and manipulate three-dimensional data and structures. For many, interacting with a computer is a cumbersome and frustrating experience. Most people would prefer more natural ways of dealing with computers, closer to face-to-face, human-to-human conversation, where talking, gesturing, hearing and seeing are part of the interaction. They would prefer computer systems that understand their verbal and non-verbal languages.

Since its inception in the 1960s, the emphasis of HCI design has been on improving the “look and feel” of a computer by incremental changes, to produce keyboards that are easier to use, and graphical input devices with bigger screens and better sounds. The advent of low-cost computing and memory, and computers equipped with tele-conferencing hardware (including cameras mounted above the display) means that video input and audio input is available with little additional cost. It is now possible to conceive of more radical changes in the way we interact with machines: of computers that are listening to and looking at their users.

Progress has already been achieved in computer speech synthesis and recognition [326]. Promising commercial products already exist that allow natural speech to be digitised and processed by computer (32 kilobytes per second) for use in dictation systems.