Sound, Speech, and Music Modeling

The sound analysis approach most commonly used in speech and music processing systems is to extract features of sound sources based on models of their acoustics, in a “front-end” process that is somewhat tuned with respect to properties of the auditory system. The presumption is that a description of the vocal tract or musical instrument that produces a sound can be reliably extracted from the sound waveform, and that such a description is very informative about the underlying speech or music communication intent.

The extraction of representations of sound sources works better when the analysis respects some of the basic properties of human hearing, such as the varying widths of critical bands with respect to frequency. All of the techniques commonly used to compress, code, or recognize speech or music have evolved to incorporate at least some influence from models of hearing. This approach—acoustic analysis influenced by hearing—stands in contrast with our more explicit hearing approach: the hearing approach is to model the kind of signal representation that the ear sends to the brain, for arbitrary sound types and sound mixtures, and to defer the interpretation into sound source features to later higher-level processes.

Although these approaches are differently motivated, they do share some conceptual basis. Understanding the nuances of the hearing approach will be easier once the acoustic approach is understood as a baseline.