Representation of physical quantities in terms of floating–point numbers allows one to cover a very wide dynamic range with a relatively small number of digits. Given this type of representation, roundoff errors are roughly proportional to the amplitude of the represented quantity. In contrast, roundoff errors with uniform quantization are bounded between ±q/2 and are not in any way proportional to the represented quantity.

Floating–point is in most cases so advantageous over fixed–point number representation that it is rapidly becoming ubiquitous. The movement toward usage of floating–point numbers is accelerating as the speed of floating–point calculation is increasing and the cost of implementation is going down. For this reason, it is essential to have a method of analysis for floating–point quantization and floating–point arithmetic.

THE FLOATING–POINT QUANTIZER

Binary numbers have become accepted as the basis for all digital computation. We therefore describe floating–point representation in terms of binary numbers. Other number bases are completely possible, such as base 10 or base 16, but modern digital hardware is built on the binary base.

The numbers in the table of Fig. 12.1 are chosen to provide a simple example. We begin by counting with nonnegative binary floating–point numbers as illustrated in Fig. 12.1. The counting starts with the number 0, represented here by 00000. Each number is multiplied by 2E, where E is an exponent. Initially, let E = 0. Continuing the count, the next number is 1, represented by 00001, and so forth.

Roundoff errors in calculations are often neglected by scientists. The success of the IEEE double precision standard makes most of us think that the precision of a simple personal computer is virtually infinite. Common sense cannot really grasp the meaning of 16 precise decimal digits.

However, roundoff errors can easily destroy the result of a calculation, even if it looks reasonable. Therefore, it is worth investigating them even for IEEE double precision representation.

COMPARISON TO REFERENCE VALUES

Investigation of roundoff errors is most straightforwardly based on comparison of the results to reference values, that is, to the outcome of an ideally precise calculation.

First of all, we need to note that the way of comparison is not well defined. It is usually done by looking at the difference of the finite precision result and of the reference value. This is very reasonable, but cannot be applied in all cases. If, for example, we investigate a stand–alone resonant system, the reference and the finite precision results could be two similar sine waves, with slightly different frequency. In such a case, the difference of the imprecise result and precise result grows significantly with time, although the outputs are still very similar. Therefore, the basis of comparison needs to be chosen very carefully.

When the input to a quantizer is a sampled time series represented by x1, x2, x3, …, the quantization noise is a time series represented by ν1, ν2, ν3, … Suppose that the input time series is stationary and that its statistics satisfy the conditions for multivariable QT II (it would be sufficient that two–variable QT II conditions were satisfied for x1 and x2, x1 and x3, x1 and x4, and so forth, because of stationarity). As such, the quantization noise will be uncorrelated with the quantizer input, and the quantization noise will be white, i.e. uncorrelated over time. The PQN model applies. The autocorrelation function of the quantizer output will be equal to the autocorrelation function of the input plus the autocorrelation function of the quantization noise.

Fig. 20.1(a) is a sketch of an autocorrelation function of a quantizer input signal. Fig. 20.1(b) shows the autocorrelation function of the quantization noise when the PQN model applies. Fig. 20.1(c) shows the corresponding autocorrelation function of the quantizer output.

Corresponding to the autocorrelation functions of Fig. 20.1, the power spectrum of the quantizer output is equal to the power spectrum of the input plus the power spectrum of the quantization noise. This spectrum is flat, with a total power of q2/12.

For many years, rumors have been circulating in the realm of digital signal processing about quantization noise:

1. (a) the noise is additive and white and uncorrelated with the signal being quantized, and

2. (b) the noise is uniformly distributed between plus and minus half a quanta, giving it zero mean and a mean square of one–twelfth the square of a quanta.

Many successful systems incorporating uniform quantization have been built and placed into service worldwide whose designs are based on these rumors, thereby reinforcing their veracity. Yet simple reasoning leads one to conclude that:

1. (a) quantization noise is deterministically related to the signal being quantized and is certainly not independent of it,

2. (b) the probability density of the noise certainly depends on the probability density of the signal being quantized, and

3. (c) if the signal being quantized is correlated over time, the noise will certainly have some correlation over time.

In spite of the “simple reasoning,” the rumors are true under most circumstances, or at least true to a very good approximation. When the rumors are true, wonderful things happen:

1. (a) digital signal processing systems are easy to design, and

2. (b) systems with quantization that are truly nonlinear behave like linear systems.

In order for the rumors to be true, it is necessary that the signal being quantized obeys a quantizing condition. There actually are several quantizing conditions, all pertaining to the probability density function (PDF) and the characteristic function (CF) of the signal being quantized.

We have seen when discussing the sampling theorem in Section 2.2 that the conditions of the theorem are exactly met only if the signal being sampled is perfectly bandlimited. This is rarely the case, since perfect bandlimitedness implies that the signal cannot be time–limited. Such a signal can be easily defined mathematically, but measured signals are always time–limited, so the condition of the sampling theorem can be met only approximately. While the sinc function wave is theoretically bandlimited, its time truncated versions are not, so the sampling theorem can be applied only approximately. However, sampling theory proved to be very powerful despite its approximate applicability.

The situation is similar with the quantizing theorems. Bandlimitedness of the CF would imply that the PDF is not amplitude–limited. Since measured signals are always amplitude–limited, the quantization theorems can be applied only approximately. Similarly to the sampling theorem, this does not prevent the quantizing theorems from being very powerful in many applications.

Most distributions that are applied in practice, like the Gaussian, exponential or chi–squared are not bandlimited. This fact does not prevent the application of the quantizing theorems if the quantum step size is significantly smaller than the standard deviation. Nevertheless, it is of interest to investigate the question whether there are distributions whose characteristic functions are perfectly bandlimited, similarly to the sinc function. In the following paragraphs we will discuss some examples of distributions whose CFs are perfectly bandlimited.

In the previous chapter, certain relations between the quantization noise and the quantizer input and output were established. These relations are in the form of moments, particularly crosscorrelations and covariances. These moments are extremely useful for the analysis of quantization and for the analysis of control and signal processing systems containing quantizers. The similarities between quantization and the addition of independent uniformly distributed noise when certain quantizing theorems are satisfied are very useful for analysis of quantization.

The properties of these moments are not totally descriptive of the behavior of the quantizer however, because the quantization noise is deterministically related to the quantizer input, and the moments do not show this. To gain a complete and deeper understanding, we dedicate this chapter to the derivation of joint PDFs and CFs of quantization noise and the quantizer input and output.

JOINT PDF AND CF OF THE QUANTIZER INPUT AND OUTPUT

In this section, we will derive the joint PDF and CF of x and x', i.e. fx,x' (x, x'), and Φx,x' (ux, ux'). The statistical relationship between x and x' is like a statistical version of an input–output “transfer function” for the quantizer.

The derivation will proceed in the following manner. Refer to Fig. 7.1. This figure compares quantization (the addition of actual quantization noise ν to x) with the PQN model (the addition of independent noise n to x).

Scientific computation is done these days with floating–point arithmetic. Generally double precision is used. Of concern are quantization effects in the solutions of equations caused by limited precision in the representation of dependent variables (equation solutions) and limited precision in the representation of coefficients (equation parameters).

The full scope of the issues raised here is so great that it would be impossible to encompass it all in a single chapter. There are too many different kinds of equations to be dealt with. Some are linear, some nonlinear. Some have a unique solution, some have solutions of many values such as sampled time functions. Some have driving functions, some are homogeneous with given initial conditions. Some have feedback from the output to decrease the error, some are open–loop. Some have a single input and a single output, some have many inputs and many outputs, etc.

In this brief chapter, we will employ what we have learned about floating–point quantization to analyze roundoff errors in a nonlinear system. The ideas will be developed by studying a few simple cases. It is hoped that the reader will be able to use these ideas in solving new and unusual problems.

DEFINITION OF THE QUANTIZER

Quantization or roundoff occurs whenever physical quantities are represented numerically. The time displayed by a digital watch, the temperature indicated by a digital thermometer, the distances given on a map etc. are all examples of analog values represented by discrete numbers.

The values of measurements may be designated by integers corresponding to their nearest numbers of units. Roundoff errors have values between plus and minus one half unit, and can be made small by choice of the basic unit. It is apparent, however, that the smaller the size of the unit, the larger will be the numbers required to represent the same physical quantities and the greater will be the difficulty and expense in storing and processing these numbers. Often, a balance has to be struck between accuracy and economy. In order to establish such a balance, it is necessary to have a means of evaluating quantitatively the distortion resulting from rough quantization. The analytical difficulty arises from the inherent nonlinearities of the quantization process.

For purposes of analysis, it has been found convenient to define the quantizer as a nonlinear operator having the input–output staircase relation shown in Fig. 1.1(a). The quantizer output x' is a single–valued function of the input x, and the quantizer has an “average gain” of unity. The basic unit of quantization is designated by q.

System identification, as a particular process of statistical inference, exploits two types of information. The first is experiment; the other, called a priori, is known before making any measurements. In a wide sense, the a priori information concerns the system itself and signals entering the system. Elements of the information are, for example:

• the nature of the signals, which may be random or nonrandom, white or correlated, stationary or not, their distributions can be known in full or partially (up to some parameters) or completely unknown,

• general information about the system, which can be, for example, continuous or discrete in the time domain, stationary or not,

• the structure of the system, which can be of the Hammerstein or Wiener type, or other,

• the knowledge about subsystems, that is, about nonlinear characteristics and linear dynamics.

In other words, the a priori information is related to the theory of the phenomena taking place in the system (a real physical process) or can be interpreted as a hypothesis (if so, results of the identification should be necessarily validated) or can be abstract in nature.

This book deals with systems consisting of nonlinear memoryless and linear dynamic subsystems, for example, Hammerstein and Wiener systems and other related structures.