A mini course on the functional analytic approach to convexity-based optimization theory. Familiarity with this appendix is a prerequisite for the main text.

Many applications require solving a system of linear equations 𝑨𝒙 = 𝒚 for 𝒙 given 𝑨 and 𝒚. In practice, often there is no exact solution for 𝒙, so one seeks an approximate solution. This chapter focuses on least-squares formulations of this type of problem. It briefly reviews the 𝑨𝒙 = 𝒚 case and then motivates the more general 𝑨𝒙 ≈ 𝒚 cases. It then focuses on the over-determined case where 𝑨 is tall, emphasizing the insights offered by the SVD of 𝑨. It introduces the pseudoinverse, which is especially important for the under-determined case where 𝑨 is wide. It describes alternative approaches for the under-determined case such as Tikhonov regularization. It introduces frames, a generalization of unitary matrices. It uses the SVD analysis of this chapter to describe projection onto a subspace, completing the subspace-based classification ideas introduced in the previous chapter, and also introduces a least-squares approach to binary classifier design. It introduces recursive least-squares methods that are important for streaming data.

This chapter is devoted to the multivariate normal and functions of it. We start by showing how linearity is essential to its definition, then we derive the main properties. These include characteristic and density functions, conditionals, and some of the normal distribution's exceptional properties: the equivalence of no-correlation and independence within the class of elliptical distributions, Cramér's deconvolution theorem, the equivalence of a random sample's normality with the independence of the sample's normal mean and chi-square variance. We also explore other properties such as fourth-order moments in multivariate normal (and elliptical) distributions, the convexity of the m.g.f., joint distributions of linear and quadratic forms and conditions for their independence, the same also for pairs of quadratic forms and their covariance, as well as decompositions of quadratic forms.