An important operation in signal processing and machine learning is dimensionality reduction. There are many such methods, but the starting point is usually linear methods that map data to a lower-dimensional set called a subspace. When working with matrices, the notion of dimension is quantified by rank. This chapter reviews subspaces, span, dimension, rank, and nullspace. These linear algebra concepts are crucial to thoroughly understanding the SVD, a primary tool for the rest of the book (and beyond). The chapter concludes with a machine learning application, signal classification by nearest subspace, that builds on all the concepts of the chapter.

We consider linear maps between normed spaces. We define what it means for a linear map to be bounded and show that this is equivalent to continuity. We define the norm of a linear operator and show that the space of all linear maps from X to Y is a vector space, which is complete if Y is complete. We give a number of examples and then discuss inverses and invertibility in some detail.

We recall some of the basic theory of linear algebra, beginning with the formal definition of a vector space. We then discuss linear maps between vector spaces and end by proving that every vector space has a basis using Zorn’s Lemma.