Linear Algebra: Projections and Least Squares

Vocabulary flashcards covering the fundamental concepts of projections in linear algebra, including the derivation of projection matrices, normal equations, and properties of orthogonality.

Projection $P$ (One-Dimensional)

The point on a line through vector $a$ that is closest to a vector $b$, calculated as $P = ax$ where $x = \frac{a^T b}{a^T a}$.

Error vector $e$

The difference between a vector $b$ and its projection $P$, represented by $e = b - P$, which is perpendicular to the subspace being projected onto.

Projection matrix $P$ (One-Dimensional)

The $n \times n$ matrix that produces a projection onto the line through $a$, defined by the formula $P = \frac{aa^T}{a^T a}$.

Symmetry property

A key characteristic of any projection matrix where the matrix equals its own transpose, expressed as $P^T = P$.

Idempotent property

The property of a projection matrix where applying it twice results in the same outcome as applying it once, expressed as $P^2 = P$.

Rank 1 Matrix

The rank of a projection matrix that projects vectors onto a one-dimensional line, formed by a column times a row ($aa^T$).

Normal Equations

The system of equations $A^T A \bar{x} = A^T b$ derived from the requirement that the error vector must be perpendicular to the column space of $A$.

$\bar{x}$ (Best estimated solution)

The combination of columns that produces the best possible solution in an overdetermined system, defined as $\bar{x} = (A^T A)^{-1} A^T b$.

Projection matrix $P$ (n-Dimensional)

The matrix that projects a vector onto a higher-dimensional subspace (the column space of $A$), calculated as $P = A(A^T A)^{-1} A^T$.

Column Space of $P$

The space consisting of all possible results when multiplying a vector by the projection matrix $P$, which is the line through $a$ or the column space of $A$.

Null Space of $A^T$

The subspace where the error vector $e$ resides, determined by the equation $A^T e = 0$, which confirms $e$ is perpendicular to the column space of $A$.

Line Fitting (Least Squares)

The application of projections to find the best straight line $b = C + Dt$ that fits a set of data points not lying on a single line.