Module 1: Basics of Linear Algebra

Flashcards reviewing key definitions, matrix operations, properties, norms, decompositions, and vector calculus from Module 1: Basics of Linear Algebra.

How are vectors and matrices represented in the module's notation?

Vectors are denoted by bold lowercase letters such as $\mathbf{x}$, while matrices are denoted by bold uppercase letters such as $\mathbf{A}$.

What do the symbols $\mathbf{0}_p$ and $\mathbf{I}_p$ represent?

$\mathbf{0}_p$ denotes the zero vector in $\mathbb{R}^p$, and $\mathbf{I}_p$ denotes the identity matrix of size $p \times p$.

What notation represents the Euclidean inner product and Euclidean norm of vectors?

$\mathbf{x}^\top \mathbf{y}$ denotes the Euclidean inner product of two vectors, and $\|\mathbf{x}\|_2$ denotes the Euclidean norm.

What is an $n \times m$ matrix $\mathbf{A}$?

An $n \times m$ matrix $\mathbf{A}$ is a rectangular array with $n$ rows and $m$ columns, where the entry in row $i$ and column $j$ is denoted $a_{ij}$.

Under what condition can two matrices $\mathbf{A}$ and $\mathbf{B}$ be added?

They can be added only if they have the same dimensions, i.e., $\mathbf{A}, \mathbf{B} \in \mathbb{R}^{m \times n}$, giving $\mathbf{A} + \mathbf{B} = [a_{ij} + b_{ij}]$.

How is the transpose of an $n \times m$ matrix $\mathbf{A}$ defined?

The transpose $\mathbf{A}^\top = [a_{ji}]$ is the $m \times n$ matrix obtained by swapping the rows and columns of $\mathbf{A}$.

What are the transposition identities for sums and products of matrices?

For conformable matrices, $(\mathbf{A} + \mathbf{B})^\top = \mathbf{A}^\top + \mathbf{B}^\top$ and $(\mathbf{A}\mathbf{C})^\top = \mathbf{C}^\top \mathbf{A}^\top$.

How is the entry $c_{ij}$ of the matrix product $\mathbf{C} = \mathbf{A}\mathbf{B}$ calculated?

Each entry $c_{ij} = \sum_{\ell=1}^{m} a_{i\ell} b_{\ell j}$ is the inner product of row $i$ of $\mathbf{A}$ with column $j$ of $\mathbf{B}$.

Is matrix multiplication commutative in general?

No, in general, $\mathbf{A}\mathbf{B} \neq \mathbf{B}\mathbf{A}$.

What defines a symmetric square matrix $\mathbf{A}$?

A square matrix $\mathbf{A}$ is symmetric if $\mathbf{A} = \mathbf{A}^\top$.

What condition characterizes an orthonormal square matrix $\mathbf{A}$?

An orthonormal square matrix satisfies $\mathbf{A}^\top \mathbf{A} = \mathbf{A}\mathbf{A}^\top = \mathbf{I}$, meaning its rows and columns are orthonormal.

What condition defines an idempotent square matrix?

A square matrix $\mathbf{A}$ is idempotent if $\mathbf{A}^2 = \mathbf{A}\mathbf{A} = \mathbf{A}$.

When is a symmetric matrix $\mathbf{A}$ positive definite?

A symmetric matrix $\mathbf{A}$ is positive definite if $\mathbf{x}^\top \mathbf{A}\mathbf{x} > 0$ for all $\mathbf{x} \neq \mathbf{0}$.

What is the rank of a matrix $\mathbf{A}$, denoted $\text{rank}(\mathbf{A})$?

The rank is the number of linearly independent columns of $\mathbf{A}$, which is equivalently the number of linearly independent rows.

What inequality relates $\text{rank}(\mathbf{A}\mathbf{B})$ to the ranks of $\mathbf{A}$ and $\mathbf{B}$?

$\text{rank}(\mathbf{A}\mathbf{B}) \leq \min\{\text{rank}(\mathbf{A}), \text{rank}(\mathbf{B})\}$.

What is the formula for the determinant of a $2 \times 2$ matrix $\mathbf{A} = \begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{bmatrix}$?

The determinant is $|\mathbf{A}| = a_{11}a_{22} - a_{12}a_{21}$.

How does scalar multiplication affect the determinant of an $n \times n$ matrix $\mathbf{A}$?

For an $n \times n$ matrix $\mathbf{A}$ and scalar $\alpha \in \mathbb{R}$, $$|\alpha \mathbf{A}| = \alpha^n |\mathbf{A}|$.

When is a square matrix considered singular?

A square matrix is singular if and only if its determinant is zero.

How is the matrix inverse $\mathbf{A}^{-1}$ defined for a full-rank square matrix $\mathbf{A} \in \mathbb{R}^{n \times n}$?

It is defined by $\mathbf{A}^{-1}\mathbf{A} = \mathbf{A}\mathbf{A}^{-1} = \mathbf{I}_n$.

What is the inverse of a matrix product $(\mathbf{A}\mathbf{B})^{-1}$ for full-rank matrices?

$(\mathbf{A}\mathbf{B})^{-1} = \mathbf{B}^{-1}\mathbf{A}^{-1}$, where $\mathbf{B} \in \mathbb{R}^{n \times n}$ is of full rank.

What is the trace of a square matrix $\mathbf{A} \in \mathbb{R}^{n \times n}$, denoted $\text{tr}(\mathbf{A})$?

The trace is the sum of its diagonal entries: $\text{tr}(\mathbf{A}) = \sum_{i=1}^n a_{ii}$.

What cyclic property does the trace satisfy for matrix products?

$\text{tr}(\mathbf{A}\mathbf{B}) = \text{tr}(\mathbf{B}\mathbf{A})$ whenever both products are defined.

What is the Kronecker product $\mathbf{C} = \mathbf{A} \otimes \mathbf{B}$ for $\mathbf{A} \in \mathbb{R}^{n \times m}$ and $\mathbf{B} \in \mathbb{R}^{k \times l}$?

It is the block matrix $\mathbf{C} \in \mathbb{R}^{nk \times ml}$ consisting of sub-blocks $a_{ij}\mathbf{B}$.

What three properties define a norm $\parallel \cdot \parallel : V \rightarrow \mathbb{R}$ on a vector space $V$?

1. Positive definiteness ($\|\mathbf{x}\| \geq 0$ and $\|\mathbf{x}\| = 0 \iff \mathbf{x} = \mathbf{0}$); 2. Absolute homogeneity ($\|\lambda \mathbf{x}\| = |\lambda| \|\mathbf{x}\|$); 3. Triangle inequality ($\|\mathbf{x} + \mathbf{y}\| \leq \|\mathbf{x}\| + \|\mathbf{y}\|$).

What are the definitions of the $\ell_1$ norm and $\ell_2$ norm for $\mathbf{x} \in \mathbb{R}^n$?

The $\ell_1$ norm is $\|\mathbf{x}\|_1 = \sum_{i=1}^n |x_i|$, and the $\ell_2$ norm is $\|\mathbf{x}\|_2 = \sqrt{\sum_{i=1}^n x_i^2}$.

What is the formula for the $\ell_p$ norm for $p \geq 1$?

$\|\mathbf{x}\|_p = \left( \sum_{i=1}^n |x_i|^p \right)^{1/p}$.

What is the formula for the maximum norm $\|\mathbf{x}\|_\infty$?

$\|\mathbf{x}\|_\infty = \max_{1 \leq i \leq n} |x_i|$.

What conditions must a matrix $\mathbf{A} \in \mathbb{R}^{n \times n}$ satisfy for a Cholesky decomposition to exist?

The matrix $\mathbf{A}$ must be square, symmetric, and positive definite.

What is the Cholesky decomposition of a matrix $\mathbf{A}$?

$\mathbf{A} = \mathbf{L}\mathbf{L}^\top$, where $\mathbf{L}$ is a lower-triangular matrix with positive diagonal entries called the Cholesky factor.

How is $\det(\mathbf{A})$ computed efficiently from its Cholesky factor $\mathbf{L}$?

For $\mathbf{A} = \mathbf{L}\mathbf{L}^\top$, $\det(\mathbf{A}) = \det(\mathbf{L})^2 = \left( \prod_{i=1}^n l_{ii} \right)^2$.

What two triangular steps are used to solve $\mathbf{A}\mathbf{x} = \mathbf{b}$ using the Cholesky factor $\mathbf{L}$?

1. Solve $\mathbf{L}\mathbf{z} = \mathbf{b}$; 2. Solve $\mathbf{L}^\top \mathbf{x} = \mathbf{z}$.

What is the definition of an eigenvector $\mathbf{p}$ and eigenvalue $\lambda$ for a matrix $\mathbf{A}$?

A nonzero vector $\mathbf{p} \in \mathbb{R}^n$ is an eigenvector of $\mathbf{A}$ with eigenvalue $\lambda \in \mathbb{R}$ if $\mathbf{A}\mathbf{p} = \lambda \mathbf{p}$.

What polynomial equation is solved to find the eigenvalues of $\mathbf{A}$?

The characteristic equation $\det(\mathbf{A} - \lambda \mathbf{I}) = 0$.

What is the spectral decomposition of a real symmetric matrix $\mathbf{A}$?

$\mathbf{A} = \mathbf{P}\mathbf{\Lambda}\mathbf{P}^\top$, where $\mathbf{P}$ is an orthonormal matrix of eigenvectors ($\mathbf{P}^\top\mathbf{P} = \mathbf{I}$) and $\mathbf{\Lambda}$ is a diagonal matrix of corresponding eigenvalues.

How are matrix powers $\mathbf{A}^k$ computed using eigendecomposition?

$\mathbf{A}^k = \mathbf{P}\mathbf{\Lambda}^k \mathbf{P}^{-1}$, where $\mathbf{\Lambda}^k = \text{diag}(\lambda_1^k, \dots, \lambda_n^k)$.

What is the Singular Value Decomposition (SVD) of a real matrix $\mathbf{A} \in \mathbb{R}^{m \times n}$?

$\mathbf{A} = \mathbf{U}\mathbf{\Sigma}\mathbf{V}^\top$, where $\mathbf{U} \in \mathbb{R}^{m \times m}$ and $\mathbf{V} \in \mathbb{R}^{n \times n}$ are orthonormal matrices, and $\mathbf{\Sigma} \in \mathbb{R}^{m \times n}$ contains non-negative singular values on its diagonal.

How are the singular vectors in SVD related to the eigendecomposition of $\mathbf{A}^\top\mathbf{A}$ and $\mathbf{A}\mathbf{A}^\top$?

The right singular vectors $\mathbf{V}$ are eigenvectors of $\mathbf{A}^\top\mathbf{A}$, and the left singular vectors $\mathbf{U}$ are eigenvectors of $\mathbf{A}\mathbf{A}^\top$.

What is the gradient of a quadratic form $f(\mathbf{x}) = \mathbf{x}^\top \mathbf{A} \mathbf{x}$ when $\mathbf{A}$ is symmetric?

$\nabla_\mathbf{x}(\mathbf{x}^\top \mathbf{A} \mathbf{x}) = 2\mathbf{A}\mathbf{x}$.

In R, why is solve(A, b) preferred over solve(A) %*% b for solving linear systems?

In R, solve(A, b) is numerically better and more efficient than calculating the explicit inverse.