Determinants, Eigenvectors and Eigenvalues

What does $\overline{M_{ij}}$ mean?

The (n-1)x(n-1) matrix obtained from M by deleting the ith row and jth column.

What’s the formula for the determinant of a matrix M?

If M is a 1×1 matrix then $detM=M_{11}$. If M is an $n\times n$ matrix with n>1 then the determinant of $M$ is:

$$detM=\sum_{k=1}^{n}\left(-1\right)^{k+1}M_{1k}\det\overline{M_{1k}}$$

What is the Laplace Expansion Theorem?

For any $n\times n$ matrix M and any natural numbers $1\le i\le n$ and $1\le j\le n$ the following hold:

• $$detM=\sum_{k=1}^{n}\left(-1\right)^{i+k}M_{ik}\det\overline{M_{ik}}$$

• $$detM=\sum_{k=1}^{n}\left(-1\right)^{k+j}M_{kj}\det\overline{M_{kj}}$$

Remark: The value $\det\overline{M_{ij}}$ is called the $(i,j)$ minor of M, while the value $(-1)^{i+j}det\overline{M_{ij}}$ is the $(i,j)$ cofactor of $M$.

If M is a $n\times n$ upper triangular matrix, what is $detM$?

$$detM=\prod_{i=1}^{n}M_{ii}$$

Let $A$ and $B$ be square matrices, what is the determinant of B equal to if A is transformed into B using each of the elementary row operations?

$A\underrightarrow{r_{i}\to\alpha r_{i}}B$ then $detB=\alpha detA$

$A\underrightarrow{r_{i}\to r_{i}+\lambda r_{j}}B$ then $detB=detA$

$A\underrightarrow{r_{i}\leftrightarrow r_{j}}B$ then $detB=-detA$

Proof that a matrix A is only invertible if and only if $detA\ne 0$.

Let R=RREF(A). No elementary row operations changes whether the determinant is 0.

Since R is obtained from A via a sequence of elementary row operations, $detA\ne 0 \iff detR\ne 0$. However, R is $n\times n$ and is in RREF, so it follows either $R=I_n$ or $R$ contains a row of zeros. It follows that $R=I_n$ iff $detR\ne 0$.

Hence $A$ is invertible $\iff R=I_n$

$$\iff detR\ne 0$$

$\iff detA \ne 0$.

What rules are the determinants of elementary matrices given by?

$$detE_{r_{i}\to\alpha r_{i}}=\alpha$$

$$detE_{r_i→r_i+\lambda r_j}=1$$

$$detE_{r_i\leftrightarrow r_j}=-1$$

What’s the adjugate of A, where A is a square matrix, and what are its properties?

let B be the matrix given by $B_{ij}=(-1)^{i+j}det\overline{A_{ji}}$

B is the adjugated of A.

$$AB=detAI_{n}=BA$$

in particular if $detA\ne 0$ then A is invertible and $A^{-1}=\frac{1}{detA} B$

What does $det(AB)=?$ and the proof of this

$$\det(AB)=\det(A)\det(B).$$

Proof:

1. Suppose $A$ is singular. Then $AB$ is singular, since if $AB$ were invertible, say $C=AB$, then $A(BC^{-1})=I_n$, so $A$ would be invertible. Hence $\det(A)=0$ and $\det(AB)=0$, so $\det(AB)=0=\det(A)\det(B).$

2. Case 2: Suppose $A$ is invertible. Then $A$ is a product of elementary matrices, so $A=E_kE_{k-1}\cdots E_2E_1$. For any elementary matrix $E$, $\det(EB)=\det(E)\det(B)$. Therefore $\det(A)=\det(E_k)\det(E_{k-1})\cdots\det(E_1)$ and $\det(AB)=\det(E_kE_{k-1}\cdots E_1B)=\det(E_k)\det(E_{k-1})\cdots\det(E_1)\det(B)=\det(A)\det(B).$

How can we work out the determinant if the matrix is in LU decomposition.

If $PA=LU$, where $P$ is a permutation matrix, $L$ has diagonal entries $1$, and $U$ is upper triangular, then $\det(A)=\pm\prod_{i=1}^{n}U_{ii}$. The sign depends on whether the number of row swaps in $P$ is even or odd (+ if even, - if odd).

What’s the definition of an eigenvector with eigenvalue $\lambda?$

Let $V$ be a v.s over $K$ and $T:V→V$ be a linear transformation.

A vector $v\in V$ is an eigenvector of $T$ with eigen value $\lambda \in K$ if $v\ne 0$ and $T(v)=\lambda v$.

If $A\in M_n(K)$, then eigenvalues and eigenvectors of $A$ are the eigenvalues and eigenvectors of the linear map $T_A:V→V$, $Av=\lambda v$.

Remark: The zero vector is never an eigenvector.

How do we find eigenvalues and eigenvectors of a matrix?

To find eigenvalues:

• evaluate $det(A-\lambda I_n)=0$.

To find eigenvectors:

• $v$ is a eigenvector if it is a non-trivial solution to the system $(A-\lambda I_n)x=0$.

What’s the characteristic polynomial?

$P_A(x)=det(A-xI_n)$ is the characteristic polynomial of A.

What is an eigenspace?

The $\lambda$-eigenspace of linear map $T:V→V$ is the subspace $E_{\lambda}=\{v\in V|T(v)-\lambda v\}$

or with matrices

the $\lambda$-eigenspace of A is the $\lambda$-eigenspace of the linear map $T_A(v)=AV$.

In other words the collection of all eigenvectors together with the zero vector.

What is the geometric multiplicity of an eigenvalue?

The dimension of the corresponding $\lambda$-eigenspace $E_\lambda$.

Proof that $\{v_1,\ldots,v_k\}$ is linearly independent where $v_1,\ldots,v_k$ are eigenvectors of $T$ with distinct eigenvalues $\lambda_1,\ldots,\lambda_k$ respectively.

Assume for contradiction that $\{v_1,\ldots,v_k\}$ is linearly dependent.

Then choose $j$ such that $\{v_1,\ldots,v_{j-1}\}$ is linearly independent but $v_j=\sum_{i=1}^{j-1}\alpha_i v_i$, where the $\alpha_i$ are not all zero.

Applying $T$ gives $\lambda_jv_j=T(v_j)=\sum_{i=1}^{j-1}\alpha_iT(v_i)=\sum_{i=1}^{j-1}\alpha_i\lambda_i v_i$.

Also, multiplying $v_j=\sum_{i=1}^{j-1}\alpha_i v_i$ by $\lambda_j$ gives $\lambda_jv_j=\sum_{i=1}^{j-1}\lambda_j\alpha_i v_i$.

Subtracting gives $0=\sum_{i=1}^{j-1}(\lambda_i-\lambda_j)\alpha_i v_i$.

Since $\{v_1,\ldots,v_{j-1}\}$ is linearly independent, $(\lambda_i-\lambda_j)\alpha_i=0$ for all $i$.

But the eigenvalues are distinct, so $\lambda_i-\lambda_j\neq 0$. Hence $\alpha_i=0$ for all $i$, contradicting that the $\alpha_i$ are not all zero.

Therefore $\{v_1,\ldots,v_k\}$ is linearly independent.

What does it mean for a field to be algebraically closed?

If every polynomial $p(x)=a_0+a_1x+…+a_nx^n$ where $a_i\in K$ and $a_{i}\ne0$ can be factorised in the form $p(x)=b(x-c_1)…(x-c_n)$ for some $b,c_1,…,c_n\in K$.

What is the algebraic multiplicity of an eigenvalue $\lambda$?

How many times the eigenvalue appears as an root of the characteristic polynomial.

What is the trace of A?

$$tr(a)=A_{11}+\ldots+A_{nn}=\sum_{i=1}^{n}A_{ii}$$

If K is an algebraically closed field and $\lambda_1,..,\lambda_n$ are the eigenvalues of $A\in M_n(K)$, counted with multiplicity, what is the trace and determinant?

$$tr(A)=\lambda_1+…+\lambda_n$$

$$det(A)=\lambda_1…\lambda_n$$

What does it mean for a matrix $A\in M_n(k)$ to be similar $B\in M_n(K)$?

There is an invertible matrix $P\in M_n(K)$ such that $B=P^{-1}AP$

Similar matrices have the same rank, trace, determinant and eigenvalues.

What does it mean for a matrix to be diagonalisable?

If $A$ is similar to a diagonal matrix.

When is a matrix diagonalizable?

if and only if there is a basis $\{v_1,\ldots,v_n\}$ of $K^n$ such that each $v_i$ is an eigenvector of $A$. If $Av_i=\lambda_i v_i$ and $P=(v_1\ \cdots\ v_n)$, then $A=PDP^{-1}$ where $D=\begin{pmatrix}\lambda_1&0&\cdots&0\\0&\lambda_2&\cdots&0\\\vdots&\vdots&\ddots&\vdots\\0&0&\cdots&\lambda_n\end{pmatrix}$.

Proof:

Suppose $A$ is diagonalizable.

Then there is an invertible matrix $P$ and a diagonal matrix $D$ such that $D=P^{-1}AP$, so $A=PDP^{-1}$.

Since $P$ is invertible, its columns $v_1,\ldots,v_n$ form a basis of $K^n$.

Also $AP=PD$, so $(Av_1\ \cdots\ Av_n)=(\lambda_1v_1\ \cdots\ \lambda_nv_n)$.

Hence $Av_i=\lambda_i v_i$, so each $v_i$ is an eigenvector.

Conversely, suppose $\{v_1,\ldots,v_n\}$ is a basis of eigenvectors with $Av_i=\lambda_i v_i$. Let $P=(v_1\ \cdots\ v_n)$ and let $D$ be the diagonal matrix with diagonal entries $\lambda_1,\ldots,\lambda_n$. Since the columns of $P$ form a basis, $P$ is invertible. Then $AP=(Av_1\ \cdots\ Av_n)=(\lambda_1v_1\ \cdots\ \lambda_nv_n)=PD$. Therefore $A=PDP^{-1}$, so $A$ is diagonalizable.

What can we say about if A is diagonalizable, about geometric and algebraic multiplicities?

A is diagonalisable if and only if the algebraic and geometric multiplicities of each of its eigenvalues coincide.

What is an orthogonal matrix?

A matrix $P\in M_n(\mathbb{R})$ is orthogonal if and only if its columns form an orthonormal basis of $\mathbb{R}^n$.

Note: columns must form an orthonormal set.

What does it mean for A and B to be orthogonally similar?

$P^TAP=B$ where $P\in M_n(\mathbb{R})$ is an orthogonal matrix.

What does it mean for a matrix to be orthogonally diagonalizable.

It is orthogonally similar to a diagonal matrix.

What can we say about the structure of $A\in M(\mathbb{R})$ if it is orthogonally diagonalisable and the proof?

$A$ is symmetric.

Proof.

Assume $A$ is orthogonally diagonalizable. Then there is an orthogonal matrix $P$ and a diagonal matrix $D$ such that $D=P^TAP$. Rearranging gives $A=PDP^T$.

Then transpose: $A^T=(PDP^T)^T=(P^T)^TD^TP^T=PD P^T=A$.

Therefore $A$ is symmetric.

Note the converse is true (but not examinable).

How do you diagonalise a matrix?

To diagonalise $A$,

1. find the eigenvalues by solving $\det(A-\lambda I)=0$.

2. Then find a basis for each eigenspace by solving $(A-\lambda I)x=0$.

3. If you get enough linearly independent eigenvectors to form a basis, put them as the columns of $P$.

4. Put the matching eigenvalues in the same order down the diagonal of $D$. Then $A=PDP^{-1}$.

What’s the process of orthogonally diagonalising a matrix?

1. Find eigenvalues of A using the characteristic polynomial

2. For each eigenvalue, find an orthonormal basis $B_\lambda$ for the eigen space $E_\lambda$.

   1. To do this find any basis $B$ of $E_\lambda$ and

      1. if $dimE_\lambda$ =1 then normalise it

      2. if dimE_\lambda>1 apply the Gram Schmitt process to find an orthonormal basis $B_\lambda$

3. Let $P=(v_1…v_n)$ where $v_i$ are the orthonormal basis vectors of the eigenspaces.

Then $P^TAP=D$ where $D$ is a diagonal matrix with eigenvalues (with multiplicities) along the main diagonal?

What is a unitary matrix?

$U\in M_n(\mathbb{C})$ is a unitary matrix if the columns form an orthonormal basis for $\mathbb{C}^n$

A matrix if unitary if and only if $U^{-1}$=?

$$U^*$$

This is the complex analogue of an orthogonal matrix.

What does it mean for

• A to be unitarily similar to B?

• A to be unitarily diagonalisable?

• There exists a unitary matrix $U\in M_n(\mathbb{C})$ such that $U^*AU=B$

  • A is unitarily similar to a diagonal matrix.

What does it mean for $A\in M_n(\mathbb{C})$ to be Hermitian?

$$A^*=A$$

If $A\in M_n(\mathbb{C})$ has real eigenvalues and A is unitarily diagonalisable, what else can we say about A?

A is Hermitian

Proof that is $A$ is Hermitian then all eigenvalues of A are real?

Let $\lambda$ be an eigenvalue of A

$$\lambda\langle v,v\rangle=\langle\lambda v,v\rangle=\langle Av,v\rangle=\langle v,A^{*}v\rangle$$

$$=\langle v,Av\rangle=\langle v,\lambda v\rangle=\overline{\lambda}\langle v,v\rangle$$

Proof that If A is Hermitian, then if $\lambda_1, \lambda_2$ are distinct eigenvalues with corresponding eigenvectors, then $v_1,v_2$ are orthogonal?

$\langle Av_1,v_2\rangle=\langle v_1,A^{*}v_2\rangle=\langle v_1,Av_2\rangle$ since it is Hermitian

Since $Av_1=\lambda_1 v_1$ etc we have

$$\lambda_1\langle v_1,v_2\rangle=\langle\lambda_1v_1,v_2\rangle=\langle v_1,\lambda_2v_2\rangle=\overline{\lambda_2}\langle v_1,v_2\rangle=\lambda_2\langle v_1,v_2\rangle$$

so $\left(\lambda_1-\lambda_2\right)\langle v_1,v_2\rangle=0$ and $\lambda_1 \ne \lambda_2$ so inner product is 0 and the vectors are orthogonal.

What does it mean for $A\in M_n(\mathbb{C})$ to be normal?

$$AA^*=A^*A$$

Prove that $A\in M_n(\mathbb{C})$ is unitarily diagonalisable if and only if it is normal ($\implies$ direction only, other direction non-examinable).

Let $A\in M_n(\mathbb{C})$ be unitarily diagonalisable. That is $U^*AU=D$ where $U$ is unitary and $D$ is diagonal.

Then $D^*=U^*A^*U$.

Since $D$ and $D^*$ are diagonal, we have $DD^*=D^*D$. Therefore

$$U^*AUU^*A^*U=U^*A^*U^*AU$$

$$\implies U^*AA^*U=U^*A^*AU$$

$\implies AA^*=A^*A$ by multiplying on the left by $U$ and on the right by $U^*$.

What is Schur’s Theorem (no proof)?

If $A\in M_n(\mathbb{C})$, then A is unitarily similar to an upper triangular matrix $T$. More over, the diagonal entries of $T$ are eigenvalues of $A$.

The matrix decomposition $A=UTU^*$ with $U$ unitary and $T$ upper triangular is called a Schur decomposition of A.

Remark: If $A$ is a real matrix then it is orthogonally similar to an upper triangular matrix $T$. Sometimes called the Triangularisation theorem.

What is the Spectral Theorem for Hermitian matrices?

If $A\in M_{n}\left(K\right)$ is Hermitian ($\mathbb{C}$) or symmetric ($\mathbb{R}$) then there exists a unitary or orthogonal $U\in M_n(K)$ and $\lambda_1,…,\lambda_n \in \mathbb{R}$ such that

$A = U \begin{pmatrix} \lambda_1 & 0 & \cdots & 0 \\ 0 & \lambda_2 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & \lambda_n \end{pmatrix} U^*$.

The factorisation shows that a Hermitian matrix can be diagonalised using a unitary matrix $U$. In the real symmetric case, the equivalent factorisation uses an orthogonal matrix $P$.

Prove the Spectral Theorem in the complex case.

$A \in M_n(\mathbb{C})$ is Hermitian. By Schur’s theorem, $\exists$ a unitary matrix $U$ and an upper triangular matrix $T$ such that $U^*AU=T$.

Since $A^*=A$, we have $T^*=(U^*AU)^*=U^*A^*(U^*)^*=U^*AU=T$.

Since $T$ is upper triangular, $T^*$ is lower triangular. But $T^*=T$, so $T$ is both upper and lower triangular, hence diagonal. Therefore $A$ is unitarily diagonalisable. Since $T=T^*$, the diagonal entries of $T$ are real.

Prove the Spectral Theorem in the $\mathbb{R}$ case.

Assume $A \in M_n(\mathbb{R})$ is symmetric. Since $A$ is symmetric, it is Hermitian, so it has real eigenvalues. By the triangularisation theorem, there exists an orthogonal matrix $P$ and an upper triangular matrix $T$ such that $P^TAP=T$. Since $A^T=A$, we have $T^T=(P^TAP)^T=P^TA^T(P^T)^T=P^TAP=T$. Since $T$ is upper triangular, $T^T$ is lower triangular. But $T^T=T$, so $T$ is both upper and lower triangular, hence diagonal. Therefore $P$orthogonally diagonalises $A$.

What is the Single Value Decomposition theorem?

Let $A \in M_{mn}(K)$. Then there exist matrices $U \in M_m(K)$ and $V \in M_n(K)$, which are unitary if $K=\mathbb{C}$ and orthogonal if $K=\mathbb{R}$, such that $A=U\Sigma V^*$, where $\Sigma \in M_{mn}(\mathbb{R})$.

The singular values are the positive diagonal entries of $\Sigma$. They satisfy \sigma_1 \geq \sigma_2 \geq \cdots \geq \sigma_r > 0.

The number $r$ is the dimension of the vector space spanned by the columns of $A$. Equivalently, $r=\operatorname{rank}(A)$:

NOTES:

• The $\sigma_i$ are called the nonzero singular values of A

• The $u_i$ are called left singular vectors of A

• The $v_i$ are called right singular vectors of A

What are $u_{1,}\ldots,u_{m}$ and $v_1,\ldots,v_{n}$ eigenvectors of in SVD

$u_i$ are eigen vectors of $AA^*$ of unit norm

$v_i$ are eigenvectors of $A^*A$ of unit norm

NOTE:

• $Av_{i}=\sigma_{i}u_{i}$ for $i=1,…,r$, 0 for $i=r+1,\ldots,n$

• $A^*u_i=\sigma_iv_i$ for $i=1,…,r$, 0 for $i=r+1,\ldots,m$

• $AA^*u_i=\sigma²_i u_i$ for $i=1,…,r$, 0 for $i=r+1,\ldots,m$

• $A^*Av_i=\sigma²_i v_i$ for $i=1,…,r$, 0 for $i=r+1,\ldots,n$

What is the Rank-Nullity Theorem?

Let $T:V→W$ be a linear transformation from a finite-dimensional vector space $V$ to an arbitrary space $W$. Then $rank(T)+nullity(T)=dim(V)$

Proof of the Rank-nullity theorem?

Let $\dim(V)=n$ and let $\mathcal{B}_1=\{v_1,\ldots,v_k\}$ be a basis for $\ker(T)$, so $\operatorname{nullity}(T)=k$. Since $\mathcal{B}_1$ is linearly independent, we can extend this to a basis of $V$: $\mathcal{B}=\{v_1,\ldots,v_k,v_{k+1},\ldots,v_n\}$.

We claim that $\mathcal{B}_2=\{T(v_{k+1}),\ldots,T(v_n)\}$ is a basis for $\operatorname{Im}(T)$.

First, it spans $\operatorname{Im}(T)$: if $w\in\operatorname{Im}(T)$, then $w=T(v)$ for some $v\in V$. Write $v=a_1v_1+\cdots+a_nv_n$. Since $v_1,\ldots,v_k\in\ker(T)$, $T(v_1)=\cdots=T(v_k)=0$, so $w=a_{k+1}T(v_{k+1})+\cdots+a_nT(v_n)$.

Now show linear independence.

Suppose $c_{k+1}T(v_{k+1})+\cdots+c_nT(v_n)=0$. Then $T(c_{k+1}v_{k+1}+\cdots+c_nv_n)=0$, so $c_{k+1}v_{k+1}+\cdots+c_nv_n\in\ker(T)$.

Therefore it is a linear combination of $v_1,\ldots,v_k$.

We can write $c_{k+1}v_{k+1}+\cdots+c_nv_n=c_1v_1+\cdots+c_kv_k$ for some $c_1,\ldots,c_k\in K$. Rearranging gives $-c_1v_1-\cdots-c_kv_k+c_{k+1}v_{k+1}+\cdots+c_nv_n=0$ so all coefficients are zero since $\mathcal{B}$ is linearly independent.

Hence $c_{k+1}=\cdots=c_n=0$, so $\mathcal{B}_2$ is linearly independent. Therefore $\mathcal{B}_2$ is a basis for $\operatorname{Im}(T)$ and has $n-k$ vectors. Hence $\operatorname{rank}(T)=n-k$ and $\operatorname{nullity}(T)=k$, so $\operatorname{rank}(T)+\operatorname{nullity}(T)=n=\dim(V)$.

What is a Row and Column Space of a matrix?

Let $A \in M_{mn}(K)$.

The Row Space of $A$ is the subspace of $K^n$ spanned by the rows of $A$, denoted by $Row(A)$ .

The column space of $A$ is the subspace of $$K^M spanned by the columns of A, denoted by $Col(A)$

Proof that if $A,B\in M_{mn}(K)$ are row equivalent, then $Row(B)=Row(A)$ and if they’re column equivalent, then $Col(B)=Col(A)$.

The matrix $A$ can be transformed into $B$ by applying a sequence of elementary row operations. Therefore the rows of $B$ are linear combinations of the rows of $A$ and so $Row(B)\subseteq Row(A)$. Reversing the row operations we can transform $B$ into $A$ so we get $Row(A) \subseteq Row(B)$.

$\implies Row(A)=Row(B)$.

Applying same argument with elementary column operations gives the second part.

Proof that for $A\in M_{mn}(K)$, the dimension of $Row(A)$ is the number of non-zero rows in RREF of $A$ and the non-zero rows of the RREF of $A$ for a basis for $Row(A)$.

Let $R$=RREF$A$. Since they are row equivalent, $Row(A)=Row(R)$ and so $dim(Row\left(A)\right)=dim(Row(R))$ .

Suppose $R$ has $k$ non-zero rows. Each of these has a leading 1 with all other entries in the same column equal to 0. Therefore the non-zero rows of $R$ are linearly independent. Hence the non-zero rows form a basis of $Row(R)=Row(A)$ and $dim(Row(A))=dim(row(R))=k$ as required.

What is the null space of $A$?

$Null(A)$ is the subset of $K^n$ consisting of solutions of the linear system $Av=0$, i.e

$$Null(A)=\{v\in K^n|Av=0\}$$

The dimension of Null(A) is called the nullity of A>

Remark: