Diagonalisation and the Spectral Theorem

What does it mean for a matrix $A\in M_n(k)$ to be similar $B\in M_n(K)$?

There is an invertible matrix $P\in M_n(K)$ such that $B=P^{-1}AP$

Similar matrices have the same rank, trace, determinant and eigenvalues.

What does it mean for a matrix $A$ to be diagonalisable?

It is similar to a diagonal matrix

When is a matrix diagonalisable? State and prove the eigen basis criterion.

A matrix $A\in M_n(K)$ is diagonalisable if and only if there is a basis $\{v_1,\ldots,v_n\}$ of $K^n$ consisting of eigenvectors of $A$.

If $Av_i=\lambda_i v_i$ and $P=(v_1\ \cdots\ v_n)$, then $A=PDP^{-1}$, where $D=\operatorname{diag}(\lambda_1,\ldots,\lambda_n)$.

Proof sketch:

If $A$ is diagonalisable, then $A=PDP^{-1}$ for some invertible $P$ and diagonal $D$. Since $P$ is invertible, its columns $v_1,\ldots,v_n$ form a basis.

From $AP=PD$, we get $(Av_1\ \cdots\ Av_n)=(\lambda_1v_1\ \cdots\ \lambda_nv_n)$, so each $v_i$ is an eigenvector.

Conversely, if $\{v_1,\ldots,v_n\}$ is a basis of eigenvectors, let $P=(v_1\ \cdots\ v_n)$ and $D=\operatorname{diag}(\lambda_1,\ldots,\lambda_n)$.

Then $P$ is invertible and $AP=PD$, so $A=PDP^{-1}$. Hence $A$ is diagonalisable.

What’s an orthogonal matrix?

A matrix $P\in M_n(\mathbb{R})$ is orthogonal if and only if its columns form an orthonormal basis of $\mathbb{R}^n$.

Note: columns must form an orthonormal set.

$$PP^T=I_n$$

$$P^TP=I_n$$

What does it mean for A and B to be orthogonally similar?

$P^TAP=B$ where $P\in M_n(\mathbb{R})$ is an orthogonal matrix.

What does it mean for a matrix to be Orthogonally diagonalisable?

It is orthogonally similar to a diagonal matrix.

What can we say about the structure of $A\in M(\mathbb{R})$ if it is orthogonally diagonalisable and the proof?

$A$ is symmetric.

Proof.

Assume $A$ is orthogonally diagonalizable. Then there is an orthogonal matrix $P$ and a diagonal matrix $D$ such that $D=P^TAP$. Rearranging gives $A=PDP^T$.

Then transpose: $A^T=(PDP^T)^T=(P^T)^TD^TP^T=PD P^T=A$.

Therefore $A$ is symmetric.

Note the converse is true (but not examinable).

How do you diagonalise a matrix?

To diagonalise $A$,

1. find the eigenvalues by solving $\det(A-\lambda I)=0$.

2. Then find a basis for each eigenspace by solving $(A-\lambda I)x=0$.

3. If you get enough linearly independent eigenvectors to form a basis, put them as the columns of $P$.

4. Put the matching eigenvalues in the same order down the diagonal of $D$. Then $A=PDP^{-1}$.

How do you orthogonally diagonalise a matrix?

1. Find eigenvalues of A using the characteristic polynomial

2. For each eigenvalue, find an orthonormal basis $B_\lambda$ for the eigen space $E_\lambda$.

   1. To do this find any basis $B$ of $E_\lambda$ and

      1. if $dimE_\lambda$ =1 then normalise it

      2. if dimE_\lambda>1 apply the Gram Schmitt process to find an orthonormal basis $B_\lambda$

3. Let $P=(v_1…v_n)$ where $v_i$ are the orthonormal basis vectors of the eigenspaces.

Then $P^TAP=D$ where $D$ is a diagonal matrix with eigenvalues (with multiplicities) along the main diagonal?

What does it mean for

• A to be unitarily similar to B?

  • A to be unitarily diagonalisable?

• There exists a unitary matrix $U\in M_n(\mathbb{C})$ such that $U^*AU=B$

  • A is unitarily similar to a diagonal matrix.

What does it mean for $A$ to be Hermitian?

$$A^*=A$$

If $A\in M_n(\mathbb{C})$ has real eigenvalues and A is unitarily diagonalisable, what else can we say about A?

A is Hermitian

Proof that is $A$ is Hermitian then all eigenvalues of A are real?

Let $\lambda$ be an eigenvalue of A

$$\lambda\langle v,v\rangle=\langle\lambda v,v\rangle=\langle Av,v\rangle=\langle v,A^{*}v\rangle$$

$$=\langle v,Av\rangle=\langle v,\lambda v\rangle=\overline{\lambda}\langle v,v\rangle$$

State and prove the result about eigenvectors of a Hermitian matrix with distinct eigenvalues.

If $A$ is Hermitian and $v_1,v_2$ are eigenvectors with distinct eigenvalues $\lambda_1\neq\lambda_2$, then $v_1$ and $v_2$ are orthogonal.

Proof sketch: since $A$ is Hermitian, $\langle Av_1,v_2\rangle=\langle v_1,Av_2\rangle$. Using $Av_1=\lambda_1v_1$ and $Av_2=\lambda_2v_2$ gives $\lambda_1\langle v_1,v_2\rangle=\lambda_2\langle v_1,v_2\rangle$ since Hermitian matrices have real eigenvalues. Hence $(\lambda_1-\lambda_2)\langle v_1,v_2\rangle=0$. Since $\lambda_1\neq\lambda_2$, we get $\langle v_1,v_2\rangle=0$, so the eigenvectors are orthogonal.

What does it mean for $A\in M_n(\mathbb{C})$ to be normal?

$$AA^*=A^*A$$

Prove that if $A\in M_n(\mathbb C)$ is unitarily diagonalisable if and only if $A$ is normal ($\implies$ only).

Suppose $A$ is unitarily diagonalisable.

Then there is a unitary matrix $U$ and a diagonal matrix $D$ such that $U^*AU=D$. Rearranging gives $A=UDU^*$.

Taking adjoints gives $A^*=UD^*U^*$.

Since $D$ and $D^*$ are diagonal, they commute, so $DD^*=D^*D$. Therefore $AA^*=(UDU^*)(UD^*U^*)=UDD^*U^*$ and $A^*A=(UD^*U^*)(UDU^*)=UD^*DU^*$. Since $DD^*=D^*D$, we get $AA^*=A^*A$. Hence $A$ is normal.

What is Schur’s Theorem?

If $A\in M_n(\mathbb{C})$, then A is unitarily similar to an upper triangular matrix $T$. More over, the diagonal entries of $T$ are eigenvalues of $A$.

The matrix decomposition $A=UTU^*$ with $U$ unitary and $T$ upper triangular is called a Schur decomposition of A.

Remark: If $A$ is a real matrix then it is orthogonally similar to an upper triangular matrix $T$. Sometimes called the Triangularisation theorem.

What is the Spectral Theorem for Hermitian matrices?

If $A\in M_{n}\left(K\right)$ is Hermitian ($\mathbb{C}$) or symmetric ($\mathbb{R}$) then there exists a unitary or orthogonal $U\in M_n(K)$ and $\lambda_1,…,\lambda_n \in \mathbb{R}$ such that

$A = U \begin{pmatrix} \lambda_1 & 0 & \cdots & 0 \\ 0 & \lambda_2 & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & \lambda_n \end{pmatrix} U^*$.

The factorisation shows that a Hermitian matrix can be diagonalised using a unitary matrix $U$. In the real symmetric case, the equivalent factorisation uses an orthogonal matrix $P$.

Prove the Spectral Theorem in the complex case.

Let $A\in M_n(\mathbb C)$ be Hermitian, so $A^*=A$.

By Schur's theorem, there exists a unitary matrix $U$ and an upper triangular matrix $T$ such that $U^*AU=T$ . Taking adjoints gives $T^*=(U^*AU)^*=U^*A^*U=U^*AU=T$, since $A^*=A$.

Thus $T$ is Hermitian. But $T$ is upper triangular, while $T^*$ is lower triangular. Since $T=T^*$, it must be both upper and lower triangular, hence diagonal. Therefore $U^*AU=T$ is diagonal, so $A$ is unitarily diagonalisable. Also, because $T=T^*$, the diagonal entries of $T$ are real.

The real case proof follows by using orthogonal matrices instead of unitary matrices.

What’s the Single Value Decomposition theorem?.

Let $A \in M_{mn}(K)$. Then there exist matrices $U \in M_m(K)$ and $V \in M_n(K)$, which are unitary if $K=\mathbb{C}$ and orthogonal if $K=\mathbb{R}$, such that $A=U\Sigma V^*$ , where $\Sigma \in M_{mn}(\mathbb{R})$.

The singular values are the positive diagonal entries of $\Sigma$. They satisfy \sigma_1 \geq \sigma_2 \geq \cdots \geq \sigma_r > 0.

The number $r$ is the dimension of the vector space spanned by the columns of $A$. Equivalently, $r=\operatorname{rank}(A)$:

NOTES:

• The $\sigma_i$ are called the nonzero singular values of A

• The $u_i$ are called left singular vectors of A

  • The $v_i$ are called right singular vectors of A

What are $u_{1,}\ldots,u_{m}$ and $v_1,\ldots,v_{n}$ eigenvectors of in SVD. $A=U\Sigma V^{*}$

$u_i$ are eigen vectors of $AA^*$ of unit norm

$v_i$ are eigenvectors of $A^*A$ of unit norm

NOTE:

• $Av_{i}=\sigma_{i}u_{i}$ for $i=1,…,r$, 0 for $i=r+1,\ldots,n$

• $A^*u_i=\sigma_iv_i$ for $i=1,…,r$, 0 for $i=r+1,\ldots,m$

• $AA^*u_i=\sigma²_i u_i$ for $i=1,…,r$, 0 for $i=r+1,\ldots,m$

• $A^*Av_i=\sigma²_i v_i$ for $i=1,…,r$, 0 for $$

How do you find the singular values of a matrix $A$ for an SVD?

Form $A^*A$ or $A^*A$ , ($A^TA$ or $AA^T$ if $A$ is real). Find its eigenvalues $\lambda_i$. The singular values are $\sigma_i=\sqrt{\lambda_i}$, using the non-zero eigenvalues and listing them in decreasing order on the diagonal of $\Sigma$.

How do you find the left singular vectors in an SVD?

For $A=U\Sigma V^*$, the left singular vectors are the columns $u_i$ of $U$. Find them as unit eigenvectors of $AA^*$. Equivalently, if you already know the right singular vectors $v_i$ and non-zero singular values $\sigma_i$, use the shortcut $u_i=\frac{1}{\sigma_i}Av_i$.

How do you find the right singular vectors in an SVD?

For $A=U\Sigma V^*$, the right singular vectors are the columns $v_i$ of $V$. Find them as unit eigenvectors of $A^*A$, or $A^TA$ if $A$ is real.

Shortcut: if you already know the left singular vectors $u_i$ and non-zero singular values $\sigma_i$, use $v_i=\frac{1}{\sigma_i}A^*u_i$."