Linear Algebra Lecture Notes - Comprehensive Lecture Notes Semester Lecture Notes

Systems of Linear Equations and Row Operations

• Definition of Linear Equation: A linear equation in $n$ variables, $x_1, x_2, \dots, x_n$, is an equation of the form $a_1x_1 + a_2x_2 + \dots + a_nx_n = b$, where the coefficients $a_1, \dots, a_n$ and the constant term $b$ are constants.

• Definition of Linear System: A finite collection of linear equations in the variables $x_1, x_2, \dots, x_n$ is called a system of linear equations or a linear system.

  • Example:

    • $x_1 + 5x_2 + 6x_3 = 100$

    • $x_2 - x_3 = -1$

    • $-x_1 + x_3 = 11$

• Homogeneous Linear System: A system is called homogeneous if all the constants on the right-hand side are zero ($b_i = 0$ for all equations).

• Solution of a System: A set of values for the variables $x_1, \dots, x_n$ that satisfies every equation in the system.

Matrix Representations of Linear Systems

• Matrix: A rectangular array of numbers. An $m \times n$ matrix has $m$ rows and $n$ columns.

• Coefficient Matrix: Formed from the coefficients of the variables in the equations.

• Augmented Matrix: Formed from the coefficients and the constant terms, typically separated by a vertical line: $[A | b]$.

• Elementary Row Operations: Operations that simplify the augmented matrix without changing the solution set:

  1. Interchanging two rows ($R_i \leftrightarrow R_j$).

  2. Multiplying a row by a non-zero constant ($R_i \rightarrow kR_i$, where $k \neq 0$).

  3. Adding a multiple of one row to another row ($R_i \rightarrow R_i + kR_j$).

• Equivalence: Matrices produced after row operations are equivalent, denoted by $\sim$, meaning they share the same solution set.

Row-Echelon Form and Elimination

• Leading Entry: The first non-zero entry in each row of a matrix.

• Row-Echelon Form (REF): A matrix is in REF if:

  1. The leading entry of a lower row is further to the right than the leading entry in the higher row.

  2. Zero rows are grouped at the bottom.

• Reduced Row-Echelon Form (RREF): A matrix is in RREF if:

  1. It is in REF.

  2. Each leading entry is exactly $1$.

  3. In each column containing a leading $1$, all other entries are zero.

• Gaussian Elimination: A systematic way to reduce a matrix to REF. It involves creating zeros below leading entries. The solution is then found via back substitution.

• Gauss-Jordan Elimination: Reduces a matrix to RREF uniquely. It involves creating leading ones and zeros above each leading one.

Consistency and Solutions

• Existence of Solutions: A system has zero solutions (inconsistent), one solution (consistent), or infinitely many solutions (consistent).

• Inconsistency Condition: A system is inconsistent if and only if the REF of its augmented matrix has a row with all zero entries except for a non-zero final constant term ($0 = b$ where $b \neq 0$).

• Homogeneous System Consistency: Every homogeneous system is consistent because it always has the trivial solution ($x_1 = 0, \dots, x_n = 0$).

• Parameters for Infinite Solutions: If a consistent system has $n$ variables and $r$ non-zero rows in the REF, then $n - r$ parameters are needed to specify the solution set. Columns without leading entries correspond to free variables.

• Superposition of Solutions: If $Ax = b$ is consistent, the general solution is $x = p + h$, where $p$ is a particular solution to $Ax = b$ and $h$ is the general solution to the homogeneous system $Ax = 0$.

Matrix Operations and Properties

• Special Matrices:

  • Square Matrix: Number of rows equals number of columns.

  • Diagonal Matrix: $A_{ij} = 0$ for $i \neq j$.

  • Upper Triangular: $A_{ij} = 0$ for $i > j$.

  • Lower Triangular: $A_{ij} = 0$ for $i < j$.

  • Zero Matrix ($0$): All entries are zero.

  • Identity Matrix ($I_n$): $A_{ij} = 1$ if $i=j$ and $0$ if $i \neq j$.

• Matrix Multiplication: The product $AB$ is defined if columns in $A$ equals rows in $B$. Entry $(AB)_{ij} = \sum_{k=1}^n A_{ik}B_{kj}$.

• Properties:

  • $A(B + C) = AB + AC$

  • $A(BC) = (AB)C$

  • $A I_n = A$

  • Note: Generally, $AB \neq BA$.

• Trace: $\text{Tr}(A) = \sum_{i=1}^n A_{ii}$.

• Transpose ($A^T$): Interchanging rows and columns ($A^T_{ij} = A_{ji}$).

  • $(AB)^T = B^T A^T$.

Matrix Inverses

• Definition: An $n \times n$ matrix $A$ is invertible if there exists a matrix $B$ such that $AB = BA = I_n$. $B$ is denoted $A^{-1}$.

• 2x2 Inverse Formula: If $A = \begin{pmatrix} a & b \\ c & d \end{pmatrix}$, then $A^{-1} = \frac{1}{ad-bc} \begin{pmatrix} d & -b \\ -c & a \end{pmatrix}$, provided $ad - bc \neq 0$.

• Invertibility Condition: $A$ is invertible if and only if its RREF is $I_n$ or its determinant is non-zero.

• Properties:

  • $(AB)^{-1} = B^{-1} A^{-1}$.

  • $(A^T)^{-1} = (A^{-1})^T$.

• Elementary Matrices: Matrices obtained by performing a single row operation on $I_n$. Multiplying $A$ on the left by an elementary matrix $E$ is equivalent to performing the row operation on $A$.

Rank and Determinants

• Rank: The number of non-zero rows in the REF of a matrix.

  • $\text{rank}(A) \le m$ and $\text{rank}(A) \le n$.

• Determinant (n x n): A function mapping square matrices to scalars.

  • Rules: Swapping two rows multiplies determinant by $-1$. Multiplying a row by $\alpha$ multiplies it by $\alpha$. Adding a multiple of one row to another doesn't change it. $\det(I_n) = 1$.

  • For triangular matrices, $\det(A)$ is the product of diagonal entries.

• Properties:

  • $\det(AB) = \det(A)\det(B)$.

  • $\det(A^T) = \det(A)$.

  • $\det(\alpha A) = \alpha^n \det(A)$.

• Cofactor Expansion:

  • Cofactor: $C_{ij} = (-1)^{i+j} \det(A(i,j))$ where $A(i,j)$ is $A$ without row $i$ and column $j$.

  • $\det(A) = \sum_{k=1}^n A_{ik}C_{ik}$ (expansion along $i$-th row).

Euclidean Vector Spaces

• Vectors in $\mathbb{R}^n$: Ordered n-tuples of real numbers $(v_1, v_2, \dots, v_n)$.

• Norm (Length): $\|u\| = \sqrt{u_1^2 + u_2^2 + \dots + u_n^2}$.

• Dot Product: $u \cdot v = \sum u_k v_k = \|u\| \|v\| \cos(\theta)$.

  • Cauchy-Schwarz Inequality: $|u \cdot v| \le \|u\| \|v\|$.

• Orthogonality: Two vectors are orthogonal if $u \cdot v = 0$.

• Vector Projection: $\text{proj}_u v = \frac{u \cdot v}{\|u\|^2} u$.

• Cross Product ($\mathbb{R}^3$ only): Result is a vector perpendicular to both $u$ and $v$. Magnitude equals area of the parallelogram spanned by $u$ and $v$.

• Scalar Triple Product: $u \cdot (v \times w)$ is the volume of the parallelepiped spanned by them (absolute value).

Lines and Planes

• Lines: Vector equation $r = r_0 + tv$. Parametric: $x = x_0 + ta, y = y_0 + tb, z = z_0 + tc$.

• Planes: Vector equation $r = r_0 + su + tv$. Point-normal form: $(r - r_0) \cdot n = 0$. Cartesian: $ax + by + cz = d$, where $n = (a, b, c)$.

• Geometric Intersection: Linear algebra methods (Gaussian elimination) are used to find intersection points of multiple planes or lines/planes.

General Vector Spaces

• Definition: A set $V$ with vector addition and scalar multiplication satisfying 10 axioms (closure, associativity, commutativity, existence of zero, additive inverse, distributivity, etc.).

• Examples: $\mathbb{R}^n$, $M_{m,n}$, $P_n$ (polynomials of degree at most $n$), and function spaces $F(S, \mathbb{R})$.

• Complex Vector Spaces: Use complex scalars $\mathbb{C}$.

• Integers Modulo 2 ($\mathbb{F}_2$): Field containing $\{0, 1\}$, used in coding theory (e.g., Hamming code).

Subspaces, Spans, and Independence

• Subspace: A subset that is itself a vector space. Must contain $0$ and be closed under addition and scalar multiplication.

• Span: The set of all linear combinations of vectors in a set.

• Linear Independence: A set of vectors is independent if $\sum \alpha_i v_i = 0$ implies all $\alpha_i = 0$.

  • In $\mathbb{R}^n$, if number of vectors $k > n$, they are linearly dependent.

• Basis: A linearly independent set that spans the space.

• Dimension ($\dim(V)$): The number of vectors in a basis.

  • $\dim(\mathbb{R}^n) = n$, $\dim(M_{m,n}) = mn$, $\dim(P_n) = n+1$.

Rank-Nullity and Matrix Spaces

• Column Space: Subspace of $F^m$ spanned by columns of $A$.

• Row Space: Subspace of $F^n$ spanned by rows of $A$.

• Solution Space (Nullspace): Subspace formed by solutions to $Ax = 0$.

• Rank-Nullity Theorem: For an $m \times n$ matrix, $\text{rank}(A) + \text{nullity}(A) = n$ (total columns).

• Coordinates: If $B = \{v_1, \dots, v_n\}$ is an ordered basis, any $v$ has unique coordinates $[v]_B$.

Linear Transformations

• Definition: Function $T: U \rightarrow V$ such that $T(u+v) = T(u)+T(v)$ and $T(\alpha u) = \alpha T(u)$.

• Standard Matrix Representation: $[T] = [T(e_1) | T(e_2) | \dots | T(e_n)]$

• Kernel (ker(T)): Set of vectors $u$ such that $T(u) = 0$.

• Image (im(T)): Set of vectors $v$ such that $v = T(u)$.

• Composition: Matrix representation of $R \circ S$ is the product $[R][S]$.

• Change of Basis: Transition matrix $P_{C,B}$ such that $[v]_C = P_{C,B}[v]_B$.

  • $[T]_B = P_{B,C} [T]_C P_{C,B}$.

Eigenvalues and Eigenvectors

• Definitions: $Av = \lambda v$ for non-zero $v$. $\lambda$ is the eigenvalue, $v$ is the eigenvector.

• Characterisic Equation: $\det(A - \lambda I) = 0$.

• Multiplicity: Algebraic (multiplicity of root in poly) vs. Geometric (dimension of eigenspace).

• Diagonalization: $A = PDP^{-1}$ where $D$ is diagonal ($D_{ii} = \lambda_i$) and $P$ contains eigenvectors as columns.

  • Requires $n$ linearly independent eigenvectors.

• Matrix Powers: $A^k = P D^k P^{-1}$.

• Genetics Application: Transition matrices can predict genotype frequencies over generations (Markov chains).

Inner Product Spaces

• Inner Product (Real): Generalizes dot product. Requisite properties: symmetry, linearity, and positivity.

• Complex (Hermitian) Inner Product: $\langle u, v \rangle = \overline{\langle v, u \rangle}$. Requires complex conjugates: $u \cdot v = \sum u_k \overline{v_k}$.

• Orthonormal Sets: Orthogonal set where each vector has unit length.

• Gram-Schmidt Procedure: Converts any basis into an orthonormal basis step-by-step.

• Least Squares: Solving $A^T A u = A^T y$ to find the line of best fit by minimizing $\sum (y_i - (a+bx_i))^2$.

• Orthogonal Matrices ($Q$): $Q^T = Q^{-1}$. Preserves distances and angles.

• Symmetric Matrices: Real symmetric matrices are always orthogonally diagonalizable ($A = Q D Q^T$).

• Singular Value Decomposition (SVD): $A = U S V^T$, where $U, V$ are orthogonal and $S$ contains singular values (square roots of eigenvalues of $A^T A$).

• Hermitian Matrices: $A^* = A$. All eigenvalues are real and they are unitary diagonalizable ($A = U D U^*$).