Comprehensive Notes on Linear Algebra, Ordinary Differential Equations, and Numerical Methods

General Information and Authors

• These notes are on Mathematics - 1021, authored by Peeyush Chandra, A. K. Lal, V. Raghavendra, and G. Santhanam.

• Supported by a grant from MHRD.

Part I: Linear Algebra

Chapter 1: Matrices

Definition and Notation

• Matrix: A rectangular array of numbers, mostly concerned with real numbers.

• Rows and Columns: Horizontal arrays are rows; vertical arrays are columns.

• Order: A matrix with $m$ rows and $n$ columns has order $m imes n$.

• Representation: Denoted by $A = [a_{ij}]$ where $a_{ij}$ is the entry at the $i^{th}$ row and $j^{th}$ column.

• Vectors: A matrix with one column is a column vector; one row is a row vector.

• Equality: Matrices $A = [a_{ij}]$ and $B = [b_{ij}]$ of the same order are equal if $a_{ij} = b_{ij}$ for all $i, j$.

Special Matrices

• Zero Matrix: Each entry is zero, denoted by $0$.

• Square Matrix: Number of rows equals the number of columns ($m imes m$).

• Diagonal Entries: In a square matrix of order $n$, entries $a_{11}, a_{22}, ext{…}, a_{nn}$ form the principal diagonal.

• Diagonal Matrix: Entry $a_{ij} = 0$ for all $i eq j$. Denoted by $ext{diag}(d_1, ext{…}, d_n)$.

• Scalar Matrix: A diagonal matrix where all diagonal entries are equal ($d_i = d$).

• Identity Matrix ($I_n$): A square matrix where $a_{ii} = 1$ and $a_{ij} = 0$ for $i eq j$.

• Triangular Matrices:   - Upper Triangular: $a_{ij} = 0$ for i > j.   - Lower Triangular: $a_{ij} = 0$ for i < j.

Operations on Matrices

• Transpose: For an $m imes n$ matrix $A$, the transpose $A^t$ is an $n imes m$ matrix where entry $b_{ij} = a_{ji}$.   - Property: $(A^t)^t = A$.

• Addition: Defined for matrices of the same order as $C = [a_{ij} + b_{ij}]$.   - Properties: Commutative ($A + B = B + A$) and Associative ($(A + B) + C = A + (B + C)$).

• Scalar Multiplication: For $k ext{ in } ext{R}$, $kA = [ka_{ij}]$.

• Additive Inverse: $-A = (-1)A$ such that $A + (-A) = 0$.

• Matrix Multiplication:   - For $A$ ($m imes n$) and $B$ ($n imes r$), the product $AB = C$ ($m imes r$) is defined by $c_{ij} = ext{∑}<em>{k=1}^{n} a</em>{ik}b_{kj}$.   - Product $AB$ is defined only if the number of columns in $A$ equals the number of rows in $B$.   - Commutativity: Generally $AB eq BA$. If $AB = BA$, they commute.   - Associativity: $(AB)C = A(BC)$.   - Distributive law: $A(B + C) = AB + AC$.   - Multiplication by Identity: $AI_n = I_n A = A$.   - Multiplication by Diagonal Matrix: In $DA$, the $i^{th}$ row of $A$ is multiplied by $d_i$. In $AD$, the $j^{th}$ column of $A$ is multiplied by $d_j$.

• Transpose Laws: $(A + B)^t = A^t + B^t$ and $(AB)^t = B^t A^t$.

More Special Matrices

• Symmetric: $A^t = A$.

• Skew-Symmetric: $A^t = -A$. Diagonal entries must be zero.

• Orthogonal: $AA^t = A^t A = I$.

• Nilpotent: There exists a positive integer $k$ such that $A^k = 0$. The least such $k$ is the order of nilpotency.

• Idempotent: $A^2 = A$.

• Trace: For a square matrix, $ext{tr}(A) = a_{11} + a_{22} + ext{…} + a_{nn}$.   - Laws: $ext{tr}(A + B) = ext{tr}(A) + ext{tr}(B)$ and $ext{tr}(AB) = ext{tr}(BA)$.

Block Matrices

• Matrices can be decomposed into smaller blocks (submatrices).

• If $A = [P ext{ } Q]$ and B = egin{pmatrix} H \ K egin{pmatrix}, then $AB = PH + QK$.

• Block addition and multiplication require compatible partitions.

Matrices over Complex Numbers

• Conjugate ($Ā$): Replacing entries $a_{ij}$ with their complex conjugate $ā_{ij}$.

• Conjugate Transpose ($A^<em>$): Transpose of the conjugate matrix, $a_{ji}^</em> = ā_{ji}$.

• Hermitian: $A^* = A$.

• Skew-Hermitian: $A^* = -A$.

• Unitary: $A^<em>A = AA^</em> = I$.

• Normal: $AA^* = A^*A$.

Chapter 2: Linear System of Equations

Introduction to Linear Systems

• A linear system of $m$ equations in $n$ unknowns is $Ax = b$.

• $A$ is the coefficient matrix; $[A ext{ } b]$ is the augmented matrix.

• Homogeneous System: $Ax = 0$. Always has the trivial solution $x = 0$.

• Solution Set: Can have a unique solution, infinite solutions, or no solution.

Elementary Operations and Equivalent Systems

• Elementary Row Operations:   - $R_{ij}$: Interchange of rows $i$ and $j$.   - $R_k(c)$: Multiply row $k$ by non-zero constant $c$.   - $R_{kj}(c)$: Replace row $k$ by (row $k$ + $c imes$ row $j$).

• Row-Equivalent: Two matrices are row-equivalent if one is obtained from the other via elementary row operations.

• Lemma: Equivalent linear systems have the same set of solutions.

Gauss Elimination and Echelon Forms

• Gauss Elimination (Forward Elimination): Reducing the augmented matrix to an upper triangular form.

• Row Reduced Form:   - First non-zero entry in each row is 1 (Leading Term).   - Column containing the leading 1 has zeros elsewhere.

• Row Reduced Echelon Form (RREF):   - Satisfies row reduced form.   - Zero rows are at the bottom.   - Leading 1s appear in a staircase pattern (left to right).

• Variables:   - Basic Variables: Variables corresponding to leading columns.   - Free Variables: Variables not corresponding to leading columns.

• Gauss-Jordan Elimination: Includes forward elimination and back substitution to reach RREF.

Elementary Matrices

• Obtained by applying a single elementary row operation to the Identity matrix.

• Multiplying $A$ on the left by an elementary matrix performs the corresponding operation on $A$.

• Multiplying on the right performs column transformations.

Rank of a Matrix

• Row-rank: Number of non-zero rows in the row reduced form.

• Rank: Row-rank equals Column-rank. Denoted $ext{rank}(A)$.

• Theorem: If $ext{rank}(A) = r$, there exist elementary matrices such that PAQ = egin{pmatrix} I_r & 0 \ 0 & 0 egin{pmatrix}.

Consistency Theorem (Ax = b)

• Let $ext{rank}(A) = r$ and $ext{rank}([A ext{ } b]) = r_a$.

• Infinite Solutions: If r = r_a < n. Solution space format: $ext{{}u_0 + k_1u_1 + ext{…} + k_{n-r}u_{n-r} ext{}}$.

• Unique Solution: If $r = r_a = n$.

• No Solution (Inconsistent): If r < r_a.

• Homogeneous Systems: $Ax = 0$ has non-trivial solutions if and only if ext{rank}(A) < n.

Invertible Matrices

• Definition: Square matrix $A$ is invertible if there exists $B$ such that $AB = BA = I$.

• Uniqueness: The inverse $A^{-1}$ is unique.

• Properties:   - $(A^{-1})^{-1} = A$.   - $(AB)^{-1} = B^{-1} A^{-1}$.   - $(A^t)^{-1} = (A^{-1})^t$.

• Equivalent conditions for Invertibility:   - $A$ is invertible.   - $ext{rank}(A) = n$ (Full rank).   - RREF of $A$ is $I_n$.   - $A$ is a product of elementary matrices.   - $Ax = 0$ has only the trivial solution.   - $Ax = b$ is consistent for every $b$.

• Calculating Inverse: Apply Gauss-Jordan to $[A ext{ } I_n]$ to get $[I_n ext{ } A^{-1}]$.

Determinants

• Definition: Inductively defined for square matrices.

• Notation: $ext{det}(A) = ext{∑}<em>{j=1}^{n} (-1)^{1+j} a</em>{1j} ext{det}(A(1|j))$ where $A(1|j)$ follows deleting row 1 and column $j$.

• Minor ($A_{ij}$): Determinant of submatrix obtained by deleting row $i$ and column $j$.

• Cofactor ($C_{ij}$): $(-1)^{i+j} A_{ij}$.

• Properties:   - Interchanging two rows changes the sign.   - Multiplying a row by $c$ multiplies the determinant by $c$.   - If two rows are equal, $ext{det} = 0$.   - $ext{det}(AB) = ext{det}(A) ext{det}(B)$.   - $ext{det}(A^t) = ext{det}(A)$.

• Adjoint ($ext{Adj}(A)$): The transpose of the cofactor matrix.

• Standard Inverse Formula: $A^{-1} = rac{1}{ ext{det}(A)} ext{Adj}(A)$.

• Singular Matrix: $ext{det}(A) = 0$. Non-singular if $ext{det}(A) eq 0$.

• Cramer’s Rule: For non-singular $A$, solution coordinates are $x_j = rac{ ext{det}(A_j)}{ ext{det}(A)}$ where $A_j$ replaces column $j$ with vector $b$.

Chapter 3: Finite Dimensional Vector Spaces

Definitions and Examples

• Vector Space $V(F)$: A set with vector addition and scalar multiplication satisfying 8 axioms (associativity, commutativity of addition, zero vector, inverse, distributive laws).

• Examples:   - $ext{R}^n$: n-tuples of real numbers.   - $ext{P}_n( ext{R})$: Polynomials of degree $ext{≤ } n$.   - $ext{M}_n( ext{R})$: $n imes n$ matrices.   - $ext{C}([a, b])$: Continuous functions on $[a, b]$.

• Subspace: A subset $S$ of $V$ is a subspace if $ext{α}u + ext{β}v ext{ in } S$ for all vectors $u, v ext{ in } S$ and scalars $ext{α, β}$.

Linear Combination and Span

• Linear Combination: $ext{α}_1u_1 + ext{…} + ext{α}_nu_n$.

• Linear Span ($L(S)$): The set of all linear combinations of elements in $S$. It is the smallest subspace containing $S$.

• Row Space: Spanned by rows of a matrix. $ext{dim}( ext{RowSpace}) = ext{row rank}$.

• Column Space / Range ($ext{Im}(A)$): Spanned by columns of a matrix.

Linear Independence and Bases

• Linearly Dependent: Non-zero scalars exist such that $ext{∑ α}_iu_i = 0$.

• Linearly Independent: $ext{∑ α}_iu_i = 0$ implies all $ext{α}_i = 0$.

• Basis: A linearly independent set that spans $V$.

• Dimension ($ext{dim}(V)$): Number of vectors in a basis of a finite-dimensional space.

• Important Theorems:   - Any two bases have the same number of vectors.   - A linearly independent set can be extended to form a basis.   - $ext{dim}(M) + ext{dim}(N) = ext{dim}(M + N) + ext{dim}(M ext{ ∩ } N)$.   - $ext{Row Rank} = ext{Column Rank}$.

Ordered Bases and Coordinates

• Ordered Basis: A basis where elements have a fixed sequence.

• Coordinates ($[v]_B$): Column vector of coefficients required to represent $v$ in ordered basis $B$.

• Change of Basis Matrix: Let $B_1 = (u_1, ext{…}, u_n)$, $B_2 = (v_1, ext{…}, v_n)$. Then $[v]<em>{B1} = A[v]</em>{B2}$ where the $i^{th}$ column of $A$ is $[v_i]_{B1}$.

Chapter 4: Linear Transformations

Definitions and Basic Properties

• Linear Transformation: A map $T: V → W$ such that $T( ext{α}u + ext{β}v) = ext{α}T(u) + ext{β}T(v)$.

• Properties:   - $T(0) = 0$.   - A linear transformation is determined by its value on a basis.

• Inverse Transform: If $T$ is one-one and onto, $T^{-1}$ exists and is linear.

Matrix of a Linear Transformation

• Let $A = T[B_1, B_2]$. This matrix maps coordinates from basis $B_1$ of $V$ to basis $B_2$ of $W$.

• Identity: $[T(x)]<em>{B2} = A [x]</em>{B1}$.

Rank-Nullity Theorem

• Range ($R(T)$): Subspace of images $ext{{}T(v) ext{}}$.

• Null Space / Kernel ($N(T)$): Subspace of vectors mapping to 0.

• Rank ($ρ(T)$): dimension of $R(T)$.

• Nullity ($ν(T)$): dimension of $N(T)$.

• Theorem: $ρ(T) + ν(T) = ext{dim}(V)$.

• Invertibility: For $T: V → V$, $T$ is one-one ↔ $T$ is onto ↔ $T$ is invertible.

Similarity

• Matrices $B$ and $C$ are similar if $B = PCP^{-1}$.

• Similar matrices represent the same linear transformation in different bases.

• Theorem: If $B = T[B_1, B_1]$ and $C = T[B_2, B_2]$, then $C = A^{-1}BA$ where $A = I[B_2, B_1]$.

Chapter 5: Inner Product Spaces

Basic Definition and Norms

• Inner Product ($⟨u, v⟩$): Map $V imes V → F$ satisfying conjugate symmetry, linearity in first component, and positive definiteness.

• Norm / Length ($‖u‑$): $√⟨u, u⟩$.

• Cauchy-Schwartz Inequality: $|⟨u, v⟩| ≤ ‖u‑ ‖v‑$.

• Angle (θ): Defined in real spaces by ext{cos}(θ) = rac{⟨u, v⟩}{‖u‑ ‖v‑}}.

• Orthogonality: vectors are orthogonal if $⟨u, v⟩ = 0$.

• Pythagoras Theorem: If $⟨u, v⟩ = 0$, then $‖u - v‑^2 = ‖u‑^2 + ‖v‑^2$.

Orthonormal Sets and Gram-Schmidt

• Orthonormal Set: vectors are unit vectors and mutually orthogonal.

• Gram-Schmidt Process: Converts independent set $ext{{}u_1, u_2, ext{…}, u_n ext{}}$ to orthonormal set $ext{{}v_1, ext{…}, v_n ext{}}$.   - $w_i = u_i - ext{∑}_{j=1}^{i-1} ⟨u_i, v_j⟩v_j$   - $v_i = rac{w_i}{‖w_i‑}$

• QR Decomposition: Any square matrix $A = QR$ where $Q$ is orthogonal/unitary and $R$ is upper triangular.

Orthogonal Projections

• Subspace Orthogonal Complement ($W^⊥$): Set of all vectors orthogonal to every vector in $W$.

• Orthogonal Projection ($P_W$): Maps $v$ to its nearest vector in $W$.

• Matrix of Projection: If $A = [v_1, ext{…}, v_k]$, the matrix is $A A^t$.

• Self-Adjoint Operator: $⟨T(u), v⟩ = ⟨u, T(v)⟩$. Real symmetric matrices generate self-adjoint operators.

Chapter 6: Eigenvalues and Diagonalization

Definitions and Characteristics

• Characteristic Equation: $ext{det}(A - ext{λ}I) = 0$.

• Eigenvalue (λ): A root of the characteristic equation.

• Eigenvector ($x$): A non-zero vector such that $Ax = ext{λ}x$.

• Trace and Determinant: $ext{tr}(A) = ext{∑ λ}_i$ and $ext{det}(A) = ∏ λ_i$.

• Cayley Hamilton Theorem: A matrix satisfies its own characteristic equation, $p(A) = 0$.

• Independency: Eigenvectors corresponding to distinct eigenvalues are linearly independent.

Diagonalization

• A matrix is diagonalizable if there exists non-singular $P$ such that $P^{-1} AP = D$.

• Equivalent to having $n$ linearly independent eigenvectors.

• Unitary Diagonalizability:   - Hermitian matrices have real eigenvalues and orthonormal eigenvectors.   - Normal matrices ($AA^* = A^*A$) are unitarily diagonalizable.

• Schur’s Lemma: Every square matrix is unitarily similar to an upper triangular matrix.

Sylvester’s Law of Inertia and Quadratic Forms

• Quadratic Form: $Q(x) = x^t Ax$.

• Hermitian Form: $H(x) = x^* Ax$.

• Sylvester’s Law: Any Hermitian form can be represented as $H(x) = ext{∑}<em>{i=1}^p |y_i|^2 - ext{∑}</em>{j=p+1}^r |y_j|^2$ where $p$ and $r$ are invariant.

• Conic Sections: Uses eigenvalues of the associated quadratic form to classify ellipses, parabolas, and hyperbolas.

Part II: Ordinary Differential Equations

Chapter 7: First Order Differential Equations

Preliminaries

• Ordinary Differential Equation (ODE): Relation $f(x, y, y', ext{…}, y^{(n)}) = 0$.

• Order: Highest derivative power present.

• Solution: A function $y = f(x)$ satisfying the equation on interval $I$.

• General Solution: A family of solutions involving one or more arbitrary constants.

Solution Methods

• Separable Equations: $y' = g(y)h(x)$. Solve by integration: $∫ rac{dy}{g(y)} = ∫ h(x) dx$.

• Homogeneous Reducible: $y' = g(y/x)$. Use substitution $y = xu(x)$.

• Exact Equations: $M(x, y) dx + N(x, y) dy = 0$ is exact if $rac{∂M}{∂y} = rac{∂N}{∂x}$. General solution is $G(x, y) = c$.

• Integrating Factors ($Q$): Multiplier to make a non-exact equation exact.

• First Order Linear Equations: $y' + p(x)y = q(x)$.   - Integrating factor: $P(x) = e^{∫ p(x) dx}$.   - Solution: $y = P(x)^{-1} [∫ P(x)q(x) dx + c]$.

• Bernoulli Equations: $y' + p(x)y = q(x)y^a$. Reduced to linear via $u = y^{1-a}$.

Initial Value Problems (IVP)

• Problem involving an ODE and initial conditions ($y(x_0) = y_0$).

• Picard’s Successive Approximations: $y_n(x) = y_0 + ∫<em>{x_0}^x f(s, y</em>{n-1}(s)) ds$.

• Existence and Uniqueness: Picard's Theorem guarantees a unique local solution if $f$ and its derivative are continuous.

Applications

• Orthogonal Trajectories: Curves that intersect a given family at right angles. Found by replacing $y'$ with $-1/y'$ in the differential equation of the family.

• Euler’s Method: Numerical approximation defined by $y_{k+1} ≈ y_k + hf(x_k, y_k)$.

Chapter 8: Higher Order Linear Equations

Homogeneous Equations

• Superposition Principle: Linear combinations of solutions are also solutions.

• Wronskian ($W$): Determinant of functions and their derivatives. $W(y_1, y_2) = y_1y'_2 - y'_1y_2$.

• Independence: Solutions are independent if and only if the Wronskian is non-zero.

• Fundamental System: A set of $n$ linearly independent solutions for an $n^{th}$-order equation.

• Reduction of Order: If one solution $y_1$ is known, another is $y_2 = u(x)y_1$.

Constant Coefficients

• Characteristic equation: $p( ext{λ}) = 0$.

• Roots define solutions:   - Distinct real: $e^{ ext{λ}x}$.   - Repeated real: $e^{ ext{λ}x}, xe^{ ext{λ}x}, ext{…}$.   - Complex conjugates ($ext{α} ± i ext{β}$): $e^{ ext{α}x} ext{cos}( ext{β}x), e^{ ext{α}x} ext{sin}( ext{β}x)$.

Non-Homogeneous Equations

• General solution: $y = y_h + y_p$.

• Method of Undetermined Coefficients:   - Guess $y_p$ based on the forcing function $f(x)$.   - Handles exponentials, sines/cosines, and polynomials.

• Variation of Parameters: General method for finding $y_p$ using the Wronskian.   - $y_p = -y_1 ∫ rac{y_2 f}{W} dx + y_2 ∫ rac{y_1 f}{W} dx$.

Chapter 9: Power Series Solutions

Basic Theory

• Power Series: $∑ a_n(x - x_0)^n$.

• Radius of Convergence ($R$): Defined by $ext{lim} | rac{a_{n+1}}{a_n}|$ or the root test.

• Ordinary Point: A point $x_0$ where coefficients are analytic. Guarantees power series solutions.

Legendre Equations and Polynomials

• Equation: $(1 - x^2)y'' - 2xy' + n(n + 1)y = 0$.

• Legendre Polynomials ($P_n(x)$): Polynomial solutions satisfying $P_n(1) = 1$.

• Rodrigues’ Formula: $P_n(x) = rac{1}{2^n n!} rac{d^n}{dx^n} (x^2 - 1)^n$.

• Orthogonality: $∫_{-1}^1 P_n(x) P_m(x) dx = 0$ for $n eq m$.

• Normalization: $∫_{-1}^1 P_n^2(x) dx = rac{2}{2n + 1}$.

• Generating Function: $(1 - 2xt + t^2)^{-1/2} = ∑ P_n(x)t^n$.

Part III: Laplace Transform

Chapter 10: Laplace Transform

Definitions and Properties

• Definition: $L(f(t)) = F(s) = ∫_0^∞ e^{-st} f(t) dt$.

• Linearity: $L(af + bg) = aL(f) + bL(g)$.

• Shifting Theorems:   - s-Shifting: $L(e^{at}f(t)) = F(s - a)$.   - t-Shifting: $L(U_a(t)f(t - a)) = e^{-as}F(s)$.

• Derivatives: $L(f') = sF(s) - f(0)$.

• Integrals: $L(∫_0^t f(τ) dτ) = F(s)/s$.

• Convolution ($f ⋆ g$): $∫_0^t f(τ)g(t - τ) dτ$. Result is $L(f ⋆ g) = F(s)G(s)$.

Applications

• Solving Differential Equations: Transforms calculus operations into algebraic operations.

• Unit Step Function ($U_a(t)$): 0 for t < a, 1 for $t ≥ a$.

• Dirac Delta Function (δ(t)): Unit-impulse function with $L(δ(t - a)) = e^{-as}$.

Part IV: Numerical Applications

Chapters 11-13: Interpolation, Differentiation, and Integration

Difference Operators

• Forward ($Δ$): $Δy_k = y_{k+1} - y_k$.

• Backward ($∇$): $∇y_k = y_k - y_{k-1}$.

• Central ($δ$): $δy_k = y_{k+1/2} - y_{k-1/2}$.

• Shift ($E$): $Ey_k = y_{k+1}$.

• Averaging ($μ$): $μy_k = rac{1}{2}(y_{k+1/2} + y_{k-1/2})$.

Interpolation Formulae

• Newton’s Forward: Best used for values near the start of a table.

• Newton’s Backward: Best used for values near the end of a table.

• Lagrange’s Formula: Used for unequally spaced tabular points.

• Sterling’s Formula: Used for values near the middle of a table.

• Divided Differences: Recursive ratio $δ[x_i, x_j] = rac{f(x_i) - f(x_j)}{x_i - x_j}$.

Numerical Differentiation and Integration

• Differentiation: Derived by differentiating interpolating polynomials.

• Integration (Quadrature):   - Trapezoidal Rule: Integrates using linear approximation. Error related to $Δ^2 y$.   - Simpson’s Rule: Integrates using quadratic approximation. Requires an even number of intervals. Form: $rac{h}{3}[y_0 + y_n + 4(∑ y_{odd}) + 2(∑ y_{even})]$.