Applied Math and Machine Learning Basics for Deep Learning

Pages 1–7: Linear Algebra Foundations for Deep Learning

Linear algebra provides the mathematical framework for representing and transforming multidimensional data across deep learning pipelines, including input processing, hidden layer operations, convolutions, attention mechanisms, and optimization.

  • Scalar (0D0\text{D}): A single numerical value without direction, representing hyperparameters or metrics (e.g., learning rate =0.001= 0.001, accuracy =92.5%= 92.5\%).

  • Vector (1D1\text{D}): An ordered array of numbers storing input features, layer activations, weights, or word embeddings.

  • Matrix (2D2\text{D}): A two-dimensional grid of numbers structured into rows and columns, used to represent structured datasets and layer weight matrices.

  • Tensor (nDn\text{D}): The general multidimensional generalization of vectors and matrices, representing higher-order inputs such as color images (height×width×channels\text{height} \times \text{width} \times \text{channels}).

Pages 8–16: Vectors, Vector Operations, Norms, and Similarity

Vectors represent directed quantities or feature lists arranged as column vectors (n×1n \times 1) or row vectors (1×n1 \times n).

  • Addition & Scalar Multiplication: Vector addition is strictly defined element-wise for vectors of identical dimension. Scalar multiplication scales each element uniformly by scalar cc: c⋅v=[cv1,cv2,…,cvn]Tc \cdot \mathbf{v} = [c v_1, c v_2, \dots, c v_n]^T.

  • Linear Combination: For vectors v1,v2,…,vp∈Rn\mathbf{v}_1, \mathbf{v}_2, \dots, \mathbf{v}_p \in \mathbb{R}^n and scalars c1,c2,…,cpc_1, c_2, \dots, c_p, the combination vector is:

v=∑i=1pcivi\mathbf{v} = \sum_{i=1}^p c_i \mathbf{v}_i

  • Dot Product (Inner Product): Multiplies matching elements and aggregates the sum to produce a scalar output:

a⋅b=∑i=1naibi\mathbf{a} \cdot \mathbf{b} = \sum_{i=1}^n a_i b_i

  • Outer Product: Multiplies each element of a\mathbf{a} (m×1m \times 1) by every element of b\mathbf{b} (n×1n \times 1) to form an m×nm \times n matrix:

(a⊗b)ij=aibj(\mathbf{a} \otimes \mathbf{b})_{ij} = a_i b_j

  • Hadamard Product (Element-wise): Multiplies matching components of identical-dimension vectors:

a∘b=[a1b1,a2b2,…,anbn]T\mathbf{a} \circ \mathbf{b} = [a_1 b_1, a_2 b_2, \dots, a_n b_n]^T

  • Vector Norms:

    • L2L_2 Norm (Euclidean): Measures geometric length: ∥v∥2=∑i=1nvi2\lVert\mathbf{v}\rVert_2 = \sqrt{\sum_{i=1}^n v_i^2}

    • L1L_1 Norm (Manhattan): Sum of absolute values: ∥v∥1=∑i=1n∣vi∣\lVert\mathbf{v}\rVert_1 = \sum_{i=1}^n |v_i|

    • L∞L_\infty Norm (Max Norm): Largest absolute value component: ∥v∥∞=max⁡i∣vi∣\lVert\mathbf{v}\rVert_\infty = \max_i |v_i|

  • Cosine Similarity: Measures the directional alignment between two non-zero vectors regardless of scale:

Cosine Similarity=A⋅B∥A∥2∥B∥2\text{Cosine Similarity} = \frac{\mathbf{A} \cdot \mathbf{B}}{\lVert\mathbf{A}\rVert_2 \lVert\mathbf{B}\rVert_2}

Values range between 11 (identical orientation), 00 (orthogonal / perpendicular), and −1-1 (diametrically opposed).

Vector angle and cosine similarity diagram

Pages 17–21: Matrix Representation and Operations

Matrices hold layer transformations, dense weight connections, and mini-batches of data.

  • Matrix Addition & Subtraction: Computed element-wise across matrices sharing identical row and column dimensions.

  • Matrix-Vector Multiplication: Transforms vector x\mathbf{x} (n×1n \times 1) via matrix A\mathbf{A} (m×nm \times n) into vector y\mathbf{y} (m×1m \times 1), where each component yiy_i is the inner product of row ii of A\mathbf{A} with x\mathbf{x}.

  • Matrix-Matrix Multiplication: For A∈Rm×n\mathbf{A} \in \mathbb{R}^{m \times n} and B∈Rn×p\mathbf{B} \in \mathbb{R}^{n \times p}, the resulting matrix C=AB∈Rm×p\mathbf{C} = \mathbf{A}\mathbf{B} \in \mathbb{R}^{m \times p} is computed as:

Cij=∑k=1nAikBkjC_{ij} = \sum_{k=1}^n A_{ik} B_{kj}

  • Matrix Transpose: Swaps row and column indices such that B=AT\mathbf{B} = \mathbf{A}^T satisfies Bji=AijB_{ji} = A_{ij}.

Pages 22–30: Tensors, Special Matrices, and Array Operations

Tensors organize multi-dimensional structures: 1D1\text{D} tensors represent single-feature arrays, 2D2\text{D} tensors store sequence steps, 3D3\text{D} tensors represent image volumes (height×width×channels\text{height} \times \text{width} \times \text{channels}), and 4D4\text{D} tensors encode video clips (frames×height×width×channels\text{frames} \times \text{height} \times \text{width} \times \text{channels}).

1D vector, 2D matrix, and 3D tensor cube representations
  • Diagonal Matrix: A square matrix where all off-diagonal entries are zero (aij=0a_{ij} = 0 for all i≠ji \neq j).

  • Symmetric Matrix: A square matrix equal to its transpose (A=AT\mathbf{A} = \mathbf{A}^T, meaning aij=ajia_{ij} = a_{ji}).

  • Orthogonal Matrix: A square matrix whose rows and columns form orthonormal vectors:

ATA=AAT=I\mathbf{A}^T \mathbf{A} = \mathbf{A} \mathbf{A}^T = \mathbf{I}

  • Matrix Inversion: A square matrix A\mathbf{A} is invertible if det⁡(A)≠0\det(\mathbf{A}) \neq 0, satisfying AA−1=A−1A=I\mathbf{A}\mathbf{A}^{-1} = \mathbf{A}^{-1}\mathbf{A} = \mathbf{I}.

  • For 2×22 \times 2:

A−1=1ad−bc(d−b−ca)\mathbf{A}^{-1} = \frac{1}{ad - bc} \begin{pmatrix} d & -b \\ -c & a \end{pmatrix}

  • For 3×33 \times 3:

A−1=1det⁡(A)adj⁡(A)\mathbf{A}^{-1} = \frac{1}{\det(\mathbf{A})} \operatorname{adj}(\mathbf{A})

  where adj⁡(A)=CT\operatorname{adj}(\mathbf{A}) = \mathbf{C}^T is the transpose of the cofactor matrix C\mathbf{C}.

  • Concatenation: Joins arrays along a specified axis: Axis 0 stacks rows vertically, while Axis 1 concatenates columns horizontally.

  • Broadcasting: Automatically expands singleton or missing dimensions of smaller tensors to perform element-wise arithmetic across differing shapes without copying data in memory.

Pages 31–41: Probability Foundations for Machine Learning

Probability quantifies uncertainty in model predictions, classifications, and stochastic parameter initializations.

  • Fundamental Terms: Experiment (random process), Sample Space (SS, set of all outcomes), Event (E⊆SE \subseteq S), Probability (P(E)=favorable outcomestotal outcomesP(E) = \frac{\text{favorable outcomes}}{\text{total outcomes}}, 0≤P(E)≤10 \le P(E) \le 1).

  • Conditional Probability: The likelihood of event AA occurring given event BB has occurred:

P(A∣B)=P(A∩B)P(B)P(A \mid B) = \frac{P(A \cap B)}{P(B)}

  • Joint Probability: The likelihood of events AA and BB occurring simultaneously:

P(A∩B)=P(A)×P(B∣A)P(A \cap B) = P(A) \times P(B \mid A)

  • Marginal Probability: The unconditional probability of an isolated event (P(A)P(A)), obtained by summing over all conditioning factors.

  • Probability Distributions in Deep Learning:

    • Bernoulli Distribution (Discrete): Models binary outcomes (22 states); used in binary classification with Sigmoid activations and Binary Cross-Entropy loss.

    • Categorical Distribution (Discrete): Models multi-class outcomes (>2> 2 states); used in multi-class classification with Softmax activations and Categorical Cross-Entropy loss.

    • Gaussian / Normal Distribution (Continuous): Governs bell-shaped distributions; central to weight initialization strategies, batch normalization layers, and noise injection.

    • Uniform Distribution (Continuous): Allocates equal likelihood over an interval; used in bounded random weight initializations.

Pages 42–47: Machine Learning Paradigms and Learning Workflow

Machine learning enables systems to infer predictive functions directly from empirical distributions rather than explicit programmed logic.

  • Supervised Learning: Learns mapping functions from labeled datasets (y=f(x)y = f(\mathbf{x})) for discrete classification or continuous regression.

  • Unsupervised Learning: Discovers intrinsic structure, clusters, or latent subspaces from unlabeled data via clustering (K-Means, Hierarchical, DBSCAN), association rules (Apriori), or dimensionality reduction (PCA, t-SNE).

  • End-to-End Learning Protocol:

    1. Collect training data

    2. Clean data, impute missing values, and extract features

    3. Select algorithm

    4. Optimize parameters on training samples

    5. Generate forward predictions

    6. Compute empirical loss against ground truth

    7. Backpropagate error gradients to update parameters

    8. Validate model convergence

    9. Deploy for inference on unseen data

Pages 48–56: Simple Linear Regression

Simple linear regression models a linear relationship between an individual explanatory feature xx and a continuous target variable yy.

y^=b0+b1x\hat{y} = b_0 + b_1 x

  • Ordinary Least Squares Parameter Solutions:

b1=∑i=1n(xi−xˉ)(yi−yˉ)∑i=1n(xi−xˉ)2b_1 = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2}

b0=yˉ−b1xˉb_0 = \bar{y} - b_1 \bar{x}

  • Cost Function (Mean Squared Error): Quantifies average squared variance from the regression line:

J=1n∑i=1n(y^i−yi)2J = \frac{1}{n} \sum_{i=1}^n (\hat{y}_i - y_i)^2

Scatter plot showing line of regression, observed values, predicted values, and random error

Pages 57–66: Multiple Linear Regression

Multiple linear regression generalizes simple regression across nn distinct independent variables:

y=a0+a1x1+a2x2+⋯+anxn+ϵy = a_0 + a_1 x_1 + a_2 x_2 + \dots + a_n x_n + \epsilon

  • Matrix Formulation & Normal Equation:   Expressing input design matrix X\mathbf{X} (with a leading column of 11s for bias) and target vector Y\mathbf{Y}, the optimal closed-form parameter vector a^\hat{\mathbf{a}} is derived directly via:

a^=(XTX)−1XTY\hat{\mathbf{a}} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{Y}

Pages 67–76: Logistic Regression for Binary Classification

Logistic regression adapts linear formulation to binary classification by passing a linear combination through the Sigmoid activation to produce calibrated class probabilities.

z=b0+b1x1+⋯+bnxnz = b_0 + b_1 x_1 + \dots + b_n x_n

P=σ(z)=11+e−zP = \sigma(z) = \frac{1}{1 + e^{-z}}

  • Decision Boundary: Standard classification maps predictions using threshold 0.50.5:

Class={1if P≥0.50if P<0.5\text{Class} = \begin{cases} 1 & \text{if } P \ge 0.5 \\ 0 & \text{if } P < 0.5 \end{cases}

  • Loss Function: Evaluated via Binary Cross-Entropy (Log Loss) and minimized iteratively through Gradient Descent.

Pages 77–82: Biological Foundations, Neural Networks, and Perceptrons

Artificial neural networks mirror biological neural signaling pathways: dendrites receive incoming stimulus, the soma aggregates inputs, and the axon transmits activations to downstream synapses.

Biological neuron structure with dendrites, nucleus, cell body, axon, and synapses
  • Artificial Neural Network Architecture: Consists of an input layer (feature ingestion), hidden layers (feature extraction via affine transformations and non-linearities), and an output layer (task prediction).

Artificial neural network architecture with input, hidden, and output layers
  • Perceptron Model (Rosenblatt, 1958): The elementary unit of neural computing for linearly separable binary decisions:

z=∑i=1nwixi+bz = \sum_{i=1}^n w_i x_i + b

y={1if z≥threshold0otherwisey = \begin{cases} 1 & \text{if } z \ge \text{threshold} \\ 0 & \text{otherwise} \end{cases}

  • Perceptron Learning Rule: Weights and bias update iteratively on misclassified samples (t≠yt \neq y ) using learning rate η\eta:

wi←wi+η(t−y)xiw_i \leftarrow w_i + \eta(t - y)x_i

b←b+η(t−y)b \leftarrow b + \eta(t - y)

Pages 83–92: Perceptron Capabilities and Inherent Limitations

A single-layer perceptron successfully constructs linear hyperplanes to solve linearly separable logical operators like AND and OR.

Linear decision boundary separating classes for OR logic function
  • Fundamental Limitation: Single-layer perceptrons cannot classify non-linearly separable functions (e.g., XOR), as no single straight line can partition the space.

  • Resolution: Multi-Layer Perceptrons (MLP) containing hidden layers equipped with non-linear activation functions are required to form complex, non-linear decision boundaries.

Pages 93–105: Activation Functions in Neural Networks

Activation functions inject non-linear properties into neural architectures; without non-linear activations, multi-layer networks collapse mathematically into a single linear transformation.

  • Sigmoid: Maps real inputs to the range (0,1)(0, 1).

f(x)=11+e−xf(x) = \frac{1}{1 + e^{-x}}

Limitations: Suffers from vanishing gradients at saturation extremes, high computational cost, and non-zero-centered outputs.

  • Tanh (Hyperbolic Tangent): Maps inputs symmetrically to (−1,1)(-1, 1).

f(x)=tanh⁡(x)=ex−e−xex+e−xf(x) = \tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}

Zero-centered, but still exhibits vanishing gradients at large positive and negative saturations.

  • ReLU (Rectified Linear Unit): Standard activation in hidden feedforward and convolutional layers.

f(x)=max⁡(0,x)f(x) = \max(0, x)

Computationally efficient and avoids vanishing gradients for positive activations. Limitation: Prone to "Dying ReLU" when inputs consistently fall below zero.

  • Leaky ReLU: Mitigates the dying neuron defect by providing a persistent, small slope α\alpha (α≈0.01\alpha \approx 0.01 or 0.10.1) for negative inputs:

f(x)={xif x>0αxif x≤0f(x) = \begin{cases} x & \text{if } x > 0 \\ \alpha x & \text{if } x \le 0 \end{cases}

  • ELU (Exponential Linear Unit): Uses an exponential curve for negative values to ensure smooth transitions towards a saturation plateau:

f(x)={xif x>0α(ex−1)if x≤0f(x) = \begin{cases} x & \text{if } x > 0 \\ \alpha(e^x - 1) & \text{if } x \le 0 \end{cases}

  • GELU (Gaussian Error Linear Unit): Multiplies inputs by the Gaussian cumulative distribution function, smoothing the gating transition; standard in Transformer architectures (BERT, GPT, ViT):

GELU(x)=x⋅Φ(x)≈0.5x(1+tanh⁡(2π(x+0.044715x3)))\text{GELU}(x) = x \cdot \Phi(x) \approx 0.5x \left(1 + \tanh\left(\sqrt{\frac{2}{\pi}}\left(x + 0.044715x^3\right)\right)\right)

  • Softmax: Normalizes an unconstrained logit vector z\mathbf{z} into a probability distribution over nn mutually exclusive classes:

Softmax⁡(zi)=ezi∑j=1nezj\operatorname{Softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^n e^{z_j}}

Pages 106–111: Objective and Loss Functions

Loss functions evaluate model prediction errors against ground truth targets, providing the scalar objective minimized during training.

  • Mean Squared Error (MSE): Applied to continuous regression tasks:

MSE=1n∑i=1n(yi−y^i)2\text{MSE} = \frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2

  • Binary Cross-Entropy Loss (BCE / Log Loss): Used for two-class classification (y∈{0,1}y \in \{0, 1\}):

L=−1n∑i=1n[yiln⁡(pi)+(1−yi)ln⁡(1−pi)]\mathcal{L} = -\frac{1}{n} \sum_{i=1}^n \left[y_i \ln(p_i) + (1 - y_i) \ln(1 - p_i)\right]

  • Categorical Cross-Entropy Loss (CCE): Evaluated against one-hot encoded targets across CC discrete classes:

CCE=−1n∑i=1n∑j=1Cyijln⁡(pij)\text{CCE} = -\frac{1}{n} \sum_{i=1}^n \sum_{j=1}^C y_{ij} \ln(p_{ij})

Pages 112–119: Calculus, Partial Derivatives, and the Chain Rule

Optimization in neural networks uses differential calculus to calculate gradient vectors that guide parameter updates in gradient descent.

  • Fundamental Derivative Rules:

    • Constant Rule: ddx(c)=0\frac{d}{dx}(c) = 0

    • Power Rule: ddx(xn)=nxn−1\frac{d}{dx}(x^n) = n x^{n-1}

    • Constant Multiple Rule: ddx[cf(x)]=cf′(x)\frac{d}{dx}[c f(x)] = c f'(x)

    • Sum & Difference Rules: ddx[f(x)±g(x)]=f′(x)±g′(x)\frac{d}{dx}[f(x) \pm g(x)] = f'(x) \pm g'(x)

    • Product Rule: ddx(uv)=udvdx+vdudx\frac{d}{dx}(uv) = u \frac{dv}{dx} + v \frac{du}{dx}

    • Exponential Rule: ddx(ex)=ex\frac{d}{dx}(e^x) = e^x

    • Logarithmic Rule: ddx(ln⁡(x))=1x\frac{d}{dx}(\ln(x)) = \frac{1}{x}

  • Partial Differentiation: Measures the instantaneous rate of change of a multivariable function with respect to one specified variable while holding all other variables constant (∂z∂x\frac{\partial z}{\partial x}, ∂z∂y\frac{\partial z}{\partial y}).

  • Chain Rule: Enables differentiation of nested composite functions (y=f(u)y = f(u) where u=g(x)u = g(x)) by multiplying outer and inner rates of change:

dydx=dydu⋅dudx\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}

  • Backpropagation Mechanism: In deep networks, the chain rule propagates error signals backwards from the loss function LL through layer activations aa and pre-activations zz to calculate parameter gradients:

∂L∂w=∂L∂a⋅∂a∂z⋅∂z∂w\frac{\partial L}{\partial w} = \frac{\partial L}{\partial a} \cdot \frac{\partial a}{\partial z} \cdot \frac{\partial z}{\partial w}