Applied Math and Machine Learning Basics for Deep Learning
Pages 1–7: Linear Algebra Foundations for Deep Learning
Linear algebra provides the mathematical framework for representing and transforming multidimensional data across deep learning pipelines, including input processing, hidden layer operations, convolutions, attention mechanisms, and optimization.
Scalar (): A single numerical value without direction, representing hyperparameters or metrics (e.g., learning rate , accuracy ).
Vector (): An ordered array of numbers storing input features, layer activations, weights, or word embeddings.
Matrix (): A two-dimensional grid of numbers structured into rows and columns, used to represent structured datasets and layer weight matrices.
Tensor (): The general multidimensional generalization of vectors and matrices, representing higher-order inputs such as color images ().
Pages 8–16: Vectors, Vector Operations, Norms, and Similarity
Vectors represent directed quantities or feature lists arranged as column vectors () or row vectors ().
Addition & Scalar Multiplication: Vector addition is strictly defined element-wise for vectors of identical dimension. Scalar multiplication scales each element uniformly by scalar : .
Linear Combination: For vectors and scalars , the combination vector is:
Dot Product (Inner Product): Multiplies matching elements and aggregates the sum to produce a scalar output:
Outer Product: Multiplies each element of () by every element of () to form an matrix:
Hadamard Product (Element-wise): Multiplies matching components of identical-dimension vectors:
Vector Norms:
Norm (Euclidean): Measures geometric length:
Norm (Manhattan): Sum of absolute values:
Norm (Max Norm): Largest absolute value component:
Cosine Similarity: Measures the directional alignment between two non-zero vectors regardless of scale:
Values range between (identical orientation), (orthogonal / perpendicular), and (diametrically opposed).

Pages 17–21: Matrix Representation and Operations
Matrices hold layer transformations, dense weight connections, and mini-batches of data.
Matrix Addition & Subtraction: Computed element-wise across matrices sharing identical row and column dimensions.
Matrix-Vector Multiplication: Transforms vector () via matrix () into vector (), where each component is the inner product of row of with .
Matrix-Matrix Multiplication: For and , the resulting matrix is computed as:
Matrix Transpose: Swaps row and column indices such that satisfies .
Pages 22–30: Tensors, Special Matrices, and Array Operations
Tensors organize multi-dimensional structures: tensors represent single-feature arrays, tensors store sequence steps, tensors represent image volumes (), and tensors encode video clips ().

Diagonal Matrix: A square matrix where all off-diagonal entries are zero ( for all ).
Symmetric Matrix: A square matrix equal to its transpose (, meaning ).
Orthogonal Matrix: A square matrix whose rows and columns form orthonormal vectors:
Matrix Inversion: A square matrix is invertible if , satisfying .
For :
For :
where is the transpose of the cofactor matrix .
Concatenation: Joins arrays along a specified axis: Axis 0 stacks rows vertically, while Axis 1 concatenates columns horizontally.
Broadcasting: Automatically expands singleton or missing dimensions of smaller tensors to perform element-wise arithmetic across differing shapes without copying data in memory.
Pages 31–41: Probability Foundations for Machine Learning
Probability quantifies uncertainty in model predictions, classifications, and stochastic parameter initializations.
Fundamental Terms: Experiment (random process), Sample Space (, set of all outcomes), Event (), Probability (, ).
Conditional Probability: The likelihood of event occurring given event has occurred:
Joint Probability: The likelihood of events and occurring simultaneously:
Marginal Probability: The unconditional probability of an isolated event (), obtained by summing over all conditioning factors.
Probability Distributions in Deep Learning:
Bernoulli Distribution (Discrete): Models binary outcomes ( states); used in binary classification with Sigmoid activations and Binary Cross-Entropy loss.
Categorical Distribution (Discrete): Models multi-class outcomes ( states); used in multi-class classification with Softmax activations and Categorical Cross-Entropy loss.
Gaussian / Normal Distribution (Continuous): Governs bell-shaped distributions; central to weight initialization strategies, batch normalization layers, and noise injection.
Uniform Distribution (Continuous): Allocates equal likelihood over an interval; used in bounded random weight initializations.
Pages 42–47: Machine Learning Paradigms and Learning Workflow
Machine learning enables systems to infer predictive functions directly from empirical distributions rather than explicit programmed logic.
Supervised Learning: Learns mapping functions from labeled datasets () for discrete classification or continuous regression.
Unsupervised Learning: Discovers intrinsic structure, clusters, or latent subspaces from unlabeled data via clustering (K-Means, Hierarchical, DBSCAN), association rules (Apriori), or dimensionality reduction (PCA, t-SNE).
End-to-End Learning Protocol:
Collect training data
Clean data, impute missing values, and extract features
Select algorithm
Optimize parameters on training samples
Generate forward predictions
Compute empirical loss against ground truth
Backpropagate error gradients to update parameters
Validate model convergence
Deploy for inference on unseen data
Pages 48–56: Simple Linear Regression
Simple linear regression models a linear relationship between an individual explanatory feature and a continuous target variable .
Ordinary Least Squares Parameter Solutions:
Cost Function (Mean Squared Error): Quantifies average squared variance from the regression line:

Pages 57–66: Multiple Linear Regression
Multiple linear regression generalizes simple regression across distinct independent variables:
Matrix Formulation & Normal Equation: Expressing input design matrix (with a leading column of s for bias) and target vector , the optimal closed-form parameter vector is derived directly via:
Pages 67–76: Logistic Regression for Binary Classification
Logistic regression adapts linear formulation to binary classification by passing a linear combination through the Sigmoid activation to produce calibrated class probabilities.
Decision Boundary: Standard classification maps predictions using threshold :
Loss Function: Evaluated via Binary Cross-Entropy (Log Loss) and minimized iteratively through Gradient Descent.
Pages 77–82: Biological Foundations, Neural Networks, and Perceptrons
Artificial neural networks mirror biological neural signaling pathways: dendrites receive incoming stimulus, the soma aggregates inputs, and the axon transmits activations to downstream synapses.

Artificial Neural Network Architecture: Consists of an input layer (feature ingestion), hidden layers (feature extraction via affine transformations and non-linearities), and an output layer (task prediction).

Perceptron Model (Rosenblatt, 1958): The elementary unit of neural computing for linearly separable binary decisions:
Perceptron Learning Rule: Weights and bias update iteratively on misclassified samples ( ) using learning rate :
Pages 83–92: Perceptron Capabilities and Inherent Limitations
A single-layer perceptron successfully constructs linear hyperplanes to solve linearly separable logical operators like AND and OR.

Fundamental Limitation: Single-layer perceptrons cannot classify non-linearly separable functions (e.g., XOR), as no single straight line can partition the space.
Resolution: Multi-Layer Perceptrons (MLP) containing hidden layers equipped with non-linear activation functions are required to form complex, non-linear decision boundaries.
Pages 93–105: Activation Functions in Neural Networks
Activation functions inject non-linear properties into neural architectures; without non-linear activations, multi-layer networks collapse mathematically into a single linear transformation.
Sigmoid: Maps real inputs to the range .
Limitations: Suffers from vanishing gradients at saturation extremes, high computational cost, and non-zero-centered outputs.
Tanh (Hyperbolic Tangent): Maps inputs symmetrically to .
Zero-centered, but still exhibits vanishing gradients at large positive and negative saturations.
ReLU (Rectified Linear Unit): Standard activation in hidden feedforward and convolutional layers.
Computationally efficient and avoids vanishing gradients for positive activations. Limitation: Prone to "Dying ReLU" when inputs consistently fall below zero.
Leaky ReLU: Mitigates the dying neuron defect by providing a persistent, small slope ( or ) for negative inputs:
ELU (Exponential Linear Unit): Uses an exponential curve for negative values to ensure smooth transitions towards a saturation plateau:
GELU (Gaussian Error Linear Unit): Multiplies inputs by the Gaussian cumulative distribution function, smoothing the gating transition; standard in Transformer architectures (BERT, GPT, ViT):
Softmax: Normalizes an unconstrained logit vector into a probability distribution over mutually exclusive classes:
Pages 106–111: Objective and Loss Functions
Loss functions evaluate model prediction errors against ground truth targets, providing the scalar objective minimized during training.
Mean Squared Error (MSE): Applied to continuous regression tasks:
Binary Cross-Entropy Loss (BCE / Log Loss): Used for two-class classification ():
Categorical Cross-Entropy Loss (CCE): Evaluated against one-hot encoded targets across discrete classes:
Pages 112–119: Calculus, Partial Derivatives, and the Chain Rule
Optimization in neural networks uses differential calculus to calculate gradient vectors that guide parameter updates in gradient descent.
Fundamental Derivative Rules:
Constant Rule:
Power Rule:
Constant Multiple Rule:
Sum & Difference Rules:
Product Rule:
Exponential Rule:
Logarithmic Rule:
Partial Differentiation: Measures the instantaneous rate of change of a multivariable function with respect to one specified variable while holding all other variables constant (, ).
Chain Rule: Enables differentiation of nested composite functions ( where ) by multiplying outer and inner rates of change:
Backpropagation Mechanism: In deep networks, the chain rule propagates error signals backwards from the loss function through layer activations and pre-activations to calculate parameter gradients: