ECS708 Machine Learning: Introduction to Probability

Course Overview and Foundations

• Course Module: ECS708 Machine Learning: Introduction to Probability.

• Academic Context: School of Electronic Engineering and Computer Science (EECS), Queen Mary University of London.

• Primary Lecturer: Ioannis Patras.

• Contributors: Mark Plumbley, Mike Davies, Kevin Murphy, and Sam Roweis.

• Core Subject: Probability Theory forms the foundational mathematical framework for handling phenomena characterized by randomness, uncertainty, and chance in machine learning systems.

Fundamentals of Probability Theory and Axioms

• Empirical Concept of Probability:

  • Probability addresses non-deterministic experiments where exact outcomes cannot be predicted beforehand.

  • The loose or relative frequency definition of probability is expressed as:     $\text{Prob}(\text{Outcome}) = \frac{\text{Number of Favorable Outcomes}}{\text{Total Number of Trials}}$

  • Coin Tossing Example:

  • In an experiment of $250$ coin trials, heads is observed $140$ times:       $\text{Prob}(\text{Heads}) = \frac{\text{No. of Heads}}{\text{No. of Trials}} = \frac{140}{250} = 0.56$

  • Although individual coin tosses are unpredictable, repeated trials yield predictable expected averages. For an additional $100$ spins, the expected number of heads is approximately:       $100 \times \text{Prob}(\text{Heads}) = 100 \times 0.56 = 56$

• Sample Space ($\Omega$):

  • The sample space, denoted by $\Omega$, is defined as the set of all possible outcomes of an experiment.

  • Coin Toss Sample Space: Discrete set containing heads ($H$) and tails ($T$):     $\Omega = \{H, T\}$

  • Die Roll Sample Space: Discrete set containing integers from $1$ to $6$:     $\Omega = \{1, 2, 3, 4, 5, 6\}$

  • Spinning Pointer Sample Space: Continuous space representing the angle $\theta$ in radians:     $\Omega = \{\theta : 0 \le \theta < 2\pi\}$

  • Note: The spinning pointer sample space is not discrete.

• Events:

  • An event $A$ is a subset of the sample space $\Omega$ ($A \subseteq \Omega$).

  • Die Roll Event Examples:

  • Elementary events: $\{1\}, \{2\}, \{3\}, \{4\}, \{5\}, \{6\}$ (shorthand: $'1'$, $'2'$, etc.).

  • Compound event 'odd': $\text{'odd'} = \{1, 3, 5\}$

  • Compound event 'even': $\text{'even'} = \{2, 4, 6\}$

  • Compound event 'less than 3': $\text{'less than 3'} = \{1, 2\}$

• Axioms of Probability:   Let $\mathcal{A}$ be the collection of valid events in $\Omega$, and let $A$ and $B$ be possible events. The measure $P$ is a function mapping events in $\mathcal{A}$ to real numbers in the closed interval $[0, 1]$ satisfying three fundamental axioms:

  1. Non-negativity: For any event $A$,      $P(A) \ge 0$

  2. Normalization (Certain Event): For the full sample space $\Omega$,      $P(\Omega) = 1$

  3. Additivity for Mutually Exclusive Events: If events $A$ and $B$ are mutually exclusive (meaning they cannot occur simultaneously, $A \cap B = \emptyset$), then:      $P(A \cup B) = P(A) + P(B)$

  • The notation $P(A \cup B)$ represents the probability that event $A$ occurs or event $B$ occurs.

• Construction of Probability Models:

  • A complete formal probability model is defined by the axiomatic triple:     $\{\Omega, \mathcal{A}, P\}$     where $\Omega$ is the sample space, $\mathcal{A}$ is the event space ($\sigma$-algebra), and $P$ is the probability measure.

  • Establishing an accurate probability model for real-world physical systems requires statistical estimation techniques.

• Fundamental Corollaries of Probability Axioms:

  1. Probability of the Empty Set:      $P(\emptyset) = 0$

  • Proof: Since $\Omega$ and $\emptyset$ are mutually exclusive ($\Omega \cap \emptyset = \emptyset$) and $\Omega \cup \emptyset = \Omega$, applying Axiom 3 gives $P(\Omega \cup \emptyset) = P(\Omega) + P(\emptyset)$. Therefore, $P(\Omega) = P(\Omega) + P(\emptyset)$, implying $P(\emptyset) = 0$

  1. General Addition Rule (Inclusion-Exclusion Principle):      $P(E_1 \cup E_2) = P(E_1) + P(E_2) - P(E_1 \cap E_2)$

  • Proof: Decompose $E_1 \cup E_2$ into mutually exclusive subsets: $E_1 \cup E_2 = E_1 \cup (E_2 \setminus (E_1 \cap E_2))$. Applying additivity yields $P(E_1 \cup E_2) = P(E_1) + P(E_2 \setminus (E_1 \cap E_2))$. Since $E_2 = (E_2 \setminus (E_1 \cap E_2)) \cup (E_1 \cap E_2)$ as a disjoint union, $P(E_2) = P(E_2 \setminus (E_1 \cap E_2)) + P(E_1 \cap E_2)$. Substituting $P(E_2 \setminus (E_1 \cap E_2)) = P(E_2) - P(E_1 \cap E_2)$ completes the proof.

Conditional Probability, Total Probability, and Bayesian Inference

• Conditional Probability:

  • The conditional probability of an event $A$ given that event $B$ has occurred is defined as:     $P(A|B) = \frac{P(A \cap B)}{P(B)}$     provided $P(B) > 0$

  • Real-world Example: The probability that a person is Greek ($A$) given that the person has blond hair ($B$).

  • Important Caveat: Conditional probability imposes no causal or temporal order between events $A$ and $B$.

• Corollaries of Conditional Probability:

  • Conditioned Certain Event: The conditioning event $B$ becomes the new sample space universe:     $P(B|B) = 1$

  • Mutually Exclusive Conditioning: If $A \cap B = \emptyset$, then:     $P(A|B) = 0$     If $A$ and $B$ cannot occur together, the occurrence of $B$ guarantees that $A$ did not occur.

  • Subset Conditioning: If $B \subset A$, then:     $P(A|B) = 1$     If $B$ is a subset of $A$, occurrence of $B$ guarantees occurrence of $A$

• Validity of Conditional Probability as a Probability Measure:

  • The conditioned measure $P(\cdot | B)$ satisfies all three probability axioms over $\Omega$:

  1. Non-negativity: $P(A|B) = \frac{P(A \cap B)}{P(B)} \ge 0$ (since $P(A \cap B) \ge 0$ and $P(B) > 0$)

  2. Certainty: $P(\Omega|B) = \frac{P(\Omega \cap B)}{P(B)} = \frac{P(B)}{P(B)} = 1$

  3. Additivity: If $A \cap C = \emptyset$, then $(A \cap B) \cap (C \cap B) = \emptyset$, so:        $P(A \cup C | B) = \frac{P((A \cup C) \cap B)}{P(B)} = \frac{P((A \cap B) \cup (C \cap B))}{P(B)} = \frac{P(A \cap B) + P(C \cap B)}{P(B)} = P(A|B) + P(C|B)$

• Total Probability Rule:

  • Let $A_1, A_2, \dots, A_n$ be a partition of the sample space $\Omega$, satisfying:

  1. $A_i \cap A_j = \emptyset$ for all $i \neq j$ (Mutually exclusive)

  2. \bigcup_{i=1}^n A_i = \Omega (Exhaustive cover)\n - For any event B, its marginal probability is calculated as:\n    P(B) = \sum_{k=1}^n P(B \cap A_k) = \sum_{k=1}^n P(B|A_k) P(A_k)\n - **Practical Utility**: The Total Probability Rule is used when conditional probabilities P(B|A_k)$and prior probabilities$P(A_k)$are known, but the marginal probability$P(B) is unobserved.\n\n- **Bayes' Rule**:\n - From the definition of conditional probability:\n    P(A \cap B) = P(A|B) P(B) = P(B|A) P(A)\n - Equating terms and isolating P(B|A)$or$P(A|B) yields Bayes' Rule:\n    P(B|A) = \frac{P(A|B) P(B)}{P(A)}\n    P(A|B) = \frac{P(B \cap A)}{P(B)}\n\n- **Bayesian Inference Terminology**:\n - Bayes' Rule is formulated for inference problems as:\n    P(A|B) = \frac{P(B|A) P(A)}{P(B)}\n - **Components**:\n - **P(A)$(Prior Probability of$A$)**: Probability of class/hypothesis$A$before observing data$B\n - **P(B|A)$(Likelihood)**: Probability of observing data$B$given hypothesis$A\n - **P(B)$(Prior Probability of$B$/ Evidence)**: Marginal probability of observing data$B\n - **P(A|B)$(Posterior Probability)**: Updated probability of hypothesis$A$after observing data$B\n - In classification systems, B$typically represents observed sensor feature data, while$A represents the target class or decision state.\n\n- **Worked Practical Example (Loan Application Risk Analysis)**:\n - **Problem Statement**:\n - A bank uses an automated classifier for loan approval.\n - 0.1\% of all loan applications will default (not be repaid).\n - The automated system correctly identifies defaulted loans 99\% of the time.\n - The system correctly identifies repaid loans 99\% of the time.\n - Target calculation: Find the probability that a loan will actually default given that the system flags it as default.\n - **Mathematical Formulation**:\n - Let D$be the event that a loan defaults ($D^c = loan repaid).\n - Let F_D be the event that the system flags the loan as default.\n - Prior probabilities:\n      P(D) = 0.001\n      P(D^c) = 1 - 0.001 = 0.999\n - System performance metrics (Likelihoods):\n      P(F_D | D) = 0.99\n      P(F_D^c | D^c) = 0.99 \implies P(F_D | D^c) = 1 - 0.99 = 0.01\n - **Step 1: Compute Marginal Probability P(F_D) via Total Probability Rule**:\n    P(F_D) = P(F_D | D) P(D) + P(F_D | D^c) P(D^c)\n    P(F_D) = (0.99 \times 0.001) + (0.01 \times 0.999) = 0.00099 + 0.00999 = 0.01098\n - **Step 2: Compute Posterior Probability P(D | F_D) via Bayes' Rule**:\n    P(D | F_D) = \frac{P(F_D | D) P(D)}{P(F_D)} = \frac{0.00099}{0.01098} = \frac{99}{1098} \approx 0.0901639\n - **Conclusion**: Given a default classification flag, there is only approximately a 9.02\% chance that the loan will actually default, illustrating the significant impact of low base-rate prior probabilities.\n\n# Independence in Probability\n\n- **Definition of Event Independence**:\n - Two events A$and$B$are independent if knowledge of$B$provides no information about the probability of$A:\n    P(A|B) = P(A)\n - Equivalent product definition:\n    P(A \cap B) = P(A) P(B)\n\n- **Mutual Independence for Multiple Events**:\n - Events A_1, A_2, A_3 are mutually independent if and only if every pair is independent AND:\n    P(A_1 \cap A_2 \cap A_3) = P(A_1) P(A_2) P(A_3)\n - **Critical Distinction**: Pair-wise independence across all pairs is a necessary condition, but is **not sufficient** to establish mutual independence.\n\n- **Properties of Independent Events**:\n - If events A$and$B are independent, then:\n - Complementary events A^c$and$B^c are mutually independent.\n - A^c$and$B are independent.\n - A$and$B^c are independent.\n - If events A, B, C are mutually independent, then:\n - A$and$(B \cap C) are independent.\n - A$and$(B \cup C) are independent.\n\n- **Independent Repeated Experiments Example (Two Fair Coins)**:\n - Sample space composed of ordered pairs:\n    \Omega = {(H, H), (H, T), (T, H), (T, T)}\n - Classical probabilities assigned equally:\n    P({(H, H)}) = P({(H, T)}) = P({(T, H)}) = P({(T, T)}) = \frac{1}{4}\n - Conditional evaluation for Coin 1 outcome given Coin 2 outcome:\n    P(\text{Coin 1} = H | \text{Coin 2} = H) = \frac{P({(H, H)})}{P(\text{Coin 2} = H)} = \frac{1/4}{1/2} = \frac{1}{2} = P(\text{Coin 1} = H)\n - Because conditional probabilities equal marginal probabilities across all combinations, trials are mutually independent.\n - Relative frequency interpretations of repeated experiments hold strictly under the assumption of independent trials.\n\n# Random Variables, CDFs, and PDFs\n\n- **Random Variables (RVs)**:\n - A Random Variable X is a function mapping outcomes of a random process to real numbers, representing states of the world.\n - Particular realized instantiations are denoted using lower case letters (x).\n\n- **Discrete Random Variables**:\n - Defined over countable outcome spaces.\n - The probability mass function (pmf) specifies the probability of exact point values:\n    p(x) = P(X = x)\n\n- **Continuous Random Variables**:\n - Defined over continuous real intervals.\n - **Cumulative Distribution Function (CDF)**: Denoted F(x)$or$P(X \le x)$, measures the probability that the random variable$X$assumes a value less than or equal to$x:\n    F(x) = P(X \le x)\n - **Probability Density Function (PDF)**: Denoted p(x), defined as the derivative of the CDF:\n    p(x) = \frac{d}{dx} P(X \le x) = \frac{d}{dx} F(x)\n - **Interval Approximation**:\n    p(x) \approx \frac{P(x \le X < x + \Delta x)}{\Delta x}\n    This relationship becomes exact as \Delta x \to 0\n\n- **Example PDF: The Continuous Uniform Distribution**:\n - A uniform distribution over interval [1, 2]:\n  \n  \n\n - **CDF Specification**:\n    F(x) = \begin{cases} 0 & x < 1 \ x - 1 & 1 \le x \le 2 \ 1 & x > 2 \end{cases}\n - **PDF Specification**:\n    p(x) = \begin{cases} 1 & 1 \le x \le 2 \ 0 & \text{otherwise} \end{cases}\n\n- **Fundamental Properties of PDFs**:\n 1. Non-negativity everywhere:\n     f(x) \ge 0\n 2. Integration over total space equals unity:\n     \int_{-\infty}^{\infty} f(x) dx = 1\n - **Mantra**: "Densities are for integrating." Probability over an interval [a, b]$is obtained by integration:$P(a \le X \le b) = \int_a^b p(x) dx.\n\n# Expectations, Sample Means, and Higher Order Statistics\n\n- **Ensemble Averages and Expectations**:\n - Characterizes random variables via summary statistical measures.\n - **Sample Characterization**: Computed directly from data points.\n - **Model Characterization**: Theoretical expectations derived from probability density functions.\n\n- **Sample Mean vs. Expected Value**:\n - **Sample Mean (\hat{m}_N$)**: Computed from$N$independent samples$x_1, x_2, \dots, x_N drawn from an experiment:\n    \hat{m}N = \frac{x_1 + x_2 + \dots + x_N}{N} = \frac{1}{N} \sum{i=1}^N x_i\n - **Expected Value (E{X}$or$m_X$or$\mu_X)**:\n - Continuous case:\n      E{X} = \int_{-\infty}^{\infty} x \cdot p(x) dx\n - Discrete case:\n      E{X} = \sum_{x} x \cdot P(X = x)\n\n- **Law of Large Numbers**:\n - States that as the number of independent sample draws approaches infinity (N \to \infty), the sample mean converges to the theoretical expected value:\n    \hat{m}N \to E{X}\n - **Fair Die Roll Example**:\n - Rolling an independent fair 6-sided die N$times ($x_i \in {1, 2, 3, 4, 5, 6}):\n    \hat{m}_N = \frac{1}{N} \sum{i=1}^N x_i \approx E{X} = \sum_{k=1}^6 \frac{k}{6} = \frac{1 + 2 + 3 + 4 + 5 + 6}{6} = \frac{21}{6} = 3.5\n\n- **Expectation of Functions of a Random Variable**:\n - Let g(x)$be a real-valued mapping function$g: \mathbb{R} \to \mathbb{R}$. The transformed variable$Y = g(X) is itself a random variable.\n - **Expected Value of Transformed Variable**:\n    E{Y} = E{g(X)} = \int_{-\infty}^{\infty} g(x) p(x) dx\n - **Sample Mean of Transformed Variable**:\n    \hat{m}N = \frac{y_1 + y_2 + \dots + y_N}{N} = \frac{g(x_1) + g(x_2) + \dots + g(x_N)}{N}\n - **Application Example**: p(x)$represents the probability density of population age$x$, and$g(x)$represents medical costs associated with age$x$.$E{g(X)} yields expected medical costs per person.\n\n- **Variance and Standard Deviation**:\n - Variance measures the statistical spread or dispersion of outcomes around the mean:\n    \text{Var}{X} = \sigma_X^2 = E{(X - E{X})^2}\n - Standard Deviation is the square root of variance:\n    \sigma_X = \sqrt{\text{Var}{X}}\n - **Mathematical Derivation of Variance Formula**:\n    \text{Var}{X} = \int{-\infty}^{\infty} (x - E{X})^2 p_X(x) dx\n    \text{Var}{X} = \int_{-\infty}^{\infty} (x^2 - 2x E{X} + E{X}^2) p_X(x) dx\n    \text{Var}{X} = \int_{-\infty}^{\infty} x^2 p_X(x) dx - 2 E{X} \int_{-\infty}^{\infty} x p_X(x) dx + E{X}^2 \int_{-\infty}^{\infty} p_X(x) dx\n    Because \int_{-\infty}^{\infty} p_X(x) dx = 1$and$\int_{-\infty}^{\infty} x p_X(x) dx = E{X}:\n    \text{Var}{X} = E{X^2} - 2 E{X}^2 + E{X}^2 = E{X^2} - E{X}^2\n\n- **Higher Order Statistics (HOS)**:\n - Many machine learning problems only require second-order statistics (mean m_X$and variance$\sigma_X^2).\n - Polynomial moments E{X^3}, E{X^4}, \dots provide refined statistical descriptions:\n - **Skewness** (3rd standardized moment):\n    \text{Skewness} = E\left{\left(\frac{X - m_X}{\sigma_X}\right)^3\right}\n - Measures distribution asymmetric lopsidedness.\n - **Kurtosis** (4th standardized moment shift):\n    \text{Kurtosis} = E\left{\left(\frac{X - m_X}{\sigma_X}\right)^4\right} - 3\n - Measures probability concentration in distribution tails.\n - Subtraction of 3 normalizes Gaussian distribution kurtosis to zero.\n - Positive kurtosis indicates a "heavy-tailed" distribution relative to Gaussian.\n\n# Parametric Continuous Distributions: The Gaussian Distribution\n\n- **Gaussian (Normal) Distribution**:\n - A ubiquitous continuous distribution parameterised entirely by its mean \mu$and variance$\sigma^2.\n - **Probability Density Function**:\n    p(x) = \frac{1}{\sigma \sqrt{2\pi}} \exp\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)\n\n  \n\n - The distribution is symmetric around \mu$, with peak density value$\frac{1}{\sigma \sqrt{2\pi}}$occurring at$x = \mu.\n\n# Bivariate and Multivariate Random Variables\n\n- **Joint Probability Density Function**:\n - For two continuous random variables X$and$Y$, individual marginal distributions$p_X(x)$and$p_Y(y) do not characterize dependency relationships between them.\n - The joint density p_{XY}(x, y) measures probability over incremental regions:\n    p_{XY}(x, y) \approx \frac{P({x < X \le x + \Delta x}, {y < Y \le y + \Delta y})}{\Delta x \Delta y}\n - Probability over any two-dimensional domain D \subset \mathbb{R}^2 is computed by double integration:\n    P((X, Y) \in D) = \iint_D p_{XY}(x, y) dx dy\n\n  \n\n- **Marginal Probability Densities**:\n - Derived from joint density by integrating out (marginalizing) unwanted variables:\n    p_X(x) = \int_{-\infty}^{\infty} p_{XY}(x, y) dy\n    p_Y(y) = \int_{-\infty}^{\infty} p_{XY}(x, y) dx\n - Geometrically, p_X(x)$equals the cross-sectional area slice through$p_{XY}(x, y)$along$X = x\n\n  \n\n- **Conditional Probability Densities**:\n - The conditional density of Y$given$X = x is defined as:\n    p_{Y|X}(y|x) = \frac{p_{XY}(x, y)}{p_X(x)}\n - Geometrically, conditional densities are normalized cross-sectional slices taken through the joint density volume:\n    \iint p(x, y | z) dx dy = 1\n\n  \n\n- **Continuous Total Probability Rule and Bayes' Rule**:\n - **Total Probability Rule for Densities**:\n    p_Y(y) = \int_{-\infty}^{\infty} p_{XY}(x, y) dx = \int_{-\infty}^{\infty} p_{Y|X}(y|x) p_X(x) dx\n - **Bayes' Rule for Densities**:\n    p(y|x) p(x) = p(x|y) p(y) = p(x, y)\n    p(y|x) = \frac{p(x|y) p(y)}{p(x)}\n\n- **Independent Random Variables**:\n - Random variables X$and$Y are independent if and only if their joint PDF factors into the product of marginal PDFs:\n    p_{XY}(x, y) = p_X(x) p_Y(y)\n\n  \n\n- **Distribution of the Sum of Two Independent RVs**:\n - Let Z = X + Y$, where$X$and$Y$are independent random variables with PDFs$p_X(x)$and$p_Y(y).\n - Express p_Z(z) using marginalization:\n    p_Z(z) = \int_{-\infty}^{\infty} p_{XZ}(x, z) dx = \int_{-\infty}^{\infty} p_{Z|X}(z|x) p_X(x) dx\n - For a fixed X = x$,$Z = x + Y \implies Y = Z - x$. The conditional density$p_{Z|X}(z|x)$represents a translation of$p_Y:\n    p_{Z|X}(z|x) = p_Y(z - x)\n - Substituting yields the **Convolution Integral**:\n    p_Z(z) = \int_{-\infty}^{\infty} p_Y(z - x) p_X(x) dx\n\n- **Expectation of the Sum of Two RVs**:\n - **Theorem**: E{X + Y} = E{X} + E{Y}\n - **Proof**:\n    E{X + Y} = \iint (x + y) p_{XY}(x, y) dx dy\n    Assuming independence (p_{XY}(x,y) = p_X(x)p_Y(y)):\n    E{X + Y} = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} (x + y) p_X(x) p_Y(y) dx dy\n    E{X + Y} = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} x p_X(x) p_Y(y) dx dy + \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} y p_X(x) p_Y(y) dx dy\n    E{X + Y} = \left( \int_{-\infty}^{\infty} x p_X(x) dx \right) \left( \int_{-\infty}^{\infty} p_Y(y) dy \right) + \left( \int_{-\infty}^{\infty} p_X(x) dx \right) \left( \int_{-\infty}^{\infty} y p_Y(y) dy \right)\n    Because \int p_X(x) dx = 1$and$\int p_Y(y) dy = 1:\n    E{X + Y} = \int_{-\infty}^{\infty} x p_X(x) dx + \int_{-\infty}^{\infty} y p_Y(y) dy = E{X} + E{Y}\n\n- **Variance of the Sum of Independent RVs**:\n - **Theorem**: If X$and$Y$are independent,$\text{Var}{X + Y} = \text{Var}{X} + \text{Var}{Y}\n - **Proof**:\n    \text{Var}{X + Y} = E{((X + Y) - E{X + Y})^2} = E{((X - m_X) + (Y - m_Y))^2}\n    \text{Var}{X + Y} = E{(X - m_X)^2 + 2(X - m_X)(Y - m_Y) + (Y - m_Y)^2}\n    \text{Var}{X + Y} = E{(X - m_X)^2} + 2 E{(X - m_X)(Y - m_Y)} + E{(Y - m_Y)^2}\n    \text{Var}{X + Y} = \text{Var}{X} + 2 \text{Cov}(X, Y) + \text{Var}{Y}\n    For independent variables, \text{Cov}(X, Y) = 0$, yielding$\text{Var}{X + Y} = \text{Var}{X} + \text{Var}{Y}.\n\n# Covariance, Correlation, and Independence vs. Uncorrelatedness\n\n- **Covariance**:\n - The covariance of two random variables X$and$Y measures their joint variability:\n    \text{Cov}(X, Y) \equiv \sigma_{XY} = E{(X - m_X)(Y - m_Y)} = E{XY} - m_X m_Y\n\n- **Pearson Correlation Coefficient**:\n - Normalized covariance measuring linear relation scale:\n    \rho_{XY} = \frac{\text{Cov}(X, Y)}{\sigma_X \sigma_Y}\n - **Properties**:\n - Bounds: -1 \le \rho_{XY} \le 1\n - If \rho_{XY} = 0$,$X$and$Y are uncorrelated.\n - If \rho_{XY} = \pm 1$,$Y$is a deterministic linear rescaling of$X$($Y = aX + b).\n\n- **Signal Processing Context Note**:\n - In signal processing literature, correlation is often defined directly as uncentered joint expectation: \text{Cor}(X, Y) = E{XY}.\n - Covariance relates to this cross-correlation via: \text{Cov}(X, Y) = \text{Cor}(X, Y) - m_X m_Y.\n\n- **Independence vs. Uncorrelatedness**:\n - **Theorem**: Independence implies uncorrelatedness.\n - *Proof*: If X$and$Y$are independent,$p_{XY}(x, y) = p_X(x) p_Y(y). Thus:\n      E{XY} = \iint x y p_X(x) p_Y(y) dx dy = \left( \int x p_X(x) dx \right) \left( \int y p_Y(y) dy \right) = E{X} E{Y} = m_X m_Y\n      Consequently, \text{Cov}(X, Y) = E{XY} - m_X m_Y = m_X m_Y - m_X m_Y = 0$, so$\rho_{XY} = 0\n - **Non-reversibility**: Uncorrelatedness **does not** imply independence (\text{Uncorrelated} Rightarrow \text{Independent}).\n - Covariance evaluates only second-order linear statistical dependence.\n - Independence requires all linear and non-linear function expectations to be decoupled (E{f(X) g(Y)} = E{f(X)} E{g(Y)}).\n - Uncorrelatedness is widely utilized due to its computational tractability.\n\n# N-Dimensional Random Variables and Multivariate Gaussian Distributions\n\n- **Multivariate Random Variables**:\n - Generalizes scalar random variables to N-dimensional column vectors:\n    \mathbf{X} = \begin{pmatrix} X_1 \ X_2 \ \vdots \ X_N \end{pmatrix}\n\n- **Multi-Dimensional Expectations**:\n - Expectation vector \mathbf{m}{\mathbf{x}} \in \mathbb{R}^N:\n    \mathbf{m}{\mathbf{x}} = E{\mathbf{X}} = \int_{-\infty}^{\infty} \dots \int_{-\infty}^{\infty} \mathbf{x} p_{\mathbf{X}}(\mathbf{x}) dx_1 \dots dx_N\n\n- **Covariance Matrix**:\n - Measures variance along individual dimensions and pairwise covariances across dimensions. Formally defined as an N \times N symmetric matrix:\n    \text{Cov}(\mathbf{X}) = E{(\mathbf{X} - \mathbf{m}{\mathbf{X}})(\mathbf{X} - \mathbf{m}{\mathbf{X}})^T}\n - Expanded matrix form:\n    \text{Cov}(\mathbf{X}) = \begin{pmatrix} \text{Var}{X_1} & \text{Cov}(X_1, X_2) & \dots & \text{Cov}(X_1, X_N) \ \text{Cov}(X_2, X_1) & \text{Var}{X_2} & \dots & \text{Cov}(X_2, X_N) \ \vdots & \vdots & \ddots & \vdots \ \text{Cov}(X_N, X_1) & \text{Cov}(X_N, X_2) & \dots & \text{Var}{X_N} \end{pmatrix}\n\n- **Cross-Covariance Matrix**:\n - For two multivariate random vectors \mathbf{X} \in \mathbb{R}^N$and$\mathbf{Y} \in \mathbb{R}^M:\n    \text{Cov}(\mathbf{X}, \mathbf{Y}) = E{(\mathbf{X} - \mathbf{m}{\mathbf{X}})(\mathbf{Y} - \mathbf{m}{\mathbf{Y}})^T}\n\n- **Covariance Matrix Exercise**:\n - Let X$and$Y$be uncorrelated scalar random variables with variances$\sigma_X^2$and$\sigma_Y^2$. Let$\mathbf{Z} = \begin{pmatrix} X \ Y \end{pmatrix}.\n - The covariance matrix \text{Cov}(\mathbf{Z}) is:\n    \text{Cov}(\mathbf{Z}) = \begin{pmatrix} \text{Var}{X} & \text{Cov}(X, Y) \ \text{Cov}(Y, X) & \text{Var}{Y} \end{pmatrix} = \begin{pmatrix} \sigma_X^2 & 0 \ 0 & \sigma_Y^2 \end{pmatrix}\n\n- **N-Dimensional Gaussian (Multivariate Normal) Distribution**:\n - A multivariate continuous random vector \mathbf{X} \in \mathbb{R}^N$follows a Gaussian distribution parameterized by mean vector$\mathbf{m}{\mathbf{x}}$and positive-definite covariance matrix$\text{Cov}(\mathbf{X}):\n    p{\mathbf{X}}(\mathbf{x}) = \frac{1}{(2\pi)^{N/2} \det(\text{Cov}(\mathbf{X}))^{1/2}} \exp\left( -\frac{1}{2} (\mathbf{x} - \mathbf{m}{\mathbf{x}})^T \text{Cov}(\mathbf{X})^{-1} (\mathbf{x} - \mathbf{m}{\mathbf{x}}) \right)\n\n- **Key Mathematical Properties of Multivariate Gaussians**:\n 1. **Linear Combinations**: If \mathbf{X}$and$\mathbf{Y}$are jointly Gaussian random vectors, any linear combination$\mathbf{Z} = a\mathbf{X} + b\mathbf{Y} is strictly Gaussian with mean:\n     \mathbf{m}{\mathbf{z}} = a\mathbf{m}{\mathbf{x}} + b\mathbf{m}_{\mathbf{y}}\n 2. **Equivalence of Uncorrelatedness and Independence**: If jointly Gaussian random vectors \mathbf{X}$and$\mathbf{Y}$are uncorrelated ($\text{Cov}(\mathbf{X}, \mathbf{Y}) = \mathbf{0}$), they are <strong>statistically independent</strong>.</p></li></ol><ul><li><p>For Gaussian distributions, linear independence (zero covariance) completely specifies total statistical independence.</p></li></ul></li></ul><p></p><h4>Key Mathematical Symbols and Notation</h4><ul><li><p><strong>$\Omega$(Omega)</strong>: The sample space; the set of all possible outcomes of an experiment.</p></li><li><p><strong>$\omega$</strong>: A single outcome within the sample space$\Omega$.</p></li><li><p><strong>$\emptyset$</strong>: The empty set; represents an impossible event with probability$0$</p></li><li><p><strong>$\cup$</strong>: Union operator;$A \cup B$denotes the event that event$A$or event$B$(or both) occur.</p></li><li><p><strong>$\cap$</strong>: Intersection operator;$A \cap B$denotes the event that both event$A$and event$B$occur.</p></li><li><p><strong>$A^c$</strong>: Complement of event$A$; denotes the event that$A$does not occur.</p></li><li><p><strong>$\subset$</strong>: Subset operator;$B \subset A$means event$B$is a subset of event$A$.</p></li><li><p><strong>$P(A)$</strong>: Marginal probability of event$A$occurring.</p></li><li><p><strong>$P(A|B)$</strong>: Conditional probability of event$A$occurring given that event$B$has occurred.</p></li><li><p><strong>$X$</strong>: Random variable (capital letter representing a random phenomenon or process).</p></li><li><p><strong>$x$</strong>: Realized value or instantiation of the random variable$X$(lowercase letter).</p></li><li><p><strong>$F(x)$</strong>: Cumulative Distribution Function (CDF), defined as$P(X \le x)$.</p></li><li><p><strong>$p(x)$</strong>: Probability Mass Function (PMF) for discrete random variables, or Probability Density Function (PDF) for continuous random variables.</p></li><li><p><strong>$\mu$or$\mu_X$or$E{X}$</strong>: Mean or expected value of a random variable$X$.</p></li><li><p><strong>$\sigma^2$or$\text{Var}{X}$</strong>: Variance of a random variable$X$, measuring statistical dispersion/spread.</p></li><li><p><strong>$\sigma$or$\sigma_X$</strong>: Standard deviation of a random variable$X$($\sqrt{\text{Var}{X}}$).</p></li><li><p><strong>$\text{Cov}(X, Y)$or$\sigma_{XY}$</strong>: Covariance between two random variables$X$and$Y$.</p></li><li><p><strong>$\rho_{XY}$</strong>: Pearson correlation coefficient between$X$and$Y$, bounded in$[-1, 1]$.</p></li><li><p><strong>$\mathbf{X}$</strong>: Vector-valued random variable in multi-dimensional space.</p></li><li><p><strong>$\text{Cov}(\mathbf{X})$</strong>: Covariance matrix for multi-dimensional vector random variables.</p></li></ul><h4>Course Overview and Foundations</h4><ul><li><p><strong>Course Module</strong>: ECS708 Machine Learning: Introduction to Probability.</p></li><li><p><strong>Academic Context</strong>: School of Electronic Engineering and Computer Science (EECS), Queen Mary University of London.</p></li><li><p><strong>Primary Lecturer</strong>: Ioannis Patras.</p></li><li><p><strong>Contributors</strong>: Mark Plumbley, Mike Davies, Kevin Murphy, and Sam Roweis.</p></li><li><p><strong>Core Subject</strong>: Probability Theory forms the foundational mathematical framework for handling phenomena characterized by randomness, uncertainty, and chance in machine learning systems.</p></li></ul><h4>Fundamentals of Probability Theory and Axioms</h4><ul><li><p><strong>Empirical Concept of Probability</strong>:</p><ul><li><p>Probability addresses non-deterministic experiments where exact outcomes cannot be predicted beforehand.</p></li><li><p>The loose or relative frequency definition of probability is expressed as:<br>$\text{Prob}(\text{Outcome}) = \frac{\text{Number of Favorable Outcomes}}{\text{Total Number of Trials}}$</p></li></ul></li><li><p><strong>Coin Tossing Example</strong>:</p><ul><li><p>In an experiment of$250$coin trials, heads is observed$140$times:<br>$\text{Prob}(\text{Heads}) = \frac{\text{No. of Heads}}{\text{No. of Trials}} = \frac{140}{250} = 0.56$</p></li><li><p>Although individual coin tosses are unpredictable, repeated trials yield predictable expected averages. For an additional$100$spins, the expected number of heads is approximately:<br>$100 \times \text{Prob}(\text{Heads}) = 100 \times 0.56 = 56$</p></li></ul></li><li><p><strong>Sample Space ($\Omega$)</strong>:</p><ul><li><p>The sample space, denoted by$\Omega$, is defined as the set of all possible outcomes of an experiment.</p></li><li><p><strong>Coin Toss Sample Space</strong>: Discrete set containing heads ($H$) and tails ($T$):<br>$\Omega = {H, T}$</p></li><li><p><strong>Die Roll Sample Space</strong>: Discrete set containing integers from$1$to$6$:<br>$\Omega = {1, 2, 3, 4, 5, 6}$</p></li><li><p><strong>Spinning Pointer Sample Space</strong>: Continuous space representing the angle$\theta$in radians:<br>$\Omega = {\theta : 0 \le \theta < 2\pi}$</p></li></ul></li></ul><h4>Conditional Probability, Total Probability, and Bayesian Inference</h4><ul><li><p><strong>Conditional Probability</strong>:</p><ul><li><p>The conditional probability of an event$A$given that event$B$has occurred is defined as:<br>$P(A|B) = \frac{P(A \cap B)}{P(B)}$<br>provided$P(B) > 0$</p></li><li><p><strong>Real-world Example</strong>: The probability that a person is Greek ($A$) given that the person has blond hair ($B$).</p></li><li><p><strong>Important Caveat</strong>: Conditional probability imposes no causal or temporal order between events$A$and$B$.</p></li></ul></li><li><p><strong>Corollaries of Conditional Probability</strong>:</p><ul><li><p><strong>Conditioned Certain Event</strong>: The conditioning event$B$becomes the new sample space universe:<br>$P(B|B) = 1$</p></li><li><p><strong>Mutually Exclusive Conditioning</strong>: If$A \cap B = \emptyset$, then:<br>$P(A|B) = 0$<br>If$A$and$B$cannot occur together, the occurrence of$B$guarantees that$A$did not occur.</p></li><li><p><strong>Subset Conditioning</strong>: If$B \subset A$, then:<br>$P(A|B) = 1$<br>If$B$is a subset of$A$, occurrence of$B$guarantees occurrence of$A$</p></li></ul></li><li><p><strong>Validity of Conditional Probability as a Probability Measure</strong>:</p><ul><li><p>The conditioned measure$P(\cdot | B)$satisfies all three probability axioms over$\Omega$:</p></li></ul><ol><li><p>Non-negativity:$P(A|B) = \frac{P(A \cap B)}{P(B)} \ge 0$(since$P(A \cap B) \ge 0$and$P(B) > 0$)</p></li><li><p>Certainty:$P(\Omega|B) = \frac{P(\Omega \cap B)}{P(B)} = \frac{P(B)}{P(B)} = 1$</p></li><li><p>Additivity: If$A \cap C = \emptyset$, then$(A \cap B) \cap (C \cap B) = \emptyset$, so:<br>$P(A \cup C | B) = \frac{P((A \cup C) \cap B)}{P(B)} = \frac{P((A \cap B) \cup (C \cap B))}{P(B)} = \frac{P(A \cap B) + P(C \cap B)}{P(B)} = P(A|B) + P(C|B)$</p></li></ol></li><li><p><strong>Total Probability Rule</strong>:</p><ul><li><p>Let$A1, A2, \dots, A_n$be a partition of the sample space$\Omega$, satisfying:</p></li></ul><ol><li><p>$Ai \cap Aj = \emptyset$for all$i \neq j$(Mutually exclusive)</p></li><li><p>$\bigcup{i=1}^n Ai = \Omega$(Exhaustive cover)</p></li></ol><ul><li><p>For any event$B$, its marginal probability is calculated as:<br>$P(B) = \sum{k=1}^n P(B \cap Ak) = \sum{k=1}^n P(B|Ak) P(A_k)$</p></li><li><p><strong>Practical Utility</strong>: The Total Probability Rule is used when conditional probabilities$P(B|Ak)$and prior probabilities$P(Ak)$are known, but the marginal probability$P(B)$is unobserved.</p></li></ul></li><li><p><strong>Bayes' Rule</strong>:</p><ul><li><p>From the definition of conditional probability:<br>$P(A \cap B) = P(A|B) P(B) = P(B|A) P(A)$</p></li><li><p>Equating terms and isolating$P(B|A)$or$P(A|B)$yields Bayes' Rule:<br>$P(B|A) = \frac{P(A|B) P(B)}{P(A)}$<br>$P(A|B) = \frac{P(B \cap A)}{P(B)}$</p></li></ul></li><li><p><strong>Bayesian Inference Terminology</strong>:</p><ul><li><p>Bayes' Rule is formulated for inference problems as:<br>$P(A|B) = \frac{P(B|A) P(A)}{P(B)}$</p></li><li><p><strong>Components</strong>:</p></li><li><p><strong>$P(A)$(Prior Probability of$A$)</strong>: Probability of class/hypothesis$A$before observing data$B$</p></li><li><p><strong>$P(B|A)$(Likelihood)</strong>: Probability of observing data$B$given hypothesis$A$</p></li><li><p><strong>$P(B)$(Prior Probability of$B$/ Evidence)</strong>: Marginal probability of observing data$B$</p></li><li><p><strong>$P(A|B)$(Posterior Probability)</strong>: Updated probability of hypothesis$A$after observing data$B$</p></li><li><p>In classification systems,$B$typically represents observed sensor feature data, while$A$represents the target class or decision state.</p></li></ul></li><li><p><strong>Worked Practical Example (Loan Application Risk Analysis)</strong>:</p><ul><li><p><strong>Problem Statement</strong>:</p></li><li><p>A bank uses an automated classifier for loan approval.</p></li><li><p>$0.1\%$of all loan applications will default (not be repaid).</p></li><li><p>The automated system correctly identifies defaulted loans$99\%$of the time.</p></li><li><p>The system correctly identifies repaid loans$99\%$of the time.</p></li><li><p>Target calculation: Find the probability that a loan will actually default given that the system flags it as default.</p></li><li><p><strong>Mathematical Formulation</strong>:</p></li><li><p>Let$D$be the event that a loan defaults ($D^c$= loan repaid).</p></li><li><p>Let$F_D$be the event that the system flags the loan as default.</p></li><li><p>Prior probabilities:<br>$P(D) = 0.001$<br>$P(D^c) = 1 - 0.001 = 0.999$</p></li><li><p>System performance metrics (Likelihoods):<br>$P(FD | D) = 0.99$$P(FD^c | D^c) = 0.99 \implies P(F_D | D^c) = 1 - 0.99 = 0.01$</p></li><li><p><strong>Step 1: Compute Marginal Probability$P(FD)$via Total Probability Rule</em></strong><em>:<br>$P(FD) = P(FD | D) P(D) + P(FD | D^c) P(D^c)$<br>$P(F_D) = (0.99 \times 0.001) + (0.01 \times 0.999) = 0.00099 + 0.00999 = 0.01098$</p></li><li><p><strong>Step 2: Compute Posterior Probability$P(D | FD)$via Bayes' Rule</em></strong><em>:<br>$P(D | FD) = \frac{P(FD | D) P(D)}{P(FD)} = \frac{0.00099}{0.01098} = \frac{99}{1098} \approx 0.0901639$</p></li><li><p><strong>Conclusion</strong>: Given a default classification flag, there is only approximately a$9.02\%$chance that the loan will actually default, illustrating the significant impact of low base-rate prior probabilities.</p></li></ul></li></ul><h4>Independence in Probability</h4><ul><li><p><strong>Definition of Event Independence</strong>:</p><ul><li><p>Two events$A$and$B$are independent if knowledge of$B$provides no information about the probability of$A$:<br>$P(A|B) = P(A)$</p></li><li><p>Equivalent product definition:<br>$P(A \cap B) = P(A) P(B)$</p></li></ul></li><li><p><strong>Mutual Independence for Multiple Events</strong>:</p><ul><li><p>Events$A1, A2, A3$are mutually independent if and only if every pair is independent AND:$P(A1 \cap A2 \cap A3) = P(A1) P(A2) P(A_3)$</p></li><li><p><strong>Critical Distinction</strong>: Pair-wise independence across all pairs is a necessary condition, but is <strong>not sufficient</strong> to establish mutual independence.</p></li></ul></li><li><p><strong>Properties of Independent Events</strong>:</p><ul><li><p>If events$A$and$B$are independent, then:</p></li><li><p>Complementary events$A^c$and$B^c$are mutually independent.</p></li><li><p>$A^c$and$B$are independent.</p></li><li><p>$A$and$B^c$are independent.</p></li><li><p>If events$A, B, C$are mutually independent, then:</p></li><li><p>$A$and$(B \cap C)$are independent.</p></li><li><p>$A$and$(B \cup C)$are independent.</p></li></ul></li><li><p><strong>Independent Repeated Experiments Example (Two Fair Coins)</strong>:</p><ul><li><p>Sample space composed of ordered pairs:<br>$\Omega = {(H, H), (H, T), (T, H), (T, T)}$</p></li><li><p>Classical probabilities assigned equally:<br>$P({(H, H)}) = P({(H, T)}) = P({(T, H)}) = P({(T, T)}) = \frac{1}{4}$</p></li><li><p>Conditional evaluation for Coin 1 outcome given Coin 2 outcome:<br>$P(\text{Coin 1} = H | \text{Coin 2} = H) = \frac{P({(H, H)})}{P(\text{Coin 2} = H)} = \frac{1/4}{1/2} = \frac{1}{2} = P(\text{Coin 1} = H)$</p></li><li><p>Because conditional probabilities equal marginal probabilities across all combinations, trials are mutually independent.</p></li><li><p>Relative frequency interpretations of repeated experiments hold strictly under the assumption of independent trials.</p></li></ul></li></ul><h4>Random Variables, CDFs, and PDFs</h4><ul><li><p><strong>Random Variables (RVs)</strong>:</p><ul><li><p>A Random Variable$X$is a function mapping outcomes of a random process to real numbers, representing states of the world.</p></li><li><p>Particular realized instantiations are denoted using lower case letters ($x$).</p></li></ul></li><li><p><strong>Discrete Random Variables</strong>:</p><ul><li><p>Defined over countable outcome spaces.</p></li><li><p>The probability mass function (pmf) specifies the probability of exact point values:<br>$p(x) = P(X = x)$</p></li></ul></li><li><p><strong>Continuous Random Variables</strong>:</p><ul><li><p>Defined over continuous real intervals.</p></li><li><p><strong>Cumulative Distribution Function (CDF)</strong>: Denoted$F(x)$or$P(X \le x)$, measures the probability that the random variable$X$assumes a value less than or equal to$x$:<br>$F(x) = P(X \le x)$</p></li><li><p><strong>Probability Density Function (PDF)</strong>: Denoted$p(x)$, defined as the derivative of the CDF:<br>$p(x) = \frac{d}{dx} P(X \le x) = \frac{d}{dx} F(x)$</p></li><li><p><strong>Interval Approximation</strong>:<br>$p(x) \approx \frac{P(x \le X < x + \Delta x)}{\Delta x}$$