ST2133 Advanced Statistics: Distribution Theory

Comprehensive vocabulary flashcards covering probability spaces, measure theory, combinatorial counting, univariate and multivariate distributions, moments, generating functions, convergence, and conditional distributions from ST2133 Advanced statistics: distribution theory.

Experiment

A repeatable procedure which has a well-defined set of possible outcomes.

Sample Space

The set of all possible outcomes of an experiment, denoted by $\Omega$, such that any sample outcome $\omega$ is a member of the set ($\omega \in \Omega$).

Event

A set of outcomes which is of interest; a subset of the sample space, denoted $A \subseteq \Omega$ (or $A \subset \Omega$ if it is a strict subset).

$\sigma$-algebra (Borel field)

A collection of subsets $\mathcal{B}$ of a set $\Psi$ satisfying: (i) $\emptyset \in \mathcal{B}$; (ii) if $A \in \mathcal{B}$, then $A^c \in \mathcal{B}$ (closed under complementation); and (iii) if $A_1, A_2, \dots \in \mathcal{B}$, then $\bigcup_{i=1}^\infty A_i \in \mathcal{B}$ (closed under countable unions).

Measurable Space

The pair $(\Psi, \mathcal{B})$ consisting of a set $\Psi$ and a $\sigma$-algebra $\mathcal{B}$ defined on that set.

Measure

A function $m : \mathcal{B} \rightarrow \mathbb{R}^+$ on a measurable space $(\Psi, \mathcal{B})$ such that: (i) $m(A) \ge 0$ for all $A \in \mathcal{B}$; (ii) $m(\emptyset) = 0$; and (iii) if $A_1, A_2, \dots \in \mathcal{B}$ are mutually exclusive, then $m\left(\bigcup_{i=1}^\infty A_i\right) = \sum_{i=1}^\infty m(A_i)$.

Measure Space

The triple $(\Psi, \mathcal{B}, m)$ consisting of a set $\Psi$, a $\sigma$-algebra $\mathcal{B}$ defined on that set, and a measure $m$.

Indicator Function

The function $I_\Omega(x)$ for a set $\Omega \subseteq \mathbb{R}$ defined as $I_\Omega(x) = 1$ for $x \in \Omega$ and $I_\Omega(x) = 0$ otherwise.

Probability Measure

Given a measurable space $(\Omega, \mathcal{F})$, a measure $P : \mathcal{F} \rightarrow [0, 1]$ with the property that $P(\Omega) = 1$.

Probability Space

The triple $(\Omega, \mathcal{F}, P)$ consisting of a sample space $\Omega$, a collection of events $\mathcal{F}$ (forming a $\sigma$-algebra on $\Omega$), and a probability measure $P$.

Boole Inequality

A theorem stating that for a probability space $(\Omega, \mathcal{F}, P)$ with events $A_1, A_2, \dots \in \mathcal{F}$, $P\left(\bigcup_{i=1}^\infty A_i\right) \le \sum_{i=1}^\infty P(A_i)$.

Permutations

The number of possible ordered sequences of $k$ objects selected without replacement from $n$ objects, defined as $^nP_k = \frac{n!}{(n - k)!}$.

Combinations (Binomial Coefficient)

The number of unordered sequences of $k$ objects selected without replacement from $n$ objects, defined as $^nC_k = \binom{n}{k} = \frac{n!}{k!(n - k)!}$.

Multinomial Coefficient

The number of ways of arranging $n$ objects where $n_1$ are of type 1, $n_2$ are of type 2, …, and $n_r$ are of type $r$ (with $\sum_{i=1}^r n_i = n$), given by $(n_1, n_2, \dots, n_r)! = \frac{n!}{n_1! n_2! \cdots n_r!}$.

Conditional Probability

For events $A, B \in \mathcal{F}$ with $P(B) > 0$, the updated probability of $A$ given that $B$ has occurred, defined by $P(A \mid B) = \frac{P(A \cap B)}{P(B)}$.

Partition

A set of events $\left\{B_1, B_2, \dots, B_n\right\}$ on a probability space $(\Omega, \mathcal{F}, P)$ that is exhaustive ($\bigcup_{i=1}^n B_i = \Omega$), mutually exclusive ($B_i \cap B_j = \emptyset$ for all $i \ne j$), and satisfies $P(B_i) > 0$ for $i = 1, 2, \dots, n$.

Total Probability Formula

A formula stating that for a partition $\left\{B_1, B_2, \dots, B_n\right\}$ of $\Omega$ and any event $A \in \mathcal{F}$, $P(A) = \sum_{i=1}^n P(A \mid B_i) P(B_i)$.

Bayes' Theorem

A mechanism for reversing the order of conditioning, expressed for events $A, B \in \mathcal{F}$ as $P(B \mid A) = \frac{P(A \mid B) P(B)}{P(A)}$.

Independent Events

Two events $A$ and $B$ in a probability space $(\Omega, \mathcal{F}, P)$ such that $P(A \cap B) = P(A) P(B)$.

Random Variable

A function $X : \Omega \rightarrow \mathbb{R}$ such that $A_x = \{\omega \in \Omega : X(\omega) \le x\} \in \mathcal{F}$ for all $x \in \mathbb{R}$.

Positive Random Variable

A random variable $X$ that takes a positive (or non-negative) value for every possible outcome, meaning $X(\omega) \ge 0$ for all $\omega \in \Omega$.

Cumulative Distribution Function (cdf)

The function $F_X : \mathbb{R} \rightarrow [0, 1]$ of a random variable $X$ given by $F_X(x) = P(X \le x)$.

Survival Function

For a random variable $X$ with distribution function $F_X$, the function defined as $\bar{F}_X(x) = P(X > x) = 1 - F_X(x)$.

Hazard Function

For a non-negative continuous random variable $X$, the function $h_X(x) = \frac{f_X(x)}{\bar{F}_X(x)} = -\frac{d}{dx}\ln\left(\bar{F}_X(x)\right)$.

Support of a Function

The subset of the real line where a positive real-valued function $f$ takes values strictly greater than zero: $\{x \in \mathbb{R} : f(x) > 0\}$.

Probability Mass Function (pmf)

The function $p_X : \mathbb{R} \rightarrow [0, 1]$ of a discrete random variable $X$ given by $p_X(x) = P(X = x)$, which satisfies $\sum_x p_X(x) = 1$.

Continuous Random Variable

A random variable $X$ whose distribution function can be expressed as $F_X(x) = \int_{-\infty}^x f_X(t)\,dt$ for all $x \in \mathbb{R}$, where $f_X : \mathbb{R} \rightarrow [0, \infty)$ is an integrable probability density function.

Mean (Expectation) of a Random Variable

The measure of central tendency defined by $\mu = E(X) = \sum_x x p_X(x)$ for discrete $X$, and $E(X) = \int_{-\infty}^\infty x f_X(x)\,dx$ for continuous $X$, whenever the sum or integral of absolute values is finite.

Variance of a Random Variable

The measure of dispersion defined by $\sigma^2 = \text{Var}(X) = E\left((X - E(X))^2\right) = E(X^2) - (E(X))^2$.

Markov Inequality

A bound stating that if $X$ is a positive random variable with $E(X) < \infty$, then $P(X \ge a) \le \frac{E(X)}{a}$ for any constant $a > 0$.

Chebyshev Inequality

A bound stating that if $X$ is a random variable with $\text{Var}(X) < \infty$, then $P(|X - E(X)| \ge a) \le \frac{\text{Var}(X)}{a^2}$ for any constant $a > 0$.

Convex Function

A function $g : \mathbb{R} \rightarrow \mathbb{R}$ such that for any real constant $a$, there exists a constant $k$ satisfying $g(x) \ge g(a) + k(x - a)$ for all $x \in \mathbb{R}$.

Jensen Inequality

An inequality stating that if $X$ is a random variable with $E(X) < \infty$ and $g$ is a convex function such that $E(g(X)) < \infty$, then $E(g(X)) \ge g(E(X))$.

$r$th Moment

For a random variable $X$ and a positive integer $r$, the expected value $\mu_r = E(X^r)$, whenever it is well-defined.

$r$th Central Moment

For a random variable $X$ and a positive integer $r$, the expected value about the mean given by $\mu'_r = E\left((X - E(X))^r\right)$, whenever it is well-defined.

Coefficient of Skewness

A standardized measure of the departure from symmetry of a distribution, defined as $\gamma_1 = E\left(\left(\frac{X - \mu}{\sigma}\right)^3\right) = \frac{\mu'_3}{(\mu'_2)^{3/2}}$.

Coefficient of Kurtosis (Excess Kurtosis)

A standardized measure of the fatness of tails relative to a normal distribution, defined as $\gamma_2 = E\left(\left(\frac{X - \mu}{\sigma}\right)^4\right) - 3 = \frac{\mu'_4}{(\mu'_2)^2} - 3$.

Moment Generating Function (mgf)

The function $M_X(t) = E(e^{tX})$, required to satisfy $M_X(t) < \infty$ for all $t \in [-h, h]$ for some $h > 0$, whose $r$th derivative evaluated at zero gives the $r$th moment $E(X^r)$.

Cumulant Generating Function (cgf)

The function defined as $K_X(t) = \ln\left(M_X(t)\right)$, whose polynomial expansion coefficients $\kappa_r$ satisfy $\kappa_1 = E(X)$ and $\kappa_2 = \text{Var}(X)$.

Convergence in Distribution

A mode of convergence where a sequence of random variables $\left\{X_n\right\}$ satisfies $F_{X_n}(x) \rightarrow F_X(x)$ as $n \rightarrow \infty$ for all $x$ at which the limiting cdf $F_X$ is continuous, denoted $X_n \xrightarrow{d} X$.

Convergence in Probability

A mode of convergence where a sequence of random variables $\left\{X_n\right\}$ satisfies $P(|X_n - X| < \epsilon) \rightarrow 1$ as $n \rightarrow \infty$ for any $\epsilon > 0$, denoted $X_n \xrightarrow{p} X$.

Convergence Almost Surely

A mode of convergence where a sequence of random variables $\left\{X_n\right\}$ satisfies $P\left(\lim_{n \rightarrow \infty} |X_n - X| < \epsilon\right) = 1$ for any $\epsilon > 0$, denoted $X_n \xrightarrow{\text{a.s.}} X$.

Convergence in Mean Square

A mode of convergence where a sequence of random variables $\left\{X_n\right\}$ satisfies $E\left((X_n - X)^2\right) \rightarrow 0$ as $n \rightarrow \infty$, denoted $X_n \xrightarrow{\text{m.s.}} X$.

Joint Cumulative Distribution Function

A function $F_{X,Y} : \mathbb{R} \times \mathbb{R} \rightarrow [0, 1]$ defined by $F_{X,Y}(x, y) = P(X \le x, Y \le y)$.

Covariance

A measure of the joint variability of two random variables $X$ and $Y$, defined as $\text{Cov}(X, Y) = E\left((X - E(X))(Y - E(Y))\right) = E(XY) - E(X)E(Y)$.

Correlation Coefficient

A scaled measure of linear association between two random variables $X$ and $Y$, defined as $\text{Corr}(X, Y) = \frac{\text{Cov}(X, Y)}{\sqrt{\text{Var}(X)\text{Var}(Y)}}$, satisfying $|\text{Corr}(X, Y)| \le 1$.

Variance-Covariance Matrix

For an $n$-dimensional random vector $\mathbf{X} = (X_1, X_2, \dots, X_n)^T$, the $n \times n$ symmetric, non-negative definite matrix $\mathbf{\Sigma} = \text{Var}(\mathbf{X}) = E\left((\mathbf{X} - E(\mathbf{X}))(\mathbf{X} - E(\mathbf{X}))^T\right)$.

Convolution

The operation, denoted $f_Z = f_X * f_Y$, that yields the density of the sum $Z = X + Y$ of two independent continuous random variables via $f_Z(z) = \int_{-\infty}^\infty f_X(u) f_Y(z - u)\,du$.

Conditional Expectation

The expected value of $Y$ given $X = x$, defined as $E(Y \mid X = x) = \int_{-\infty}^\infty y f_{Y \mid X}(y \mid x)\,dy$ for continuous variables and $E(Y \mid X = x) = \sum_y y p_{Y \mid X}(y \mid x)$ for discrete variables.

Law of Iterated Expectations

A theorem stating that the expected value of a conditional expectation equals the unconditional expectation: $E(Y) = E(E(Y \mid X))$.

Conditional Variance

The variance of $Y$ conditional on $X = x$, defined as $\text{Var}(Y \mid X = x) = E\left((Y - E(Y \mid X = x))^2 \mid X = x\right) = E(Y^2 \mid X = x) - (E(Y \mid X = x))^2$.

Decomposition of Variance

The identity stating that the unconditional variance of $Y$ can be partitioned as $\text{Var}(Y) = E(\text{Var}(Y \mid X)) + \text{Var}(E(Y \mid X))$.

Random Sum

A sum of the form $Y = \sum_{i=1}^N X_i$, where $\left\{X_i\right\}$ is a sequence of independent and identically distributed random variables and $N$ is an independent random variable taking only positive integer values.