Unit 3: Joint Distributions and Probability Modeling Fundamentals and Applications

REVIEW OF CONTINUOUS PROBABILITY DISTRIBUTIONS

• Summary of Key Distributions: Important continuous probability distributions include the uniform, normal, Student's T, exponential, beta, gamma, and Weibull distributions.

• PDF and CDF: Probability Density Functions (PDFs) were defined for all these distributions. In several cases, explicit formulas for the Cumulative Distribution Function (CDF) were provided.

• Parameters: Distributions vary by the number of parameters they utilize:

  • One parameter: Bernoulli, geometric, and Poisson distributions.

  • Two parameters: Binomial and negative binomial distributions.

• The Normal Distribution:

  • The Standard Normal Distribution is a special case denoted as $N(0, 1)$.

  • Transformation to Standard Normal (Standardizing): If a random variable $X$ follows a normal distribution $X \sim N(\mu, \sigma^2)$ for $\mu \in \mathbb{R}$ and \sigma > 0, then the random variable $Z = \frac{X - \mu}{\sigma}$ holds the distribution $Z \sim N(0, 1)$.

  • Inverse Transformation: If $X \sim N(0, 1)$, then the random variable $Z = \mu + \sigma X$ follows the distribution $Z \sim N(\mu, \sigma^2)$.

• Importance of Support: When defining the PDF of any continuous distribution, the support must be explicitly stated. The support consists of all values where the PDF is positive.

  • Beta Distribution Support: The interval $[0, 1]$.

  • Gamma Distribution Support: The interval $(0, \infty)$.

INTRODUCTION TO JOINT DISTRIBUTIONS

• Rationale: Real-world data is usually multidimensional. To analyze such data, joint probability distributions are required to model the probability of value tuples.

• Key Definitions:

  • Joint Probability Mass (Density) Function: A function of several variables providing the probability (or density) of tuples of random variables.

  • Joint Probability Measure: A probability measure for a tuple (pair, triple, etc.) of random variables.

  • Joint Cumulative Distribution Function (CDF): A function of several variables giving the accumulated probability across a range of values for tuples of random variables.

• Sample Spaces and Cross-Products: Given two sample spaces $\Omega_1$ and $\Omega_2$, the cross-product is defined as $\Omega_1 \times \Omega_2 = \{ (x, y) \mid x \in \Omega_1, y \in \Omega_2 \}$, consisting of all pairs $(x, y)$.

  • Example: If $\Omega_1 = \{ 1, 2, 3 \}$ and $\Omega_2 = \{ a, b \}$, the cross-product is $\Omega_1 \times \Omega_2 = \{ (1, a), (1, b), (2, a), (2, b), (3, a), (3, b) \}$.

  • If a probability measure $P$ assigns $P(x) = \frac{1}{6}$ for each $x \in \Omega_1 \times \Omega_2$, then each pair is equally likely.

DISCRETE JOINT DISTRIBUTIONS

• Joint Probability Mass Function (PMF): For sample spaces $\Omega_1, \Omega_2$ with a probability measure $P$ on $\Omega_1 \times \Omega_2$, consider discrete random variables $X: \Omega_1 \rightarrow \{ x_1, x_2, x_3, \dots \} \subseteq \mathbb{R}$ and $Y: \Omega_2 \rightarrow \{ y_1, y_2, y_3, \dots \} \subseteq \mathbb{R}$. A function $f: \{ x_1, x_2, \dots \} \times \{ y_1, y_2, \dots \} \rightarrow [0, 1]$ is a joint PMF if it satisfies:

  1. $P(\omega_1, \omega_2 \in \Omega_1 \times \Omega_2 \mid X(\omega_1) = x_i, Y(\omega_2) = y_k) = P(X = x_i, Y = y_k) = f(x_i, y_k)$ for all $i, k \in \mathbb{N}$.

  2. $\sum_{i=1}^{\infty} \sum_{k=1}^{\infty} f(x_i, y_k) = 1$.

• Support: The set of all points where f(x_i, y_k) > 0.

• Joint CDF (Discrete): For fixed $x \in \Omega_1, y \in \Omega_2$:     $F_{X,Y}(x, y) = \sum_{\forall i: x_i \le x} \sum_{\forall k: y_k \le y} f_{X,Y}(x_i, y_k)$

CONTINUOUS JOINT DISTRIBUTIONS

• Joint Probability Density Function (PDF): For sample spaces $\Omega_1, \Omega_2$ with piecewise continuous random variables $X, Y: \Omega \rightarrow \mathbb{R}$, a piecewise continuous function $f: \mathbb{R} \times \mathbb{R} \rightarrow [0, 1]$ is a joint PDF if it satisfies:

  1. $P(X \le x, Y \le y) = \int_{-\infty}^x \int_{-\infty}^y f(t, s) \,ds \,dt = \int_{-\infty}^y \int_{-\infty}^x f(t, s) \,dt \,ds$ for all $x, y \in \mathbb{R}$.

  2. $\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} f(t, s) \,ds \,dt = 1$.

• Support: The set of all points where f(x, y) > 0 for $x, y \in \mathbb{R}$.

• Joint CDF (Continuous):     $F_{X,Y}(x, y) = \int_{-\infty}^x \int_{-\infty}^y f(t, s) \,ds \,dt$

• Geometric Interpretation: While a one-dimensional integral measures area under a curve, the joint integral measures the volume below the function $f$ and above the rectangle defined by $(-\infty, x] \times (-\infty, y]$ for fixed $x, y \in \mathbb{R}$.

• Fubini’s Theorem: This theorem proves that for these types of functions, the order of integration does not affect the value of the integral.

EXAMPLES OF JOINT DISTRIBUTIONS

Example 3.1: Rolling a Fair Die

• Variables:

  • $X = 1$ for the event $(1, 2)$, otherwise $0$.

  • $Y = 1$ for an odd result $(1, 3, 5)$, otherwise $0$.

• Mapping Outcomes to $(x, y)$:

  • Outcome 1: $(1, 1)$

  • Outcome 2: $(1, 0)$

  • Outcome 3: $(0, 1)$

  • Outcome 4: $(0, 0)$

  • Outcome 5: $(0, 1)$

  • Outcome 6: $(0, 0)$

• Inverse Mapping (Events):

  • $(0, 0) \rightarrow \{ 4, 6 \}$

  • $(0, 1) \rightarrow \{ 3, 5 \}$

  • $(1, 0) \rightarrow \{ 2 \}$

  • $(1, 1) \rightarrow \{ 1 \}$

• Joint PMF Table:

  • $f_{X,Y}(0, 0) = \frac{2}{6} = \frac{1}{3}$

  • $f_{X,Y}(0, 1) = \frac{2}{6} = \frac{1}{3}$

  • $f_{X,Y}(1, 0) = \frac{1}{6}$

  • $f_{X,Y}(1, 1) = \frac{1}{6}$

Example 3.2: Drawing Marbles Without Replacement

• Setup: Urn with 3 Green (G) and 6 Blue (B) marbles. Two are drawn without replacement.

• Outcome Mapping:

  • $GG \rightarrow (2, 0)$

  • $GB \rightarrow (1, 1)$

  • $BG \rightarrow (1, 1)$

  • $BB \rightarrow (0, 2)$

• Probability Calculations:

  • $P(GG) = \frac{3}{9} \times \frac{2}{8} = \frac{6}{72}$

  • $P(GB) = \frac{3}{9} \times \frac{6}{8} = \frac{18}{72}$

  • $P(BG) = \frac{6}{9} \times \frac{3}{8} = \frac{18}{72}$

  • $P(BB) = \frac{6}{9} \times \frac{5}{8} = \frac{30}{72}$

• Joint PMF Results:

  • $f_{X,Y}(2, 0) = \frac{6}{72} = \frac{1}{12}$

  • $f_{X,Y}(1, 1) = \frac{18+18}{72} = \frac{36}{72} = \frac{1}{2}$

  • $f_{X,Y}(0, 2) = \frac{30}{72} = \frac{5}{12}$

MULTIVARIATE HYPER-GEOMETRIC DISTRIBUTION

• Context: A generalization of sampling without replacement from a population with multiple categories (colors).

• Parameters:

  • $K$: Number of categories (colors).

  • $n_k$: Number of items of the $k^{th}$ color.

  • $n$: Total number of items drawn.

  • $X_k$: Random variable counting the number of items of color $k$ in the sample.

• Support Constraints:

  • $x_1 + x_2 + \dots + x_K = n$

  • $0 \le x_k \le \min(n, n_k)$ for $k = 1, 2, \dots, K$.

• Joint PMF Formula:     $f_{X_1, \dots, X_K}(x_1, \dots, x_K) = \frac{\binom{n_1}{x_1} \binom{n_2}{x_2} \dots \binom{n_K}{x_K}}{\binom{\sum n_k}{n}}$

• Revisiting Example 3.2: With $K=2$, $n=2$, $n_1=3$ (green), and $n_2=6$ (blue):     $f_{X_1, X_2}(x_1, x_2) = \frac{\binom{3}{x_1} \binom{6}{x_2}}{\binom{9}{2}}$     for $x_1 + x_2 = 2$.

BIVARIATE UNIFORM DISTRIBUTION

• Definition: A generalization of the univariate uniform distribution over a rectangular region $[a, b] \times [c, d]$.

• Joint PDF:     $f_{X,Y}(x, y) = \frac{1}{(b-a)(d-c)}$ if $(x, y) \in [a, b] \times [c, d]$, and zero otherwise.

• Measure Principle: The constant value of the PDF is the reciprocal of the area (measure) of the support.

• Example Calculation over Non-Rectangular Region:

  • Given support $[0, 4] \times [0, 2]$, then $f_{X,Y}(x, y) = \frac{1}{(4-0)(2-0)} = \frac{1}{8}$.

  • To find $P(X + 3Y \le 3)$, integrate over the triangular region with vertices $(0, 0), (0, 1), (3, 0)$.

  • Parametrize: $0 \le y \le 1$ and $0 \le x \le 3 - 3y$.

  • $P = \int_0^1 \int_0^{3-3y} \frac{1}{8} \,dx \,dy = \int_0^1 \frac{1}{8}(3 - 3y) \,dy = \frac{3}{8} \left[ y - \frac{y^2}{2} \right]_0^1 = \frac{3}{8}\left(\frac{1}{2}\right) = \frac{3}{16}$.

MULTIVARIATE NORMAL DISTRIBUTION

• Bivariate Case Definition: Defined by a mean vector $\mu = (\mu_1, \mu_2)^T$ and a covariance matrix $\Sigma = \begin{pmatrix} \sigma_1^2 & \sigma_{12} \\ \sigma_{12} & σ_2^2 \end{pmatrix}$.

• Determinant and Inverse:

  • $\det(\Sigma) = \sigma_1^2 \sigma_2^2 - \sigma_{12}^2$

  • $\Sigma^{-1} = \frac{1}{\det(\Sigma)} \begin{pmatrix} \sigma_2^2 & -\sigma_{12} \\ -\sigma_{12} & \sigma_1^2 \end{pmatrix}$

• Joint PDF:     $f_{X,Y}(x, y) = \frac{1}{2\pi \sqrt{\det(\Sigma)}} e^{-\frac{1}{2} (\mathbf{x} - \mu)^T \Sigma^{-1} (\mathbf{x} - \mu)}$

• Covariance Effects:

  • Identity Covariance ($\Sigma = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix}$): Circular contours, no relationship between variables.

  • Positive Covariance ($\sigma_{12} = 0.85$): Elliptical contours with positive slope. Higher values of one variable suggest higher values of the other.

  • Negative Covariance ($\sigma_{12} = -0.85$): Elliptical contours with negative slope. Higher values of one variable suggest lower values of the other.

MARGINAL DISTRIBUTIONS

• Concept: Determining the probability distribution of a single variable within a multivariate system by "summing out" or "integrating out" the other variables.

Discrete Marginal Distributions

• Marginal PMF of X: $f_X(x) = \sum_{j=1}^{\infty} f_{X,Y}(x, y_j)$.

• Marginal PMF of Y: $f_Y(y) = \sum_{i=1}^{\infty} f_{X,Y}(x_i, y)$.

• Tabular Method: In a two-way table, the marginals are found by summing the rows or columns.

Continuous Marginal Distributions

• Marginal PDF of X: $f_X(x) = \int_{-\infty}^{\infty} f_{X,Y}(x, y) \,dy$.

• Marginal PDF of Y: $f_Y(y) = \int_{-\infty}^{\infty} f_{X,Y}(x, y) \,dx$.

• Example 3.3: Given $f_{X,Y}(x, y) = e^{-(x+y)}$ for x, y > 0:

  • $f_X(x) = \int_0^{\infty} e^{-(x+y)} \,dy = e^{-x} \int_0^{\infty} e^{-y} \,dy = e^{-x} \cdot 1 = e^{-x}$.

  • Both marginals follow an Exponential distribution with parameter 1.

• Example 3.4: Given $f_{X,Y}(x, y) = \frac{x+y}{3}$ for $x \in [0, 1], y \in [0, 2]$:

  • $f_X(x) = \int_0^2 \frac{x+y}{3} \,dy = \left[ \frac{xy}{3} + \frac{y^2}{6} \right]_0^2 = \frac{2x}{3} + \frac{4}{6} = \frac{2}{3}(x + 1)$.

  • $f_Y(y) = \int_0^1 \frac{x+y}{3} \,dx = \left[ \frac{x^2}{6} + \frac{xy}{3} \right]_0^1 = \frac{1}{6} + \frac{y}{3} = \frac{1}{6}(2y + 1)$.

INDEPENDENT RANDOM VARIABLES

• Definiton (Discrete): $X$ and $Y$ are independent if $f_{X,Y}(x, y) = f_X(x) f_Y(y)$ for all possible pairs $(x, y)$.

• Definition (Continuous): $X$ and $Y$ are independent if $f_{X,Y}(x, y) = f_X(x) f_Y(y)$ for all $x, y \in \mathbb{R}$.

• Standard Normal Case: If $X \sim N(0, 1)$ and $Y \sim N(2, 1)$ are independent, their joint PDF is:     $f_{X,Y}(x, y) = \frac{1}{2\pi} e^{-\frac{x^2 + (y-2)^2}{2}}$

• Diagonal Covariance Matrix: If the covariance matrix of a multivariate normal distribution is diagonal ($\sigma_{12} = 0$), then the variables are independent.

• Testing for Independence:

  • To prove independence, the product rule must hold for every pair.

  • To prove dependence, the product rule must fail for at least one pair.

• Real-World Application: Global Warming Survey ($N=500$ Participants):

  • Education levels (None, Undergraduate, Graduate) and Belief (Yes, No).

  • Modeled as multi-noulli distributions.

  • Expected counts under independence are calculated as $E_{i,k} = N \cdot P(X=x_i) \cdot P(Y=y_k)$.

  • The $\chi^2$ (Chi-square) distribution is used quantitatively to model departures from independence.

CONDITIONAL DISTRIBUTIONS

• Core Formula: Based on the event probability $P(A \mid B) = \frac{P(A \cap B)}{P(B)}$.

• Discrete Conditional PMF: $f_{X|Y}(x \mid y) = \frac{f_{X,Y}(x, y)}{f_Y(y)}$ for f_Y(y) > 0.

• Continuous Conditional PDF: $f_{X|Y}(x \mid y) = \frac{f_{X,Y}(x, y)}{f_Y(y)}$ for f_Y(y) > 0.

• Deriving Joint from Conditional: $f_{X,Y}(x, y) = f_{Y|X}(y \mid x) f_X(x)$.

Example 3.9: Composite Distribution

• Given: $X \sim \text{Uniform}(1, 2)$ and $Y \mid X=x \sim \text{Uniform}(0, x)$.

• Joint PDF:

  • $f_X(x) = 1$ for $1 \le x \le 2$.

  • $f_{Y|X}(y \mid x) = \frac{1}{x}$ for $0 \le y \le x$.

  • $f_{X,Y}(x, y) = \frac{1}{x}$ for $0 \le y \le x$ and $1 \le x \le 2$.

• Marginal of Y:

  • For $0 \le y \le 1$: $f_Y(y) = \int_1^2 \frac{1}{x} \,dx = \ln(2) - \ln(1) = \ln(2)$.

  • For 1 < y \le 2: $f_Y(y) = \int_y^2 \frac{1}{x} \,dx = \ln(2) - \ln(y)$.

Example 3.10: Ecological Predation Data

• Data Provided:

  • $P(\text{Eagle}) = 0.2$

  • $P(\text{Rabbit}) = 0.5$

  • $P(\text{Eagle} \cap \text{Rabbit}) = 0.05$

• Completed Joint PMF Table:

  • $f_{E,R}(1, 1) = 0.05$ (Eagle and Rabbit)

  • $f_{E,R}(1, 0) = 0.2 - 0.05 = 0.15$ (Eagle, no Rabbit)

  • $f_{E,R}(0, 1) = 0.5 - 0.05 = 0.45$ (Rabbit, no Eagle)

  • $f_{E,R}(0, 0) = 1 - (0.05 + 0.15 + 0.45) = 0.35$ (Neither)

• Selected Conditionals:

  • $P(E=1 \mid R=1) = \frac{0.05}{0.5} = 0.1$

  • $P(E=1 \mid R=0) = \frac{0.15}{0.5} = 0.3$

  • $P(R=1 \mid E=1) = \frac{0.05}{0.2} = 0.25$

  • $P(R=1 \mid E=0) = \frac{0.45}{0.8} = 0.5625$