MAS 102: Probability and Distribution Theory I

MAS 102: Probability and Distribution Theory I

Course Details

• Lecturers: Edgar Otumba, Samuel Baffoe & Seth Opoku Larbi

• Institution: Maseno University

• Date: February 16, 2026

Table of Contents

1. Expected Learning Outcomes

2. Core Reading Material for the Course

3. Special Continuous Distributions I – Gamma and Chi-Square

4. Special Continuous Distributions II – Weibull, Beta, and Cauchy

5. Joint Distributions of Two Random Variables

6. Covariance and Correlation Between Two Random Variables

7. The Bivariate Normal Distribution

8. Distribution of Functions of Random Variables – CDF Technique

9. Distribution of Functions – Moment Generating Function (MGF)

10. Distribution of Functions – Change of Variable (Transformation)

Expected Learning Outcomes

• Understanding Probability Extensions: Extend the definition of a probability distribution of one random variable to the joint probability distribution of two random variables.

• Correlation Coefficient: Use the correlation coefficient as a way of quantifying the extent to which two random variables are linearly related.

• Investigation of Bivariate Normal Distribution: Investigate a particular joint probability distribution, namely the bivariate normal distribution.

• Distribution of Functions: Determine the probability distribution of functions of random variables using the distribution function, moment generating function, and change of variable techniques.

Core Reading Material for the Course

• Textbook 1: Morris H DeGroot & Mark J Schervish (2012), Probability and Statistics, 4th Edition, Pearson Education, Inc. ISBN 978-0-321-50046-5

• Textbook 2: Robert V Hogg & Allen T Craig (2004), Introduction to Mathematical Statistics, 5th Edition, Pearson Education, Inc.

Recommended Reference Materials

• Reference 1: S C Gupta & V K Kapoor (2000), Fundamentals of Mathematical Statistics (A Modern Approach), 10th Edition, Sultan Chand & Sons. ISBN 81-7014-791-3

• Reference 2: Jay L Devore & Kenneth N Berk (2012), Modern Mathematical Statistics with Applications, 2nd Edition, Springer Texts in Statistics. ISBN 978-1-4614-0390-6

• Reference 3: William Mendenhall, III, Robert J Beaver & Barbara M Beaver (2013), Introduction to Probability and Statistics, 14th Edition, Brooks/Cole Cengage Learning.

Special Continuous Distributions I – Gamma and Chi-Square

Objectives

• Define the gamma distribution and identify its parameters.

• Derive and use the probability density function (PDF) of a gamma distribution.

• Use the incomplete gamma function and tables.

• Define the chi-square distribution as a special case of the gamma distribution.

• Compute probabilities involving gamma and chi-square variables.

The Gamma Distribution

• Definition: A continuous random variable X follows a gamma distribution with shape parameter $\alpha > 0$ and scale parameter $\theta > 0$, written as:
    $X \sim \text{Gamma}(\alpha, \theta)$

• Probability Density Function (PDF):
    $f(x; \alpha, \theta) = \frac{1}{\Gamma(\alpha) \theta^{\alpha}} x^{\alpha - 1} e^{-x/\theta}, \quad x > 0$

Where:

• x is a possible value of the random variable X, with $x > 0$.

• $\alpha$ is the shape parameter of the distribution, with $\alpha > 0$.

• $\theta$ is the scale parameter of the distribution, with $\theta > 0$.

• $\Gamma(\alpha)$ is the gamma function, defined as:
    $\Gamma(\alpha) = \int_{0}^{\infty} t^{\alpha - 1} e^{-t} dt$

Properties of the Gamma Distribution

• Mean: $\mu = \alpha \theta$

• Variance: $\sigma^2 = \alpha \theta^2$

• When $\alpha = 1$, the Gamma distribution becomes the exponential distribution:
    $\text{Gamma}(1, \theta) = \text{Exponential}(\theta)$

Gamma Function and Incomplete Gamma Function

• Gamma Function: Generalizes the factorial function:
    $\Gamma(n) = (n - 1)!$ for positive integers n.

• Incomplete Gamma Function:
    $\gamma(\alpha, x) = \int_{0}^{x} t^{\alpha - 1} e^{-t} dt$

Cumulative Distribution Function (CDF) of the Gamma Distribution

• $P(X \leq x) = \frac{\gamma(\alpha, x/\theta)}{\Gamma(\alpha)}$

• Values of the incomplete gamma function are typically obtained from tables or computed numerically.

Using Gamma and Incomplete Gamma Tables

• To compute probabilities involving the Gamma distribution, such as $P(X \leq x)$, we use the incomplete gamma function:
    $P(X \leq x) = \frac{\gamma(\alpha, x/\theta)}{\Gamma(\alpha)}$

• Steps for using the table:
    1. Identify the shape parameter $\alpha$ and scale parameter $\theta$.
    2. Compute $y = \frac{x}{\theta}$, where $x$ is the value of interest.
    3. Look up the value of $P(\alpha, y)$ in the table corresponding to your value of $\alpha$ and $y$.
    4. That value gives $P(X \leq x)$.

Examples of Gamma Distribution

Example 1: Find $P(X \leq 4$ for $X \sim \text{Gamma}(\alpha = 2, \theta = 3)$.

• Approach 1 (Direct Formula): For a Gamma distribution with integer shape, we use the Erlang form.
    $F_X(x) = P(X \leq x) = 1 - e^{-x/\theta} \sum_{k=0}^{\alpha-1} \frac{(x/\theta)^k}{k!}$
    - Substituting $\alpha = 2, \theta = 3$ gives:
    $F_X(x) = 1 - e^{-x/3} \left(1 + \frac{x}{3}\right)$

• At $x = 4$:
    $y = \frac{x}{\theta} = \frac{4}{3} \approx 1.333$
    - Thus,
    F_X(4) = 1 - e^{-4/3} \left(1 + \frac{4}{3}
  ight)
    - Numerical values: $e^{-4/3} \approx 0.2636, 1 + \frac{4}{3} \approx 2.3333$
    Thus,
    $F_X(4) \approx 1 - (0.2636) (2.3333) \approx 0.3849$.

Example 2: Find $P(X \leq 5$ for $X \sim \text{Gamma}(\alpha = 2, \theta = 1)$ using the incomplete gamma function.

• First, find the complete gamma function: $\Gamma(2) = 1$

• Evaluate the lower incomplete gamma function $\gamma(2, 5)$ through integration.
    $\gamma(2, 5) = \int_{0}^{5} te^{-t} dt$

• Use integration by parts to solve the integral.

The Chi-Square Distribution

• Definition: If a random variable $X$ follows a chi-square distribution with $k$ degrees of freedom, then:
    $X \sim \chi^2(k) \Leftrightarrow X \sim \text{Gamma}\left(\frac{k}{2}, 2\right)$.

• Probability Density Function (PDF):
    $f(x; k) = \frac{1}{2^{k/2} \Gamma(k/2)} x^{(k/2)-1} e^{-x/2}, \quad x > 0$

Key Properties of the Chi-Square Distribution

• Mean: $\mu = k$

• Variance: $\sigma^2 = 2k$

• Additivity Property: If $X_1 \sim \chi^2(k_1)$ and $X_2 \sim \chi^2(k_2)$ are independent, then
    $X_1 + X_2 \sim \chi^2(k_1 + k_2)$

• Applications: Commonly used in hypothesis testing, confidence intervals, and tests of independence.

How to Determine Critical Values from the Chi-Square Table

• Step 1: Identify the degrees of freedom (df): $df = n - 1$ for sample size $n$.

• Step 2: Choose your significance level (α). For a two-sided confidence interval with confidence level $1 - \alpha$, split $\alpha$ into two tails.

• Step 3: Use the chi-square table to locate the appropriate chi-square values for your degrees of freedom.

Example: Confidence Interval for Population Variance

1. Identify the distribution:
      Since the data is from a normal population, the test statistic $\chi^2 = \frac{(n - 1)S^2}{\sigma^2}$ follows a chi-square distribution with $n - 1$ degrees of freedom.

2. Determine critical values using the chi-square table.

3. Construct the confidence interval.

Visualizing the Gamma Distributions

• The gamma distribution is right-skewed for low values of the shape parameter $\alpha$, and as the parameter increases, it tends toward a symmetry.

Practice Problems

1. Show that the exponential distribution is a special case of the gamma distribution when the shape parameter $\alpha = 1$.

2. Let $X \sim \text{Gamma}(3, 2)$, calculate the mean and variance:
      - Mean: $\mu = \alpha \theta = 3 \times 2 = 6$
      - Variance: $\sigma^2 = \alpha \theta^2 = 3 \times 2^2 = 12$

3. For $X \sim \chi^2(6)$, find $P(X < 5.35)$ and $P(X > 12.59)$ using the cumulative distribution function (CDF).

4. Let $X \sim \text{Gamma}(2, 1)$. Compute $P(X < 5)$ using the incomplete gamma function.

Special Continuous Distributions II – Weibull, Beta, and Cauchy

The Weibull Distribution

• Definition: A random variable $X$ has a Weibull distribution with shape parameter $k > 0$ and scale parameter $\lambda > 0$, denoted $X \sim \text{Weibull}(\lambda, k)$, if it has the PDF:
    $f(x; \lambda, k) = \begin{cases} \frac{k}{\lambda} \left(\frac{x}{\lambda}\right)^{k-1} e^{-(x/\lambda)^{k}} & x \geq 0 \ 0 & x < 0 \end{cases}$

• Mean: $\mu = \lambda \cdot \Gamma\left(1 + \frac{1}{k}\right)$

• Variance: $\sigma^2 = \lambda^2 \left(\Gamma\left(1 + \frac{2}{k}\right) - \left(\Gamma\left(1 + \frac{1}{k}\right)\right)^2 \right)$

Key Properties and Applications of the Weibull Distribution

• When $k = 1$, the Weibull distribution reduces to the exponential distribution.

• Widely used in reliability engineering, survival analysis, and failure time analysis.

• Hazard rate shape:
    - $k < 1$: Decreasing hazard rate   - $k = 1$: Constant hazard rate (exponential case)   - $k > 1$: Increasing hazard rate (aging or wear-out failures)

The Beta Distribution

• Definition: A random variable $X \in (0, 1)$ is said to follow a Beta distribution with shape parameters $\alpha > 0$ and $\beta > 0$ if its PDF is given by:
    $f(x; \alpha, \beta) = \frac{1}{B(\alpha, \beta)} x^{\alpha-1}(1-x)^{\beta-1}, \quad 0 < x < 1$
    Where:
  $B(\alpha, \beta) = \int_0^1 t^{\alpha-1}(1-t)^{\beta-1} dt = \frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha + \beta)}$

• Mean: $\mu = \frac{\alpha}{\alpha + \beta}$

• Variance: $\sigma^2 = \frac{\alpha \beta}{(\alpha + \beta)^2(\alpha + \beta + 1)}$

Applications of the Beta Distribution

• Useful for modeling variables bounded between 0 and 1, such as proportions and probabilities.

• Widely used in Bayesian statistics as a conjugate prior for binomial and Bernoulli distributions.

• Shape flexibility:
    - Symmetric when $\alpha = \beta$
    - Left-skewed when $\alpha < \beta$   - Right-skewed when $\alpha > \beta$

Worked Examples and Practice Problems

1. Let $X \sim \text{Beta}(\alpha = 2, \beta = 3)$, compute $P(X < 0.4)$.

2. **Suppose $X \sim \text{Weibull}(k = 1.5, \lambda = 2)$, find $P(X < 2.5)$.

3. For $X \sim Cauchy(x_0, \gamma)$, find the PDF and key properties.

Joint Distributions of Two Random Variables

Joint Distributions Definition

• A joint distribution describes the probability behavior of two random variables $X$ and $Y$ simultaneously.

• Discrete Case:
    - Joint PMF: $p(x, y) = P(X = x, Y = y)$
    - Must satisfy: $p(x, y) \geq 0$ and $\sum_{x}\sum_{y} p(x, y) = 1$

• Continuous Case:
    - Joint PDF: $f(x, y)$
    - Must satisfy: $f(x, y) \geq 0$ and $\iint f(x,y) dx dy = 1$

• For any region $A \subset \mathbb{R}^2,$ $P((X, Y) \in A) = \iint_A f(x, y) dx dy$

Marginal Distributions

• Marginal distributions describe the individual behavior of a single random variable irrespective of the other.

• Discrete Case:
    - Marginal PMFs:
    $p_X(x) = \sum_y p(x, y)$
    $p_Y(y) = \sum_x p(x, y)$

• Continuous Case:
    - Marginal PDFs:
    $f_X(x) = \int_{-\infty}^{\infty} f(x, y) dy$
    $f_Y(y) = \int_{-\infty}^{\infty} f(x, y) dx$

Conditional Distributions

• Conditional distributions describe the probability distribution of one variable given the value of another.

• Discrete Case:
    - Conditional PMF:
    $P(Y = y | X = x) = \frac{p(x, y)}{p_X(x)}, \text{ if } p_X(x) > 0$

• Continuous Case:
    - Conditional PDF:
    $f_{Y|X}(y|x) = \frac{f(x,y)}{f_X(x)}, \text{ if } f_X(x) > 0$

Independence of Random Variables

• Two random variables $X$ and $Y$ are independent if the joint distribution factorizes into the product of their marginal distributions:

• Continuous Case: $f(x, y) = f_X(x)f_Y(y)$

• Discrete Case: $p(x, y) = p_X(x)p_Y(y)$

• When independent, knowledge of one variable does not affect the distribution of the other.

Worked Example 1 (Discrete Case)

Problem: Let the joint PMF of $X$ and $Y$ be:

• $p_X(0) = 0.1 + 0.2 = 0.3$

• $p_X(1) = 0.3 + 0.4 = 0.7$

• $p_Y(0) = 0.1 + 0.3 = 0.4$

• $p_Y(1) = 0.2 + 0.4 = 0.6$

Covariance and Correlation Between Two Random Variables

Covariance

• Definition: Let $X$ and $Y$ denote two random variables, then the covariance of $X$ and $Y$ is defined as:
    $Cov(X, Y) = E[(X - E[X])(Y - E[Y])]$

• Alternatively:
    $Cov(X, Y) = E[XY] - E[X]E[Y]$

• Interpretation:
    - $Cov(X, Y) > 0$: X and Y tend to increase together.
    - $Cov(X, Y) < 0$: As X increases, Y tends to decrease.
    - $Cov(X, Y) = 0$: No linear relationship between X and Y.
    - Note: Covariance depends on the units of $X$ and $Y$, thus not always meaningful alone.

Correlation Coefficient

• Definition: The correlation coefficient $\rho_{XY} = \frac{Cov(X,Y)}{\sigma_X \sigma_Y}$ eliminates scale effects and standardizes values.

• Properties:
    - $-1 \leq \rho_{XY} \leq 1$
    - $\rho = 1$: Perfect positive linear relationship.
    - $\rho = -1$: Perfect negative linear relationship.
    - $\rho = 0$: No linear relationship.
    - Zero correlation does not imply independence unless the joint distribution is bivariate normal.

Worked Example 1 (Discrete Case)

• 
  
  

• Problem: Given joint distribution of $X$ and $Y$:
  
  

  
  

  	$Y=1$	$Y=2$
  $X=1$	0.1	0.2
  $X=2$	0.3	0.4
  Calculating Expectations

  (a) Compute: $E[X]$ and $E[Y]$.

  • Marginals:
      - $P(X=1) = 0.1 + 0.2 = 0.3$
      - $P(X=2) = 0.3 + 0.4 = 0.7$

  • Thus,
      $E[X] = 1 imes 0.3 + 2 imes 0.7 = 1.7$
    (b) Compute $E[Y]$:
      - $P(Y=1) = 0.1 + 0.3 = 0.4$
      - $P(Y=2) = 0.2 + 0.4 = 0.6$
      - $E[Y] = 1 imes 0.4 + 2 imes 0.6 = 1.6$

  (c) Compute $Cov(X,Y)$:

  • $E[XY]$:
      - $E[XY] = (1 imes 1 imes 0.1) + (1 imes 2 imes 0.2) + (2 imes 1 imes 0.3) + (2 imes 2 imes 0.4) = 2.7$

  • Covariance Calculation:
      $Cov(X,Y) = E[XY] - E[X]E[Y] = 2.7 - (1.7)(1.6) = 2.7 - 2.72 = -0.02$.

  (d) **Compute correlation coefficient: **

  • First compute variance:
      - $E[X^2] = 1^2 imes 0.3 + 2^2 imes 0.7 = 3.1$
      - $Var(X) = E[X^2] - (E[X])^2 = 3.1 - (1.7)^2 = 3.1 - 2.89 = 0.21$

  • Similarly for $Y$:

  • $E[Y^2] = 1^2 imes 0.4 + 2^2 imes 0.6 = 2.8$

  • $Var(Y) = E[Y^2] - (E[Y])^2 = 2.8 - (1.6)^2 = 0.24$

  • Finally:
      $\rho_{XY} = \frac{Cov(X,Y)}{\sqrt{Var(X)} \sqrt{Var(Y)}} \rightarrow \rho_{XY} \approx -0.089$

  ---

  The Bivariate Normal Distribution

  Definition

  • A pair of continuous random variables $X$ and $Y$ is said to have a bivariate normal distribution if their joint PDF is given as:
      $f(x, y) = \frac{1}{2 \pi \sigma_X \sigma_Y \sqrt{1-\rho^2}} \exp\left(-\frac{1}{2(1-\rho^2)} \left(\frac{(x-\mu_X)^2}{\sigma_X^2} - \frac{2\rho(x - \mu_X)(y - \mu_Y)}{\sigma_X\sigma_Y} + \frac{(y-\mu_Y)^2}{\sigma_Y^2}\right)\right)$

  • Where:
      - $\mu_X, \mu_Y$ are the means of $X$ and $Y$,
      - $\sigma_X, \sigma_Y$ are the standard deviations of $X$ and $Y$,
      - $\rho$ is the correlation coefficient between $X$ and $Y$.

  Properties

  1. Marginal Distributions:
        $X \sim N(\mu_X, \sigma_X^2) \text{ and } Y \sim N(\mu_Y, \sigma_Y^2)$.

  2. Conditional Distribution:
        $Y|X=x \sim N\left(\mu_Y + \rho \frac{\sigma_Y}{\sigma_X}(x - \mu_X), \sigma_Y^2(1 - \rho^2)\right)$.

  3. Independence Condition: If $\rho = 0$, then $X$ and $Y$ are independent (only in the bivariate normal case).

  4. Contours: The contours of constant density are ellipses, whose orientation depends on $\rho$.

  ---

  Worked Examples

  1. Find conditional distribution: Suppose $X, Y \sim Bivariate Normal$ with $\mu_X = 5, \mu_Y = 10, \sigma_X = 2, \sigma_Y = 3, \rho = 0.6$. Find:

  • a) The conditional distribution of $Y$ given $X=7$.

  • b) The mean and variance of $Y | X = 7$.

  Solution:

  • Using the conditional distribution formula:
      a)
       $\mu_{Y | X=7} = 10 + 0.6 \left(\frac{3}{2}(7 - 5)\right) = 11.8$

  • b)
       $\sigma^2_{Y | X=7} = 9(1 - 0.36) = 5.76$. Thus,
      $Y | X = 7 \sim N(11.8, 5.76)$.

  ---

  Functions of Random Variables – CDF Technique

  Overview

  • CDF Technique: Used to derive the distribution of a transformed random variable $W = g(X)$, where $X$ is a continuous random variable, known PDF $f_X(x)$.
      1. Find the cumulative distribution function (CDF):
       $F_W(w) = P(W \leq w) = P(g(X) \leq w)$.
      2. Solve for $X$ and write as $F_W(w) = \int_{x \in A(w)} f_X(x) dx$ where $A(w) = {x: g(x) \leq w}$.
      3. Differentiate to find the PDF:
       $f_W(w) = \frac{d}{dw} F_W(w)$.

  ---

  Worked Example: Linear Transformation

  Problem: Let $X \sim U(0, 1)$ and define $W = 3X + 2$. Find the PDF of $W$.

  1. Find CDF:

  • Solve:
    $FW(w) = P(3X + 2 \leq w) \Rightarrow X \leq \frac{w - 2}{3}$

  1. Substitute:

  • Using $FX(x) = \begin{cases} 0, & x < 0 \ x, & 0 \leq x \leq 1 \ 1, & x > 1 \end{cases}$

  1. Differentiate to obtain PDF:

  • Final answer: $f_W(w) = \begin{cases} 0, & w < 2 \ \frac{1}{3}, & 2 \leq w \leq 5 \ 0, & w > 5 \end{cases}$

  ---

  Functions of Random Variables – Moment Generating Function (MGF)

  Definition

  • Moment Generating Function: The MGF of a random variable $X$ is defined as
      $M_X(t) = E[e^{tX}]$

  • Unique properties:

  1. Uniqueness: MGF uniquely determines the distribution of $X$.

  2. Linear Transformations: If $W = aX + b$, then
         $M_{aX+b}(t) = e^{bt}M_X(at)$.

  3. Sum of Independent Variables: For independent $X$ and $Y$,
         $M_{X + Y}(t) = M_X(t)M_Y(t)$.

  ---

  Example: Sum of Two Independent Exponential Random Variables

  Step 1: MGF of Exponential:
  For $X \sim Exp(\lambda)$, $M_X(t) = \frac{\lambda}{\lambda - t}, \, t < \lambda$
  Step 2: MGF of Sum

  • For independent $X_1$ and $X_2$:
        $M_S(t) = M_{X_1}(t)M_{X_2}(t) = \left(\frac{\lambda}{\lambda - t}\right)^2.$
    Step 3: Identify Distribution:

  • Recognized as MGF of a Gamma with parameters $\alpha = 2, \lambda$.

  Conclusion

  • The sum of independent exponential random variables follows a Gamma distribution.

  ---

  Practice Problems:

  1. If $X \sim Gamma(3, \lambda)$, calculate its moments using MGF.

  2. Derive the distribution of the sum of independent uniform variables using the CDF technique.

  3. Verify the properties of the moment generating functions of major distributions.

  ---

  Distribution of Functions – Change of Variable (Transformation)

  Univariate Case

  • If $Y = g(X)$ where $g$ is a strictly monotonic and differentiable function, the PDF of $Y$ is given by:
      $f_Y(y) = f_X(g^{-1}(y)) \left| \frac{d}{dy} g^{-1}(y) \right|$.

  ---

  Conclusion

  • Each section provides detailed insights into the fundamental concepts of probability distributions, essential for a comprehensive understanding in fields such as statistics and data analysis. The well-structured examples and problems ensure practical understanding of theoretical components.

  ---

  Additional Details

  Note:

  All examples, definitions, and explanations encapsulated herein are aimed to assist students in mastering the course content thoroughly, ultimately providing a robust foundation in probability and distribution theory.