Probability Theory - Refresher

Sample Space ( Ω )

• Represents all possible outcomes of an experiment.

Event

• A specific subset of outcomes from the sample space.

Probability Function

• Represents the mapping from an event to a real number, denoted as ( P(event) ).

  • Conditions: ( 0 \leq P(event) \leq 1 )

  • Sure event: ( P(Sure event) = 1 )

  • Impossible event: ( P(Impossible event) = 0 )

Intersection and Union of Events

• Intersection ( P(A \cap B) ): Probability of both events A and B occurring.

• Union ( P(A \cup B) ): Probability of either event A or B occurring.

  • ( P(A \cup B) = P(A) + P(B) - P(A \cap B) )

Disjoint Events

• Events A and B are disjoint if they have no outcomes in common ( (A \cap B = \emptyset) ).

  • Example: Probability that either A or not A occurs must total 1: ( P(A \cup
    eg A) = P(A) + P(
    eg A) = 1 )

Practical Examples

• For a fair die, ( P({1, 2, 3, 4, 5, 6}) = 1 )

  • Each outcome of the die is equally likely: ( P({k}) = \frac{1}{6} ) for ( k \in {1, 2, 3, 4, 5, 6} )

• Probability of an even number appearing: ( P({2, 4, 6}) = \frac{1}{2} )

• Mixed events example: ( P(\text{even number or } 6) ) not disjoint so calculation changes accordingly.

Interpretations of Probability

Frequentist Interpretation

Subjectivist Interpretation

• Represents personal belief regarding the likelihood of an event.

Conditional Probability

• , given prior knowledge of another event:

  • General formula: ( P(A|B) = \frac{P(A, B)}{P(B)} ) (for ( P(B) > 0 ))

Multiplicative and Chain Rules

• Multiplicative Rule:

  • Probability of both A and B: P(A, B) = P(B) \ P(A|B)

• Chain Rule:

  • For multiple events: ( P(A1, A2, …, An) = P(A1) \cdot P(A2|A1) \cdots P(An|A1,…,An−1) )

Theorem of Total Probability

• For events A and B:

  • ( P(A) = P(A, B) + P(A,
    eg B) ) where B and (
    eg B ) cover all outcomes.

Independence of Events

• Events A and B are independent if: ( P(A, B) = P(A) \cdot P(B) )

  • Happenstance of one does not affect the other.

Conditional Independence

• Events A and B are conditionally independent given event C:

  • ( P(A, B|C) = P(A|C) \cdot P(B|C) )

Random Variable

• A variable representing outcomes of random phenomena (e.g., numbers on die throws).

  • Example: ( X ) is the outcome from one throw; ( Z ) is the sum from two throws.

Probability Distribution

• Specifies the probability for each outcome of a random variable, summing to 1.

  • Example distribution for a biased die shown.

Joint Probability Distribution

• Defines probabilities across combinations of multiple random variables.

• Example: Joint distribution for conditions involving Fever, Headache, and Flu.

Marginal Probability Distributions

• Extracting probabilities for subsets of variables from joint distributions by summing relevant combinations.

Conditional Probability Distributions

• Computation of conditional probabilities based on previously calculated marginal distributions.

  • Example: Probability of Flu given Fever and Headache.

Probability Density Functions

• Describes continuous random variables:

  • Must meet requirements: ( f(x) \geq 0 ) and ( \int_{-\infty}^{\infty} f(x) dx = 1 )

• Represents areas rather than exact values.

Common PDFs

• Uniform Distribution: Equal probability across its range.

• Normal Distribution (Gaussian): Characterized by mean (( \mu )) and standard deviation (( \sigma )).

• Exponential Distribution: Models the time until an event occurs.

• Mixture of Gaussians: Approximation of complex distributions by combining multiple normal distributions.