Probability for Discrete Variables

Foundations and Core Axioms of Discrete Probability

• Historical Foundation of Probability Theory:

  • As stated by Pierre-Simon Laplace: "It is remarkable that a science which began with the consideration of games of chance should have become the most important object of human knowledge."

  • The formal study of probability originated from the mathematical analysis of gambling and games of chance, later evolving into the primary framework for quantifying uncertainty across all scientific domains.

• Core Terminology:

  • Experiment: Any specific action, trial, or procedure that gives rise to an event or a set of events.

  • Event / Outcome ($A$): A particular result or collection of results generated by an experiment (for example, drawing a blue ball from a container).

  • Sample Space ($S$): The comprehensive universal set containing all possible elementary events or outcomes that can occur in an experiment.

  • Variable Classification ($X$): In discrete models such as drawing colored balls ($X = \text{blue, red, green balls}$), the variable is classified as a nominal-level discrete variable.

• Axiomatic Rules of Probability:

  • Non-negativity and Bounded Probability (Rule i):     $0 \le P(A) \le 1$     The probability of any event $A$ occurring within the sample space $S$ is strictly bounded between $0$ (impossible event) and $1$ (certain event).

  • Total Probability of the Sample Space (Rule ii):     $P(S) = 1$     The probability assigned to the entirety of the sample space is equal to $1$, indicating that one of the possible outcomes within $S$ must occur.

Marginal Probability and Complements

• Marginal (Unconditional) Probability:

  • Definition: The probability of a single discrete event occurring without reference to or conditioning upon any other event, represented mathematically as $P(A)$.

  • Equally Likely Outcomes Formula:

  • If a sample space contains a finite number of $N$ total elements or occurrences belonging to a set of discrete events ($A_i$), and every individual element has an equal probability of occurring:       $P(A_i) = \frac{N_{A_i}}{N}$

  • In this expression, $N_{A_i}$ denotes the count of elements belonging to the specific event of interest $A_i$, and $N$ denotes the total count of elements within sample space $S$.

• Complement of an Event:

  • Definition: The complement of an event represents the probability of that event not occurring.

  • Mathematical Representation: Denoted as $P(A_i')$ or $P(A^c)$.

  • Formula:     $P(A_i') = 1 - P(A_i)$

• Calculation Example (Ball Selection Model):

  • Experiment Setup: An urn contains a total of $N = 20$ balls consisting of green, red, and blue balls. The subset of blue balls contains $N_{\text{blue}} = 6$ items.

  • Marginal Probability Calculation:     $P(\text{blue}) = \frac{6}{20} = 0.30$

  • Complement Probability Calculation:     $P(\text{blue}') = 1 - P(\text{blue}) = 1 - 0.30 = 0.70$

Set Operations and Event Classifications

• Intersection and Joint Probability ($A \cap B$):

  • The intersection represents the joint occurrence of both event $A$ and event $B$.

  • Referred to conceptually and operationally as Joint Probability, written as $P(A \cap B)$.

  • Associated Venn Diagram Regions:

  • $A \cap B$: The overlapping area containing elements belonging simultaneously to both event $A$ and event $B$.

  • $AB'$: All elements within the sample space that belong to event $A$ but do not belong to event $B$.

  • $BA'$: All elements within the sample space that belong to event $B$ but do not belong to event $A$.

• Union ($A \cup B$):

  • The union refers to the occurrence of either event $A$, event $B$, or both (i.e., the occurrence of at least one of the events).

  • Constituent possibilities included within the union set:

  • $A$ is a possibility

  • $B$ is a possibility

  • $A \cap B$ is a possibility

• Conditional Probability ($A|B$):

  • The probability of event $A$ occurring given the prior knowledge or condition that event $B$ has occurred.

  • Represented symbolically as $P(A|B)$.

• Classifications of Event Relationships:

  • Mutually Exclusive (Disjoint) Events:

  • Events that cannot occur simultaneously; the occurrence of one completely precludes the occurrence of the other.

  • The intersection is the empty set:       $P(A \cap B) = 0$

  • Independent Events:

  • Two events are independent if the occurrence of event $A$ provides no information about the occurrence of event $B$, and vice versa.

  • The occurrence of one event does not alter the probability of the other event occurring.

  • Dependent Events:

  • Two events are dependent if the occurrence of event $A$ provides information regarding the occurrence of event $B$ (i.e., it alters the probability of $B$ occurring), and vice versa.

Sampling Strategies and Dependency

• Sampling With Replacement:

  • Items are extracted from the sample space, observed, and returned to the population prior to the next draw.

  • The composition and size of the sample space remain invariant across successive draws.

  • Maintains strict statistical independence between successive draws.

• Sampling Without Replacement:

  • Items are drawn from the sample space and removed permanently without being returned.

  • The total number of available elements and the relative proportions of remaining categories change after each extraction.

  • Fundamental Principle: Sampling without replacement directly creates statistical dependencies whenever operating on a finite sample space.

Discrete Probability Calculation Rules and Formula Matrix

• Fundamental Calculation Rules:

  • Additive Rule of Probability: If two discrete events, $A$ and $B$, are mutually exclusive:     $P(A \cup B) = P(A) + P(B)$

  • Multiplicative Rule of Probability: If two discrete events, $A$ and $B$, are independent:     $P(A \cap B) = P(A) \times P(B)$

• Comprehensive Formula Matrix (Event Types vs. Probability Types):

  • Mutually Exclusive Events:

  • Union Probability:       $P(A \cup B) = P(A) + P(B)$

  • Joint Probability:       $P(A \cap B) = 0$

  • Conditional Probability:       $P(A|B) = 0$

  • Independent Events:

  • Union Probability:       $P(A \cup B) = P(A) + P(B) - P(A \cap B)$

  • Joint Probability:       $P(A \cap B) = P(A) \times P(B)$

  • Conditional Probability:       $P(A|B) = P(A)$

  • Dependent Events:

  • Union Probability:       $P(A \cup B) = P(A) + P(B) - P(A \cap B)$

  • Joint Probability:       $P(A \cap B) = P(A|B) \times P(B)$

  • Conditional Probability:       $P(A|B) = \frac{P(A \cap B)}{P(B)}$

  • Information Requirements for Dependent Events:

  • For dependent events, calculating probabilities requires either being explicitly given $P(A \cap B)$ or $P(A|B)$, or being provided with sufficient information to mathematically deduce them.

Probability Mass Functions and Joint Distributions

• Probability Mass Function (PMF):

  • A mathematical function that gives the probability that a discrete random variable $X$ is exactly equal to some value $x_i$:     $P(X = x_i) \quad \text{for } i = 1, 2, 3, \dots, N$

  • Empirical Likert Distribution Example:

  • Category 1: $P(X = 1) \approx 0.68$

  • Category 2: $P(X = 2) \approx 0.27$

  • Category 3: $P(X = 3) \approx 0.03$

  • Category 4: $P(X = 4) \approx 0.02$

  • Category 5: $P(X = 5) \approx 0.00$

• Joint Probability Mass Function (Two Independent Events $A$ and $B$):

  • Structured as a contingency matrix crossing events $A$ and $A'$ with $B$ and $B'$:

  • Cell $(B, A)$:       $P(A \cap B) = P(A) \times P(B)$

  • Cell $(B, A')$:       $P(A' \cap B) = P(A') \times P(B)$

  • Row $B$ Marginal Sum:       $P(A \cap B) + P(A' \cap B) = P(B)$

  • Cell $(B', A)$:       $P(A \cap B') = P(A) \times P(B')$

  • Cell $(B', A')$:       $P(A' \cap B') = P(A') \times P(B')$

  • Row $B'$ Marginal Sum:       $P(A \cap B') + P(A' \cap B') = P(B')$

  • Column $A$ Marginal Sum:       $P(A \cap B) + P(A \cap B') = P(A)$

  • Column $A'$ Marginal Sum:       $P(A' \cap B) + P(A' \cap B') = P(A')$

Interpreting Probabilities and Common Fallacies

• Fallacy of Individual Probability from Aggregate Rates:

  • Scenario: A clinical research study demonstrates that a novel pharmaceutical drug was effective for $90\%$ of the participants in the sample.

  • Deduction Regarding an Individual: This finding does not mean that there is a $90\%$ probability of the drug being effective for an arbitrary individual outside or inside the study.

  • Participant-Level Knowledge:

  • What were the chances of the drug being effective for participant $4$ in the study? Unknown.

  • What are the chances that the drug will be effective for a new individual? Unknown.

  • Population-level or aggregate sample frequencies cannot be equated with an individual's deterministic outcome or unconditional probability.

• Real-World Lifetime Risks and Perceptions:

  • Epidemiology of Depression: Over a lifetime, an individual faces a $20\%$ (or $1\text{ in }5$) probability of experiencing an episode of depression.

  • Comparative Threat Assessment (Andrew Shaver, November 23, 2015, The Washington Post):

  • Human risk perception frequently overemphasizes salient, dramatic hazards while ignoring statistically greater everyday risks.

  • An individual, their family, and their community are statistically far more vulnerable to ignored domestic threats than to terrorism.

  • Comparative Example: An individual is statistically more likely to be fatally crushed by falling furniture than killed in a terrorist attack.

  • Quantitative Comparison (Paris Attacks, November 2015): While the terrorist attacks in Paris resulted in approximately $130$ fatalities, roughly three times that number ($\approx 390$) of French citizens died on that exact same day from cancer.