Basic Probability Rules and Foundations of Statistics

Role of Probability in Inferential Statistics

• Statistics relies heavily on probability theory to draw conclusions about large populations based on data collected from smaller groups.

• A central challenge for statisticians is that gathering data from every individual in a population is often impossible or impractical. Consequently, data is collected from a sample (a smaller group), and probability principles are used to extrapolate these results to the larger group.

• Probability provides the mathematical foundation and specific conditions under which these sample-to-population inferences are valid.

Fundamental Vocabulary and Concepts of Probability

• Probability: A quantitative measure of the likelihood that a specific event will occur due to randomness or chance.

• Probability Experiment: Any process with uncertain outcomes where all possible outcomes can be listed in advance, but the actual outcome of any given trial cannot be known until the experiment is performed.

  • Flipping Coins: Flipping a coin has two potential outcomes (heads or tails); the specific result is unknown until flipped.

  • Rolling Dice: Rolling a die produces a random outcome from the available sides.

  • Random Selection of Individuals: Choosing a person from a group at random and measuring a variable (e.g., asking for their age).

  • Waiting for Public Transit: Observing whether a bus arrives early, on time, or late when the exact schedule adherence is unknown.

  • Weather Forecasting: Predicting atmospheric conditions, such as stating a $30\%$ or $50\%$ chance of rain.

• Trial: A single performance of a probability experiment.

• Outcome: The specific result obtained from a single trial of a probability experiment.

  • Example: Flipping a coin once and getting heads makes "heads" the outcome. Rolling a die and getting a $3$ makes "$3$" the outcome.

• Sample Space ($S$): The set of all possible outcomes for a probability experiment.

  • Typically expressed using set notation with curly brackets ($\{\}$).

Constructing Sample Spaces and Outcome Counting

• Flipping One Coin:

  • $S = \{\text{heads}, \text{tails}\}$ or $\{H, T\}$

• Rolling One Standard Die:

  • Unless specified otherwise, standard dice are assumed to have $6$ sides.

  • $S = \{1, 2, 3, 4, 5, 6\}$

• Flipping Two Coins:

  • Each outcome consists of the combined results of both coins.

  • $S = \{HH, HT, TH, TT\}$

  • The outcomes $HT$ (heads on first coin, tails on second) and $TH$ (tails on first coin, heads on second) represent distinct physical outcomes because each coin displays a different side.

• Flipping Three Coins:

  • $S = \{HHH, HHT, HTH, HTT, THH, THT, TTH, TTT\}$

  • Total outcomes = $8$.

  • Detailed breakdown of combinations:

  • $3$ heads: $\{HHH\}$ ($1$ outcome)

  • $2$ heads and $1$ tail: $\{HHT, HTH, THH\}$ ($3$ outcomes)

  • $1$ head and $2$ tails: $\{HTT, THT, TTH\}$ ($3$ outcomes)

  • $3$ tails: $\{TTT\}$ ($1$ outcome)

• Determining the Number of Outcomes (Fundamental Counting Principle):

  • When a probability experiment consists of multiple sequential actions, the total number of outcomes is found by multiplying the number of possible outcomes for each individual action (do not add).

  • Three Coin Flips:     $\text{Total Outcomes} = 2 \times 2 \times 2 = 2^3 = 8$

  • Rolling Two Six-Sided Dice:     $\text{Total Outcomes} = 6 \times 6 = 6^2 = 36$

Events and Core Probability Rules

• Event ($E$): A subset of a sample space consisting of one or a combination of possible outcomes.

  • Three Coins (Event: Exactly One Head):     $E = \{HTT, THT, TTH\}$ ($3$ outcomes out of $8$ possible).

  • Two Dice (Event: Sum of Dice Equals 4):     $E = \{(1,3), (2,2), (3,1)\}$ ($3$ outcomes out of $36$ possible). Note that $(1,3)$ and $(3,1)$ are distinct outcomes based on which die displays which number.

  • Three Coins (Event: All Three Heads):     $E = \{HHH\}$ ($1$ outcome out of $8$ possible).

• Mathematical Nature of Probabilities:

  • Probabilities represent proportions (part of a whole) and are expressed as fractions or decimals.

  • Numerical range: Bounded strictly between $0$ and $1$, inclusive ($0 \le P(E) \le 1$).

  • A probability of $1$ corresponds to a $100\%$ chance (certainty; one whole).

  • Probabilities cannot exceed $1$ ($100\%$) and cannot be negative.

  • Fractions: The numerator must be less than or equal to the denominator.

  • Decimals: Must be in the format $0.xxx$.

  • Percentages: Calculated by multiplying the proportion or decimal by $100$ ($P(\text{percent}) = P(E) \times 100$), ranging from $0\%$ to $100\%$.

• Two Fundamental Rules of Probability:

  • Rule 1 (Probability Bounds): For any event $E$, $0 \le P(E) \le 1$.

  • $P(E) = 0$ indicates an impossible event (cannot occur).

  • $P(E) = 1$ indicates a certain event (always occurs).

  • Rule 2 (Sum of Probabilities in a Sample Space): The sum of the probabilities of all distinct simple outcomes in a sample space equals $1$ ($100\%$):     $\sum P(e_i) = 1$

• Calculational Answer Check & Critical Thinking:

  • Always evaluate calculated probabilities to ensure they lie within the interval $[0, 1]$.

  • Any calculated result outside the range $[0, 1]$ (e.g., negative numbers or values greater than $1$) indicates a calculation error.

Classical Probability

• Definition: Classical probability applies to experiments where every outcome in the sample space is equally likely to occur.

  • Applicable to fair coins, fair dice, and standard decks of cards.

• Classical Probability Formula:   $P(E) = \frac{\text{Number of outcomes in event } E}{\text{Total number of outcomes in sample space } S} = \frac{N(E)}{N(S)}$

• Exhaustive Calculation Examples:

  • Example 1: Rolling a 4 on a Single Six-Sided Die

  • Experiment: Rolling one standard die ($N(S) = 6$).

  • Event $E$: Getting a $4$ ($N(E) = 1$).

  • Calculation:       $P(\text{getting a 4}) = \frac{1}{6} \approx 0.167 \text{ or } 16.7\%$

  • Example 2: Drawing a 9 from a Standard Deck of Cards

  • Standard deck total cards (excluding jokers): $N(S) = 52$.

  • Cards that are a $9$: $9$ of clubs, $9$ of diamonds, $9$ of hearts, $9$ of spades ($N(E) = 4$).

  • Calculation:       $P(\text{drawing a 9}) = \frac{4}{52} = \frac{1}{13} \approx 0.0769 \text{ or } 7.7\%$

• Formatting and Rounding Standards:

  • Fractional probabilities must always be fully reduced to lowest terms.

  • Standard decimal rounding for probabilities is to the nearest thousandth ($0.001$) or ten-thousandth ($0.0001$) place unless directed otherwise.

Interpretation of Probabilities and Unusual Events

• Long-Term Average Interpretation (Law of Large Numbers):

  • A probability indicates what happens on average over a large number of repeated trials, not a guarantee for a small number of trials.

  • Fractional Interpretation: $P(\text{4}) = \frac{1}{6}$ indicates that if a die is rolled $6$ times, on average, one of those rolls will result in a $4$. Individual samples of $6$ rolls may yield zero $4$s, one $4$, multiple $4$s, or all $4$s due to random chance.

  • Decimal/Large-Sample Interpretation: $0.167 = \frac{167}{1000}$. If a die is rolled $1000$ times, on average, approximately $167$ of those rolls will result in a $4$.

• Likelihood Bounding:

  • Probabilities closer to $1$ ($100\%$) indicate highly likely events.

  • Probabilities closer to $0$ ($0\%$) indicate highly unlikely events.

• Definition of an Unusual Event:

  • An event $E$ is formally defined as unusual if its probability is less than or equal to $0.05$ ($P(E) \le 0.05$ or $5\%$).

  • This $0.05$ threshold is a standard convention used in statistical inference to denote unexpected outcomes.

  • An event with a probability of $0.06$ ($6\%$) is not formally classified as unusual under this threshold, despite having a low probability.

Empirical Probability

• Definition: Empirical probability is used when the outcomes of a probability experiment are not equally likely.

  • Common in real-world scenarios and observational data (e.g., age distribution, bus timeliness).

  • Relies on relative frequencies observed in real data.

• Empirical Probability Formula:   $P(E) = \frac{\text{Frequency of event } E}{\text{Total number of observed data values or trials}} = \frac{f}{n}$

• Exhaustive Calculation Examples:

  • Example 1: College Class Academic Level Distribution

  • Observed Data:

    • Freshmen: $18$

    • Sophomores: $12$

    • Juniors: $4$

    • Seniors: $6$

    • Total students ($n$): $18 + 12 + 4 + 6 = 40$

  • Question A: Probability of randomly selecting a Sophomore:       $f = 12, \quad n = 40$       $P(\text{Sophomore}) = \frac{12}{40} = \frac{3}{10} = 0.30 \text{ or } 30\%$

  • Question B: Probability of randomly selecting a Junior or Senior:       $f = 4 + 6 = 10, \quad n = 40$       $P(\text{Junior or Senior}) = \frac{10}{40} = \frac{1}{4} = 0.25 \text{ or } 25\%$

  • Example 2: Bus Arrival Timeliness

  • Observed Data: Waiting for a bus $50$ times ($n = 50$); the bus arrived late $3$ times ($f = 3$).

  • Calculation:       $P(\text{Late}) = \frac{3}{50} = 0.06 \text{ or } 6\%$

  • Evaluation against unusual event criteria: Since 0.06 > 0.05, the bus being late is not classified as an unusual event.

Probability Models

• Definition: A mathematical representation (typically formatted as a table) that lists every possible outcome in a sample space alongside its corresponding probability.

• Requirements for a Valid Probability Model:

  1. Every listed probability $P(x_i)$ must be between $0$ and $1$ inclusive ($0 \le P(x_i) \le 1$).

  2. The sum of all listed probabilities must equal exactly $1$ ($\sum P(x_i) = 1$).

• Complete Probability Model Construction (College Class Dataset):

• Model Verification:

  • Individual bounds check: $0.45, 0.30, 0.10, 0.15$ all lie within $[0, 1]$.

  • Summation check: $0.45 + 0.30 + 0.10 + 0.15 = 1.00$.