Introductory Statistics Explained: Comprehensive Review

Principles of Data Collection, Sampling, and Study Design

Statistics is fundamentally an empirical discipline dedicated to learning about an unknown underlying reality through the disciplined collection and analysis of data.

Populations, Parameters, Samples, and Statistics

• Individuals (Units / Cases): The distinct entities or objects on which measurements are recorded.
• Population: The comprehensive collection of all individuals or units of interest. Populations can be finite (e.g., the $25$ children in a specific preschool class) or conceptual/infinite (e.g., all present and future bags of popcorn manufactured by a standardized industrial process).
• Parameter: A fixed numerical summary describing a population. Parameters are designated using Greek letters (such as the population mean $\mu$ and the population variance $\sigma^2$). In practice, parameters are almost universally unknown because measuring an entire population is impossible, destructive, or prohibitively costly.
• Sample: A subset of individuals selected from the population that is directly observed.
• Statistic: A numerical value computed from sample observations. Statistics are designated using Latin letters (such as the sample mean $\bar{x}$ and sample variance $s^2$). A statistic is a random variable whose observed value serves as a point estimate of an unknown population parameter.

Motivating Case Studies in Sampling and Parameters

• Popcorn Nutritional Claims: In $1994$, the Center for Science in the Public Interest (CSPI) found theatre popcorn loaded with saturated fat. A $2009$ follow-up revealed actual calorie and fat counts exceeded corporate claims. Consider an establishment claiming an average of $920\,\text{calories}$ per large untopped bag. Testing a sample of $n = 8$ bags yields a sample mean of $\bar{x} = 1210\,\text{calories}$. Whether this provides conclusive evidence that the true parameter $\mu > 920\,\text{calories}$ depends on the sampling distribution of the sample mean.
• Raccoon Pathogen Surveillance: Trapping $210$ wild raccoons in a conservation area near Guelph, Ontario revealed that $20\%$ tested positive for Salmonella on their paws. The $210$ raccoons represent the sample, and $20\%$ is a sample statistic. Defining the target population is challenging: does it encompass all raccoons in Southern Ontario, or solely raccoons prone to entering traps? Trapping behavior may correlate with health status or foraging habits, introducing subtle selection bias.

• COVID-19 Clinic Projections: During early $2020$, clinic physicians extrapolated the proportion of symptomatic patients testing positive for SARS-CoV-2 directly to the general population of California. Because clinical patients sought testing due to symptoms or known exposures, their positive rate grossly overestimated the general population prevalence.

Bias and Random Sampling Methods

A sampling design is unbiased if the sample values mirror the target population on average, without systematically over- or under-representing specific sub-demographics.

Failure Modes of Non-Random Sampling

• Voluntary Response Bias: Arises when individuals self-select into the sample (e.g., call-in polls on partisan broadcasts or voluntary student course evaluations). Respondents with extreme negative or positive emotions participate disproportionately, while indifferent individuals abstain.
• Convenience Sampling: Involves sampling units that are easiest to reach, often distorting sample composition relative to the target population.

Probability Sampling Designs

1. Simple Random Sampling (SRS): A design where every possible sample of size $n$ has the exact same probability of being selected. Sampling is typically conducted without replacement. If drawing a sample of size $n$ from a finite population of size $N$, the total number of distinct samples is given by the combination formula $\binom{N}{n}$, and the marginal inclusion probability for any individual is $\frac{n}{N}$.

   Worked Small-Population Example: For a population of $N = 8$ individuals (Mary, Hal, Eddy, Tony, Tom, Pete, Twiddy, Jemmy), there are exactly:

   $\binom{8}{3} = \frac{8!}{3!(8-3)!} = 56$

   possible samples of size $n = 3$. Each combination has a selection probability of $\frac{1}{56}$, and each individual has a selection probability of $\frac{3}{8}$.

2. Stratified Random Sampling: The population is partitioned into mutually exclusive, internally homogeneous subgroups called strata. Independent simple random samples are drawn within each stratum, and stratum-specific estimates are combined mathematically. Stratification guarantees proportional representation across critical demographic or geographic boundaries and reduces the variance of population parameter estimators compared to an SRS of equivalent total size.

3. Cluster Sampling: Applied when the population naturally aggregates into heterogeneous units (clusters). Rather than sampling individuals across all clusters, researchers randomly select a subset of clusters, and then survey all (or a sample of) individuals within the chosen clusters. This design reduces logistics costs when sampling geographically dispersed populations.

   Examples: Assessing rubella immunity across schoolchildren by randomly selecting $4$ out of $12$ regional schools; surveying ecological features across lake islands by randomly sampling whole lakes (clusters of islands).

4. Systematic Sampling ($\text{Sampling Every } k\text{th Unit}$): From an ordered list or continuous stream of items, a random starting point within the first $k$ units is selected, after which every $k\text{th}$ subsequent item is included. While operationally convenient in industrial assembly lines or admissions streams, systematic sampling is vulnerable to severe bias if a latent periodicity in the process aligns with the sampling interval $k$.

Observational Studies Versus Randomized Experiments

• Response Variable: The primary outcome measured to answer a research question.
• Explanatory Variable: A factor hypothesized to explain or cause changes in the response variable.
• Observational Study: Researchers measure variables of interest without assigning treatments or manipulating conditions. Groups are pre-existing.

  Crucial Limitation: Observational studies cannot isolate cause-and-effect relationships due to lurking variables—unmeasured variables related to both the explanatory and response variables that produce spurious associations.

  Case Study (Maternal Cocaine Exposure): A study tracked $361$ cocaine-exposed neonates and $387$ unexposed neonates. While exposed infants exhibited lower birth weights ($\text{p} < 0.05$), maternal cocaine usage was confounded with socioeconomic status, nutritional deficit, alcohol intake, and prenatal medical care.

• Randomized Controlled Experiment: Researchers actively impose specific treatments on experimental units via random assignment.

  Mechanism of Randomization: Random allocation ensures that all lurking variables—both known and unmeasured—are balanced across treatment groups at baseline up to random chance. Observed differences in the response variable can therefore be attributed directly to the experimental treatments.

  Case Study (Caloric Restriction in Mice): An experiment randomly assigned $n = 338$ mice across $6$ distinct nutritional regimens: a standard unrestricted lab diet ($\text{NP}$) and five progressively restricted diets ($\text{R1}$, $\text{R2}$, $\text{R3}$, $\text{R4}$, $\text{R5}$). Lifespans in months showed systematic increases with caloric restriction, establishing a causal relationship between diet restriction and longevity.

• Confounding: Two variables are confounded when their individual effects on the response variable cannot be distinguished. Confounding arises from non-randomized observational studies, poorly constructed factorial experiments, or static temporal comparisons (e.g., comparing educational test scores across successive school years, where cohort differences, exam difficulty, and curricular changes are inextricably blended).

• Causal Inference Criteria in Observational Frameworks: Establishing causality from observational data requires rigorous triangulation: consistent replication across diverse demographics, strong biological/mechanistic plausibility, dose-response gradients, and alignment with natural experiments (e.g., examining ground temperatures after air traffic groundings following September 11, or evaluating state-level employment shifts across contiguous borders following local minimum wage hikes).

Principles of Sample Size and Study Reliability

• Absolute Sample Size Governs Precision: The reliability and margin of error of a sample statistic depend almost entirely on the absolute sample size $n$, not on the sampling fraction $\frac{n}{N}$, provided the population size $N$ is substantially larger than the sample ($N > 10n$ or $N > 20n$). A random sample of $n = 1000$ individuals selected from a nation of $300\,\text{million}$ provides virtually the same statistical precision as $n = 1000$ drawn from a city of $100,000$.
• Sample Size Cannot Cure Bias: Large sample sizes shrink sampling variability but do not reduce systematic bias. An enormous sample drawn via a biased design yields a very precise, highly confident, yet completely incorrect conclusion.
• Unequal Sample Sizes: Comparing groups with disparate sample sizes (e.g., $n_1 = 126$ versus $n_2 = 50$) is statistically valid provided appropriate variance models are applied.
• Publication Bias and Replicability: Empirical literature exhibits publication bias because novel, statistically significant findings ($\text{p} < 0.05$) are accepted for publication far more frequently than null results. Replicating findings across independent labs is essential to verify genuine scientific effects.

Descriptive Statistics: Visualizing and Summarizing Data

Descriptive statistics provides numerical and graphical methods to summarize the central features, spread, and shape of observed data.

Categorical Data Summaries

A categorical (qualitative) variable allocates individuals into mutually exclusive qualitative groups.

• Nominal Variables: Categories have no inherent rank or order (e.g., blood types $\text{A}, \text{B}, \text{AB}, \text{O}$; modes of transport).
• Ordinal Variables: Categories possess an inherent progression (e.g., gun calibers categorized as Small, Medium, Large; survey scales ranging from Strongly Disagree to Strongly Agree).
• Frequency: The raw count of observations in category $i$.
• Relative Frequency: The proportion of total observations belonging to category $i$:

$\text{Relative Frequency} = \frac{\text{Frequency}}{n}$

• Percent Relative Frequency: $\text{Relative Frequency} \times 100\%$.

Case Study: Road Rage Victim Transport Modes (Hamburg, Germany)

A clinical study assessed $n = 108$ physical assault road rage victims:

Bar charts depict category proportions through lengths along a standardized scale. Pie charts encode proportions through two-dimensional angular slices. Human visual perception judges linear differences along a shared axis with substantially higher accuracy than angular areas, making bar charts superior for comparative analysis.

Comparing Groups: Gun Buyback Programs (Milwaukee)

Comparing gun calibers surrendered during a city buyback program against guns recovered from homicides and suicides demonstrates how categorical distributions differ across contexts:

Citizens selectively surrendered inexpensive, small-caliber firearms ($78\%$ small) to collect financial incentives, while criminal homicides involved predominantly medium-caliber weapons ($64\%$ medium).

Quantitative Data Summaries

Quantitative variables represent numeric measurements where standard arithmetic operations are meaningful.

Frequency Tables and Histograms

To construct a frequency table, the numeric domain is partitioned into contiguous, non-overlapping intervals called classes or bins of uniform width:

$\text{Class Width} \approx \frac{\text{Maximum} - \text{Minimum}}{\text{Number of Classes}}$

• Cumulative Frequency: The count of observations falling in a given class or any lower class.
• Cumulative Relative Frequency: The cumulative frequency divided by the sample size $n$.

Worked Example (Midterm Exam Scores, $n = 40$):

Scores: $21, 32, 32, 33, 37, 39, 41, 44, 46, 48, 48, 48, 54, 55, 57, 57, 63, 64, 65, 65, 66, 67, 67, 69, 69, 71, 72, 72, 73, 73, 75, 77, 77, 78, 82, 87, 87, 88, 94, 97$.

Histograms plot bin frequencies (or relative frequencies) as contiguous vertical bars against bin intervals. Bin width selection is critical: too many bins produce noisy, jagged plots; too few bins obscure structural features.

Structural Characteristics of Distributions

1. Modality: The number of prominent peaks in the distribution. Distributions can be unimodal (one peak), bimodal (two peaks), or multimodal. Bimodality frequently signals that data from two distinct sub-populations have been pooled together.
2. Symmetry and Skewness:
   • Symmetric: The left and right halves form approximate mirror images.
   • Right-Skewed (Positively Skewed): Characterized by a long tail extending toward higher positive values. Commonly seen in variables with a hard physical lower bound (such as zero) but no restrictive upper bound (e.g., survival times, housing prices, individual wealth).
   • Left-Skewed (Negatively Skewed): Characterized by a long tail extending toward lower values. Arises when data cluster near an upper limit with occasional small values (e.g., scores on an easy exam) or due to specific pathological mechanisms (e.g., human birth weights, where premature births form an extended lower tail).

3. Outliers: Isolated observations that lie far from the central pattern of the data.

Additional Visual Displays

• Empirical Cumulative Distribution Function (eCDF): Plots the running proportion of sample observations less than or equal to $x$, forming a non-decreasing step function from $0$ to $1$.

• Time Plots: Sequential line charts displaying quantitative values against chronological time. Useful for identifying temporal trends, structural regime shifts, and ecological collapses (e.g., Newfoundland northern cod harvests dropping toward zero prior to the $1992$ moratorium).

• Stemplots (Stem-and-Leaf Displays): Tabular arrangements where each value is split into a leading digit (stem) and trailing digit (leaf). Stemplots retain raw data values while approximating the shape of a histogram on its side. Stemplots require an explicit key/legend detailing the placement of the decimal point relative to the stem.

Numerical Measures of Central Tendency

• Sample Mean ($\bar{x}$): The arithmetic balance point of all observations:

$\bar{x} = \frac{\sum_{i=1}^n x_i}{n}$

• Median: The physical middle observation when values are ordered from smallest to largest. If $n$ is odd, it is the value at rank $\frac{n+1}{2}$. If $n$ is even, it is the arithmetic average of the two middle values at ranks $\frac{n}{2}$ and $\frac{n}{2} + 1$.
• Mode: The most frequently occurring value in the sample. For continuous variables, it corresponds to the modal class containing the highest frequency.

Sensitivity to Extreme Values

The mean incorporates the numerical magnitude of every observation, making it sensitive to extreme values and skewness. The median depends on rank order, making it resistant to outliers.

Comparative Example (Oldest NHL Players, $n = 5$):

Ages: $33, 34, 34, 35, 37$.

$\bar{x} = \frac{33 + 34 + 34 + 35 + 37}{5} = 34.6\,\text{years}$

$\text{Median} = 34\,\text{years}, \quad \text{Mode} = 34\,\text{years}$

If the maximum value had been recorded erroneously as $370$ instead of $37$, the median remains $34$, whereas the sample mean becomes:

$\bar{x} = \frac{33 + 34 + 34 + 35 + 370}{5} = 101.2\,\text{years}$

Relationship Between Mean, Median, and Distribution Shape

• Symmetric Distributions: The mean and median are equal: $\text{Mean} = \text{Median}$.
• Right-Skewed Distributions: Extreme high values pull the mean into the right tail, resulting in: $\text{Mean} > \text{Median}$.
• Left-Skewed Distributions: Extreme low values pull the mean into the left tail, resulting in: $\text{Mean} < \text{Median}$.

Case Study (Guinea Pig Lifetimes, $n = 60$): In guinea pigs inoculated with tubercle bacilli, the survival times are right-skewed, yielding a median of $214.5\,\text{days}$ and a mean of $241.2\,\text{days}$.

Alternative Measures of Center

• Geometric Mean: $\left(\prod_{i=1}^n x_i\right)^{1/n}$
• Harmonic Mean: $\frac{n}{\sum_{i=1}^n \frac{1}{x_i}}$
• Trimmed Mean: Computes the arithmetic mean after removing a fixed percentage (e.g., $5\%$ or $10\%$) of the lowest and highest observations, reducing sensitivity to outliers.

Numerical Measures of Dispersion (Variability)

• Range: $\text{Range} = \text{Maximum} - \text{Minimum}$
• Deviations from the Mean: The directed differences $(x_i - \bar{x})$. The sum of deviations about the sample mean is identically zero for any dataset:

$\sum_{i=1}^n (x_i - \bar{x}) = \sum_{i=1}^n x_i - n\bar{x} = n\bar{x} - n\bar{x} = 0$

• Mean Absolute Deviation (MAD): Measures the average linear distance from the mean:

$\text{MAD} = \frac{\sum_{i=1}^n |x_i - \bar{x}|}{n}$

• Sample Variance ($s^2$): The average squared deviation using $n-1$ degrees of freedom:

$s^2 = \frac{\sum_{i=1}^n (x_i - \bar{x})^2}{n - 1} = \frac{\sum_{i=1}^n x_i^2 - \frac{(\sum_{i=1}^n x_i)^2}{n}}{n - 1}$

• Sample Standard Deviation ($s$): The positive square root of the variance, restoring units to the original measurement scale:

$s = \sqrt{s^2} = \sqrt{\frac{\sum_{i=1}^n (x_i - \bar{x})^2}{n - 1}}$

Degrees of Freedom and the $n-1$ Denominator

Using $n$ in the denominator creates a sample variance that systematically underestimates the true population variance $\sigma^2$. Because the sample mean $\bar{x}$ mathematically minimizes the sum of squared differences $\sum (x_i - c)^2$ across any choice of $c$, the sample sum of squares satisfies:

$\sum_{i=1}^n (x_i - \bar{x})^2 < \sum_{i=1}^n (x_i - \mu)^2$

Dividing by $n-1$ inflates the denominator by a factor of $\frac{n}{n-1}$, exactly correcting this negative bias and making $s^2$ an unbiased estimator of $\sigma^2$. The denominator $n-1$ reflects the degrees of freedom—the number of independent pieces of information remaining after estimating the single parameter $\mu$ with $\bar{x}$.

Interpreting the Standard Deviation

• The Empirical Rule: For unimodal, symmetric, bell-shaped (approximately normal) distributions:

  • Approximately $68.3\%$ of observations lie within $1$ standard deviation: $\bar{x} \pm 1s$
  • Approximately $95.4\%$ of observations lie within $2$ standard deviations: $\bar{x} \pm 2s$
  • Approximately $99.7\%$ of observations lie within $3$ standard deviations: $\bar{x} \pm 3s$
  • The full data range typically spans between $4s$ and $6s$.

• Chebyshev's Inequality: For any distribution regardless of shape, the proportion of observations lying within $k$ standard deviations of the mean ($k > 1$) is at least:

$1 - \frac{1}{k^2}$

• For $k = 2$: at least $1 - \frac{1}{4} = 0.75$ ($75\%$) of observations lie within $\bar{x} \pm 2s$.
• For $k = 3$: at least $1 - \frac{1}{9} \approx 0.889$ ($88.9\%$) of observations lie within $\bar{x} \pm 3s$.

Linear Transformations

A linear transformation transforms a variable $x$ into $x^*$ via:

$x^* = a + bx$

Applying this transformation affects summary statistics as follows:

• Measures of Center and Position: Transform directly by applying the function:

$\bar{x}^* = a + b\bar{x}, \quad \text{Median}^* = a + b(\text{Median})$

• Measures of Spread: Are invariant to the additive constant $a$ and scale by the magnitude of $b$:

$s_{x^*} = |b|s_x, \quad s_{x^*}^2 = b^2 s_x^2, \quad \text{IQR}_{x^*} = |b|\text{IQR}_x$

Temperature Conversion Example (Death Valley, July 2021):

Converting Fahrenheit ($x$) to Celsius ($x^*$) via $x^* = -\frac{160}{9} + \frac{5}{9}x$:

• Mean: $116.4^\circ\text{F} \implies -\frac{160}{9} + \frac{5}{9}(116.4) = 46.9^\circ\text{C}$
• Median: $117.0^\circ\text{F} \implies -\frac{160}{9} + \frac{5}{9}(117.0) = 47.2^\circ\text{C}$
• Standard Deviation: $3.1^\circ\text{F} \implies \frac{5}{9}(3.1) = 1.72^\circ\text{C}$
• Variance: $3.1^2 = 9.61 \implies \left(\frac{5}{9}\right)^2(9.61) = 2.96\,(^\circ\text{C})^2$

Measures of Relative Standing

Standardized Scores ($z$-Scores)

A $z$-score indicates how many standard deviations an individual observation $x_i$ falls above or below the mean:

$z_i = \frac{x_i - \bar{x}}{s}$

Because $z_i = -\frac{\bar{x}}{s} + \left(\frac{1}{s}\right)x_i$, a set of $z$-scores is a linear transformation with $a = -\frac{\bar{x}}{s}$ and $b = \frac{1}{s}$. As a consequence:

$\bar{z} = -\frac{\bar{x}}{s} + \left(\frac{1}{s}\right)\bar{x} = 0, \quad s_z = \left|\frac{1}{s}\right|s = 1$

$z$-scores are dimensionless quantities. Under empirical normality, $z$-scores outside $[-3, 3]$ represent extreme outliers.

Comparative Height Example: Adult American male heights follow $\mu \approx 176.3\,\text{cm}, \sigma \approx 7.6\,\text{cm}$.

• Barack Obama ($185\,\text{cm}$): $z = \frac{185 - 176.3}{7.6} \approx +1.1$
• Shaquille O'Neal ($216\,\text{cm}$): $z = \frac{216 - 176.3}{7.6} \approx +5.2$
• Danny DeVito ($147\,\text{cm}$): $z = \frac{147 - 176.3}{7.6} \approx -3.9$

Percentiles, Quartiles, and Boxplots

The $k\text{th}$ percentile is the value such that $k\%$ of the ordered data values fall at or below it.

• Quartiles:
  • First Quartile ($Q_1$): $25\text{th}$ percentile
  • Second Quartile ($Q_2$): $50\text{th}$ percentile (the median)
  • Third Quartile ($Q_3$): $75\text{th}$ percentile
• Interquartile Range (IQR): $\text{IQR} = Q_3 - Q_1$. A robust measure of spread.
• Five-Number Summary: Minimum, $Q_1$, Median, $Q_3$, Maximum.

Boxplot Construction Rules

1. Draw a central box extending from $Q_1$ to $Q_3$, with a line across the box at the median.
2. Compute the step fences: $\text{Lower Fence} = Q_1 - 1.5 \times \text{IQR}$; $\text{Upper Fence} = Q_3 + 1.5 \times \text{IQR}$.
3. Draw the lower whisker to the smallest sample observation that is $\ge Q_1 - 1.5 \times \text{IQR}$.
4. Draw the upper whisker to the largest sample observation that is $\le Q_3 + 1.5 \times \text{IQR}$.
5. Plot any observations outside these fences individually as outliers.

Worked Boxplot Calculation: Sample of $n = 9$ values: $112, 114, 120, 126, 134, 139, 142, 159, 179$.

• Minimum = $112$, $Q_1 = 120$, Median = $134$, $Q_3 = 142$, Maximum = $179$
• $\text{IQR} = 142 - 120 = 22$
• Lower Fence: $120 - 1.5(22) = 87$. Smallest observation $\ge 87$ is $112$ (lower whisker endpoint).
• Upper Fence: $142 + 1.5(22) = 175$. Largest observation $\le 175$ is $159$ (upper whisker endpoint).
• Value $179 > 175$ is identified and plotted as an outlier.

Advanced Displays: Jittered Boxplots and Violin Plots

• Jittered Boxplot: Superimposes raw data points onto the boxplot with minor random horizontal noise to reveal local point densities.
• Violin Plot: Uses kernel density estimation to depict the continuous probability profile of the data. Violin plots can distinguish between uniform spreads and bimodal distributions that share identical five-number summaries.

Probability Foundations and Theoretical Frameworks

Probability provides the mathematical foundation for quantifying uncertainty in statistical inference.

Interpretations of Probability

• Frequentist Interpretation: The probability of an event is the limiting relative frequency with which it occurs in an infinite sequence of identical, independent trials:

$P(A) = \lim_{N \to \infty} \frac{n_A}{N}$

• Bayesian Interpretation: Probability represents a subjective degree of belief regarding an outcome or proposition, updated systematically as new evidence emerges.

Sample Spaces, Events, and Set Operations

• Sample Space ($S$): The set of all possible mutually exclusive and collectively exhaustive outcomes of an experiment. Can be discrete finite, countably infinite, or continuous.
• Sample Point: An individual elementary outcome in $S$.
• Event: Any subset of the sample space $S$.
• Equally Likely Outcomes: If a discrete sample space contains $N$ equally likely sample points, the probability of event $A$ is:

$P(A) = \frac{\text{Number of sample points in } A}{N}$

Fundamental Probability Axioms and Set Relations

1. For any event $A$, $0 \le P(A) \le 1$.
2. For the entire sample space, $P(S) = 1$.
3. Intersection ($A \cap B$): The event that both $A$ and $B$ occur simultaneously.
4. Mutually Exclusive (Disjoint) Events: Two events are mutually exclusive if they share no sample points: $A \cap B = \emptyset$, implying $P(A \cap B) = 0$.
5. Union ($A \cup B$): The event that either $A$ occurs, $B$ occurs, or both occur.

6. General Addition Rule:

$P(A \cup B) = P(A) + P(B) - P(A \cap B)$

7. Complement ($A^c$): The event that $A$ does not occur. Since $A \cup A^c = S$ and $A \cap A^c = \emptyset$:

$P(A^c) = 1 - P(A)$

Diagnostic Medical Comorbidity Example

Consider a population where $10\%$ have diabetes ($D$), $30\%$ have hypertension ($H$), and $7\%$ have both:

• $P(D) = 0.10, \quad P(H) = 0.30, \quad P(D \cap H) = 0.07$
• Diabetes only: $P(D \cap H^c) = P(D) - P(D \cap H) = 0.10 - 0.07 = 0.03$
• Hypertension only: $P(D^c \cap H) = P(H) - P(D \cap H) = 0.30 - 0.07 = 0.23$
• Union: $P(D \cup H) = 0.10 + 0.30 - 0.07 = 0.33$
• Neither condition: $P((D \cup H)^c) = 1 - 0.33 = 0.67$
• Exactly one condition: $P(D \cap H^c) + P(D^c \cap H) = 0.03 + 0.23 = 0.26$

Conditional Probability and Independence

Conditional Probability Formula

The conditional probability of event $A$ given that event $B$ has occurred ($P(B) > 0$) represents the probability of $A$ within the reduced sample space defined by $B$:

$P(A \mid B) = \frac{P(A \cap B)}{P(B)}$

Worked Example: Using the diabetes-hypertension population:

$P(H \mid D) = \frac{P(H \cap D)}{P(D)} = \frac{0.07}{0.10} = 0.70$

Knowing an individual has diabetes increases the probability that they also have hypertension from $0.30$ to $0.70$.

Independence of Events

Events $A$ and $B$ are independent if and only if:

$P(A \cap B) = P(A)P(B)$

When $P(A) > 0$ and $P(B) > 0$, this is mathematically equivalent to:

$P(A \mid B) = P(A) \quad \text{and} \quad P(B \mid A) = P(B)$

If $A$ and $B$ are independent, then their complements are also pairwise independent: $(A, B^c)$, $(A^c, B)$, and $(A^c, B^c)$.

Warning Regarding Mutual Exclusivity and Independence: Mutually exclusive events with positive probabilities cannot be independent. If $A \cap B = \emptyset$, then $P(A \cap B) = 0$, whereas $P(A)P(B) > 0$. If event $A$ occurs, event $B$ cannot occur ($P(B \mid A) = 0$), demonstrating complete dependency.

The Multiplication Rule and Applications

Rearranging the conditional probability identity gives the general multiplication rule:

$P(A \cap B) = P(A)P(B \mid A) = P(B)P(A \mid B)$

For three events $A, B, C$:

$P(A \cap B \cap C) = P(A)P(B \mid A)P(C \mid A \cap B)$

Sampling Without Replacement (Urn Problem)

An urn contains $3\,\text{red}$ and $6\,\text{blue}$ balls ($9\,\text{total}$). Two balls are drawn sequentially without replacement:

$P(R_1 \cap R_2) = P(R_1)P(R_2 \mid R_1) = \frac{3}{9} \times \frac{2}{8} = \frac{6}{72} = \frac{1}{12}$

If three balls are drawn without replacement:

$P(R_1 \cap R_2 \cap R_3) = \frac{3}{9} \times \frac{2}{8} \times \frac{1}{7} = \frac{6}{504} = \frac{1}{84}$

Multi-Stage Demographic Intersection

Calculating the probability that a randomly chosen Canadian is a senior grandparent living in Alberta:

• Probability lives in Alberta ($A$): $P(A) = 0.12$
• Probability senior given lives in Alberta ($E \mid A$): $P(E \mid A) = 0.14$
• Probability grandparent given senior in Alberta ($G \mid A \cap E$): $P(G \mid A \cap E) = 0.78$

$P(A \cap E \cap G) = P(A)P(E \mid A)P(G \mid A \cap E) = 0.12 \times 0.14 \times 0.78 = 0.013104$

Reliability in Engineering Systems

• Parallel Redundancy: A satellite guidance system incorporates three independent parallel components, each with a failure probability of $p = 0.02$. The system functions as long as at least one component operates:

$P(\text{System Operates}) = 1 - P(\text{All } 3 \text{ Fail}) = 1 - (0.02)^3 = 1 - 0.000008 = 0.999992$

• Series Configuration: A string of $20$ older holiday lights is wired in series; if any bulb fails, the circuit opens. If each bulb has an independent failure probability of $p = 0.03$:

$P(\text{String Functions}) = P(\text{All } 20 \text{ Function}) = (1 - 0.03)^{20} = (0.97)^{20} \approx 0.5438$

The Law of Total Probability and Bayes' Theorem

Let $B_1, B_2, \dots, B_k$ form a partition of the sample space $S$ (mutually exclusive and collectively exhaustive events, so $\sum P(B_i) = 1$).

• Law of Total Probability: For any event $A$:

$P(A) = \sum_{i=1}^k P(A \cap B_i) = \sum_{i=1}^k P(B_i)P(A \mid B_i)$

• Bayes' Theorem: Inverts the conditioning to update the prior probability $P(B_j)$ to the posterior probability $P(B_j \mid A)$:

$P(B_j \mid A) = \frac{P(B_j \cap A)}{P(A)} = \frac{P(B_j)P(A \mid B_j)}{\sum_{i=1}^k P(B_i)P(A \mid B_i)}$

Case Study: Rare Disease Diagnostic Screening

A disease has a population prevalence of $0.5\%$ ($P(D) = 0.005$, $P(D^c) = 0.995$). A test provides a true positive rate (sensitivity) of $94\%$ ($P(T \mid D) = 0.94$) and a false positive rate of $2\%$ ($P(T \mid D^c) = 0.02$).

1. Probability an individual tests positive ($P(T)$) via the Law of Total Probability:

$P(T) = P(D)P(T \mid D) + P(D^c)P(T \mid D^c) = (0.005)(0.94) + (0.995)(0.02) = 0.0047 + 0.0199 = 0.0246$

2. Probability an individual who tests positive actually has the disease via Bayes' Theorem:

$P(D \mid T) = \frac{P(D)P(T \mid D)}{P(T)} = \frac{0.0047}{0.0246} \approx 0.191$

Despite testing positive on a test with $94\%$ sensitivity, the probability of having the disease is only $19.1\%$. Because the disease is rare, false positives ($0.0199$) outnumber true positives ($0.0047$) by more than four to one.

Case Study: Factory Defect Source Attribution

Three machines manufacture parts for a factory:

• Machine 1: makes $60\%$ of parts ($P(M_1) = 0.60$); defect rate $7\%$ ($P(D \mid M_1) = 0.07$)
• Machine 2: makes $30\%$ of parts ($P(M_2) = 0.30$); defect rate $15\%$ ($P(D \mid M_2) = 0.15$)
• Machine 3: makes $10\%$ of parts ($P(M_3) = 0.10$); defect rate $30\%$ ($P(D \mid M_3) = 0.30$)

1. Overall probability a randomly selected part is defective:

$P(D) = (0.60)(0.07) + (0.30)(0.15) + (0.10)(0.30) = 0.042 + 0.045 + 0.030 = 0.117$

2. Probability a defective part originated from Machine 1:

$P(M_1 \mid D) = \frac{P(M_1)P(D \mid M_1)}{P(D)} = \frac{0.042}{0.117} \approx 0.359$

Combinatorial Counting Rules

• Fundamental Counting Principle: If a process comprises $k$ successive stages with $n_1, n_2, \dots, n_k$ possible choices, there are $\prod_{i=1}^k n_i$ total pathways.
• Factorial: The number of ways to arrange $n$ distinct items:

$n! = n(n-1)(n-2)\cdots(2)(1), \quad \text{with } 0! = 1$

• Permutations: The number of ordered arrangements of $x$ items chosen from $n$ distinct items:

$P^n_x = \frac{n!}{(n - x)!}$

  Bus Driver Example: Assigning $4$ drivers from a pool of $9$ to $4$ distinct routes:

  $P^9_4 = \frac{9!}{(9 - 4)!} = \frac{362880}{120} = 3024$

• Combinations: The number of unordered selections of $x$ items chosen from $n$ distinct items:

$C^n_x = \binom{n}{x} = \frac{n!}{x!(n - x)!}$

  Anti-Doping Example: Selecting $4$ athletes from $60$:

  $\binom{60}{4} = \frac{60!}{4!56!} = \frac{60 \times 59 \times 58 \times 57}{24} = 487635$

  Lotto 6/49 Example: Drawing $6$ winning numbers from $49$:

  $\binom{49}{6} = \frac{49!}{6!43!} = 13983816 \implies P(\text{Jackpot}) = \frac{1}{13983816}$

The Law of Large Numbers and the Fallacy of "Evening Out"

• The Law of Large Numbers (LLN): As the number of independent, identically distributed trials $n \to \infty$, the empirical sample proportion $\hat{p}_n$ converges in probability to the true theoretical probability $p$:

$P(|\hat{p}_n - p| < \epsilon) \to 1 \quad \text{as } n \to \infty$

• Misconception (Gambler's Fallacy): The law of large numbers does not operate via compensatory mechanics. Nature does not generate tails to balance an early run of heads. Early imbalances are not cancelled; they are simply diluted by the denominator as the number of trials increases.
• Divergence of Absolute Differences: While the relative proportion $\frac{\text{Heads}}{n}$ converges to $0.5$, the absolute difference $|\text{Heads} - \text{Tails}|$ typically drifts outward on the order of $\sqrt{n}$.

Coin Betting Simulation: Two players wager $\$1$ on the outcome of a fair coin. Over $100,000\,\text{tosses}$, Player B leads by $\$608$ (heads count = $49696$, tails count = $50304$). Although the empirical proportion of tails ($0.50304$) is close to $0.5$, Player A faces an absolute loss of $\$608$. The probability that two bettors remain within a fixed dollar boundary ($\$100, \$500, \$1000$) decreases toward zero as the number of trials increases.

Continuous Probability Distributions

Continuous random variables assume any value across a real interval, modeling quantities such as time, weight, concentration, and length.

Properties of Continuous Distributions

Every continuous random variable $X$ is characterized by a continuous probability density function (PDF), denoted $f(x)$.

1. Height is Not Probability: The vertical coordinate $f(x)$ represents density, not probability.
2. Probabilities Correspond to Areas: Probability over an interval $[a, b]$ is given by the integral of the PDF:

$P(a \le X \le b) = \int_a^b f(x)\,dx$

3. Probability of a Single Point is Zero: Because a single mathematical point has width zero, the area above it is zero:

$P(X = c) = \int_c^c f(x)\,dx = 0 \quad \text{for any constant } c$

   Consequently, boundary inclusion does not alter probabilities for continuous distributions:

$P(a \le X \le b) = P(a < X \le b) = P(a \le X < b) = P(a < X < b)$

4. Non-negativity: The density function cannot fall below the horizontal axis: $f(x) \ge 0$ for all $x$.
5. Total Area Equals One:

$\int_{-\infty}^{\infty} f(x)\,dx = 1$

6. Cumulative Distribution Function (CDF): Gives the cumulative area to the left of $x$:

$F(x) = P(X \le x) = \int_{-\infty}^x f(t)\,dt$

   By the Fundamental Theorem of Calculus, the PDF is the first derivative of the CDF:

$f(x) = \frac{d}{dx}F(x)$

7. $k\text{th}$ Percentile: The value $x_k$ such that the area to its left under the PDF equals $\frac{k}{100}$:

$F(x_k) = \int_{-\infty}^{x_k} f(t)\,dt = \frac{k}{100}$

8. Expected Value (Theoretical Mean): Represents the physical center of mass of the density:

$E(X) = \mu = \int_{-\infty}^{\infty} x f(x)\,dx$

9. Variance:

$\text{Var}(X) = \sigma^2 = E((X - \mu)^2) = \int_{-\infty}^{\infty} (x - \mu)^2 f(x)\,dx = E(X^2) - (E(X))^2$

   where $E(X^2) = \int_{-\infty}^{\infty} x^2 f(x)\,dx$.

Worked Integration Example

Consider the parameterized density function:

$f(x) = \begin{cases} cx^2 & \text{for } 1 < x < 2 \\ 0 & \text{elsewhere} \end{cases}$

1. Find constant $c$:

$\int_1^2 c x^2\,dx = 1 \implies c \left[ \frac{x^3}{3} \right]_1^2 = 1 \implies c \left( \frac{8}{3} - \frac{1}{3} \right) = 1 \implies \frac{7}{3}c = 1 \implies c = \frac{3}{7}$

2. Compute $P(X < 1.5)$:

$P(X < 1.5) = \int_1^{1.5} \frac{3}{7}x^2\,dx = \left[ \frac{x^3}{7} \right]_1^{1.5} = \frac{(1.5)^3 - 1^3}{7} = \frac{3.375 - 1}{7} = \frac{2.375}{7} \approx 0.3393$

3. Derive the CDF, $F(x)$:

   • For $x \le 1$: $F(x) = 0$
   • For $1 < x < 2$: $F(x) = \int_1^x \frac{3}{7}t^2\,dt = \left[ \frac{t^3}{7} \right]_1^x = \frac{x^3 - 1}{7}$
   • For $x \ge 2$: $F(x) = 1$

4. Determine the $75\text{th}$ percentile ($a$):

$F(a) = 0.75 \implies \frac{a^3 - 1}{7} = 0.75 \implies a^3 - 1 = 5.25 \implies a^3 = 6.25 \implies a = (6.25)^{1/3} \approx 1.8420$

5. Calculate the expected value ($E(X)$):

$E(X) = \int_1^2 x \left(\frac{3}{7}x^2\right)\,dx = \frac{3}{7} \int_1^2 x^3\,dx = \frac{3}{7} \left[ \frac{x^4}{4} \right]_1^2 = \frac{3}{7} \left( \frac{16 - 1}{4} \right) = \frac{45}{28} \approx 1.6071$

Special Continuous Probability Distributions

The Continuous Uniform Distribution

A continuous uniform random variable $X \sim \text{Unif}(L, U)$ has constant probability density over a bounded interval $[L, U]$.

• Probability Density Function:

$f(x) = \begin{cases} \frac{1}{U - L} & \text{for } L \le x \le U \\ 0 & \text{elsewhere} \end{cases}$

• Cumulative Distribution Function:

$F(x) = \begin{cases} 0 & \text{for } x < L \\ \frac{x - L}{U - L} & \text{for } L \le x \le U \\ 1 & \text{for } x > U \end{cases}$

• Moments:

$\mu = \frac{L + U}{2}, \quad \sigma^2 = \frac{(U - L)^2}{12}$

Worked Example ($X \sim \text{Unif}(100, 150)$): Here $L = 100, U = 150$, so $f(x) = \frac{1}{50} = 0.02$ for $100 \le x \le 150$.

• Mean and Median: $\mu = \text{Median} = \frac{100 + 150}{2} = 125$
• $P(85 \le X \le 137.25) = P(100 \le X \le 137.25) = (137.25 - 100) \times 0.02 = 0.745$
• $P(X > 130) = (150 - 130) \times 0.02 = 0.40$
• $P(113.2 \le X \le 127.4) = (127.4 - 113.2) \times 0.02 = 14.2 \times 0.02 = 0.284$
• $17\text{th}$ Percentile ($a$): $\frac{a - 100}{50} = 0.17 \implies a = 100 + 50(0.17) = 108.5$

The Normal (Gaussian) Distribution

The normal distribution $X \sim N(\mu, \sigma^2)$ is symmetric, unimodal, and bell-shaped.

• Probability Density Function:

$f(x) = \frac{1}{\sqrt{2\pi}\sigma} e^{-\frac{1}{2\sigma^2}(x - \mu)^2} \quad \text{for } -\infty < x < \infty$

• Parameters: Mean $\mu \in (-\infty, \infty)$ controls distribution location; standard deviation $\sigma > 0$ controls spread. The curve exhibits inflection points at $x = \mu - \sigma$ and $x = \mu + \sigma$.
• Standard Normal Distribution ($Z \sim N(0, 1)$): A normal distribution with $\mu = 0$ and $\sigma = 1$. The linear transformation:

$Z = \frac{X - \mu}{\sigma}$

converts any normal random variable $X$ into a standard normal variable $Z$.

Worked Clinical Example (Infant Arm Length): Upper arm length in one-year-old American girls follows $X \sim N(16.1, 1.3^2)$.

1. Find $P(X < 18.0)$.
   • Standardize: $z = \frac{18.0 - 16.1}{1.3} = \frac{1.9}{1.3} \approx 1.462$
   • Using the standard normal CDF: $P(Z < 1.462) \approx 0.9281$
2. Determine the $20\text{th}$ percentile.
   • Find $z$ such that $\Phi(z) = 0.20$, which yields $z \approx -0.8416$.
   • Unstandardize: $x = \mu + z\sigma = 16.1 + (-0.8416)(1.3) = 16.1 - 1.094 = 15.006\,\text{cm}$.

The Chi-Square ($\chi^2$) Distribution

• Mathematical Origin: If $Z_1, Z_2, \dots, Z_\nu$ are independent standard normal random variables ($Z_i \sim N(0, 1)$), then the sum of their squares follows a chi-square distribution with $\nu$ degrees of freedom: