AP Statistics - Section 3C: Experiments

Fundamentals of Statistical Data Collection: Populations, Samples, and Studies

• Core Definitions and Distinctions:

  • Population: The entire group of individuals or objects about which information is desired in a statistical study.
  • Census: An observational or data collection process that gathers data from every single individual in the entire population.
  • Practical Limitations of a Census: Collecting data from an entire population is rarely feasible because it requires excessive time, prohibitive financial costs, and immense logistical resources.
  • Sample: A subset of individuals selected from the population from which data are actually collected.
  • Sample Survey: A statistical study that collects data from a sample to make inferences about the larger population from which the sample was chosen.

• Identifying Populations and Samples in Practice:

  • Quality Control in Manufacturing: A manager selects $10$ computer monitors produced during a specific hour to inspect for construction defects and performance issues.
  • Population: All computer monitors produced in the factory during that specific hour.
  • Sample: The $10$ inspected computer monitors.
  • Pre-Election Polling: A news organization surveys $1000$ registered voters to predict presidential election outcomes.
  • Population: All registered voters in the country.
  • Sample: The $1000$ registered voters who completed the survey.
  • Hospital Emergency Room Throughput: Staff record the wait times (in minutes) for $50$ patients before treatment by a doctor or nurse during a single week.
  • Population: All patients who visited that hospital's emergency room during that week.
  • Sample: The $50$ recorded emergency room patients.
  • Lifeguard Readiness Testing: A safety company evaluates lifeguard shifts over the next $2$ months by selecting $20$ unannounced shifts for skill testing.
  • Population: All lifeguard shifts scheduled at the community pool over the upcoming $2$ months.
  • Sample: The $20$ unannounced shifts selected for testing.

• Three Essential Steps in Planning a Sample Survey:

  • 1. Identify the Population: Clearly define the entire group of individuals to be described.
  • 2. Identify the Variables: Specify exactly what measurements, characteristics, or questions are being investigated.
  • 3. Select the Sampling Method: Formulate a detailed plan for choosing a representative sample from the population.

• Study Architectures: Observational Studies vs. Experiments:

  • Observational Study: A study that observes individuals and measures variables of interest without attempting to influence or alter the responses.
  • Retrospective Observational Study: An observational study that examines existing historical data or medical records for a sample of individuals.
  • Prospective Observational Study: An observational study that identifies individuals in advance and tracks them into the future to collect data over time.
  • Purpose: To describe a situation, compare groups, or establish correlations and associations between variables.
  • Experiment: A study that deliberately imposes specific conditions (treatments) on experimental units (individuals, animals, or items) to observe and measure their responses.
  • Purpose: To determine whether a treatment directly causes a change in the response variable (establishing cause-and-effect).

• Evaluating Study Designs in Applied Contexts:

  • Family Dinners and Academic Performance: A survey of $1000$ randomly selected U.S. teenagers found that those eating dinner with family at least $5$ times per week earned higher grades. This is an observational study because researchers merely recorded existing family habits without imposing or assigning dinner frequencies.
  • Background Music Pace and Eating Speed: Volunteers ate chocolate while listening to either a slow-paced or fast-paced musical composition assigned at random. Eating time was shorter with fast music. This is an experiment because researchers manipulated the explanatory variable (tempo of music) by deliberately assigning it to subjects.
  • Cartoons and Child Executive Function: Sixty $4$-year-old children were randomly divided into three groups: watching a fast-paced cartoon, watching an educational cartoon, or drawing with art supplies for $9$ minutes, followed by four cognitive function tests. This is an experiment because treatments were deliberately imposed.
  • Coastal Visits and General Health: In a multi-country study of $14{,}702$ individuals, those who never visited the coast were $2.13$ times as likely to report bad health compared to those visiting weekly. This is an observational study because participants self-selected their coastal visit frequency.
  • Added Sugars and Periodontal Disease: A survey of $2437$ young adults (ages $18\text{--}25$) revealed a $73\%$ higher risk of periodontal disease among those consuming added sugars at least $5$ times weekly.
  • Generalizability: Results cannot be generalized to all U.S. adults; they only apply to the sampled subpopulation of young adults aged $18\text{ to }25$.
  • Causal Inference: Researchers cannot conclude that added sugar consumption causes periodontal disease because confounding factors (such as oral hygiene habits, dental visits, and overall nutritional balance) were not controlled.
  • Education and Depression Symptoms: In a sample of U.S. adults aged $18$ and older, $21.5\%$ with a high school diploma or less reported depression symptoms compared to $12.4\%$ with a college degree or higher. This observational study generalizes to U.S. adults aged $18$ and older, but cannot establish that higher educational attainment causes lower depression due to confounding variables like income and healthcare access.

Formulation and Components of Investigative Questions

• Functions of an Investigative Question:

  • Guides the data collection process by establishing the variables and target population.
  • Guides the choice of appropriate statistical analysis methods.
  • Dictates the scope of valid conclusions and inferences that can be drawn from the data.

• The Three Necessary Components of an Investigative Question:

  • 1. Variable(s) of Interest: Explicitly specify the characteristics being measured (categorical, quantitative, explanatory, or response).
  • 2. Type of Inference and Parameter(s):
  • Estimating Parameters: Identifying population parameters (such as the true mean $\mu$ or true proportion $p$) to be estimated via confidence intervals.
  • Testing Claims: Stating a comparative claim about parameters to be evaluated via hypothesis testing.
  • 3. Type of Conclusion and Scope: Specifying the population to which results generalize (justified by random selection) and whether a causal statement is permissible (justified by random assignment).

• Deconstructing and Formulating Examples:

  • Factory Candy Bar Weights:
  • Question: What are the plausible values for the mean weight (in grams) of all candy bars produced by a factory today?
  • Variable: Weight of candy bars (quantitative, measured in grams).
  • Type of Inference: Estimating the population mean weight $\mu$.
  • Type of Conclusion: Generalizing an interval of plausible values to the population of all candy bars produced by the factory today.
  • Pneumonia Recovery Time:
  • Question: For all pneumonia patients like the ones in this experiment, do supplemental breathing exercises cause a reduction in the mean recovery time compared to no supplemental breathing exercises?
  • Variables: Completion of supplemental breathing exercises (categorical explanatory variable) and recovery time in days (quantitative response variable).
  • Type of Inference: Testing the comparative claim that the population mean recovery time for patients performing breathing exercises is strictly less than that for patients not performing exercises.
  • Type of Conclusion: Establishing a cause-and-effect relationship valid for patients similar to those participating in the experiment.
  • Driver's License Possession (Los Angeles vs. New York City):
  • Question: Is the proportion of all adults in Los Angeles who have a driver's license greater than the proportion of all adults in New York City who have a driver's license?
  • Variables: Driver's license status (categorical: yes/no) and city of residence (categorical: Los Angeles vs. New York City).
  • Type of Inference: Testing the comparative claim regarding the difference between two population proportions ($p_1 - p_2$).
  • Type of Conclusion: Generalizing to the populations of all adults residing in Los Angeles and New York City without asserting causation.
  • Laptop Battery Performance (Brand A vs. Brand B):
  • Context: A consumer organization randomly selects $10$ Brand A laptops and $10$ Brand B laptops to evaluate runtime.
  • Variables: Laptop brand (categorical explanatory) and battery life hours (quantitative response).
  • Type of Inference: Estimating the difference between the mean battery life of Brand A and Brand B ($\mu_A - \mu_B$).
  • Type of Conclusion: Generalizing to the populations of all Brand A and Brand B laptops.
  • Formulated Investigative Question: "What is the difference in the mean battery life (in hours) between all Brand A laptops and all Brand B laptops?"
  • School Commute Duration (Palatine High School):
  • Context: An investigator wants to know about the transit time of school employees.
  • Variables: Daily commute time (quantitative) or whether commute time is under $20$ minutes (binary categorical).
  • Type of Inference: Estimating the true population proportion ($p$).
  • Type of Conclusion: Generalizing to the entire staff population of Palatine High School.
  • Formulated Investigative Question: "What proportion of all employees at Palatine High School have a daily drive to school of less than $20$ minutes?"

Random Sampling Methodologies and Field Implementations

• Random Sampling Definition and Foundational Principle:

  • Random Sampling: Involves using a chance process to determine which members of a population are selected for a sample.
  • Random sampling eliminates human selection bias and ensures that sample statistics provide unbiased estimates of population parameters.

• Simple Random Sample (SRS):

  • Definition: A simple random sample (SRS) of size $n$ is chosen in such a way that every group of $n$ individuals in the population has an equal chance of being selected as the sample.
  • Implication: An SRS guarantees not only that every individual member has an equal probability of selection, but that every combination of $n$ individuals is equally likely.
  • Sampling Without Replacement: An individual selected from the population can be chosen only once. Duplicates generated by chance processes must be discarded.
  • Sampling With Replacement: An individual selected from the population may be chosen more than once.

• Mechanisms for Executing an SRS:

  • 1. Slips of Paper in a Hat: Write each individual's name or identifier on identical slips of paper, mix them thoroughly in a container, and draw $n$ slips without replacement. Impractical for large populations.
  • 2. Digital Random Number Generators (Technology):
  • Step 1 (Label): Assign every member of the population a unique integer from $1$ to $N$.
  • Step 2 (Randomize): Use a pseudorandom algorithm to generate $n$ unique integers between $1$ and $N$.
  • Step 3 (Select): Choose the individuals corresponding to the generated numbers.
  • TI-84 Execution: Seed the calculator using [number] -> STO -> MATH -> PROB -> 1:rand. Select unique values via MATH -> PROB -> randInt(1, N, 1) (repeating and ignoring duplicates) or directly via randIntNoRep(1, N, n).
  • Desmos Execution: Command random([1...N], n) generates random values (with replacement). Command shuffle([1...N])[1...n] permutes the list to yield unique, non-repeating integers (without replacement).
  • 3. Random Digit Table (Table D):
  • Step 1 (Label): Assign each individual a label of identical digit length using as few digits as possible (e.g., $01\text{ to }19$ for $19$ units, $001\text{ to }500$ for $500$ units).
  • Step 2 (Randomize): Read groups of digits of length $k$ from left to right across a line in Table D. Skip numbers outside the labeling range and skip duplicate labels.
  • Step 3 (Select): Continue until $n$ distinct individuals are chosen.
  • School Audit Example: Auditing $3$ schools from a roster of $19$ (labeled $01\text{ to }19$) using line 62081 64816 87374 09517 84534 06489 87201 97245:
    • Digit pairs: 62 (skip), 08 (Holaway Elementary), 16 (Prince Elementary), 48 (skip), 16 (skip duplicate), 87 (skip), 37 (skip), 40 (skip), 95 (skip), 17 (Rio Vista Elementary).
    • Selected sample: Holaway Elementary, Prince Elementary, and Rio Vista Elementary.
  • Athletic Drug Screen Example: Selecting $2$ players from a roster of $12$ (Chang, Delacour, Diggory, Finnigan, Granger, Krum, Longbottom, Lovegood, Potter, Thomas, Weasley, Wood labeled $01\text{ to }12$) using line 64061 76328 17202 41052 43894 53950:
    • Digit pairs: 64 (skip), 06 (Krum), 17 (skip), 63 (skip), 28 (skip), 17 (skip), 20 (skip), 24 (skip), 10 (Thomas).
    • Selected sample: Krum and Thomas.

• Stratified Random Sample:

  • Definition: A sampling method where the population is divided into distinct, non-overlapping groups called strata that share common characteristics expected to influence the variable of interest. An independent SRS is selected from each stratum, and these subsamples are pooled into one overall sample.
  • Core Principle: Form strata that are "homogeneous within, but heterogeneous between."
  • Advantage: Substantially reduces sampling variability (the natural fluctuation of sample estimates across different samples of identical size). Grouping similar units prevents extremes and yields more precise parameter estimates than an SRS of the same total size.
  • Sunflower Field Case Study: Estimating the mean count of healthy plants across a $10 \times 10$ grid ($100$ plots) with irrigation ditches running along the top and bottom borders. True population mean is $\mu = 102.46$.

• Analysis of Sunflower Results: Stratifying by row yields vastly less sampling variability than an SRS or stratifying by column. Because plots in any single row are equidistant from the irrigation ditch, moisture availability is homogeneous within rows. Stratifying by column bundles plots with high water exposure together with plots suffering from water deficits, producing high within-stratum variability and high sampling error.

• Auditorium Stratification: For an assembly of $800$ students seated by grade ($9\text{th: } 601\text{--}800$, $10\text{th: } 401\text{--}600$, $11\text{th: } 201\text{--}400$, $12\text{th: } 1\text{--}200$), stratifying by grade level (drawing an SRS of $20$ seats from each grade block of $200$) ensures all cohorts are represented proportionally, increasing estimation precision.

  • Cluster Random Sample:

• Definition: A sampling method where the population is divided into geographic or operational groupings located near each other, termed clusters. A simple random sample of entire clusters is chosen, and every individual within each selected cluster is included in the sample.

• Core Principle: Ideal clusters are "heterogeneous within, but homogeneous between"—each cluster acts as a miniature replica of the overall population.

• High School Homeroom Application: An administration seeking opinions from $100$ students randomly selects $4$ whole homerooms (from $80$ rooms of $25$ students sorted alphabetically) and censuses all $25$ students in each selected room.

• Advantage: Practicality, time savings, and reduced administrative and travel costs.

• Drawback: Statistically less precise than stratified sampling; if individuals within clusters share unobserved similarities, cluster sampling increases sampling error.

• Auditorium Cluster Application: Defining each vertical column of seats running from stage to back as a cluster allows rapid survey distribution down complete aisles rather than tracking $80$ scattered seat numbers across the hall.

  • Systematic Random Sample:

• Definition: A sample selected from an ordered arrangement of the population by randomly selecting one of the first $k$ individuals and choosing every $k\text{th}$ individual thereafter, where $k = N/n$.

• Voter Exit Polling: Selecting an integer from $1\text{ to }20$ at random (e.g., $6$). Interviewers poll the $6\text{th}$, $26\text{th}$, $46\text{th}$, and subsequent voters leaving the polling location.

• Auditorium Implementation: To survey $80$ of $800$ students entering an assembly, set $k = 800 / 80 = 10$. Generate a random integer from $1\text{ to }10$. If $3$ is chosen, survey the $3\text{rd}$ student and every $10\text{th}$ student thereafter ($13\text{th}$, $23\text{rd}$, etc.).

• Critical Hazard: If the population exhibits an underlying periodic or cyclical pattern that aligns with the sampling interval $k$ (e.g., every $10\text{th}$ student entering is an unpunctual student arriving in a group), the sample will be systematically non-representative.

  • Multistage Sampling:

• Combines two or more distinct random sampling methods hierarchically for large-scale operations.

• U.S. Current Population Survey (CPS): Surveys approximately $60{,}000$ households monthly. Stage 1: Selects a stratified sample of neighborhoods across $756$ of the $2007$ U.S. geographic areas. Stage 2: Partitions chosen neighborhoods into clusters of four nearby households. Stage 3: Randomly selects household clusters to interview.

Sources of Bias and Errors in Sample Surveys

• The Nature of Statistical Bias:

  • Bias: The design of a statistical study shows bias if it systematically favors certain outcomes, making it very likely to consistently underestimate or overestimate the true population parameter.
  • Bias is an inherent defect in data collection design; increasing the sample size does not fix bias.
  • Framework for Explaining Bias: (1) Identify how individuals in the sample systematically differ from the non-sampled population; (2) State whether this difference leads to an overestimate or an underestimate of the parameter of interest.

• Flawed Sampling Designs:

  • Convenience Sample: A sample composed of individuals who are easiest to contact or reach.
  • Library Study: Asking the first $30$ students in a high school library how much time they spent on homework systematically overestimates homework duration because library visitors typically study more than the student body average.
  • Tomato Garden Harvest: Estimating the total yield of $20$ tomato plants by counting $53$ tomatoes on the $5$ plants in the front row closest to the house ($53 \times 4 = 212$). The southern front row receives superior sunlight and maintenance, creating an overestimate of total garden yield.
  • Classroom Calculator Ownership: Surveying an AP Statistics class to estimate schoolwide graphing calculator ownership results in an overestimate because calculator access is a prerequisite for that course.
  • Voluntary Response Sample (Self-Selected Sample): A sample consisting of individuals who choose to be in the sample by responding to an open, general invitation.
  • Vulnerability: Attracts individuals with extreme viewpoints, especially negative feelings, who are motivated to expend effort to respond.
  • Parenting Poll: An advice columnist asked readers: "If you had to do it over again, would you have children?" Approximately $70\%$ of nearly $10{,}000$ self-selected respondents answered "No." A scientifically randomized Newsday survey revealed that only $9\%$ of parents held that view.
  • Boaty McBoatface Campaign: The British Natural Environment Research Council invited online votes to name a $\$300$ million research vessel. The name "Boaty McBoatface" garnered over $124{,}000$ votes (triple the second-place entry), reflecting internet users' satirical preferences rather than the broader British public's stance.

• Nonsampling and Non-Selection Errors:

  • Undercoverage: Occurs when some members of the population are less likely to be chosen or cannot be chosen in a sample because they are omitted from the sampling frame (the working list of individuals from which a sample is drawn).
  • Telephone Polling: Omits individuals who are unhoused, institutionalized, or lack active telephone lines.
  • Dental Survey: Polling patients in a dental clinic to estimate national flossing habits undercovers adults lacking access to dental care, leading to an overestimate of dental hygiene compliance.
  • Nonresponse: Occurs when an individual chosen for the sample cannot be contacted or refuses to participate.
  • Mechanisms: Screening unfamiliar numbers via caller ID, hanging up, or failing to return mail-in questionnaires.
  • Work-Hour Estimation: A mail survey estimating weekly work hours suffers nonresponse bias because overworked individuals working extreme hours lack the time to fill out forms, leading to an underestimate of average work hours.
  • Response Bias: A systematic pattern of inaccurate, dishonest, or erroneous responses to survey items.
  • Causative Factors: Perceived lack of anonymity, fear of social disapproval, recall failure, or characteristics of the interviewer (clothing, status, gender).
  • Handwashing Hygiene Evidence: In a telephone survey of $1006$ adults, $96\%$ stated they always wash their hands after using public restrooms. An observational study tracking $6028$ public restroom users revealed that only $85\%$ actually washed their hands.
  • Wording of Questions: The phrasing, tone, or sequencing of questions can direct or confuse respondents.
  • Landfill Diaper Inquiry: A diaper manufacturer sponsored a poll asking: "It is estimated that disposable diapers account for less than $2\%$ of the trash in today’s landfills. In contrast, beverage containers, third-class mail, and yard wastes are estimated to account for about $21\%$ of the trash in landfills. Given this, in your opinion, would it be fair to ban disposable diapers?" Only $16\%$ favored a ban. Pre-loading the prompt with comparative statistics minimized perceived waste impact, producing an underestimate of public support for a ban.

Experimental Architecture and Confounding Variables

• Variable Types in Experiments:

  • Response Variable: Measures the final outcome or result of a study.
  • Explanatory Variable: May help explain or predict changes observed in the response variable.
  • Factor: An explanatory variable that is deliberately manipulated by the experimenter.
  • Levels: The specific discrete values or intensities assigned to a factor.
  • Treatment: A specific experimental condition applied to units. If an experiment has multiple factors, a treatment is a unique combination of specific levels from each factor.
  • Experimental Unit: The smallest entity or object to which a treatment is randomly assigned. Human units are designated as subjects.

• The Concept and Mechanics of Confounding:

  • Confounding: Occurs when two variables are associated in such a way that their separate effects on a response variable cannot be distinguished from each other.
  • Confounding in Vitamin D Inquiries: Observational studies link high circulating vitamin D to reduced risk of fractures, heart disease, cognitive decline, and diabetes. However, healthy dietary habits (consuming vitamin D-rich foods and balanced calories) and outdoor cardiovascular exercise (elevating vitamin D via solar ultraviolet exposure and reducing metabolic disease) serve as confounders.
  • AP Exam Response Requirement: To properly demonstrate confounding, one must explicitly explain: (1) how the variable is associated with the explanatory variable, AND (2) how the variable is associated with the response variable.
  • Maternal Smoking and ADHD Study: In a study of over $4700$ children, smoking during pregnancy was associated with double the rate of ADHD. A confounder is maternal diet quality: mothers who smoke during pregnancy are more likely to have poor nutritional intake, and poor maternal nutrition independently impairs fetal neurological development, elevating ADHD risk. Thus, an apparent causal link between smoking and ADHD cannot be established without an experiment.
  • Experimental Resolution of Vitamin D Diabetes Hypothesis: Researchers assigned $500$ prediabetic subjects randomly to receive either high-dose weekly vitamin D ($20{,}000\text{ IU}$) or an identical placebo ($0\text{ IU}$). After $5$ years, approximately $40\%$ of participants in both groups developed diabetes, proving the observational association was spurious and driven by confounding.

• Multi-Factor Factorial Designs:

  • The Five-Second Food Rule Experiment (Rutgers University): Food was dropped onto inoculated surfaces to evaluate microbial transfer.
  • Experimental Units: The individual pieces of food dropped.
  • Factors and Levels: Factor 1: Surface material ($4$ levels: stainless steel, ceramic tile, wood, carpet). Factor 2: Contact duration ($4$ levels: $< 1\text{ s}$, $5\text{ s}$, $30\text{ s}$, $300\text{ s}$). Factor 3: Bacterial medium ($2$ levels: tryptic soy broth, peptone buffer).
  • Treatments: Total treatments = $4 \times 4 \times 2 = 32$ unique conditions.
  • Findings: Bacterial transfer occurs almost instantaneously; carpet showed the lowest transfer rates, proving surface and food moisture characteristics equal or exceed time in driving bacterial migration.
  • Hatchery Lobster Nutritional Trial: Culturing lobsters to maximum carapace length before wild release.
  • Experimental Units: Individual juvenile lobsters.
  • Factors and Levels: Factor 1: Dietary protein concentration ($2$ levels: $400\text{ g/kg}$ and $500\text{ g/kg}$). Factor 2: Lipid-to-carbohydrate (L:CHO) ratio ($3$ levels: low, medium, high).
  • Treatments: Total experimental treatments = $2 \times 3 = 6$ formulated diets, plus $1$ control diet of Antarctic krill, equaling $7$ treatments total.

• Control Groups, Placebos, and Blinding:

  • Control Group: A baseline group used for comparison against active treatments. May receive an inert treatment (placebo), standard baseline care, or no treatment.
  • Kenyan School Malaria Prophylaxis: Children across $101$ schools were evaluated, with half receiving regular diagnostic screenings and treatments while the control schools received no screening. The control group provided a baseline to verify whether reductions were due to medical intervention rather than ecological shifts (such as a regional drought suppressing mosquito breeding).
  • Rwanda Literacy Boost Project: Evaluated educational sectors by assigning them to Teacher Training (TT), Literacy Boost (LB, combining TT with community reading programs), or a Control group receiving neither intervention.
  • Placebo: A simulated treatment lacking active pharmaceutical ingredients that is otherwise visually, texturally, and administratively indistinguishable from active therapies.
  • Placebo Effect: The measurable physiological or psychological tendency of human subjects to respond favorably to any treatment, including inert placebos, driven by belief and anticipation.
  • Blinding Protocols:
  • Single-Blind: Either the experimental subjects OR the researchers who interact with them and record outcomes are unaware of treatment assignments.
  • Double-Blind: Neither the experimental subjects NOR the researchers who interact with them and measure the response variable know which treatment was administered.
  • Chronic Pain Magnet Therapy: Fifty chronic pain subjects received active or inert magnetic bars for $45$ minutes from randomly assigned, identical sealed envelopes. Double-blinding prevented patient expectations from influencing subjective pain ratings ($0\text{--}10$ scale) and prevented examining physicians from interpreting recovery differentially.
  • AzaSite Clinical Trial: Evaluated antibiotic eyedrops against placebo drops over $7$ days for bacterial conjunctivitis. Double-blinding ensured that diagnostic criteria of clinical cure were applied impartially.

The Four Principles of Experimental Design

• 1. Comparison:

  • Use an experimental design that contrasts two or more treatments.
  • Prevents confounding by establishing whether the active treatment performs differently from a control baseline or an established standard therapy.

• 2. Random Assignment:

  • Random Assignment: Experimental units are assigned to treatment groups using an objective chance process.
  • Purpose: Spreads individual differences (such as baseline health, genetics, metabolic rate, and diet) evenly across all treatment groups before treatments are administered.
  • Creates roughly equivalent groups across all extraneous variables—including those the experimenter failed to identify.
  • Implementation Techniques:
  • Slips in a Hat: Write treatment names on equal numbers of slips, shuffle, and draw one per unit.
  • Random Number Generator: Assign units labels from $1\text{ to }N$. Generate $n$ non-repeating integers to assign units to Group 1; assign remaining units to Group 2.
  • The Flaw of Coin Flipping in Fixed-Size Groups: Never flip coins to assign subjects to equal-sized groups by stopping when one group reaches capacity. Doing so alters the probability structure: the final subjects in line face a conditional probability higher than $50\%$ of landing in the under-filled group, violating completely random assignment.
  • Pfizer COVID-19 Vaccine Trial: Enrolled $43{,}548$ subjects. Numbered $1\text{ to }43{,}548$. An RNG selected $21{,}774$ unique integers for the vaccine group ($2$ doses spaced $21$ days apart); the remaining $21{,}774$ received saline placebo shots. Random assignment balanced COVID-19 viral exposure risks and comorbidities, enabling a valid $95\%$ efficacy conclusion.

• 3. Replication:

  • Replication: Imposing each treatment on a sufficiently large number of experimental units so that genuine differences between treatment effects can be distinguished from random assignment variation.
  • In statistics, replication does not mean repeating the entire study; it means using enough subjects per treatment arm to dilute individual anomalies.

• 4. Control:

  • Control: Keeping all extraneous variables identical for all experimental groups, particularly variables that could directly affect the response.
  • Benefits: Prevents confounding and minimizes residual variation in the response, making subtle treatment effects easier to detect.
  • Classroom Multitasking Study: Forty university students attended an identical lecture in the same hall, took notes on identical laptops, and took the same examination. One group performed simulated web browsing while the other took uninterrupted notes. Holding lecture content, environment, and testing instruments constant isolated the true cognitive impact of multitasking.
  • TRIKAFTA Cystic Fibrosis Trial: Tested $403$ patients receiving either $375\text{ mg}$ daily TRIKAFTA or placebo. All subjects were required to ingest their pills twice daily alongside fat-containing foods while maintaining baseline therapies. Lung function improved by $14.3$ percentage points over placebo after $24$ weeks.

Advanced Experimental Architectures: Randomized Block and Matched Pairs Designs

• Randomized Block Designs:
  • Block: A group of experimental units known prior to the experiment to be similar in a specific characteristic that is systematically expected to influence the response to treatments.
  • Randomized Block Design: An experimental design in which the random assignment of experimental units to treatments is executed completely independently within each distinct block.
  • Methodological Parallel: Blocking in experiments serves the identical analytical objective as stratifying in sample surveys ("similar within, different between").
  • Analytical Advantage: Partitions out known sources of variability. By removing the variation associated with the blocking variable, the standard error decreases, making it easier to demonstrate statistically significant treatment effects.
  • Field Experiment with Blackberries: Agricultural students tested a new cultivar against a standard cultivar across $10$ planting plots: $4$ plots in a highly fertile field and $6$ plots in an infertile field.

* *Execution:* Within the high-fertility block ($n=4$), randomly assign $2$ plots to the new cultivar and $2$ to the control cultivar. Within the low-fertility block ($n=6$), randomly assign $3$ plots to each cultivar. Soil fertility variability is prevented from confounding cultivar comparisons.

• Microwave Popcorn Efficiency: Testing microwave "popcorn buttons" versus package instructions on the number of unpopped kernels across $40$ bags ($10$ bags each of butter, cheese, natural, and kettle corn varieties).

  • Block Choice: Block by kernel variety, as oil and sugar content vary by flavor and alter heat absorption.
  • Procedure: Within each flavor block of $10$, draw $5$ bags for the popcorn button treatment and assign the remaining $5$ to the printed package time. Pop in random sequence and count remaining unpopped kernels.

• School Commute Optimization: Two students tested three travel routes over a $15$-day span ($3$ weeks). Block by time of day: Morning rush hour (heavy congestion) versus Afternoon departure (free-flowing traffic). Randomly assign the three routes within the morning block across days, and separately within the afternoon block across days.

• Experimental Golden Rule: "Control what you can, block on what you cannot control, and randomize the rest to create comparable groups."

  • Matched Pairs Design:

• Matched Pairs Design: A specialized form of randomized block design that restricts block size to exactly $2$.

• Variant 1: Two Closely Matched Units: Two distinct, highly similar experimental units (e.g., identical twins, or individuals paired by baseline exam scores) are paired together. Within each pair, treatments are assigned at random (e.g., via coin flip).

  • Classical Music Math Test: Thirty students are paired into $15$ dyads based on classroom math grades (highest two paired, next two paired, etc.). One student per pair is randomly assigned to test with classical music; the other tests in silence.

• Variant 2: Repeated Measures on a Single Unit: Each experimental unit receives both treatments in a randomized temporal order, serving as its own control.

  • Order Randomization Necessity: To prevent order effects, learning effects, or confounding environmental trends (e.g., fatigue or classroom temperature changes across days), the sequence of treatments must be randomized.

• Automotive Fuel Additive Evaluation: Twenty cars of varying makes and ages are tested for fuel efficiency ($\text{mpg}$). Each car serves as its own block and is tested under both conditions: standard fuel and fuel with additive. A coin flip determines whether a car runs standard fuel first or additive fuel first.

• Swim Team Drag Suit Trial: Twenty-four swimmers swim a $50$-meter freestyle trial twice: once in a traditional racing suit and once wearing a specialized resistance drag suit. Order is assigned randomly per swimmer with adequate recovery intervals between trials.

Statistical Significance and Inference for Experiments

• Statistical Significance Defined:

  • Statistically Significant: An observed difference in response between groups in an experiment that is so large that it is unlikely to have occurred solely due to chance variation in the random assignment.
  • Decision Rule: Differences that would occur in less than $5\%$ of simulated random assignments ($p < 0.05$) by chance alone are deemed statistically significant.

• Simulation Analysis for Experimental Significance:

  • Distracted Driving Investigation (University of Utah - Strayer et al.): Evaluated whether talking on a cell phone was more distracting than conversing with a passenger in a driving simulator. Forty-eight undergraduates were randomly split into two groups of $24$. The response was whether the driver successfully stopped at a designated rest area.

• Proportion Difference: $\hat{p}_{\text{Passenger}} - \hat{p}_{\text{Cell}} = \frac{21}{24} - \frac{12}{24} = 0.875 - 0.500 = +0.375$.
• Simulation Mechanism: Assume null hypothesis that conversation type does not affect stopping. Pool all $33$ stoppers and $15$ non-stoppers, shuffle, and redistribute them at random into two hypothetical groups of $24$ for $100$ simulated trials.
• Interpretation of Dotplots: A dot at $0.125$ indicates a trial where chance assignment alone produced an excess passenger stopping proportion of $0.125$. Across all $100$ trials, an observed difference of $0.375$ never occurred ($0\%$). Because $0\% < 5\%$, the difference is statistically significant, providing strong evidence that cell phone use is more distracting than passenger conversation.
• Botox for Chronic Low Back Pain Clinical Study: Fifty Iranian patients aged $18\text{--}55$ were randomly assigned to receive either Botox injections or saline placebo injections across $5$ lumbar locations.

• Proportions and Difference: $\hat{p}_{\text{Botox}} = \frac{17}{25} = 0.68$ ($68\%$); $\hat{p}_{\text{Saline}} = \frac{3}{25} = 0.12$ ($12\%$). Difference = $0.68 - 0.12 = +0.56$ ($56$ percentage points).

• Significance Determination: In $100$ simulated randomizations of $20$ improvers and $30$ non-improvers into two groups of $25$, an outcome of $+0.56$ never occurred by chance. The result is statistically significant, demonstrating that Botox injections cause functional improvement in patients suffering from chronic low back pain.

  • The Scope of Inference Framework:

• Random Selection of Individuals justifies making inferences about the broader population.

• Random Assignment to Groups justifies making inferences about cause and effect.

• Inference Quadrants:
  • Random Selection YES + Random Assignment YES: Inferences about population permitted; Inferences about cause and effect permitted. (Ideal, but rare in practice).
  • Random Selection NO + Random Assignment YES: Inferences about population NOT permitted; Inferences about cause and effect permitted. Standard for human clinical trials; conclusions apply strictly to individuals similar to study participants.
  • Random Selection YES + Random Assignment NO: Inferences about population permitted; Inferences about cause and effect NOT permitted. Standard for well-designed random sample surveys and observational studies.
  • Random Selection NO + Random Assignment NO: Inferences about population NOT permitted; Inferences about cause and effect NOT permitted. Characteristic of anecdotal accounts and voluntary web polls.
• Applied Scope Cases:
  • CareerStart Educational Initiative (University of North Carolina): Fourteen middle schools in Forsyth County were recruited; $7$ were randomly assigned to implement CareerStart alongside standard curriculum, and $7$ followed standard curriculum alone. CareerStart students achieved statistically significant gains in attendance, test scores, and graduation rates.
  • Causation: Yes, justified because schools were randomly assigned to treatments.
  • Generalization: Restricted to middle schools in Forsyth County or schools with highly similar socioeconomic characteristics, because participating schools were recruited volunteers, not an SRS of all U.S. middle schools.
  • Botox Trial Generalization: Causation is established via random assignment, but population generalization is restricted strictly to Iranian adults aged $18\text{ to }55$ presenting with chronic low back pain similar to the volunteers.

Ethical Standards in Statistical Experimentation

• Ethical Boundaries in Experimentation:

  • Experiments cannot be performed when deliberately exposing human subjects to known hazards (e.g., assigning subjects to smoke cigarettes, consume toxins, or text while driving in actual traffic) violates basic safety.
  • In such scenarios, researchers must rely on carefully adjusted observational studies or sample surveys.

• Three Basic Data Ethics Standards for Human Research:

  • 1. Institutional Review Board (IRB): All planned statistical studies involving humans must be reviewed and approved in advance by an independent institutional review board charged with protecting the rights, safety, and well-being of participants.
  • 2. Informed Consent: All human subjects must give voluntary, fully informed consent in writing prior to data collection, confirming they understand the study nature and potential risks.
  • 3. Confidentiality: All individual personal data must be kept strictly confidential. Only aggregated statistical summaries for groups of subjects may be made public.