AP Stat Unit 2

Introduction to Sampling and Simple Random Samples

Foundational Sampling Concepts:
- Population: The complete collection of all individuals, objects, or items that are of interest in a study.
- Sample: A subset of individuals selected from the population from which data are collected.
- Representative Sample: A sample that accurately reflects the relevant characteristics of the entire population, allowing valid generalizations to be made.
Flawed Sampling Designs (Sources of Bias):
- Convenience Sample:
  - Selection of individuals based on ease of access.
  - Leads to bias because individuals who are easy to reach typically share traits that make them unrepresentative of the entire population.
  - Example: A manager seeking to assess community opinion about a business surveys the next $10$ purchasing customers. This excludes non-purchasers, non-customers, and community members shopping at other times or locations.
- Voluntary Response Sample:
  - A sample composed of individuals who self-select into the study by responding to an open invitation or appeal.
  - Leads to bias because volunteers typically hold stronger, more polarized opinions (often extreme or incentivized) than the general population.
  - Example: A sign displays: "Rate us 5-stars on Google and get a $10$ gift card!" The manager finds that $92\%$ of respondents gave a $5$ -star rating. This sample statistic of $92\%$ is likely greater than the true proportion of all community members who believe the business deserves $5$ stars, as the financial incentive and self-selection attract favorable ratings.
Simple Random Sample (SRS):
- Definition: A sample of size $n$ chosen in such a way that every individual in the population has an equal chance of being selected, and every group of size $n$ individuals has an equal chance of being selected.
- Eliminates human selection bias and forms the baseline standard for random sampling designs.
Three-Step Procedure for Executing an SRS:
- Step 1: Label Individuals: Assign a unique integer from $1$ to $N$ to each individual in the population (or write each name on an identical slip of paper).
- Step 2: Randomize: Use a random number generator (such as randInt(1, N)) or place the paper slips into a container/hat and shuffle them thoroughly.
- Step 3: Select: Draw numbers without replacement (ignoring duplicate values) until $n$ unique individuals are chosen, or draw $n$ slips of paper without looking. Identify the individuals corresponding to the drawn numbers.

Sampling Methods and Simple Random Sample Framework

Stratified Random Samples

Concept and Mechanics:
- Stratified Random Sample: A sampling method in which the population is divided into non-overlapping subgroups called strata, followed by an independent SRS selected from within each stratum.
- Strata Composition: Each stratum consists of individuals who share similar traits, attributes, or behaviors related to the variable of interest (homogeneous grouping within strata).
- Operational Motto: "Sample some from all groups."
Concert Enjoyment Study (Eras Tour VIP Section):
- Scenario Parameters:
  - Taylor Swift surveys fans on concert enjoyment using a numerical rating scale from $1$ to $100$ , where $100$ represents maximum enjoyment.
  - The VIP seating section contains $50$ seats organized in a grid of $5$ horizontal rows and $10$ vertical columns ( $5 \times 10 = 50$ seats).
  - The performance stage runs along the northern edge of the venue (directly facing Row 1).
  - Target sample size: $n = 10$ fans.
  - True population parameter: Mean enjoyment $\mu = 78.38$ .
- VIP Seat Enjoyment Population Grid:
  - Row 1 (North, adjacent to stage): $92$ , $89$ , $90$ , $88$ , $95$ , $100$ , $98$ , $93$ , $95$ , $84$
  - Row 2: $82$ , $86$ , $90$ , $88$ , $86$ , $91$ , $90$ , $89$ , $85$ , $83$
  - Row 3: $80$ , $74$ , $80$ , $67$ , $81$ , $82$ , $76$ , $77$ , $74$ , $65$
  - Row 4: $72$ , $68$ , $74$ , $73$ , $70$ , $69$ , $72$ , $70$ , $68$ , $67$
  - Row 5 (South, farthest from stage): $69$ , $67$ , $68$ , $68$ , $64$ , $66$ , $63$ , $63$ , $70$ , $68$

Taylor Swift VIP Section Seat Enjoyment Ratings

Evaluation of Sampling Methods Across Repeated Sampling:
- Method #1: Simple Random Sample (SRS) of $10$ Fans:
  - Assign seat numbers $1$ to $50$ . Run randInt(1, 50, 10) without repeats. Survey the selected fans.
  - Sampling distribution characteristics: Unimodal, roughly symmetric. Dotplot mean $\bar{x} = 78.17$ , demonstrating low bias. However, it exhibits high variability (sample means range broadly from $70$ to $94$ ) because pure chance can overselect either high-enjoyment front rows or low-enjoyment back rows.
- Method #2: Stratify by Row:
  - Divide the population into $5$ strata by horizontal row. Select an SRS of $2$ fans from each row ( $5 \text{ rows} \times 2 = 10 \text{ fans}$ ).
  - Sampling distribution characteristics: Slightly skewed left, slightly symmetric. Dotplot mean $\bar{x} = 78.24$ . Produces low bias and low variability.
  - Optimal Performance Explanation: Distance from the stage strongly dictates enjoyment. Within any single row, ratings are similar (homogeneous within); across rows, ratings drop significantly (heterogeneous between). Stratifying guarantees representation across every distance tier, compressing sampling variability.
- Method #3: Stratify by Column:
  - Divide the population into $10$ strata by vertical column. Select an SRS of $1$ fan from each column ( $10 \text{ columns} \times 1 = 10 \text{ fans}$ ).
  - Sampling distribution characteristics: Roughly symmetric, bimodal. Dotplot mean $\bar{x} = 78.96$ . Shows low bias but substantially higher variability than row stratification because columns traverse all distances from the stage (heterogeneous within columns).

Dotplot Comparisons of Sampling Distributions for SRS and Stratified Designs

Criteria for Producing Good Statistical Estimates:
- Low Bias: The sampling distribution of the estimator is centered at the true population parameter.
- Low Variability: The individual sample statistics cluster tightly around the center with minimal spread.
Implementation: Digital Hall Pass Survey Case:
- Scenario: A high school principal explores a phone-based digital hall pass, taking a sample of $100$ students.
- With versus Without Replacement: The sampling must be conducted without replacement (no repeats). Surveying the same student more than once duplicates opinions, reduces sample diversity, and introduces artificial dependency.
- Stratified Design by Grade Level:
  - Partition the school into $4$ strata: Freshmen, Sophomores, Juniors, Seniors.
  - Assign unique numbers to students within each stratum.
  - Use a random number generator to execute an independent SRS of $25$ students within each grade level ( $25 \times 4 = 100$ students total).
- Advantage over an SRS: An SRS risks generating disproportionate grade samples by chance. Because student viewpoints on hall passes and autonomy vary markedly between seniors and freshmen, guaranteeing proportional grade representation minimizes estimate variability.

Cluster Sampling and Systematic Sampling

Cluster Random Sample:
- Definition: A sampling method in which the population is divided into geographical or location-based groups called clusters. A simple random sample of clusters is chosen, and all individuals within the selected clusters are surveyed.
- Cluster Composition: Each cluster should be as diverse as possible, serving as a microcosm of the overall population (heterogeneous grouping within clusters).
- Operational Motto: "Sample all from some groups."
- Primary Advantage: Logistical convenience, lower costs, and ease of fieldwork compared to tracking individuals scattered across an entire population.
Systematic Random Sample:
- Definition: A sampling method where individuals are selected from an ordered list using a fixed numerical interval after a random starting point.
- Sampling Interval Calculation: $k = \frac{N}{n}$ where $N$ is the population size and $n$ is the desired sample size.
- Procedure:
  - Order the population from $1$ to $N$ .
  - Select a random starting integer between $1$ and $k$ .
  - Sample that starting individual, then select every $k\text{-th}$ individual across the list until $n$ individuals are chosen.
Concert VIP Grid Comparison (Cluster vs. Systematic):
- Row Clusters (Method #1): Number rows $1$ to $5$ . Use a random number generator to select $1$ row and sample all $10$ fans in that row. Fails to produce good estimates because rows are homogeneous internally, producing high variability (e.g., Row 1 yields a sample mean of $91.4$ , while Row 5 yields $66.6$ ).
- Column Clusters (Method #2): Number columns $1$ to $10$ . Select $2$ columns at random and sample all $5$ fans in each ( $2 \times 5 = 10$ fans). Produces much better estimates because each column contains fans at varying distances from the stage, mirroring the overall venue diversity.
- Systematic Sampling (Method #3): Label seats $1$ to $50$ . Randomly pick a starting seat from $1$ to $5$ , then sample every $8\text{-th}$ seat thereafter until $10$ seats are recorded.

Comparison of Stratified, Cluster, and Systematic Sampling

Ocean-Front Hotel Case Study ( $500$ Rooms, $n = 50$ Rooms):
- Structure: $10$ floors, $50$ rooms per floor ( $25$ ocean view, $25$ street view per floor).
- Stratified Random Sample: Group rooms into $2$ strata by view (Ocean View vs. Street View). Use an SRS to select $25$ rooms from the ocean-view stratum and $25$ rooms from the street-view stratum. Ensures equal representation across view types, eliminating view-based variability.
- Cluster Random Sample: Treat each of the $10$ floors as a cluster, as each floor contains an even mix of $25$ ocean and $25$ street rooms. Randomly select $1$ floor and survey all $50$ rooms on that floor. Highly practical and minimizes surveyor travel.
- Systematic Random Sample: Number all rooms $1$ to $500$ . Calculate interval $k = \frac{500}{50} = 10$ . Generate a random integer between $1$ and $10$ as a starting room; survey that room and every $10\text{-th}$ room thereafter (e.g., rooms $4, 14, 24, \dots, 494$ ). Evenly disperses the sample across the building with minimal administrative effort.

Potential Problems with Sampling and Bias

Undercoverage:
- Occurs when members of the population are systematically omitted from the sampling frame or have a lower probability of selection.
- Example: Surveying city residents using landline directories excludes individuals with only mobile phones. Surveying registered homeowners excludes renters, unhoused individuals, and youth living with parents.
Nonresponse Bias:
- Occurs when selected individuals in a sample cannot be reached or refuse to provide data.
- Critical Distinction: In nonresponse, researchers select individuals randomly, but the individuals choose not to answer. In voluntary response, the entire initial contact relies on self-selection.
- Example: Discarding email surveys, hanging up on telephone calls.
Response Bias:
- Systematic patterns of inaccurate or untruthful responses generated by the data gathering instrument, interviewer, or process.
- Drivers of Response Bias:
  - Dishonest self-reporting on sensitive, illegal, or socially undesirable behaviors (e.g., drug use, cheating, study time).
  - Interviewer characteristics, where the identity, clothing, or authority of the interviewer intimidates or primes the respondent.
  - Question wording: Leading, unbalanced, or loaded language that steers answers.

QuickNotes on Undercoverage, Nonresponse, and Response Bias

Bias Case Scenarios:
- Mayor of Springfield Age Survey:
  - Contacts an SRS of $300$ individuals chosen strictly from a registry of homeowners to estimate the mean age of all city residents.
  - Type of Error: Undercoverage (renters and non-homeowners excluded).
  - Direction of Bias: Overestimate. Homeowners are systematically older than renters and dependent residents.
- Principal Homework Survey:
  - Sends an email survey to an SRS of $100$ students asking if they completed all homework; only $40$ reply, with $36$ ( $90\%$ ) answering yes.
  - Type of Error: Nonresponse bias ( $60\%$ nonresponse rate).
  - Direction of Bias: Overestimate. Diligent students who did their homework are motivated to respond, while students who did not complete their homework avoid responding.
- Firefighter Uniform Survey:
  - Firefighter visits an SRS of $30$ neighborhood homes in official uniform asking if residents support proposed budget cuts to the fire department.
  - Type of Error: Response bias (interviewer effect / social desirability).
  - Direction of Bias: Underestimate support for cuts. Residents are reluctant to state support for budget cuts directly to a uniformed firefighter.
- Counselor High School Program Options:
  - Surveying only students enrolled in remote learning introduces undercoverage of in-person perspectives.
  - Selecting $100$ students where only $28$ reply introduces nonresponse bias.
  - Surveying the first $50$ students submitting preference forms introduces convenience sampling bias.
- Leading Social Media Question:
  - Question text: "Social media use can negatively affect teens, distracting them, disrupting their sleep, and exposing them to bullying, rumor spreading, unrealistic views of other people's lives and peer pressure. How many hours per week do you use social media?"
  - Type of Error: Response bias via loaded question wording.
  - Direction of Bias: Underestimate. The negative framing induces guilt, causing teens to report fewer hours than they actually spend.

Observational Studies versus Experiments

Observational Study:
- Researchers observe and measure variables of interest without imposing treatments on individuals.
- Can demonstrate association/correlation, but cannot establish cause-and-effect relationships due to potential confounding variables.
Experiment:
- Researchers deliberately impose treatments on experimental units to measure their responses.
- Well-designed experiments with random assignment can establish causation.
Key Variables and Terms:
- Explanatory Variable (Factor): The variable whose manipulation explains or predicts changes in the response variable.
- Levels: The specific values or categories assigned to a factor.
- Response Variable: The outcome variable measured in the study.
- Confounding Variable: A variable related to the explanatory variable that also affects the response variable, making it impossible to determine which factor caused the observed change in the response.
- Experimental Units: The physical entities to which treatments are applied (referred to as subjects when human).
- Treatments: The specific condition applied to experimental units; formed by a specific level or combination of levels of the explanatory variable(s).

QuickNotes on Observational Studies vs Experiments and Variables

Observational vs. Experimental Analysis Examples:
- After-School SAT Prep Class:
  - $44$ high school students volunteer for an after-school SAT prep course. Their mean SAT score is $1220$ , compared to a mean score of $1050$ for students who did not take the course.
  - Classification: Observational study (students self-selected into prep; treatment was not imposed).
  - Explanatory variable: Enrollment in SAT prep class (yes/no). Response variable: SAT exam score.
  - Causation: Cannot infer that the course caused higher scores. Confounding variables such as academic motivation, prior GPA, family income, and study habits may drive both enrollment and higher scores.
- Roof Type and Cell Service (Ben and Jerry):
  - Ben (metal roof) and Jerry (shingle roof) randomly assign $50$ employees to attend a party at either Ben's home or Jerry's home to record signal strength.
  - Classification: Experiment. Treatments (home location with designated roof type) were deliberately and randomly imposed on subjects.
- Home Book Count and Academic Success:
  - A sample of $100$ students demonstrates a strong positive correlation between family home book count and academic performance.
  - Classification: Observational study.
  - Confounding: Socioeconomic status and parental education correlate with home book count and also facilitate tutoring, nutrition, and academic support.

Principles of Experimental Design

Four Core Principles of Sound Experiments:
- Comparison: Compare two or more treatments (or a treatment against a control) to establish relative effects.
- Random Assignment: Use chance procedures to assign experimental units to treatments. Balances confounding variables across treatment groups, ensuring groups are roughly equivalent prior to treatment.
- Control: Hold non-treatment variables constant across groups to avoid confounding.
- Replication: Apply each treatment to a sufficiently large number of experimental units so true treatment differences can be distinguished from random chance variation.
Blinding and Placebos:
- Placebo: A baseline, inactive dummy treatment that matches the real treatment in appearance and administration.
- Placebo Effect: Measurable positive response from subjects resulting from the psychological expectation of receiving treatment.
- Single-Blind Design: Either the subjects or the individuals measuring the response do not know which treatment was administered.
- Double-Blind Design: Neither the experimental units nor the evaluators/researchers interacting with them know which treatment was assigned, preventing both subject placebo effects and observer bias.
Critique and Correction of Flawed Experiments:
- Mr. Wilcox Beard Oil Experiment:
  - Grows beard for No Shave November using a new beard oil; gains $3\text{ inches}$ of growth and attributes it to the oil.
  - Defect: Absence of a control group or comparison. Beards grow naturally over one month.
  - Correction: Compare beard growth against a control group of men receiving no oil, or test across a large sample with a placebo oil over the identical duration.
- Track Athlete Workout Options:
  - $120$ track athletes pick between a rigorous strength workout or a relaxed stretching routine. Strength group gains $11\%$ in speed; relaxed group gains $3\%$ .
  - Defect: Lack of random assignment. Athletes self-selected, confounding motivation, baseline athleticism, and strength with workout type.
  - Random Assignment Protocol: Label athletes $1$ to $120$ . Use a random number generator to select $60$ unique integers for the strength workout; allocate the remaining $60$ athletes to the relaxed routine.
- Smart Pills Testing:
  - Students are assigned pills explicitly labeled "smart pill" or "sugar pill" upon entering class. Smart pill group scores higher.
  - Defect: No blinding. Obvious labeling induces strong placebo effects.
  - Correction: Double-blind setup with identical, indistinguishable pills coded with concealed identifiers.
African Cattle Lion Deterrence Experiment:
- Treatments: Eyes painted on rear, cross-marks painted on rear, no markings on rear (control group).
- Observed Data ( $4\text{-year}$ study):
  - Painted eyes: $0\text{ deaths}$ .
  - Painted cross-marks: $4\text{ deaths}$ .
  - No markings (control): $15\text{ deaths}$ .
- Necessity of Control Group: Establishes the baseline predatory attack rate, proving marked rears reduced deaths below the natural predation level.
- Random Assignment Procedure ( $1200$ Cattle):
  - Label cattle $1$ to $1200$ .
  - Generate $400$ unique random integers between $1$ and $1200$ to receive painted eyes.
  - Generate the next $400$ unique random integers to receive painted crosses.
  - Assign the remaining $400$ cattle to the unpainted control group.
- Completely Randomized Design Outline: $1200\text{ Cattle} \xrightarrow{\text{Random Assignment}} \begin{cases} \text{Group 1 } (400\text{ cattle}) \rightarrow \text{Treatment 1: Painted Eyes} \\ \text{Group 2 } (400\text{ cattle}) \rightarrow \text{Treatment 2: Painted Cross-Marks} \\ \text{Group 3 } (400\text{ cattle}) \rightarrow \text{Treatment 3: No Markings (Control)} \end{cases} \rightarrow \text{Compare Lion Attack Deaths}$

Advanced Experimental Designs

Completely Randomized Design (CRD):
- All experimental units are allocated across all treatment groups strictly through random assignment.
Randomized Block Design (RBD):
- Block: A group of experimental units known before the experiment to be similar in a way that is expected to systematically influence the response variable.
- Mechanics: Partition units into blocks based on the blocking variable, then carry out random assignment of treatments separately within each block.
- Guiding Principle: "Control what you can, block what you can control, and randomize the rest."
- Advantage: Isolates and accounts for the variation caused by the blocking variable, reducing unexplained variability and clarifying treatment differences.
Matched Pairs Design:
- A specialized block design comparing two treatments.
- Either subjects are paired based on common traits (e.g., matching twins, or pairing by identical pre-test scores) with one member randomly assigned to each treatment, or each subject receives both treatments in a randomized order (serving as their own control).
SAT Prep Format Experiment ( $50$ Students: $30$ Juniors, $20$ Seniors):
- Completely Randomized Design Risk: Random allocation might place disproportionately more seniors into one format. Grade level correlates with academic preparation, confounding format comparisons.
- Randomized Block Design by Grade:
  - Block 1 (Juniors, $n = 30$ ): Randomly assign $15$ to Online Prep, $15$ to Classroom Prep.
  - Block 2 (Seniors, $n = 20$ ): Randomly assign $10$ to Online Prep, $10$ to Classroom Prep.
  - Compare SAT score outcomes within blocks and across formats.
- GPA Blocking Alternative (Matched Pairs):
  - Sort the $50$ students by descending GPA.
  - Group into pairs of adjacent GPAs: Pair 1 (Ranks 1 and 2), Pair 2 (Ranks 3 and 4), …, Pair 25 (Ranks 49 and 50).
  - Within each pair, randomly assign one student to Online Prep and the other to Classroom Prep.
Presidential Candidate YouTube Ad Experiment ( $300$ Users):
- Completely Randomized Design: $300\text{ Users} \xrightarrow{\text{Random Assignment}} \begin{cases} \text{Group 1 } (100\text{ users}) \rightarrow \text{Ad 1} \\ \text{Group 2 } (100\text{ users}) \rightarrow \text{Ad 2} \\ \text{Group 3 } (100\text{ users}) \rightarrow \text{Ad 3} \end{cases} \rightarrow \text{Compare Candidate Support}$
- Randomized Block Design:
  - Block by existing political party affiliation (e.g., Democrat, Republican, Independent).
  - Within each political block, randomly assign users across Ad 1, Ad 2, and Ad 3.
  - Accounting for political affiliation eliminates pre-existing political ideology as an extraneous source of variation, making true differences between advertisements detectable.

Inference and Experiments: A/B Testing

A/B Testing Framework:
- A randomized digital experiment comparing two versions of a digital product, interface, or ad format (Format A vs. Format B) by randomly routing users to determine conversion efficacy.
Yelp Ad Click A/B Test:
- Experimental Setup: $100$ user accounts randomly allocated ( $50$ to Format A, $50$ to Format B).
- Observed Sample Results:
  - Format A Click Proportion: $\hat{p}_A = \frac{21}{50} = 0.42 = 42\%$
  - Format B Click Proportion: $\hat{p}_B = \frac{19}{50} = 0.38 = 38\%$
  - Observed Sample Difference: $\hat{p}_A - \hat{p}_B = 42\% - 38\% = 4\%$
Concept of Statistical Significance:
- An observed outcome is statistically significant if the magnitude of the effect is so large that it would rarely occur solely by chance variation in random assignment.
- Initial Assessment: A $4\%$ difference represents a gap of only $2$ clicks ( $21$ vs. $19$ ), suggesting the outcome could be pure chance.
Hypothesis Simulation Testing:
- Null Model Assumption: Formats A and B are equally effective in generating clicks.
- Simulation Process: Pool the $100$ user outcomes ( $40\text{ clicks}$ , $60\text{ non-clicks}$ ). Randomly redistribute these outcomes into two groups of $50$ . Calculate simulated click percentage differences $\hat{p}_{\text{sim A}} - \hat{p}_{\text{sim B}}$ . Repeat $100$ times.

Simulation Dotplot of Differences in Proportions for A/B Testing

Simulation Results and Conclusion:
- The simulation differences center around $0.0\%$ and spread from $-30.0\%$ to $+25.0\%$ .
- A difference of $+4.0\%$ occurs regularly across the simulation runs.
- Final Statistical Conclusion: The observed $4\%$ difference is not statistically significant. Because a difference of this size occurs frequently by chance variation alone, there is no convincing evidence that Format A is superior to Format B.