Introduction to Statistical Sampling, Experimental Design, and Sources of Bias

Observational Studies, Experiments, and Censuses

• Observational Study:

  • Definition: A study that gathers and reports data on subjects without attempting to manipulate, influence, or introduce treatments to the variable outcomes being measured.
  • Observation Impact: Researchers must acknowledge that the process of observing or measuring subjects can inadvertently exert an influence on the observed data results.
  • Example: Determining the average height of oak trees in Indiana by taking a sample of $100$ trees out of a population of $1,000,000$ and measuring their heights.
  • Field Measurement Technique: A practical method for estimating tree height and distance across a stream (e.g., during wilderness survival) involves standing at the base of the tree, stepping off a baseline distance, and sighting the angle to the treetop.

• Experiment:

  • Definition: A study where researchers deliberately apply specific conditions or treatments to subjects, control external variables, and measure the resulting responses from scratch.
  • Example: Investigating whether soil type affects oak tree growth by deliberately planting oak saplings in controlled, different soil compositions.
  • Observational vs. Experimental Boundaries: Assessing mature oak trees growing naturally across various soil types is observational and subject to confounding variables, whereas planting saplings in assigned soils under controlled conditions is an experiment.

• Census:

  • Definition: An exhaustive collection of measurement data from every single individual or element in the target population.
  • Example: Measuring the height of every single oak tree in Indiana.
  • Parameter Evaluation: When a census is successfully conducted, the calculated summary measure yields the exact population parameter, eliminating the need for inferential statistics.

• Parameters vs. Statistics:

  • Parameter: A fixed numerical summary value describing a characteristic of an entire population (e.g., the exact mean height of all $1,000,000$ oak trees in Indiana).
  • Statistic: A numerical summary calculated from sample data used to infer or estimate the value of an unknown population parameter.

• Study Design Scenarios and Categorization:

  • Food Form and Satiety Study:
  • Design: Half of a sample group is given whole apples and the other half is given apple juice containing equal calories. Subjects rate their feeling of fullness after eating.
  • Classification: Experiment.
  • Reasoning: The form of food (apple vs. apple juice) represents a controlled treatment explicitly administered to the subjects.
  • Athlete Bone Density Study:
  • Design: $30$ adult athletes are paired with $30$ non-athlete adults possessing similar physical characteristics to evaluate bone strength.
  • Classification: Observational Study.
  • Reasoning: Pre-existing group statuses (athlete vs. non-athlete) are observed without imposing treatments.
  • Experimental Alternative Design: Measure subjects' initial bone density, administer a specific physical training regimen as a treatment, and re-measure bone density post-training.
  • Rancher Horse Weight Study:
  • Design: A rancher gathers and weighs every single horse on her ranch to calculate the mean weight.
  • Classification: Census and Observational Study.
  • Reasoning: Data is obtained for the entire target population without manipulating conditions. The resulting mean is the true population parameter rather than a sample statistic.

Sample Size and Statistical Inference

• The Process of Statistical Inference:

  • Statistical inference analyzes measured sample statistics to estimate or infer information regarding unknown population parameters.
  • If complete data is gathered via a census, statistical inference is unnecessary because population parameters are directly known.

• Battery Lifetime Study Case:

  • Context: A study estimates the average lifetime of manufactured batteries. Researchers randomly select $200$ batteries from $1,000,000$ batteries produced by a factory during a single month.
  • Population: The $1,000,000$ batteries manufactured by the plant during that specific month.
  • Parameter: The true average lifetime of all $1,000,000$ batteries in the population.
  • Sample: The $200$ randomly selected batteries ($n = 200$).
  • Data Collected: The exact recorded lifetime for each individual battery among the $200$ sampled.
  • Statistic: The sample mean lifetime calculated from the $200$ selected batteries.
  • Study Classification: Observational study (monitoring time elapsed until battery failure).

• Sample Size ($n$) Significance:

  • Without knowing the sample size, a sample's utility and accuracy cannot be evaluated.
  • The sample size direct governs the level of precision, accuracy, and reliability achievable in a study.
  • Larger sample sizes yield higher reliability, with a census representing the maximum attainable sample size.

• Mathematical Underpinnings of Statistics:

  • Probabilistic models underlying statistical inferential theory assume an infinite population size.
  • The formal proofs underlying statistical inference rely on advanced graduate-level calculus (Real Analysis) rather than standard undergraduate engineering calculus.
  • Historical Timeline: Practical statistical techniques were developed during the late 19th century, but the formal mathematical and axiomatic underpinnings were established in the 1930s.

Sampling Designs and Sources of Bias

• Sampling Design Principles:

  • Sampling Design: A systematic protocol for selecting a sample from a target population, serving as the foundation for both observational studies and experiments.
  • Representative Sample: A sample whose characteristics match the exact proportions present in the underlying population (e.g., equal representation of primary colors blue, red, and green). Achieving or verifying absolute representativeness is impossible without a complete census.
  • Bias: Systematic favoritism toward specific outcomes over others within a study design. Bias must be eliminated, minimized through mathematical techniques, or explicitly noted.

• Response Bias:

  • Definition: Systematically distorted responses caused by the behavior of the interviewer, social pressure, leading questions, or confusing survey wording.
  • Coercive/Leading Question Example: A supervisor asking a subordinate: "As a new employee at the firm, surely you must agree that a lack of congruence between the management echelons leads to a degradation of morale?"
  • Flaws: Coercive authority dynamic, leading phrasing ("surely you must agree"), and confusing jargon.
  • Classroom Cheating Example: Asking students to raise their hands in class if they cheated on an exam within the past year.
  • Flaws: Severe response bias due to public exposure, lack of anonymity, and fear of academic repercussions.

• Non-Response Bias:

  • Definition: Occurs when selected individuals fail to respond, refuse to participate, or leave survey forms incomplete, creating systematic differences between respondents and non-respondents.
  • Mailed Survey Example: Sending surveys to male subjects where only $50\%$ of questionnaires are returned, with several left incomplete.
  • Food Satisfaction Example: A campus food survey that receives responses from only $98$ fans/respondents.
  • Demographic Blind Spots: Individuals who consistently decline survey participation represent a non-responder demographic blind spot. The U.S. Census Bureau addresses this by employing teams of surveyors to physically visit non-responding households.

• Undercoverage Bias:

  • Definition: Occurs when specific subgroups of the target population are systematically omitted from the sampling frame, preventing them from being selected.
  • Park Survey Example: A news headline claiming that $83\%$ of Smallville residents approve of a new park based on interviews with $210$ adults present at the park.
  • Flaws: Omits all Smallville residents who do not visit the park, invalidating claims about the overall town population.
  • Lead Poisoning Council Example: A council in metropolitan Western Tennessee randomly samples $500$ homes to test for unsafe lead levels, but accidentally omits several entire neighborhoods from the sampling frame.

• Voluntary Response Bias / Volunteer Sampling:

  • Definition: A non-probability sampling design relying on self-selected individuals who opt into a study in response to an open call.
  • Example: Posting tear-off flyers around a hospital asking for paid volunteers ($25$ compensation) to participate in a clinical drug trial.
  • Flaws: Attracts individuals with specific personal or financial motivations ($25$ incentive) while excluding those who never see the physical posting.

Probability Sampling Methods

• Simple Random Sampling (SRS):

  • Definition: A sampling design of size $n$ where every individual in the population has an equal probability of selection, AND every possible subset/combination of $n$ individuals has an equal likelihood of being chosen.
  • Implementation Steps:
  1. Assign a unique numeric identifier to every single individual in the population.
  2. Select identifiers strictly at random using pseudo-random generation algorithms.
  • Strengths and Limitations: Minimizes undercoverage bias, but does not guarantee that every individual sample draw will be perfectly representative of all sub-population traits. Response and non-response bias can still occur.

• Stratified Random Sampling:

  • Procedure:
  1. Divide the entire target population into distinct, non-overlapping, homogeneous subgroups called strata based on specific shared traits.
  2. Classify every population member into a stratum (utilizing an "Other" category if necessary).
  3. Execute a separate Simple Random Sample (SRS) within each individual stratum.
  • Advantages: Prevents undercoverage by guaranteeing that all population subgroups (including minority groups) are represented in the sample.

• Multistage Random Sampling:

  • Procedure: A sampling design that selects subjects through successive stages of random selection applied to increasingly smaller organizational or geographical units.
  • Example (Indiana Opinion Poll across $92$ Counties):
  • Stage 1 (Geographic Stratification): Partition Indiana into regional strata (North, Central, South) and randomly select representative counties (e.g., selecting $3$ counties from the $92$ total).
  • Stage 2: Randomly select $2$ towns from each selected county.
  • Stage 3: Randomly select $5$ individuals from each selected town.
  • Utility: Provides a practical sampling design when a single complete list of the entire population is unavailable.