Sampling, Estimation, and External Validity

Sampling Foundations and the Population of Interest

• Definition of Sampling:

  • Sampling is the formal act of selecting a subset of observations from which data is collected.

  • Rather than gathering data from an entire group, statisticians and data scientists systematically choose specific observations to study.

• The Sample:

  • The specific observations selected during the sampling process are collectively referred to as the sample.

• The Population of Interest:

  • The population of interest comprises all observations that researchers are interested in studying or making a statement about.

  • Before selecting a sample, researchers must explicitly specify who or what belongs to the population of interest.

• Inclusion Criteria:

  • Inclusion criteria are the explicit rules and attributes established by statisticians and data scientists to delineate who or what qualifies as an observation within the population of interest.

  • Setting precise inclusion criteria ensures clarity regarding the boundaries of the study group.

• Logistical Necessity of Sampling:

  • In almost all empirical investigations, collecting data from every single observation in the population of interest is logistically impossible.

  • Consequently, researchers collect data from a sample of observations to draw conclusions about the larger population.

Parameters, Sample Statistics, and Generalization Inferences

• Parameters:

  • A parameter is a summary statistic computed from an entire population of interest describing a specific attribute.

  • Example: The average household income computed across all Americans represents a parameter because it summarizes the entirety of the American population.

• Sample Statistics:

  • A sample statistic is a summary statistic calculated solely from the observations contained within a sample.

  • Example: If a sample of $1{,}000$ Americans is selected and their average household income is calculated, that resulting numerical figure is a sample statistic.

• Estimates:

  • An estimate is the numerical value of a sample statistic utilized as a stand-in for an unknown population parameter.

• Generalization Inference:

  • A generalization inference is the act of using a sample statistic as an estimate to determine the value of a population parameter.

• The 2022 US Bachelor's Degree Salary Case Study:

  • Objective: Determine the median salary in $2022$ of all Americans over the age of $25$ who hold a Bachelor's degree.

  • Target Population: All Americans over the age of $25$ with a Bachelor's degree, encompassing almost $18\text{ million}$ individuals ($18{,}000{,}000$ people).

  • Feasibility Constraint: Querying all $18\text{ million}$ individuals meeting the inclusion criteria is logistically impossible.

  • Methodology: The US Census Bureau recruits a sample of Americans through the American Community Survey (ACS) using complex probability sampling strategies.

  • Sample Statistic: The sample median salary calculated from the American Community Survey sample is approximately \\$65{,}000.

  • Generalization Application: Because the US Census Bureau utilizes complex probability sampling strategies, researchers use this sample statistic (\\$65{,}000) as an estimate and stand-in for the true population parameter.

External Validity and Representativeness

• External Validity Defined:

  • External validity is the degree to which a given sampling strategy supports making a sound generalization inference.

  • High external validity allows researchers to project sample findings onto the wider population with confidence.

• Representativeness Defined:

  • Representativeness refers to the extent to which observations in the sample mirror the observations in the population of interest regarding essential underlying baseline characteristics.

  • External validity depends directly on the sample's representativeness.

• UIUC Student Body Case Study on Non-Representativeness:

  • Scenario: A sample of college students selected from the University of Illinois Urbana-Champaign (UIUC) that completely lacks freshmen.

  • Underlying Characteristic Distribution: Freshmen make up a substantial proportion of the entire UIUC student body.

  • Distributional Mismatch: In this sample, the distribution of students' year in school does not match the distribution of students' year in school in the population of interest.

  • Outcome: The failure to include freshmen creates a fundamental difference in an important underlying baseline characteristic between the sample and the population, rendering the sample non-representative and invalidating general inferences about all UIUC students.

Mathematical Demonstration of Representativeness: Qualifying Exam Study Hours

• Context and Population Setup:

  • In $2023$, there were $9$ graduate students in Statistics at the University of Illinois preparing to take their qualifying exams.

  • Research Goal: Determine the average number of hours per day spent studying for qualifying exams across this population of $9$ students.

  • Baseline Population Groupings:

  • Mia, Ming, and Mindy all study the identical amount of time per day.

  • Rhea, Rina, and Rita all study the identical amount of time per day.

  • Tom, Timo, and Tito all study the identical amount of time per day.

• Sample Selection and Observed Sample Statistics:

  • A statistics professor randomly draws a sample of $3$ students: Mia, Rina, and Tito.

  • Observed daily study hours from the sample:

  • Mia: $7\,\text{hrs/day}$

  • Rina: $6\,\text{hrs/day}$

  • Tito: $8\,\text{hrs/day}$

  • Calculation of Sample Mean (Sample Statistic):     $\\bar{x} = \frac{7 + 6 + 8}{3} = \frac{21}{3} = 7\,\text{hrs/day}$

• Calculating the Population Parameter via Representation:

  • Because the sample is representative, each student in the sample serves as an exact stand-in for their peers in the population:

  • Mia ($7\,\text{hrs}$) represents Ming ($7\,\text{hrs}$) and Mindy ($7\,\text{hrs}$).

  • Rina ($6\,\text{hrs}$) represents Rhea ($6\,\text{hrs}$) and Rita ($6\,\text{hrs}$).

  • Tito ($8\,\text{hrs}$) represents Tom ($8\,\text{hrs}$) and Timo ($8\,\text{hrs}$).

  • Complete Population Data Distribution:

  • Mia: $7\,\text{hrs}$

  • Ming: $7\,\text{hrs}$

  • Mindy: $7\,\text{hrs}$

  • Rhea: $6\,\text{hrs}$

  • Rina: $6\,\text{hrs}$

  • Rita: $6\,\text{hrs}$

  • Tom: $8\,\text{hrs}$

  • Timo: $8\,\text{hrs}$

  • Tito: $8\,\text{hrs}$

  • Calculation of Population Mean (True Parameter):     $\\mu = \frac{7 + 7 + 7 + 6 + 6 + 6 + 8 + 8 + 8}{9} = \frac{21 + 18 + 24}{9} = \frac{63}{9} = 7\,\text{hrs/day}$

• Key Principles Revealed by the Exercise:

  • The sample mean ($7\,\text{hrs/day}$) matches the true population parameter ($7\,\text{hrs/day}$) because every single individual in the population of interest is represented by an observation in the sample.

  • The Impact of Missing Data: If Rina had not been selected in the sample, there would be no basis to infer the study hours for Rhea and Rita, making an accurate parameter estimate impossible.

Sample Bias and the "Who's Not Here?" Framework

• Consequences of Non-Representative Samples:

  • When a sample is non-representative, information regarding certain subsets of the population of interest is systematically absent.

  • Missing data causes parameter estimates to become inaccurate, making it inappropriate to draw generalizations about the population.

• Sample Bias:

  • Sample bias occurs when non-representativeness systematically prevents a researcher from making a valid generalization inference.

• Primacy of Defining the Population:

  • Before evaluating whether a sample is representative, researchers must explicitly define the inclusion criteria of the target population.

  • Representativeness can only be judged relative to the specific inclusion criteria established for the study.

• Illinois Farm Crop Study:

  • Research Goal: Investigate the crops grown on agricultural farms in the state of Illinois.

  • Observation Unit: Each individual farm.

  • Measured Attributes: Total farm size, total number of crops, each individual crop grown, and crop yields.

  • Inclusion Criteria: The farm must be physically situated within the state of Illinois.

  • Extraneous Observations: Farms located in Lesotho or Laos are possible global observations, but they fall outside the inclusion criteria.

  • Evaluation of Representativeness: The omission of farms from Laos or Lesotho does not impair representativeness or introduce sample bias, because they do not belong to the defined population of interest.

• The Evaluative Heuristic: "Who's Not Here?":

  • A vital diagnostic tool for assessing representativeness is asking the question: "Who's not here?"

  • Applying this question requires understanding who ought to be present based on the inclusion criteria.

  • Researchers must systematically contrast the baseline attributes and inclusion criteria of the population against the composition of the sample to identify omitted groups.

Visualizing Representativeness and External Validity via Proportions

• Population Baseline:

  • Consider a population of interest comprising two subgroups: Red Tones and Blue Tones.

  • The population is distributed evenly such that:     $\\text{Red Tones} = 50\%$     $\\text{Blue Tones} = 50\%$

• Representative Sample (High External Validity):

  • Composition: The sample consists of $50\%$ red tones and $50\%$ blue tones.

  • Property: The sample distribution precisely mirrors the underlying population distribution.

  • Result: High external validity, supporting sound generalization inferences.

• Non-Representative Sample (Low External Validity):

  • Composition: The sample consists of $25\%$ red tones and $75\%$ blue tones.

  • Property: Red tones are underrepresented in the sample ($25\%$) relative to the population ($50\%$).

  • Result: Low external validity caused by distributional disparity, producing sample bias and invalidating generalizations.