Evaluating Research Methods: Sampling, Instruments, and Procedures

Vocabulary practice flashcards covering quantitative and qualitative sampling methods, instrument standardization, reliability, validity, and procedural threats to internal and external validity.

Target Population

All individuals or objects in which the researcher is interested and to which the results of the study are intended to be applied.

Accessible Population

The realistic portion of the target population that researchers are actually able to access and draw a sample from.

Population Validity

The extent to which the sample matches the population being studied, enabling researchers to generalize results from the sample back to the population at large.

Simple Random Sampling

A quantitative probability sampling method where every single individual in the population has an equal chance of being chosen for the study.

Systematic Random Sampling

A sampling method involving a finite list of individuals in the population from which every $n\text{th}$ person is selected.

Stratified Sampling

A sampling method (proportional or nonproportional) in which individuals are selected from the population to ensure adequate representation of specific subgroups on relevant variables, such as ethnicity.

Cluster Sampling

A sampling method involving the random selection of intact groups (such as entire classrooms), utilized when individual subjects cannot be separated.

Convenience Sampling

A nonprobability sampling method selecting a section of a population that is convenient and accessible to the researcher, which severely limits generalizability and harms population validity.

Statistical Power Analysis

A mathematical procedure determining the sample size needed to ensure a specific probability that a test of statistical significance will reject a false null hypothesis.

Purposive Sampling

A qualitative sampling technique where researchers rely on their own special knowledge or expertise to handpick subjects who represent the population of interest.

Maximal Variation Sampling

A form of purposive sampling in qualitative research that selects participants who provide a wide range of different perspectives or characteristics related to the study's focus.

Typical Sampling

A variation of purposive sampling in qualitative research where participants are selected because they represent what is considered typical or "normal."

Snowball Sampling

A qualitative sampling method where researchers identify an initial set of participants with relevant attributes, who then provide referrals to others possessing the same characteristics.

Quota Sampling

A qualitative sampling technique where specific characteristics of interest are established and participants are recruited until a predetermined number is filled for each cell or category.

Sampling Error

An unintended discrepancy between a sample and its population that occurs purely by accident or chance when the individuals selected fail to represent the population accurately.

Sampling Bias

A systematic flaw introduced when researchers actively select individuals in a manner that results in intentional or unintentional partiality in the sample composition.

Standardization of Instruments

The establishment of uniform procedures and consistent scoring methods for a dependent measure, allowing scores to be converted into objective $z$-statistics or $t$-statistics.

Criterion-Referenced Scoring

An approach to scoring a formal instrument based on a predetermined level of performance criteria (e.g., demonstrating single-digit multiplication with $100\%$ accuracy) rather than relative group ranking.

Norm-Referenced Scoring

An assessment scoring method where individual scores are interpreted by comparing them against the performance of a clearly defined normative population (e.g., SAT, WAIS), typically reported as percentile ranks.

Reliability (of Instruments)

The precision, consistency, and reproducibility of a measurement procedure, quantified by correlation coefficients ranging from $-1.0$ to $+1.0$ (where $.80$ and higher denotes strong reliability).

Target Models of Reliability and Validity

A bullseye visual analogy: reliability reflects cluster tightness (shot consistency), whereas validity reflects proximity to the bullseye center (measuring the actual target construct).

Test-Retest Reliability

The stability or consistency of test scores over time, evaluated by administering the exact same instrument to the same sample at two different time points.

Alternate-Form Reliability

The degree of consistency between two different but parallel versions of an instrument administered to the same group to eliminate memory and practice effects.

Interrater Reliability

The degree of agreement between multiple independent observers or scorers evaluating the same target behavior, often measured as interobserver agreement (IOA) with an $80\%$ minimum threshold.

Internal Consistency

The most commonly reported reliability type, reflecting how consistently items within an instrument correlate with one another and measure the same underlying construct (e.g., Cronbach's alpha).

Construct Validity

The degree to which an instrument actually measures the intended theoretical concept or construct, supported by content-related sources, convergent/divergent evidence, and criterion prediction.

Content Validity

The extent to which the items of an assessment adequately sample and represent the specific domain of content, typically established through expert evaluation.

Concurrent Validity

A form of criterion-related validity evaluated by correlating the scores of a newly designed test with scores from an already validated, accepted standard test administered at roughly the same time.

Direct Observation and Behavioral Measures

An objective approach to measuring the dependent variable where trained observers systematically record observable behaviors, especially nonverbal cues, naturally occurring actions, and live processes.

Self-Report Measures and Inventories

The most common method of measuring the dependent variable in psychology and education, wherein participants rate their own internal feelings, attitudes, or cognitions using Likert-type scales, checklists, or open-ended items.

Random Assignment

The procedure of placing selected participants into experimental or control conditions purely by chance, giving each subject an equal probability of being assigned to any group.

Internal Validity

The extent to which differences observed in the dependent variable can confidently be attributed solely to the manipulation of the independent variable, free from confounding or extraneous variables.

External Validity

The extent to which the cause-and-effect findings of an experiment can be generalized across diverse populations, settings, treatment variables, and measurement conditions.

History Threat

An internal validity threat where extraneous, uncontrolled events occurring outside the experiment during the course of the study influence the dependent variable.

Maturation Threat

An internal validity threat resulting from biological, physical, or psychological changes occurring within participants over time (such as aging, fatigue, or growing wiser) rather than due to the treatment.

Testing Threat

An internal validity threat where taking a pretest improves posttest scores or makes participants sensitized to desired responses regardless of the experimental treatment.

Instrumentation Threat

An internal validity threat caused by low reliability, changes in the calibration of an instrument, or shifts in observer scoring standards between pretest and posttest.

Statistical Regression

The mathematical tendency for participants selected on the basis of extreme pretest scores to score closer to the overall mean upon posttest evaluation.

Differential Selection

An internal validity threat occurring when non-equivalent criteria are used to assign participants to control and experimental groups, commonly seen with non-random convenience samples.

Experimental Mortality (Attrition)

The differential loss of participants who drop out of the study over time, potentially biasing group equivalence and distorting outcomes.

Resentful Demoralization

A subject effect where control group members learn they are being deprived of a desirable or beneficial intervention, causing them to become resentful and perform poorly.

John Henry Effect

A subject effect occurring when control group members realize they are in a study and exert extra effort to compete with and outperform the experimental group.

Ecological Validity

A major subcategory of external validity concerning how closely the experimental conditions, materials, and settings represent the natural environment.

Hawthorne Effect

An ecological validity threat where participants alter their behavior simply because they know they are being observed and evaluated as part of an experiment.

Novelty and Disruption Effects

Threats to ecological validity where performance gains are caused by the excitement of a new stimulus (novelty) or performance drops due to unfamiliar disruption, rather than the treatment's true effect.

Experimenter Effects

Unintentional cues, actions, or communications delivered by the researcher that bias participants' behaviors or skew the scoring of subjective outcomes.

Multiple Treatment Interference

An ecological threat that occurs when two or more interventions are administered sequentially to the same participants, making it impossible to separate individual treatment effects.

Interaction of Time of Measurement and Treatment Effects

An external validity threat where the observed impact of an intervention is strongest immediately following treatment administration but fails to persist over extended time frames.