Reliability, Validity, Research Design, and Sampling

A comprehensive vocabulary review of measurement reliability, measurement validity, research replication, validity tradeoffs, and sampling methodologies based on the lecture study guide.

Reliability

The consistency or stability of a measurement procedure, yielding the same accurate measurement each time it is used under identical conditions.

Obtained Score

The actual raw score or numerical value recorded from a respondent on a measure.

True Score

A respondent's actual level or underlying capacity on the construct being measured, if it could be measured perfectly without any flaws.

Measurement Error

Any deviation from the true score caused by extraneous factors that artificially increase or decrease the obtained score.

Basic Equation of Measurement Theory

$$\text{Obtained Score} = \text{True Score} + \text{Error}$$

Random Measurement Error

Unpredictable chance fluctuations that affect scores in either direction, caused by transient participant factors (e.g., fatigue, mood) or test administration factors (e.g., environmental distractions).

Systematic Measurement Error (Bias)

Consistent error that skews scores predictably in one single direction, often caused by demographic test bias or faulty, uncalibrated equipment.

Content Sampling Error

Error resulting from the specific sample of items selected for a test compared to the vast domain of all possible items.

Time Sampling Error

Error resulting from temporary, moment-to-moment factors that change depending on when the test is administered.

Scoring Error

Errors introduced during manual scoring, typos in data entry, or subjective ambiguity when evaluating responses.

Test-Retest Reliability

A reliability procedure administering the same test to the same sample at two different points in time to calculate the correlation coefficient ($r_{12}$), assuming the measured construct is a stable trait.

Alternate / Parallel Forms Reliability

Administering two equivalent versions of a test (Form A and Form B) to the same sample at one time to calculate the correlation ($r_{ab}$), eliminating time sampling error but assuming true equivalence.

Split-Halves Reliability

An internal consistency method administering one test to one sample at one time, splitting items in half and correlating score totals between the two halves (most preferably using an odd-even split).

Cronbach’s Alpha

An internal consistency measure computing the overall inter-item correlation across all scale items simultaneously, where $\alpha \ge .70$ indicates acceptable reliability under the assumption of unidimensionality.

Inter-Rater Reliability

Having two or more independent raters score the same behavior using an operational definition to calculate correlation ($r_{12}$) or percent agreement, mitigating scoring subjectivity.

Validity

The accuracy of a measurement tool—whether a test actually measures what it purports to measure.

Construct Validity

The overarching degree to which a test measures the theoretical construct it claims to assess.

Face Validity

The superficial appearance of whether a test looks like it measures its intended construct; the least rigorous form of validity.

Criterion Validity

Evaluating a test by comparing scores on the measure to an external standard or criterion.

Convergent Validity

A subtype of criterion validity where test scores correlate strongly with measures of related constructs.

Concurrent Validity

A subtype of criterion validity where test scores correlate strongly with an established measure assessing the exact same construct at the same point in time.

Divergent / Discriminant Validity

A subtype of criterion validity where test scores show no correlation ($r \approx .00$) with measures assessing unrelated constructs.

Predictive Validity

A subtype of criterion validity where test scores successfully predict a relevant behavioral outcome or performance criterion in the future.

Evidence from Distinct Groups

Demonstrating validity by showing that groups known a priori to differ on a construct score differently on the measure.

Content Validity

The degree to which items on a test adequately sample and cover all major subdomains or dimensions of the construct being measured.

Factor Analysis

A statistical procedure identifying underlying clusters (factors) of items on which respondents score similarly, helping researchers discover subscales within a construct.

Exploratory Factor Analysis (EFA)

A statistical method used when discovering patterns or dimensions in a new test without strong a priori predictions.

Confirmatory Factor Analysis (CFA)

A statistical method used to test whether collected data fits a predefined theoretical factor structure.

Literal Replication

Repeating an experiment using the exact same methods, operationalizations, and procedures to confirm reliability.

Conceptual Replication

Testing the same underlying theoretical hypothesis using different operational definitions or methodological procedures.

Converging Evidence

A scientific principle where multiple independent studies—using distinct methodologies, samples, or operationalizations—all lead to the same overarching conclusion.

Internal Validity

The extent to which a study demonstrates that only the manipulated independent variable caused changes in the dependent variable, ruling out confounding variables.

External Validity

The extent to which research findings generalize beyond the specific sample and experimental setting to the real world.

Mundane Realism

The degree to which an experimental setting or task physically resembles situations encountered in everyday life.

Psychological Realism

The degree to which experimental tasks trigger genuine psychological processes, emotional states, or social dynamics in participants.

Population

The entire target group of interest in a study from which inferences are drawn.

Sample

A smaller subset of the population from which data is actually collected.

Sampling Plan

The explicit recruitment strategy used to select participants from the target population.

Representative Sample

A sample whose key demographic and psychological characteristics closely match those of the broader target population.

Probability Sampling

Sampling plans where every individual in the target population has a known, non-zero (equal) chance of being selected, minimizing selection bias.

Simple Random Sampling

A probability sampling plan where every member of the population has an equal chance of selection via a completely random draw.

Stratified Random Sampling

A probability sampling method where the population is divided into subgroups (strata) by demographic variables, and participants are randomly selected from each subgroup in proportion to the population.

Area Probability Sampling

A geographic probability sampling variation where target geographic areas (e.g., zip codes, city blocks) are randomly selected prior to sampling individuals within those areas.

Cluster Random Sampling

A probability sampling method where pre-existing groups (e.g., schools, classrooms) are randomly selected, and all individuals within the chosen clusters participate.

Nonprobability Sampling

Sampling plans where individuals do not have an equal chance of being selected, introducing potential selection bias.

Convenience Sampling

A nonprobability sampling method where participants are selected based entirely on easy availability and accessibility.

Quota Sampling

A nonprobability sampling method involving the non-random selection of participants to meet predetermined group numbers.

Purposive Sampling

A nonprobability sampling method where participants are hand-picked deliberately based on specific expert criteria or attributes.

Snowball Sampling

A nonprobability sampling method where initial participants recruit additional acquaintances or peers from their social networks.

Nonresponse Bias

Systematic differences between individuals who choose to participate in research and those who decline or refuse.

Abraham Wald’s Aircraft Solution

A historic example of survivorship bias where statistician Abraham Wald recognized that armor should be placed where returning WWII bombers lacked bullet holes, because planes shot in those locations did not survive to be counted.

Volunteer Participant Problem

A sampling issue where volunteers systematically differ from non-volunteers in personality, open-mindedness, and socioeconomic status, potentially overstating behavioral prevalence (e.g., Alfred Kinsey's sexuality studies).

The College Sophomore Problem

An over-reliance on undergraduate students representing a WEIRD (Western, Educated, Industrialized, Rich, Democratic) demographic, leading to questions about generalizability across broader human populations.