Research Methods: Measurement Reliability, Survey Design, Sampling, and Validity

Vocabulary practice flashcards covering calculating reliability, observational ratings, survey question construction, sampling methods, and the four big validities across psychological research claims.

Correlation Coefficient

A quantitative index ranging from $-1$ to $+1$ that indicates the direction (sign) and strength (absolute value) of the relationship between two measurements.

Test-Retest Reliability

A measure of consistency determined by administering the exact same assessment to the same participants across two different time points.

Parallel Forms Reliability

An assessment of consistency evaluated by having participants complete two equivalent but not identical test forms at the same time point.

Split-Half Reliability

A measure of internal consistency evaluated by calculating the correlation between participants' scores on one half of items and the other half of items intended to measure the same construct.

Inter-Rater Reliability

The degree of agreement and correspondence between independent raters who code or quantify observational data.

Construct Validity

The extent to which the operational variables in a study serve as accurate approximations of the conceptual variables, reflecting how well variables are measured or manipulated.

External Validity

The degree to which the results of a study can be generalized beyond the current investigation to other populations, settings, situations, or time periods.

Statistical Validity

The extent to which numbers and statistical analyses support a claim, addressing effect size strength, precision of estimates via confidence intervals, and study replication.

Internal Validity

The extent to which variable A, rather than a confounding third variable C, is responsible for observed changes in variable B in a causal relationship.

Population

The entire set of people, items, or products that are of interest to a researcher.

Sample

A subset drawn from a larger population that is selected for inclusion in a research study.

Statistical Inference

The process of drawing conclusions and generalizations back to the broader population based on observations gathered from a representative sample.

Probability Sampling

A sampling category utilizing random selection where every member of the population has an equal likelihood of being chosen, yielding high external validity and avoiding sample bias.

Non-Probability Sampling

A sampling category that does not use random selection, resulting in varying inclusion likelihoods among population members, potential sample bias, and lower external validity.

Simple Random Sampling

A probability sampling technique where every individual in the population has an entirely equal probability of selection into the sample.

Systematic Sampling

A probability sampling method where members of a population are selected according to a fixed numerical sequence interval, such as every $10^{\text{th}}$ individual.

Cluster Sampling

A probability sampling design in which pre-existing naturally occurring groups or clusters of the population are selected at random.

Stratified Random Sampling

A probability sampling technique in which the population is divided into meaningful demographic subgroups (strata) and individuals are randomly drawn proportionally from each group.

Convenience Sampling

A non-probability sampling technique where participants are selected solely because they are easily accessible and convenient for the researcher to recruit.

Consecutive Sampling

A non-probability sampling technique where every accessible subject meeting the inclusion criteria is included sequentially over time until a specified quota or target is reached.

Judgmental / Purposive Sampling

A non-probability sampling technique where the researcher selects specific participants based on professional judgment or defined subject characteristics.

Quota Sampling

A non-probability sampling method where researchers specify non-random target quotas for specific demographic groups and recruit non-randomly until those target numbers are satisfied.

Snowball Sampling

A non-probability sampling method where enrolled participants are asked to identify and recruit additional participants from their own social networks.

Random Assignment

An experimental technique occurring during a study where participants have an equal likelihood of being assigned to different experimental conditions to eliminate selection bias and protect internal validity.

WEIRD Samples

Research participant pools drawn disproportionately from Western, Educated, Industrialized, Rich, and Democratic societies, frequently limited to specific age ranges such as 18–28 or 60+.

Value Claim (Frequency Claim)

A descriptive research claim that identifies the specific rate, percentage, or level of a single variable, prioritizing construct and external validity.

Association Claim

A research claim asserting a relationship or correlation between two variables, prioritizing construct and statistical validity while leaving internal validity non-applicable.

Causal Claim

A research claim asserting that one variable directly causes change in another, prioritizing internal, construct, and statistical validity while frequently sacrificing external validity.

Observational Method

A descriptive research technique involving direct, detailed observation and recording of behavior, characterized by high external validity in natural settings but lower experimental control.

Case Study

A descriptive research design involving an in-depth examination of a single individual or unique population with distinctive characteristics, which may have limited generalizability.

Leading Question

A survey question phrased in a way that suggests a preferred answer or presumes a condition, thereby biasing respondent answers and compromising validity.

Double-Barreled Question

A flawed survey item that combines two distinct questions or constructs into a single item, preventing the participant from responding to each component individually.

Double Negative

A survey question construction containing two negative words or clauses, causing confusion and cognitive misinterpretation among respondents.

Response Set

A systematic pattern of answering survey questions based on a personal style or bias rather than the actual content of the questions.

Yea-Saying / Nay-Saying

A response set in which a respondent consistently agrees (yea-saying) or disagrees (nay-saying) with every survey item regardless of its meaning, typically mitigated by reverse-worded items.

Social Desirability

A response bias in which participants select survey answers that are perceived as socially, ethically, or morally favorable to portray themselves in a positive light.

Faking Bad

A response set where a participant purposefully over-reports negative symptoms or under-reports positive attributes to appear more impaired or distressed than they truly are.

Affective Neuroscience Personality Scales (ANPS)

A personality assessment instrument designed by Davis and Panksepp to measure behavioral traits related to six primary subcortical brain emotion systems: SEEKING, RAGE, FEAR, CARE, GRIEF, and PLAY.