External Validity and Generality in Behavioral Research

Research Approaches in Behavioral Science

• Between-Groups Research: Nomothetic approach comparing large groups to determine averages and statistical significance (Gast \, & \, Ledford, \, 2023).

• Single-Case Research: Idiographic approach where participants serve as their own control based on baseline logic ($Sidman, \, 1960$).

Views on Variation and the Average Person Doctrine

• Between-Subjects Methodology: Assumes variation conforms to a normal distribution; individuality is suppressed in favor of average performance ($Kazdin, \, 2010$).

• Single-Subject Designs: Focuses on individual patterns rather than populations; rejects the concept of the "average person" as a mathematical conceptualization rather than a biological reality.

• Average Person Doctrine: Originally termed the Average Man doctrine by $Quetelet$ , it assumes nature aims at a fixed point and deviations are errors.

Statistical vs. Experimental Control

• Statistical Control: Developed to manage intrinsic variability through tests of significance; results not significant are attributed to chance.

• Experimental Control: Demonstrates uniformities by isolating variables and identifying controlling relationships to establish universal laws.

• $Sidman's$ Meanings of Chance:

  1. Combined effect of uncontrolled variables (sloppy experiments).

  2. Unknown variables (researcher ignorance).

  3. Acceptance of unpredictability.

Replication: Foundation of Knowledge

• Within-Study Replication: Establishes internal validity and reliability of the effect (Johnston \, & \, Pennypacker, \, 2009).

• Across-Study Replication: Contributes to external validity and establishes the generality of the effect ($Sidman, \, 1960$).

• Direct Replication: Repeating an experiment with all features held constant to verify reliability.

• Systematic Replication: Intentionally changing elements to explore boundary conditions and critical features.

External, Ecological, and Construct Validity

• External Validity: The extent to which a relation holds across contexts ($Shadish \, et \, al., \, 2001$).

• Participant Characteristics:

  • Status Variables: Demographic descriptors (gender, age, race).

  • Functional Characteristics: Features related to the independent and dependent variables, often better predictors of success ($Bateman \, et \, al., \, 2023$).

• Ecological Validity: The degree to which study features relate to real-world contexts ($Fahmie \, et \, al., \, 2023$).

• Construct Validity: Represents whether labels used truly reflect the concepts of interest; threats include inadequate explication and confounding variables ($Shadish \, et \, al., \, 2001$).

Generality and Demographic Reporting

• In between-group designs, larger numbers are assumed to increase generality, though individual effects remain unknown.

• In single-subject designs, dramatic individual changes are viewed as more generalizable.

• Underreporting of Demographics: $Jones \, et \, al. \, (2020)$ found only $7\%$ of studies reported race/ethnicity, and only $2\%$ reported socioeconomic status, hindering the assessment of diversity.

Building a Science of Behavior

• Scientific progress relies on integrating discrepant data through ongoing across-study replication ($Sidman, \, 1960, \, p. \, 83$).

• Failure to replicate should be viewed as a challenge to identify alternative effective procedures rather than a reason to reject data.