Shadish et al. (2002) Ch.2 - Internal Validity and Statistical Conclusion Validity

Statistical Conclusion Validity and Internal Validity

Definitions

  • Validity (Val-id):

    • Adjective; derived from Latin validus, meaning strong.

    • 1. Well grounded; just: a valid objection.

    • 2. Producing the desired results; efficacious: valid methods.

    • 3. Having legal force; effective or binding: a valid title.

    • 4. Logic:

      • a. Containing premises from which the conclusion may logically be derived: a valid argument.

      • b. Correctly inferred or deduced from a premise: a valid conclusion.

  • Typology (Ty-pol-o-gy):

    • Noun (plural: typologies); the systematic classification of types with common characteristics.

  • Threat (Threat):

    • Noun; an expression of an intention to inflict pain, injury, evil, or punishment.

Introduction: The Importance of Validity

  • The examination of validity in inference is crucial in scientific experiments, deriving from early psychology’s famous study of Clever Hans, where the horse’s apparent mathematical ability was proven to be a response to subtle cues from researchers.

Validity Explained

  • Validity Defined:

    • Refers to the approximate truth of an inference. A valid inference is supported by relevant evidence from empirical findings and existing theories.

  • Nature of Validity:

    • Validity is a property of inferences, not of designs or methods. Each design may contribute to varying degrees of validity depending on circumstances.

    • Valid judgments of validity are fallible and contingent on human judgment; thus, they are always tentative.

Types of Validity and Their Implications

  1. Statistical Conclusion Validity:

    • Validity regarding inferences about the covariance (correlation) between treatment and outcome.

  2. Internal Validity:

    • Validity concerning whether the observed relationship reflects a causal relationship due to the manipulations in the study.

  3. Construct Validity:

    • Validity of inferences regarding operationalizations mapping to higher-order constructs (e.g., how treatments relate to theoretical constructs).

  4. External Validity:

    • Validity about generalizing causal inferences across varied populations, settings, treatments, and measures.

Table of Validity Types

  • Table 2.1: Four Types of Validity

    1. Statistical Conclusion Validity: Covariance between treatment and outcome.

    2. Internal Validity: Causal relationship between manipulated A and measured B.

    3. Construct Validity: From observed particulars to overarching constructs.

    4. External Validity: Generalizability of causal claims.

Historical Context

  • Campbell's Definitions (1957):

    • Internal validity focused on whether the treatment made a significant difference.

    • External validity pertains to generalization to populations, settings, and variables.

Threats to Validity

  • Threats to validity represent specific reasons that can lead to incorrect inferences regarding covariance and causation. This section addresses statistical conclusion validity and internal validity threats.

Statistical Conclusion Validity Threats

Threat

Description

Low Statistical Power

Inability to detect true effects; insufficient sample size may lead to Type I and Type II errors.

Violated Assumptions

Statistical test assumptions may not hold, leading to unreliable inferences.

Fishing

Repeated tests inflate significance levels if uncorrected; leads to false findings.

Unreliability of Measures

Measurement errors attenuating relationships, affecting outcome analysis.

Restriction of Range

Variables restricted to a narrow range reduce apparent relationships.

Unreliability of Treatment Implem.

Variability in treatment implementation results in diluting perceived effects.

Extraneous Variance

External factors can inflate error variance, complicating effect detection.

Heterogeneity of Units

Variability among subjects leads to obscured relationships.

Inaccurate Effect Size Estimation

Misestimation can lead to incorrect conclusions regarding relationships.

Understanding the Threats

  • Low Statistical Power:

    • Defined as the probability of rejecting a false null hypothesis. Power calculation is essential in ensuring adequate sample sizes.

  • Violated Assumptions:

    • Many tests have robust properties against deviations, but some violations can severely distort statistical results, such as non-independence of observations leading to biased estimates.

  • Fishing and Error Rate Problem:

    • Involves generating multiple hypotheses; confounding occurs if adjustments for mates are not made leading to false significance claims (using the Bonferroni correction).

  • Unreliability of Measures:

    • When measures yield variable responses, the bivariate relationships weaken; reliability assessment is necessary.

Internal Validity Threats

Threat

Description

Ambiguous Temporal Precedence

Confusion over which variable occurred first affects interpretation of relationships.

Selection

Systematic pre-existing differences in respondent characteristics yielding confounding.

History

Concurrent events outside the control of the experiment are affecting outcomes.

Maturation

Naturally occurring changes in participants over time that might mimic treatment effects.

Regression

Tendency for individuals with extreme scores to score closer to the mean on retesting; often mistaken for treatment effects.

Attrition

Loss of participants leading to systematic differences between treatment and control groups.

Testing

Previous exposure to assessments influencing subsequent test performances.

Instrumentation

Changes in the measurement tools influencing results over time.

Additive Effects

Cumulative impact of interrelated threats affecting validity.

Addressing Internal Validity

  • Internal Validity ensures that observed relationships are truly causal. Adhering to thorough random sampling practices and monitoring treatment implementation strengthens inferential confidence.

Conclusion

  • The Relationship Between Internal and Statistical Conclusion Validity:

    • Both forms of validity are interrelated. Proper statistical analysis supports causal inferences, while clearer causal conditions improve the probability of accurate statistical conclusions. Moreover, qualitative assessments and research can function without direct reliance on quantitative statistical validation yet must still demonstrate covariation between the treatment and the effect for reliability.

By rigorously addressing these facets of validity, researchers can draw stronger, more reliable conclusions from their studies, ensuring that the inferences they make resonate with the underlying theoretical frameworks and empirical findings.