External Validity and Replication

External Validity

  • Definition: Concerns whether the results of a study are true for populations beyond the sample used in the study.

  • Generalization Examples:

    • Generalizing to other colleges.

    • Generalizing across gender.

    • Generalizing to other ethnic groups.

  • Limitations:

    • A limitation on generalizability can be a type of interaction.

    • Example: Dutton and Aron (1974) study.

    • Generalizing to other situations can be limited by:

      • Effect of testing on generalizability.

      • Practice effects.

      • Reactivity.

      • Interrupted time-series design.

      • Field Studies and Naturalistic Observation

        • Time sampling.

        • Situation sampling.

      • Limits of generalizing from volunteers.

  • Best Evidence: Replication with different populations, situations, and times.

  • Acceptance: Acknowledge that no single study can answer all questions.

Replication Studies

1. Exact Replication

  • Purpose:

    • Relate results to previous results.

    • Confirm surprising or unexpected results ("").

  • Success: Provides evidence for generalizability.

  • Failure: Does not negate original findings due to the probability of an incorrect decision.

2. Conceptual Replication

  • Definition: Replication of the conceptual relationship between variables using different procedures and participants.

    • Example: "Insults increase aggressive behavior."

      • To replicate with different age groups, age-appropriate insults must be used.

      • Measure the dependent variable (DV) in operationally different ways.

  • Importance: Replications in many different situations increase confidence in the generalizability of the results.

Literature Review & Meta-Analysis

  • Literature Review: Conclusions are somewhat subjective.

  • Meta-analysis

    • A set of statistical procedures for comparing and combining results across studies.

    • Goal: Examine many studies exploring common variables to draw conclusions about the relationship between variables.

    • "File drawer phenomenon".

Interaction Examples

  • Example 1:

Subjects who studied the word list while listening to jazz music recalled more words when the list was printed on blue paper, and subjects who studied while listening to classical music recalled more words when they were printed on yellow paper. This indicates an interaction between music type and paper color.

Music Type×Paper Color\text{Music Type} \times \text{Paper Color}

  • Example 2: Right-handed and left-handed subjects both caught a larger number of large balls than small balls. No interaction exists as both groups performed similarly.

Handedness×Ball Size\text{Handedness} \times \text{Ball Size}

  • Example 3: Subjects exhibited higher levels of anxiety when the noise was loud and high-pitched and when the noise was quiet and low-pitched. Other combinations of loudness and pitch produced less anxiety. This demonstrates an interaction between loudness and pitch.

Noise Level×Pitch\text{Noise Level} \times \text{Pitch}