Research Methods: Validity, Experimental Designs, and Non-Experimental Approaches

External Validity, Internal Validity, and Experimental Settings

• Internal Validity: Focuses strictly on the internal integrity of a study by controlling for extraneous variables, stranglers, and confounding variables to establish a clear relationship between conditions and outcomes.

• External Validity: An extension of internal validity that determines whether the study's findings can be externalized, generalized, or applied to the broader public or target population.

  • Target Population Scopes: Depending on the research sample, the broad public may refer to all people, all adults, a specific age group, or a specific gender.
  • Core Principle: Maintaining strict control over study conditions allows researchers to assert that the findings apply broadly to the target population.

• Ensuring External Validity via Field Experiments:

  • Field Experiments: Observing behavior in real-world environments and situations, commonly utilized by social psychologists.
  • Trade-offs: While field settings inherently contain extraneous and confounding variables, they capture real-life context where meaningful differences actually occur without relying on artificial or manufactured scenarios.

• External Validity in Lab Experiments:

  • Laboratory experiments can maintain high external validity provided there is robust, clear manipulation of Independent Variables (IVs), distinct differentiation between experimental conditions, and thorough control over extraneous variables.
  • Achieving a realistic approximation of real-life situations within the lab setting preserves external validity.

Between-Subjects Designs: Advantages, Disadvantages, and Sample Size Constraints

• Random Assignment and Between-Subjects (Independent Groups) Designs:

  • Mechanism: Participants are randomly assigned to independent, distinct groups (e.g., treatment condition vs. no-treatment/control condition).
  • Advantages: Ensures clear distinction between experimental conditions and unbiased allocation of participants.

• Disadvantages and Source of Variance:

  • Individual Background Differences: Participants bring varying personal backgrounds, traits, and baselines into independent groups.
  • Gender Identity Variance: Divergence across males, females, nonbinary individuals, and other gender identities.
  • Baseline Differences: Variations in baseline cognitive or psychological measures, such as baseline memory performance or baseline subjective meaning in life.
  • Impact on Dependent Variables (DVs): Pre-existing participant variance can directly influence and distort the DV measurements.

• Sample Size and Allocation Issues:

  • Uneven Sample Size Risk: Improper random assignment or execution flaws can result in unequal participant counts across conditions.
  • High Sample Size Requirements: Between-subjects designs require significantly larger total participant pools. For example, needing a total sample size of $N = 50$ yields only $n = 25$ participants per experimental condition.

Within-Subjects Designs, Noise, and Carryover Effects

• Participant Variance as "Noise":

  • Variance in individual baseline states (such as varying levels of anxiety, depression, memory recall capability, or perceived meaning in life) introduces unwanted statistical "noise."
  • Participant bias similarly injects noise into a study sample.

• Purpose of Within-Subjects Designs:

  • Within-subjects designs are implemented specifically to minimize statistical noise by utilizing the same participants across all experimental conditions.
  • Because each participant serves as their own control, participant characteristics remain identical across conditions. Consequently, observed changes in the dependent variable can be confidently attributed to the experimental treatments/conditions rather than individual differences.

• Demonstration Setup Example:

  • All participants complete Treatment A (e.g., sad condition), followed by Treatment B (e.g., happy condition), after which an outcome variable (e.g., memory performance) is measured.

• Advantages of Within-Subjects Designs:

  • Controls effectively for individual extraneous variables.
  • Substantially reduces statistical noise.
  • Ensures equal sample sizes across conditions and increases overall statistical power per condition without increasing total participant count.

• Disadvantages and Threats to Internal Validity:

  • Carryover Effects: Performance or responses in subsequent conditions are influenced by participation in prior conditions.
  • Practice Effect: Participants improve at a task (such as memory recall) purely through repeated exposure to stimuli (e.g., seeing a word list multiple times).
  • Contextual Confusion: Participants experience confusion due to prolonged study duration or specific stimulus properties (e.g., perceiving illusions in list words), which is difficult to control.
  • Fatigue Effect: Participant performance declines over time due to tiredness or boredom.

• Counterbalancing as a Mitigation Tool:

  • Mechanism: Varying the order of condition presentation across different sub-groups of participants.
  • Example: If experiencing a sad condition increases memory recall compared to a happy condition, presenting the sad condition last could bias results. Counterbalancing evaluates one group receiving the sad condition first then happy condition, and another receiving happy condition first then sad condition, comparing differential order effects.

Non-Experimental Research: Correlational and Single-Variable Designs

• Defining Features of Non-Experimental Research:

  • Lacks active manipulation of independent variables.
  • Evaluates pre-existing conditions, natural distributions, or associations.

• Correlational Research:

  • Examines relationships, links, or associations between two quantitative variables without inferring direct causality.
  • Visualized standardly using scatter plots.
  • Complex Correlational Designs: Can extend beyond simple quantitative pairs to incorporate qualitative parameters (e.g., comparing individuals categorized with high stress versus no stress against secondary outcomes).

• Single-Variable Research:

  • Focuses exclusively on measuring the frequency, prevalence, or rate of a single variable, rather than assessing relationships between two variables.
  • Data Representation: Frequently uses histograms to display distributions and frequencies.
  • Case Example - Milgram Study: Examined a single primary variable: whether individuals would administer electric shocks when commanded by an authority figure, without comparing associations across manipulated conditions.
  • Case Example - National Comorbidity Survey: Evaluates the prevalence and frequency rates of mental health disorders across populations.
  • Empirical Frequency Examples (Lecture 7 Demonstrations):
    • Approximately one-third ($\frac{1}{3}$) of Americans meet criteria for a psychological disorder at some point during their lifetime.
    • Average highway driving speed measured at $87\,\text{mph}$.
    • Proportion of people reporting an alien encounter measured at $1\text{ in }3$.

Quasi-Experimental Research, Pretest-Posttest Designs, and Qualitative Methods

• Observational Non-Experimental Research:

  • Typically conducted in field settings to observe naturally occurring effects across pre-existing groups (e.g., individuals with clinical anxiety versus clinical depression).

• Qualitative Research Methods:

  • Focuses on collecting non-quantitative narrative data through focus groups, structured interviews, or open-ended responses.
  • Case Study (Syllabus Satisfaction Project):
    • Combines quantitative rating scales with qualitative open-ended text fields where participants provide detailed feedback on syllabus defects.
    • Analytical Requirements: Qualitative text data requires systematic coding and categorization into explicit thematic categories.

• Quasi-Experimental Research:

  • Defining Feature: Resembles true experimental design but explicitly lacks random assignment of participants.
  • Relies on naturally occurring pre-existing groups or non-equivalent control groups.
  • Pre-existing Group Categorizations: Gender, college major, race, ethnicity, relationship status, educational attainment, and generational status (e.g., first-generation vs. continuing-generation college students).

• Quasi-Experimental Implementation Example (Reading Curriculum Study):

  • Objective: Test the efficacy of a new third-grade reading technique on reading comprehension.
  • Design: Rather than randomly assigning individual students, pre-existing whole units are used.
  • Baseline: Both School A and School B start at the same baseline old curriculum.
  • Treatment Application: School A receives the new reading program (treatment group), while School B continues with the old reading program (control group).
  • Evaluation: Post-intervention comprehension scores are compared between School A and School B to assess statistical significance.

• Pretest-Posttest Design:

  • Involves longitudinal tracking across two distinct temporal points.
  • Procedure Structure:
    1. Administer initial baseline measurement (pretest, e.g., memory recall).
    2. Apply experimental intervention or stimulus (e.g., exposure to a sad video).
    3. Administer secondary measurement (posttest) to evaluate differential change resulting from the intervention.

Continuum of Internal Validity and Selection of Research Designs

• Internal Validity Hierarchy Across Designs:

  1. Correlational Research: Lowest internal validity; identifies associations between variables without controlling for confounds or establishing directionality/causality.
  2. Quasi-Experimental Research: Intermediate internal validity; higher than correlational due to specific treatment application, but lower than true experiments because naturally occurring group differences exist and random assignment is absent.
  3. True Experimental Research: Highest internal validity; utilizes full manipulation of IVs, strict control of extraneous variables, and random assignment.

• Criteria for Design Selection:

  • Single-Variable Design: Selected when the primary objective is purely descriptive (e.g., recording the frequency rate of Major Depressive Disorder in a given population).
  • Correlational Design: Selected when evaluating non-causal relationships or predictive links between variables across broader populations.
  • Between-Subjects / Field Design: Preferred when real-world variance and external generalizability to the general public are prioritized.

Questions and Discussion

• Participant Question (Darren): Inquired about specific examples of how diversity among individual participants causes statistical noise within a within-subjects experimental design.
• Detailed Response: Baseline diversity—such as variance in participants' underlying levels of anxiety, depression, or subjective perception of meaning in life—creates varying initial benchmarks. This baseline divergence introduces extra noise into the data, directly impacting individual scores on outcome measures like memory performance or meaning-in-life scales. Implementing a within-subjects design controls for this variance by holding individual baseline factors constant across conditions.