chapter 11
More on Experiments: Confounding and Obscuring Variables
Illustration: A group of young boys laughing and holding water guns exhibits the question, "Was it really the intervention, or something else, that caused things to improve?"
Interpreting Null Results
Illustration: Two bowls of salsa, one with a chip in it and the other with a bottle held over it poses the inquiry, "How should we interpret a null result?"
Learning Objectives
By the end of the chapter, you should be able to:
Interrogate a study and decide whether it rules out 12 potential threats to internal validity.
Describe how researchers design studies to prevent internal validity threats.
Analyze an experiment with a null result to determine if the study design obscured an effect or if there is truly no effect.
Explain how researchers can minimize possible obscuring factors in study design.
Chapter Overview
Previous Chapters: Covered the basic structure of an experiment.
This Chapter: Addresses questions about experimental design, such as:
Importance of a comparison group.
Reasons experimenters create standardized and controlled environments.
The rationale behind having many participants.
The usage of technologies for measuring variables.
The necessity for double-blind study designs.
Objective: Establish clear and responsible experimental designs to estimate real effects and understand when predictions are incorrect.
Threats to Internal Validity
Key Question: Did the independent variable really cause the difference?
Focus on Internal Validity: It is crucial to interrogate experiments for potential internal validity threats.
Common Threats:
Design Confounds: Poor design leads to other variables systematically varying with the independent variable, which complicates causal interpretations.
Example: The poster study tested if a positive norms poster improved the inclusive climate; however, differences in waiting room conditions introduced confounding.
Selection Effects: Participant characteristics systematically differ between independent variable groups, leading to biased results.
Example: The intensive therapy for autism study had parents self-select into groups, impacting outcomes.
Order Effects: Effects may arise from the order of conditions presented in a within-groups design, complicating the attribution of results to the treatment.
Example: Tiredness or boredom may affect results if the same participants undergo multiple conditions in succession.
Total Internal Validity Threats: Estimated to be about 12 potential threats in total.
The Really Bad Experiment (Cautionary Tale)
Three Fictional Experiments:
Nikhil's Study: 15 boys showed behavioral changes after diet changes, but alternative interpretations (natural behavior maturation) are possible.
Dr. Yuki's Study: 40 women with depression showed lower symptoms post-therapy, yet spontaneous remission may explain results.
Go Green Campaign: University dorm reduced energy usage; seasonal adjustment could be an alternative explanation.
Common Design Template: All examples show lack of proper controls (i.e., comparison groups) leading to unreliable conclusions on causation.
Internal Validity Threats in One-Group, Pretest/Posttest Designs
Six Potential Threats:
Maturation Threats: Changes that occur naturally over time, independent of the experimental variable.
History Threats: Changes that occur due to external factors affecting the treatment group during the study period.
Regression Threats: Tendency for extreme scores to revert closer to the mean upon retesting.
Attrition Threats: Loss of participants systematically affecting study results (e.g., people with extreme scores dropping out).
Testing Threats: Changes in participants’ scores due to repeated exposure to testing.
Instrumentation Threats: Changes in measurement instruments or procedures over time affecting results.
Maturation Effects Explained
Description: Maturation refers to improvements in participants' behavior that occur naturally as they adjust to their environment.
Examples:
Nikhil's campers could have improved behavior due to adaptation rather than diet changes.
Depression symptoms in Dr. Yuki's study might have improved over time without treatment.
Prevention Strategy: Including a control group would help rule out maturation effects by comparing with similarly aged participants not receiving the treatment.
History Effects Explained
Description: History threats arise from external events that influence the group’s behavior during the experiment.
Example: Go Green Campaign results may appear due to seasonal temperature drops rather than campaign effectiveness.
Prevention Strategy: Measuring a comparable group’s outcomes during the same period to account for external factors.
Regression to the Mean Explained
Concept: Extreme values on repeated measures are likely to move closer to the average on subsequent measures.
Example: Women's depression scores in Dr. Yuki's study might show improvement due to their initial extreme levels rather than the treatment effect.
Prevention Strategy: Include an appropriate comparison group, ensuring groups are equally extreme at pretest.
Attrition Effects Explained
Concept: Participant dropout can skew results, particularly if dropouts have extreme scores.
Example: If the most unruly campers drop out of Nikhil's study, results could falsely appear as though the intervention was effective.
Prevention Strategy: Adjust posttest averages based on participants who completed the study, analyzing dropout characteristics.
Testing Effects Explained
Concept: Familiarity with a test can improve results or fatigue may degrade posttest scores.
Example: Performance may improve on a second attempt not due to treatment but due to repetition.
Prevention Strategy: Use alternative measures across different testing sessions or omit pretests in favor of posttests only.
Instrumentation Effects Explained
Concept: Changes in measurement tools or procedures can alter findings.
Example: Coders may become more lenient over time, creating misleading data analysis.
Prevention Strategy: Ensuring all measuring tools are calibrated and retraining personnel as needed.
Observer Bias and Demand Characteristics
Observer Bias: When researchers’ interpretations are skewed by their expectations, affecting internal validity.
Demand Characteristics: Participants change behavior due to awareness of the study purpose, thus confounding results.
Solutions: Implement double-blind or masked designs to shield both participants and experimenters from biases.
Placebo Effects
Description: Improvement in participants due to belief in treatment efficacy, not the treatment itself.
Illustration: Placebo effects can lead to significant symptom reductions, complicating the interpretation of treatment effectiveness.
Design Strategy: Utilize double-blind placebo control studies to differentiate placebo influences from true effects.
Null Effects and Their Implications
Understanding Null Effects: Reflects scenarios where no statistically significant difference is found.
Investigating Null Causes: Potential obscuring factors relate to manipulations or measurements being ineffective, variability within groups being too high, or no true effects existing.
Importance of Reporting: Null results must be transparently documented to inform scientific understanding and guide future research.
Addressing Obscuring Factors
Two Main Strategies:
Increase between-group distinctions by improving manipulations and clarifying dependent measures.
Reduce within-group variability through larger sample sizes, repeated measurements, and controlled experimental environments.
Conclusion on Null Results
Importance: Recognizing and publishing null results supports scientific integrity, inviting further investigation and theory revision.