Study Notes on Threats to Internal Validity
Threats to Internal Validity
History
Definition: Refers to historical events occurring during an experiment that may influence scores on the dependent variable.
Influence on Recruitment: Historical events may impact later participants due to the differentiation of exposure based on timing.
Examples of Historical Events:
Wars
Natural disasters
Terrorist attacks
Political changes
Financial crashes
Specific Effects:
If an event is related to the experimental variables, it can threaten internal validity.
Mitigation Strategies:
Conduct testing quickly to minimize historical impact.
Utilize control groups to equalize experiences of historical events across groups.
Selection Bias
Definition: Occurs when participant selection or group allocation results in unequal groups before the experiment, causing observed differences in the dependent variable.
Potential Outcomes:
Differences may stem from selection bias rather than independent variable effects.
Pre-existing differences can interact with treatment effects.
Example of Selection Bias:
Using a private school for the intervention group and a public school for the control may reveal differences unrelated to intervention efficacy.
Minimizing Selection Bias:
Implement random allocation of participants to ensure group similarity.
For critical variables (e.g., age), match participants prior to random assignment (e.g., evenly distributing three-year-olds across groups).
Post-Experiment Checks: Measure variables before the experiment to assess group differences post hoc.
Instrumentation
Definition: Refers to variations in how instruments score over time, impacting scores for the dependent variable.
Influence of Scoring Changes:
Variations in scorers (e.g., different individuals with varying leniency) can create systematic biases.
Even the same scorer may change due to improved skills or subjective judgments over time.
Manipulation Effects:
Changes in scoring instruments or manipulations may alter the nature of the treatment being assessed.
Mitigation of Instrumentation Threats:
Use objective and standardized scoring methods to reduce bias.
Employ multiple scorers and randomize scoring order.
Consider using a control group for comparison of scoring impacts.
Attrition
Definition: Occurs when participants do not complete the experiment, leading to incomplete data.
Impacts of Attrition:
May result in systematic bias if enough participants drop out, affecting group representation.
Types of Attrition:
Premature dropout: Participants leave before completing all conditions.
Differential dropout: Varying dropout rates between conditions threaten equivalence.
Health Studies: Particularly sensitive to attrition due to treatment perceptions (e.g., unpleasantness, ineffectiveness).
Consequences:
Remaining participants may no longer reflect the original random assignment.
Creates confounds impacting the interpretation of results (e.g., motivated participants remain in treatment).
Minimizing Attrition:
Assess manipulations to ensure they aren’t overly burdensome.
Compare dropouts to finishers regarding characteristics (requires pre-measurements).
Examine scores on dependent variables for both groups, when possible. Check for differences to identify biases influencing attrition.
History
Definition: Refers to historical events occurring during an experiment that may influence scores on the dependent variable.
Influence on Recruitment: Historical events may impact later participants due to the differentiation of exposure based on timing.
Examples of Historical Events:
Wars
Natural disasters
Terrorist attacks
Political changes
Financial crashes
Specific Effects:
If an event is related to the experimental variables, it can threaten internal validity.
Example: Studying stress levels throughout a semester; if a natural disaster or political change occurs at a pivotal moment, results may reflect that external influence rather than the independent variable.
Mitigation Strategies:
Conduct testing quickly to minimize historical impact.
Utilize control groups to equalize experiences of historical events across groups.
Consider incorporating pre-experiment surveys to gauge participants' awareness of ongoing events that could skew results.
Selection Bias
Definition: Occurs when participant selection or group allocation results in unequal groups before the experiment, causing observed differences in the dependent variable.
Potential Outcomes:
Differences may stem from selection bias rather than independent variable effects.
Pre-existing differences can interact with treatment effects, leading to confounded results.
Example of Selection Bias:
Using a private school for the intervention group and a public school for the control may reveal differences in socio-economic status, educational background, or motivation unrelated to intervention efficacy.
Minimizing Selection Bias:
Implement random allocation of participants to ensure group similarity.
For critical variables (e.g., age), match participants prior to random assignment (e.g., evenly distributing three-year-olds across groups).
Use stratified random sampling if certain characteristics (e.g., gender, ethnicity) are expected to influence results.
Post-Experiment Checks:
Measure variables before the experiment to assess group differences post hoc, ensuring those measures align with the dependent variable under study.
Instrumentation
Definition: Refers to variations in how instruments score over time, impacting scores for the dependent variable.
Influence of Scoring Changes:
Variations in scorers (e.g., different individuals with varying leniency) can create systematic biases that complicate the analysis.
Even the same scorer may change due to improved skills or subjective judgments over time. This inconsistency can lead to differing treatment evaluations.
Manipulation Effects:
Changes in scoring instruments or manipulations (i.e., modification in how assessments are administered) may alter the nature of the treatment being assessed, impacting the results for both experimental and control groups.
Mitigation of Instrumentation Threats:
Use objective and standardized scoring methods to reduce bias.
Employ multiple scorers and randomize scoring order to account for potential biases in individual assessments.
Consider using a control group to compare the impacts of various scoring processes on results.
Conduct regular calibration sessions for scorers to maintain consistency.
Attrition
Definition: Occurs when participants do not complete the experiment, leading to incomplete data. It can have profound implications for the interpretability of results.
Impacts of Attrition:
May result in systematic bias if enough participants drop out, affecting group representation. Attrition influences internal and external validity when the characteristics of dropouts differ from those who remain.
Types of Attrition:
Premature dropout: Participants leave before completing all conditions, skewing results.
Differential dropout: Varying dropout rates between conditions threaten equivalence, posing risks to the reliability of comparisons between groups.
Health Studies: Particularly sensitive to attrition due to treatment perceptions (e.g., unpleasantness, ineffectiveness). Participants who leave may do so for reasons that directly correlate with their experiences in the study.
Consequences:
Remaining participants may no longer reflect the original random assignment, invalidating assumptions based on randomization.
Creates confounds impacting the interpretation of results (e.g., motivated participants may remain in treatment while those who are disinterested drop out).
Minimizing Attrition:
Assess manipulations to ensure they aren’t overly burdensome, which may induce dropout, creating a confounding variable.
Compare dropouts to finishers regarding demographics, psychological state, or other relevant characteristics (requires pre-measurements).
Examine scores on dependent variables for both groups when possible, checking for differences to identify biases influencing attrition.
Implement strategies to improve retention, such as offering incentives or providing regular updates to participants about study progress and significance, fostering a sense of involvement.