CRIM 220 - Lecture 5 - Experimental Designs

Lecture Information

Date: February 13, 2025Lecture Title: Experimental DesignsCourse: CRIM 220: Research Methods in CriminologyInstructor: Chelsey Lee

Experimental Designs

This lecture focuses on understanding various types of experimental designs specifically utilized in criminology research to determine causality and the efficacy of interventions. Experimental designs are crucial for establishing clear and precise conclusions about relationships between variables.

The Classical Experiment

Key Components:

• Independent and dependent variables: These are fundamental parts of any experiment, representing what you manipulate and what you measure.

• Pretesting and posttesting: These processes allow researchers to gauge the effects of the manipulation across time, providing insight into the changes that result from the experimental intervention.

• Experimental and control groups: Random assignment is a critical feature here to ensure groups are statistically equivalent, mitigating biases in the results.

Independent & Dependent Variables

• Independent Variables (IV): These are defined as the 'cause' in the causal relationship. Typically, they represent a dichotomous stimulus, meaning they can exist in two states (e.g., present or absent) and are manipulated by the experimenter to establish treatment conditions.

• Dependent Variables (DV): These are defined as the 'effect.' They represent the outcomes that are measured after the independent variable has been changed, allowing researchers to observe the impact of different values of the IV on the DV.

Pretesting & Posttesting

• Pretesting: This involves measuring the dependent variable before participants are exposed to the independent variable. It establishes a baseline for comparison.

• Posttesting: This involves measuring the dependent variable after exposure to the independent variable, allowing researchers to assess any changes.

Example of Pretest/Posttest

Participants are asked to watch a Public Service Announcement (PSA):

• Pretest: Measurement is taken before viewing the PSA.

• Posttest: Measurement is taken after viewing the PSA to assess any difference in responses or behaviors.

Experimental & Control Groups

• Experimental Group (EG): This group is the one exposed to the independent variable (the treatment or intervention).

• Control Group (CG): This group is not exposed to the independent variable, providing a comparison point against the experimental group.

Example of Pretest/Posttest with Groups

• Group 1 (EG):

  • Pretest: Watch PSA

  • Posttest: Measure change in attitudes or behaviors towards the message in the PSA.

• Group 2 (CG):

  • Pretest: Do not watch PSA

  • Posttest: Measure changes to evaluate the influence of the IV (PSA exposure) against no exposure.

Random Assignment

Importance: Random assignment is crucial as it ensures that every participant has an equal chance to be placed in either the experimental or control groups. This process minimizes systematic bias and ensures the groups are statistically equivalent, allowing for fair comparison of outcomes.

The Classical Experiment - The Gold Standard

The classical experiment is considered the gold standard in research due to its incorporation of all key features essential for valid experimentation:

• Clear definitions of independent and dependent variables

• Comprehensive pretesting and posttesting methodologies

• Inclusion of both experimental and control groups along with random assignment, enhancing the internal validity of the experiment.

Experiments & Causal Inference

Criteria for Causality: To establish a causal relationship, researchers must demonstrate:

• Empirical association: A relationship must be observed between the IV and DV.

• Temporal order: The cause must precede the effect in timing.

• Non-spuriousness: The relationship shouldn't be influenced by external or confounding variables.

Random assignment significantly aids in demonstrating causality, showing that changes in the dependent variable are a direct result of manipulation of the independent variable, rather than the result of other factors.

Threats to Internal Validity

Several common threats can compromise the validity of experimental findings:

• History: Events external to the experiment may confound the outcomes.

• Maturation: Natural changes in participants over time may influence results.

• Testing: The process of testing can itself affect the outcome, particularly if repeated measures are implemented.

• Instrumentation: Changes or inconsistencies in measurement tools can alter results.

• Causal Time Order: Failure to establish a clear causal sequence can lead to misleading conclusions.

• Regression to the Mean: Extreme scores tend to move towards the average upon retesting, complicating the assessment of treatment effects.

• Selection Bias: Non-random selection processes may skew sample characteristics, affecting results.

• Mortality (Attrition): Participant drop-out can impact the integrity and representativeness of data.

• Diffusion of Treatment: The treatment effect may spill over, affecting the control group.

• Compensatory Treatment: Offering compensatory measures to the control group can influence outcomes inadvertently.

• Compensatory Rivalry: Control group participants may push themselves to perform equally as treated participants.

• Demoralization: Feelings of deprivation in the control group can negatively affect study outcomes.

Theories for Each Threat

• History: External events unrelated to the study can interfere with results.

• Maturation: Growth or change in study subjects that occurs naturally over time can skew results.

• Testing: Previous testing may prime participants, altering responses in subsequent tests.

• Instrumentation: Variability in tools or methods of measurement can affect reliability.

• Causal Time Order: Inability to discern the order of cause and effect relations.

• Statistical Regression: Fluctuating scores may not reflect actual treatment effects.

• Selection Bias: Randomly assigning subjects is critical to avoid skewed results.

• Mortality: Loss of participants can bias results if dropouts are related to treatment effects.

• Diffusion of Treatment: Uncontrolled interactions between groups may contaminate results.

• Compensatory Treatment: Control groups receiving alternative aids may dilute differences.

• Compensatory Rivalry: Competition may arise between control and treatment groups.

• Demoralization: Participants in the control group may feel devalued if they are not receiving treatment.

Minimizing Threats to Internal Validity

To mitigate these threats, researchers should design and implement experiments with care, ensuring:

• Well-defined measures: Establish clear and objective metrics for outcome assessment.

• Maintaining group separation: Prevent interaction between experimental and control participants to uphold integrity.

• Balancing time between pretests and posttests: Consistent timeframes must be adhered to ensure that any changes can be reliably assessed.

• Offering incentives where necessary: Engaging participants may help in reducing dropout rates.

• Observing the potential for rivalry or demoralization: Monitor group dynamics to adjust studies proactively.

Threats to Construct Validity

It's important to consider how well the observations in an experiment reflect real-world processes and behaviors. Researchers should evaluate the reasonableness and completeness of the measures used to ensure they accurately represent the constructs being tested.

Threats to External Validity

Researchers need to examine whether results can be replicated across different contexts and populations. A balance must be struck between internal and external validity, prioritizing based on the specific goals of the study.

Threats to Statistical Conclusion Validity

Findings derived from small sample sizes tend to be less reliable and generalizable. Larger sample sizes can yield more robust and trustworthy results, minimizing the possibility of erroneous conclusions.

Issues with Experimental Designs

Ethical Considerations: Researchers often face challenges when it comes to denying treatment to participants in control groups, particularly when the treatment is believed to be beneficial. Additionally, certain variables may be unethical or impossible to manipulate.Practical Considerations: Cost implications and issues of intrusiveness in real-world applications can pose challenges to the feasibility of implementing certain experimental designs.

Building Blocks of Experiments

Research design must consider various factors, including:

• The number of treatment and control groups

• Variations in the independent variable used in the study

• The number of pre and post-test measures integrated into the design

• Established procedures for participant selection and assignment leading to balanced and fair representation of diversity in samples.

Notation for Experimental Designs

• R: Denotes random assignment

• X: Represents treatment (Independent Variable)

• O: Denotes no treatment (control)

• t: Represents time

Types of Experimental Designs

• Two-group posttest only: This is the simplest experimental format which guards against certain validity threats by comparing the outcomes of a treatment group against a control group without any pre-test measurement.

• Classical Experiment: This design incorporates pretest, treatment, and posttest methodologies, making it the gold standard for effectively controlling validity threats in research findings.

• Solomon Four-Group Design: An advanced layout combining the benefits of multiple designs by comparing results across groups, which receive both and do not receive pretests. This allows for comprehensive analysis of treatment effects.

• Factorial Design: This design allows researchers to examine multiple treatments simultaneously and their interactions, offering greater flexibility in testing various hypotheses.

Upcoming Events

• Next Week: Reading week - no class

• Research Proposal Part 1 Due: February 27 @ 11:59 PM.