Ch: 13 Quasi-Experimental and Small-N Research Designs

Overview of Quasi-Experimental Designs

Analysis of Quasi-Experiments: Quasi-experimental designs are analyzed using the same statistical methods as true experiments. * If there are two independent variables (IVs), a two-way ANOVA is used. * If there is one IV, a one-way ANOVA is used.
The Key Distinction: The primary difference that distinguishes a quasi-experiment from a true experiment is the lack of random assignment.
Reasons for Utilizing Quasi-Experimental Designs: * Participant Variables: Researchers often want to test variables that cannot be manipulated or assigned, such as gender. A person cannot be randomly assigned to be male or female. * Lack of Control over Service Access: Researchers cannot always control which group a participant joins. For example, in comparing a free clinic versus a paid clinic, participants choose the clinic based on their needs and resources; the researcher cannot assign them. * Outside Programs/Policy Allocation: Some programs are allocated by external entities. For instance, scholarships are awarded based on merit or specific criteria, meaning a researcher cannot randomly assign who receives one. * Political or Ethical Constraints: Creating a random control group can be politically incorrect or face pushback. For example, in a senior living facility, it might be unfair to allow some residents to participate in a meditation program while barring others who wish to join. In such cases, one might compare those who already participate versus those who do not.
Causal Inference: Because there is no random assignment, researchers cannot make causal inferences (cause-and-effect statements).
Selection Effects: A major concern in quasi-experiments is that groups may have been different from the start. For example, students who choose their own assignment topics might already be more motivated or persevere more than students who prefer assigned topics.

Types of Quasi-Experimental Designs

Nonequivalent Control Group Design

Definition: This design compares two groups (a control group and an experimental group) that have not been randomly assigned.
Example: High School Spanish Learning: * Scenario: A principal wants to test if an immersive experience (interacting with a Spanish-speaking family, cooking with them, and writing about it) is better for learning the language than traditional textbook-based classroom learning. * Quasi-Experimental Design: Instead of randomly assigning students within one class (which might cause pushback), the principal could compare students at one high school using the new method with students at another school in the same district using the traditional method. * Mitigating Initial Differences: To address concerns that one school may already be better at Spanish, researchers can use a pretest at both schools, followed by the program, and then a post-test to measure growth.

Time Series Designs

General Definition: Time series designs involve taking multiple measures over a period of time before and after an intervention is introduced. This is common in applied research, nonprofits, and public policy evaluation.
Interrupted Time Series Design: * Data is collected over a period of time before and after a specific "interruption" or event (like a new law). * Example: Speeding Laws: Measuring traffic fatalities from November until March, introducing a zero-tolerance law in March, and then measuring again after March. * Multiple measures are necessary to establish a trend and ensure the data isn't just an outlier due to holidays or specific accidents (like an overturned big rig).
Multiple Time Series Design (Nonequivalent Control Group's Interrupted Time Series Design): * This is an interrupted time series design that adds a nonequivalent control group to increase confidence. * Example: Philadelphia Soda Tax (2017): * Philadelphia introduced a tax on sugary soda to reduce diabetes rates. The price rose from $5.43\,\text{cents}$ per ounce to $6.24\,\text{cents}$ per ounce. * Interrupted Phase: Researchers measured sales in Philadelphia before and after the tax. Sales went down. * Multiple Phase: Researchers also measured sales in the city next door to Philadelphia (the control group). * Findings: Soda sales in the neighboring city went up because residents just crossed the border to buy cheaper soda. Without the control group, Philadelphia might have falsely concluded the health initiative was working purely due to the tax.

Threats to Internal Validity in Quasi-Experiments

Selection Effects: Groups are already different before the study begins. Pre/post-tests or matching can help mitigate this.
History Threat: An external event unrelated to the study influences the data. In the speeding fatality example, weather could be a history threat. A neighboring state with similar weather can serve as a control to manage this.
Maturation: Changes occurring naturally within the participant over time (e.g., getting tired, hungry, older, or more health-conscious). For a longitudinal smoking cessation study, older participants might quit because they naturally become more concerned about health, not because of the intervention.
Change in Criteria (Instrumentation): Changes in how a variable is defined or measured over time. * Example: Autism Diagnoses: The increase in autism cases correlates with the removal of Asperger's as a separate diagnosis in the DSM (Diagnostic and Statistical Manual). Asperger's cases went down and were grouped into Autism, changing the frequency through a shift in criteria rather than an actual increase in the condition.

Small-N Designs

Definition: Research designs using very small sample sizes, often used for rare/unique cases (e.g., split-brain patients) or in clinical settings like Applied Behavioral Analysis (ABA).

Three Types of Small-N Designs

Stable Baseline Design: * A researcher observes behavior for an extended period to establish a stable baseline before introducing the intervention. * Example: Measuring how many times per week a person has a migraine or how many times a shy child makes eye contact before treatment.
Reversal Design (ABAB Design): * A: Initial Baseline period. * B: Treatment/Intervention introduced. * A: Treatment removed (return to baseline). * B: Treatment reintroduced. * The second "AB" acts as a replication to ensure the behavior change was due to the treatment and not a "random fluke." * Ethics: Treatment should not be reversed if the behavior change is necessary for safety (e.g., stopping shoplifting) or if removing treatment would be harmful.
Multiple Baseline Design: * Across Settings/Situations: Testing if a treatment for aggressive behavior works at both school and home. * Across Behaviors: Testing if an intervention reduces both aggressiveness and distractibility in one person. * Across Participants/Subjects: Testing if a treatment that worked for one person (e.g., Tammy) also works for another (e.g., Patty).

Validity in Small-N Designs

Internal Validity: Can be very high if the researcher is highly controlled and only manipulates the intervention.
External Validity: Generally low due to small sample size; however, in many clinical settings, the goal is to treat the specific individual (e.g., Natalia or Auklia), making generalizability less of a priority.
Statistical Validity: Often not relevant because standard statistical analyses cannot be run on $n = 5$ or $n = 10$ ; researchers rely on tracking general trends via graphs.
Construct Validity: Can be very high if operational definitions are precise (e.g., clearly defining what counts as a "correct" math problem, such as whether a rounding error makes it wrong).

Questions & Discussion

Q: How was the Nonequivalent Spanish study assigned?A: Students at one school get the new method, and students at another school in the same district get the traditional method. Their scores are then compared.

Q: Does a longitudinal study negatively affect a nonequivalent control group design?A: No, the study can be done over any period. In the Spanish example, it was just measured at the beginning and the end of the year.

Q: Practice Case: New Jersey birth rates and financial assistance.

The Scenario: New Jersey stopped giving extra cash to women on public assistance who had more than one child. Birth rates for those on assistance dropped by $12\%$ . However, researchers found no difference between those subject to the new policy and those "grandfathered" in under the old policy.
Design Used: This was a multiple time series design.
Evaluation: It was a better design than an interrupted time series because the comparison group (those previously enrolled) provided extra information. A better control would have been women not on assistance at all to see if birth rates were dropping for everyone.
Additional Info needed: Demographic information, cultural shifts, health care access, and the state of the economy (history threats).