FSCI 5353 Lecture 7: Intro to Experimental Design Part 2 notes

Quasi = “to a certain degree”
Quasi-Experiment: An experiment to a certain degree
Use when randomization is not possible; results in less control over validity
When cannot randomize, cannot assume equivalency among the groups
Use nonequivalent groups designs
Need to make groups as comparable as possible
Match subjects in experimental and control groups using important variables related to DV under study

Validity: The extent to which a metric in an experiment actually measures the concept it purports to measure
Internal validity: The ability of an experimental design to document the causal relationship between an IV and a DV
Refers to possibility that conclusions drawn from experiment reflect what actually went on in experiment; the likelihood of causality
Whether observed associations between 2 (or more) variables are, in fact, causal associations
For example: I did X to A, and because of that, I got B. My application of X directly caused the impact on A to become B. This is causality.

History: External events may occur during course of study
- E.g., natural disaster occurs during data collection
Maturation: People constantly are growing/aging, and may naturally produce different responses over time
Testing: The process of testing and retesting
- E.g., If subjects take same test over and over, may be trying to recall previous responses rather than indicating responses for the present.
Regression: Though may be an initial treatment effect, effect diminishes over time, indicating IV had no long-term effect.
Experimental mortality/attrition: Subjects may drop out prior to completion of experiment
- Reasons for drop out may vary: “study requires too much time”, “it’s boring,” etc.
Instrumentation: Changes in the measurement process
- E.g., Instrument isn’t used the same way in successive measurements, due to operator error or a different person using the instrument differently/incorrectly
Selection bias: Way in which subjects are chosen (i.e., not randomly)
- E.g., Selecting only your friends to participate

Generalizability: Quality of a research finding that justifies the inference it represents something more than the specific observations on which it was based.
AKA: Do the results of an experiment really tell us what would happen in the real world (i.e., a non-experimental setting)?
External validity: Whether the results from experiments in one setting will be obtained in other settings
Threats are more significant for experiments conducted under carefully controlled (i.e., lab) conditions rather than more natural conditions (i.e., observation at the mall)
However, more controlled conditions reduce internal validity threats
Internal validity must be established before external validity is an issue

Reactivity: An awareness that they are being measured causes a change in the behavior of subjects
Interaction between selection bias and the DV: A failure to ensure that subjects assigned to the experimental and control groups are equivalent
Statistical conclusion validity: Whether we are able to determine if two variables are related
Becomes an issue when findings are based on small samples
More cases allows you to reliably detect small differences; less cases result in detection of only large differences

Benefits:
- Ability to isolate the effect of an IV on a DV
- Ability to measure how much of an effect a treatment has on an outcome
- Ability to demonstrate causality
Limitations:
- Requirement of much time, money, control
- Potential for serious ethical concerns
- Possible lack of feasibility