Experimental Design Notes

EXPERIMENTAL DESIGN NOTES

SOME BASIC DESIGN CONCEPTS

• Sir Ronald Fisher described experiments as ‘only experience carefully planned in advance, and designed to form a secure basis of new knowledge’ (Fisher, 1935: 8).

• Characteristics of experiments:

  1. Manipulation of Independent Variables: At least one independent variable is manipulated.

  2. Use of Controls: Implementing controls by random assignment of participants or experimental units to treatment levels.

  3. Observation/Measurement of Dependent Variables: Careful measurement of one or more dependent variables.

• The first two characteristics differentiate experiments from other research methods.

• The emergence of modern science was marked by the emphasis on experimentation in the sixteenth and seventeenth centuries.

• Causal Relationships (Nineteenth-century philosophers): A causal relationship exists if:

  1. The cause precedes the effect.

  2. Whenever the cause is present, the effect occurs.

  3. The cause must be present for the effect to occur.

EXPERIMENTAL DESIGN

• Definition: An experimental design is a plan for assigning experimental units to treatment levels and the statistical analysis associated with it (Kirk, 1995: 1).

• Key activities in experimental design:

  1. Formulation of Statistical Hypotheses: Forming testable formulations of scientific hypotheses, which relate to population parameters or functional forms.

  2. Determination of Treatment Levels: Setting independent variable treatment levels, dependent variable measurements, and controlling for nuisance variables.

  3. Specification of Number of Experimental Units and Population: Identifying how many units are needed and from where they will be taken.

  4. Randomization Procedure: Defining how experimental units are randomly assigned to treatment levels.

  5. Statistical Analysis: Deciding on which statistical analysis will be performed (Kirk, 1995: 1–2).

RANDOMIZATION

• Cornerstone of Experimental Design: Originated from Fisher’s work.

• Purpose of Random Assignment:

  1. Distributes participant characteristics evenly across treatment levels.

  2. Allows for unbiased estimates of error effects.

  3. Ensures statistical independence of error effects, creating probabilistically similar groups.

QUASI-EXPERIMENTAL DESIGN

• Definition: Research that mimics an experiment but lacks random assignment (e.g., due to ethical reasons).

• Significance: Often leads to ambiguous interpretations of results due to the lack of control over extraneous variables.

REPLICATION AND LOCAL CONTROL

• Replication: Refers to observing two or more experimental units under the same conditions, providing estimates of error effects and treatment effects.

• Local Control (Blocking): Isolating variation attributed to nuisance variables using:

  1. Holding the nuisance variable constant.

  2. Randomly assigning units to treatment levels to distribute the nuisance variable across conditions.

  3. Including the nuisance variable as a factor in the experiment.

• Test Statistic Formula:
  $Test Statistic = rac{f(error ext{ effects}) + f(treatment ext{ effects})}{f(error ext{ effects})}$

ANALYSIS OF COVARIANCE

• Definition: Combines regression analysis with ANOVA to control for nuisance variables.

• Process: It involves measuring concomitant variables alongside the dependent variable to adjust for unaccounted variation.

• Advantages: It increases statistical power by removing predictable error variance related to the concomitant variable.

THREATS TO INTERNAL VALIDITY IN SIMPLE EXPERIMENTAL DESIGNS

ONE-GROUP POSTTEST-ONLY DESIGN

• Structure: Participants are exposed to treatment and then the dependent variable is measured without a control group or pretest.

• Key Concerns:

  • Rival hypotheses can arise, challenging the internal validity.

• Threats to Validity:

  1. History: External events impacting results between treatment presentation and measurement.

  2. Maturation: Changes in participants over the passage of time affecting results.

  3. Selection: Differences between participants in the experiment and a comparison group.

ONE-GROUP PRETEST-POSTTEST DESIGN

• Structure: The dependent variable is measured once before and once after treatment.

• Allows measurement of changes between pretest and posttest (Y̅.1 and Y̅.2).

• Introduces additional threats:

  1. Testing Effects: Familiarity with the testing situation affecting outcomes.

  2. Statistical Regression: Effects causing measurement scores to trend toward the mean over time.

  3. Instrumentation Changes: Alterations in data collection methods between tests.

• The internal validity can be improved with an additional pretest.

ONE-GROUP DOUBLE-PRETST-POSTTEST DESIGN

• Structure: Adds an additional pretest allowing for stronger controls over threats to validity.

• Threats still include history and instrumentation but may be lessened through repeated measures.

SIMPLE EXPERIMENTAL DESIGNS WITH CONTROL GROUPS

INDEPENDENT SAMPLES T-STATISTIC DESIGN

• Description: Participants are randomly assigned to treatment and control groups enhancing internal validity.

• Example: Evaluating medication effectiveness on smoking cessation with two levels (medication vs. placebo).

• Null Hypothesis: $H<em>0: ext{µ}</em>1 - ext{µ}<em>2 = ext{δ}</em>0$

• Significance of Random Assignment: Distributes characteristics equally to prevent bias; differences can be attributed to the treatment.

DEPENDENT SAMPLES T-STATISTIC DESIGN

• Use: Pairs of matched participants assigned to treatment levels.

• Null Hypothesis applies as before but focuses on easier comparison between matched individuals.

• Involves methods such as participant matching or repeated measures for each participant under all conditions.

SOLOMON FOUR-GROUP DESIGN

• Purpose: Controls all threats to internal and some threats to external validity.

• Framework: Random assignment to four groups where two groups receive pretests and two groups do not.

• Multiple null hypotheses can be formed based on comparison between treatment/ no treatment and pretest/no pretest.

DEMAND CHARACTERISTICS AND PARTICIPANT EFFECTS

• Identifiable as cues in the experimentation leading to indirect influence on the participant’s behavior through the following:

  1. Demand Characteristics: Hypothesis cues that influence participant behavior.

  2. Participant-Predisposition Effects: Participants may either aim to please the researcher or non-cooperate.

  3. Experimenter-Expectancy Effects: Researcher expectations inadvertently affect participant performance or data interpretation.

USE OF CONTROL IN EXPERIMENTATION

• A common method of maintaining internal validity involves using single-blind or double-blind procedures, minimizing biases.

ANALYSIS OF VARIANCE DESIGNS

COMPLETELY RANDOMIZED DESIGN

• Simplest ANOVA design involving random assignment to treatments.

• Null Hypotheses tested with standard F-statistics.

RANDOMIZED BLOCK DESIGN

• Extends the ANOVA to control for nuisance variables while assessing multiple treatments.

• Facilitates more powerful tests when compared to simple designs.

LATIN SQUARE DESIGN

• Further isolates two nuisance variables simultaneously, enhancing power over simpler designs.

GENERALIZED RANDOMIZED BLOCK DESIGN

• Variation that handles heterogeneous participant groups, isolating nuisance variables.

CONFOUNDING IN ANOVA DESIGNS

• Split-Plot Factorial Design: Manages impractically large block sizes through confounding treatment with block effects.

• Fractional Factorial Design: Reduces treatment combinations to decrease complexity but creates interpretational challenges.

HIERARCHICAL DESIGNS

• Developed for complex nesting of treatments where one or more treatments occur under another treatment level.

ANALYSIS OF COVARIANCE

• Offers a statistical method to reduce error variance through control of concomitant variables to improve statistical power and minimize bias.

• It combines regression with ANOVA to adjust means for the influence of other variables.

REFERENCES

• Anderson, N.H. (2001) Empirical Direction in Design and Analysis.

• Dean, A. and Voss, D. (1999) Design and Analysis of Experiments.

• Fisher, R.A. (1935) The Design of Experiments.

• Kirk, R.E. (1995) Experimental Design: Procedures for the Behavioral Sciences (3rd edn.).

• Others referenced in the original transcript as applicable.