Comparative Experiments and Experimental Design Methodology

Observational Studies versus Comparative Experiments

  • Observational Studies vs. Experimental Studies:
    • In statistics, terminology is often used loosely. For instance, observational devices or programs are frequently referred to as "experiments," even when no treatments are applied or manipulated.
    • Observational Study Example: The Nancy Grace Roman Space Telescope gathers data from across the universe without applying experimental treatments. While scientists currently know of approximately 5,0005,000 exoplanets (planets orbiting other stars), this telescope is designed to discover potentially 100,000100,000 exoplanets and analyze them closely for atmospheric signatures. Despite its scientific scope, it remains an observational study consisting of accumulated observations.
    • Comparative Experiments: A true comparative experiment applies specific treatments to experimental units and explicitly links explanatory variables to response variables using comparison.

Terminology and Elements of Experimental Design

  • Core Concepts and Definitions:

    • Experimental Unit: The specific object, plot, or entity receiving a treatment in a study.
    • Subject: The term designated for an experimental unit when the unit receiving treatment is a human being.
    • Explanatory Variable / Factor / Independent Variable / Predictor Variable: A variable that is manipulated or categorized to determine its potential influence on the response variable.
    • Levels: The specific values, states, or qualitative/quantitative variations that a factor variable can take.
    • Treatments: The specific combinations of factor levels applied to experimental units. When an experiment involves only one factor, the treatments are identical to the levels of that single factor. When multiple factors are present, treatments consist of every combination of factor levels across those variables.
    • Response Variable / Dependent Variable: The quantitative outcome measured after treatments have been administered.
  • Example: Crop Yield and Fertilizer Application Study:

    • Research Question: Does a new fertilizer formula impact crop yields compared to an old formula?
    • Experimental Units: The plots of land on which crops are grown (or the crops themselves via the land plots).
    • Factor: Fertilizer formula (11 factor).
    • Levels: Old formula and New formula (22 levels).
    • Treatments: Application of the Old formula or the New formula.
    • Response Variable: The weight of the harvested crops after being dried.
  • Clarifications on Factor Levels:

    • Levels represent all the specific ways in which a factor variable varies. They are not necessarily numeric values; they can represent qualitative differences (e.g., distinct formulas, different instructional formats, or specific drug dosages).

Blinding, Placebos, and Control Groups

  • Placebos and the Placebo Effect:

    • Placebo: A non-treatment or inert substance (such as a inactive pill) administered in place of an active treatment.
    • Placebo Effect: A psychological phenomenon where human subjects show a positive physical or clinical outcome purely due to the belief that they are receiving a beneficial treatment.
  • Blinding Protocols:

    • Single-Blind Protocol: The subject does not know which treatment level or placebo they are receiving.
    • Double-Blind Protocol: Neither the subject receiving the treatment nor the researcher administering/measuring the outcome knows which treatment assignment the subject has received. Study codes are utilized and only revealed/decoded after all measurement data has been collected.
    • Purpose: Eliminates both psychological subject bias and subconscious non-verbal communication or cues from researchers administering treatments.
  • Control Groups:

    • Control Group: A baseline experimental group that receives either zero dosage of an active ingredient or a non-treatment placebo, allowing researchers to isolate whether observed changes are caused by the active treatment itself.
  • Example: Drug X Fever Reduction Study:

    • Research Question: Does Drug X reduce fever in adults?
    • Study Setup: A sample of 800800 adult fever patients are randomly assigned to receive either 0 mg0\,\text{mg}, 325 mg325\,\text{mg}, or 500 mg500\,\text{mg} of Drug X. Patient body temperatures are measured exactly 1 hour1\,\text{hour} post-administration.
    • Explanatory Variable / Factor: Dosage of Drug X.
    • Levels: Three dosages (0 mg0\,\text{mg}, 325 mg325\,\text{mg}, and 500 mg500\,\text{mg}).
    • Treatments: Administration of 0 mg0\,\text{mg}, 325 mg325\,\text{mg}, or 500 mg500\,\text{mg} dosage levels.
    • Experimental Units / Subjects: The 800800 adult patients.
    • Response Variable: Patient temperature measured 1 hour1\,\text{hour} after administration.
    • Comparisons: Pairwise comparisons across all three groups (325 mg325\,\text{mg} vs. 0 mg0\,\text{mg}, 500 mg500\,\text{mg} vs. 0 mg0\,\text{mg}, and 325 mg325\,\text{mg} vs. 500 mg500\,\text{mg}).

Three Fundamental Principles of Experimental Design

  • 1. Comparison:

    • Directly compares active treatments against each other or against control/placebo groups to quantify changes in the response variable.
    • Identifies potential lurking variables—unobserved or confounding factors (e.g., pre-existing health conditions) that might influence outcomes independently of the experimental treatment.
  • 2. Randomization:

    • Uses chance assignment to allocate experimental units to treatment groups, neutralizing human bias and ensuring equal probability of receiving any given treatment.
    • Example: If there are 33 possible treatment levels, each subject has exactly a 13\frac{1}{3} probability of being assigned to any single treatment.
  • 3. Repetition:

    • Applies treatments to a sufficiently large number of experimental units within a single study to minimize chance variation and anomalous outcomes.
    • Repetition vs. Replication Distinction:
      • Repetition: Administering treatments to a large sample of units within the same experiment to reduce random statistical noise.
      • Replication: Re-executing an entire experimental study from scratch to evaluate the consistency, replicability, and reproducibility of the results.

Experimental Design Schematics and Types

  • Completely Randomized Design (CRD):

    • All experimental units are assigned to treatments purely through random chance with equal probability.
    • Example: University Instructional Methods Study:
      • Setup: Evaluates which instructional method yields the best results for university students using a large sample of 9090 students.
      • Treatments: Self-study, Traditional classroom, or Online instruction (33 active treatment methods).
      • Procedure: The 9090 students are randomly allocated into 33 equal groups of 3030 students per instruction method.
      • Response Variable: Final exam scores compared across all three instructional methods.
      • Note: Control groups/placebos are not applicable or possible here because subjects actively participate in and are aware of their mode of instruction.
  • Randomized Block Design (RBD):

    • Controls for identified, known lurking variables by grouping experimental units with similar characteristics into blocks before random assignment to treatments takes place.
    • Example 1: Instructional Methods Study with Academic Status Blocking:
      • The 9090 students are first divided into two distinct blocks based on academic level: Undergraduate and Graduate.
      • Within each academic block, students are randomly assigned to one of the 33 instructional methods (Self-study, Traditional, Online).
      • Exam scores are compared both within each block and between the blocks.
    • Example 2: Dairy Cow Grain Mix and Milk Production Study:
      • Setup: A Simple Random Sample (SRS) of 9090 dairy cows is selected across multiple farms to evaluate how grain mix affects milk production.
      • Blocking Variable: Breed of cow (subjects are separated into distinct breed blocks).
      • Factor: Type of grain mix (22 levels: Mix A and Mix B).
      • Treatments: Mix A and Mix B.
      • Procedure: Within each breed block, exactly half of the cows are randomly assigned to eat Mix A, and the other half eat Mix B. Milk production is measured after 1 month1\,\text{month}.
      • Response Variable: Milk production volume recorded after 1 month1\,\text{month}.
      • Experimental Design: Randomized block design grouped by cow breed.

Matched Pairs Designs and Crossover Protocols

  • Matched Pairs Design:

    • A specialized form of experimental design that controls for lurking variables by pairing highly similar units or using each unit as its own internal control.
  • Method 1: Paired Similar Experimental Units:

    • Example: Oxygen Inhalation and Football Player Recovery Study:
      • Setup: A sample of 6060 football players is selected.
      • Pairing Procedure: All 6060 players run a 100 yard100\,\text{yard} dash 3 times3\,\text{times} to baseline their running speeds. 3030 matched pairs are formed by pairing players with nearly identical running speed records.
      • Experimental Step: Both players in each pair run 100 yards100\,\text{yards} 3 times3\,\text{times} and rest for 3 minutes3\,\text{minutes}.
      • Treatment: Player 1 rests without supplemental oxygen; Player 2 inhales supplemental oxygen during rest.
      • Response Variable: Both players run a final 100 yard100\,\text{yard} sprint, and their sprint times are directly compared against each other within each pair.
  • Method 2: Self-Control / Crossover Design (Each Unit as Its Own Control):

    • Example 1: Football Player Oxygen Study (Self-Control Variant):
      • 6060 football players are randomly divided into two groups of 3030.
      • Trial 1: Group 1 runs 100 yards100\,\text{yards} 3 times3\,\text{times}, rests without oxygen, and runs 100 yards100\,\text{yards}. Group 2 runs 100 yards100\,\text{yards} 3 times3\,\text{times}, rests with oxygen, and runs 100 yards100\,\text{yards}.
      • Crossover (Trial 2): A few days later, the experiment is repeated with reversed protocols: Group 1 receives oxygen during rest, while Group 2 rests without oxygen.
      • Response Variable: Recorded sprint times with oxygen versus without oxygen are compared for each individual player.
    • Example 2: Hand Lotion Satisfaction Study:
      • Setup: A Simple Random Sample (SRS) of 4040 women evaluates two types of lotion (Lotion A and Lotion B).
      • Group 1 (2020 women): Apply Lotion A for 1 month1\,\text{month}, refrain from using any lotion for a 2 week2\,\text{week} washout period, then apply Lotion B for 1 month1\,\text{month}.
      • Group 2 (2020 women): Apply Lotion B for 1 month1\,\text{month}, refrain from using any lotion for a 2 week2\,\text{week} washout period, then apply Lotion A for 1 month1\,\text{month}.
      • Factor: Type of hand cream (22 levels: Lotion A and Lotion B).
      • Treatments: Application of Lotion A or Lotion B.
      • Response Variable: Subjective user satisfaction score.
      • Experimental Design: Matched pairs design where each participant serves as her own control.

Questions & Discussion

  • Question: Could levels be considered similar to the stratas of different groups?
    • Response: Levels represent all the specific values or states that a factor variable can take on. While people often think of numbers when hearing