Notes on Experimental Design

Below are comprehensive, study-ready notes on experimental design based on the transcript. The notes are organized as bullet points under multiple top-level headings, with key concepts, definitions, examples, and practical guidance. All mathematical expressions are formatted in LaTeX syntax.

Key Concepts and Definitions

Experimental design and analysis focus on how to structure a study to test factors (independent variables) and their effects on outcomes.
Independent variables in many experiments are categorical (interventions, treatments, groupings).
ANOVA is used when the outcome is continuous and independent variables are categorical; it assesses mean differences across treatment groups or factor levels.
Regression typically involves continuous independent and dependent variables; it assesses relationships and correlations rather than mean differences across factor groups.
The design must be planned before data collection to control for outside variables and establish a causal link as much as possible; regression designs are often more flexible post hoc, while experimental designs require pre-specified factors and randomization.
In experiments, factors and treatments are distinct concepts:
- Factor: a variable that is considered to influence the outcome (e.g., intervention, gender).
- Level of a factor: a specific category of that factor (e.g., intervention A, intervention B, male, female).
- Treatment: the level combination actually experienced by a subject (e.g., intervention A with gender male if two factors are present).
Experimental units (subjects or other units like classrooms) are the entities from which data are collected. The unit of analysis may be an individual or an aggregate (e.g., classroom). Power is based on the number of experimental units, not necessarily the number of individuals.

Experimental Design vs Observational Studies

Experimental design assigns subjects to factor levels, typically via randomization, to control for external variables.
Observational design does not randomly assign; factor levels are observed as they occur (e.g., current behaviors or existing dosages).
Causal language is more appropriate for experimental designs due to randomization and control of extraneous variables; observational studies generally support noncausal, associative statements.
Mixed designs can combine experimental and observational components (e.g., randomize at a higher level like classrooms but observe individual-level variation within classrooms).
Practical takeaway: if you can randomize and control, you typically interpret results with a stronger causal frame; if not, interpretation is more cautious and noncausal.

Components of an Experiment (Structure)

Explanatory factors (independent variables) and their levels.
Treatments: the specific combination of factor levels assigned to a subject.
Experimental units: the unit at which data are collected (could be classroom, student, etc.).
Randomization rules and procedures: how treatments are assigned to experimental units.
Outcome variables: what is measured (continuous outcomes for ANOVA).
Time points: number of measurements and the timing of data collection.
Blocking and control variables: strategies to account for known sources of variability without focusing on them as primary factors.
Power considerations: degree of freedom (df) management, sample size, and the number of factors.
Design type and plan: selecting from complete randomization, factorial designs, blocking, nesting, and cross- or partial-cross designs.

Factors, Levels, Treatments, and Experimental Units

Factor: a variable used in the study as an independent variable (categorical often).
Factor level: a specific category within a factor (e.g., dose 5 mg, dose 10 mg).
Treatment: the combination of factor levels that an experimental unit experiences.
Single-factor design: one factor, straightforward treatment classification.
Multi-factor design: two or more factors; treatments are combinations of factor levels (e.g., temperature × recipe).
Crossed factors (completely crossed): every level of one factor is combined with every level of the other factor (e.g., 3 levels of temperature × 2 cupcake recipes = 6 treatments).
Nested design: levels of one factor are nested within levels of another (e.g., teachers nested within schools).
Partial crossing: not all possible cross combinations are used to save time or resources while preserving power.
Completely randomized design (CRD): all experimental units are randomly assigned to treatment levels.
Factorial design: a study with multiple factors, typically analyzed with ANOVA to assess main effects and interactions.
Higher-order/factorial order: the number of factors included in the design; more factors reduce degrees of freedom and can reduce power if sample size is limited.
Blocking: grouping units into blocks to control for known sources of variability; a block is a control-like grouping that is not a factor of primary interest.

Randomization: Methods and Constraints

Randomization is the process of randomly assigning subjects to treatment levels to control for outside variability.
Simple randomization: random assignment of individuals to treatments.
Constrained randomization: randomization performed within constraints (e.g., randomizing within gender strata because gender cannot be randomized across subjects).
Blocking variable: a known confounder that is controlled for in the design (akin to a control variable but used in randomization/blocking).
Example 1: If classroom assignment limits randomization at the student level, randomize at the classroom level (constrained randomization) so all students in a class receive the same method.
Example 2: When randomization is not possible for a factor like gender, constrain randomization within gender groups.
Practical note: In small samples or complex designs, randomization may be approximated or replaced with quasi-random or constrained approaches; the goal is to minimize bias and balance confounders across treatments.

Design Structures: CRD, Factorial, Crossed, Nested, Partial Cross, Blocking

Completely randomized design (CRD): All factors are randomized; typically a one-way design or a fully randomized factorial design when multiple factors are involved.
Completely randomized factorial design: All factor levels are fully crossed and randomized.
Crossed designs (two-way, three-way, etc.): All levels of one factor are paired with all levels of the others.
Partially crossed designs: Some but not all cross combinations are realized to reduce the number of treatments while maintaining the ability to detect effects and interactions.
Nested designs: One factor’s levels are nested within another factor’s levels (e.g., teachers within schools). Nested factors are not crossed across higher-level factors.
Blocking: Control for categorical extraneous variables by grouping units, then randomizing within blocks.
Higher-order designs: More than two factors; the order refers to the number of factors included.
Interaction effects: In factorial designs, interactions show that the effect of one factor depends on the level of another factor.

Cross-Sectional vs Repeated Measures; Mixed Effects

Cross-sectional design: Data collected at a single time point for each unit.
Repeated measures design: Data collected from the same units across multiple time points.
Mixed effects models: Analyzing data with fixed effects (factors of interest) and random effects (random variation across units or clusters).
When crossing over time and groups, you typically analyze with methods that accommodate within-unit correlations (e.g., mixed ANOVA, repeated measures ANOVA, MANOVA).

Qualitative vs Quantitative Factors; Levels and Measurement

Qualitative (categorical) factors: Levels are categories with no natural order (e.g., gender: male, female).
Quantitative (ordinal or continuous) factors: Levels have a meaningful order and potentially equal spacing (e.g., temperature in degrees, dose levels).
Coding in data entry (e.g., SPSS): qualitative factors may be coded as 0/1, which does not imply numeric ordering.
When factor levels are quantitative, post hoc trend analyses can assess whether increasing levels correspond to systematic changes in the outcome.
For qualitative levels, interpretation focuses on differences between categories rather than directional trends.

Determining the Number of Levels and Levels Selection

Practical constraints: time, budget, complexity, subject availability, and sample size limit the number of levels/treatments.
When continuous factors are involved, consider adding more levels if a nonlinear relationship is suspected (to capture curvature or plateau effects).
For qualitative factors, rely on prior literature to decide which categories are potentially influential and to avoid unnecessary replication of known effects.
Colors, materials, or other amenities used as factors should be chosen based on prior evidence of impact and planned combinations with other factors.
The goal is to balance informative design with feasibility; avoid overloading the study with too many levels that dilute power.

Control, Covariates, and Blocking in Analysis

Control variables (blocking variables) account for known confounding factors; they are not primary factors of interest but help reduce unexplained variance.
Covariates (in designs like ANCOVA) adjust for continuous covariates that might influence the outcome, increasing precision.
Blocking and covariates help isolate the effect of primary factors from extraneous variance.

Writing Research Questions and Hypotheses

Two aspects of research questions:
- Conceptual goal or area of interest (broad): what you want to study, without specifying direction.
- Formal research questions for the paper: clear specification of independent and dependent variables, and the time frame.
Important guidelines:
- Do not reveal the study outcome in the question (avoid giving away the answer; do not state direction in the research question).
- Include the independent variable(s) and the dependent variable(s) in the question.
- If data are collected over time, include “over time” or “over the study period” to indicate longitudinal data.
- Avoid directional language in research questions; you can state hypotheses with direction, but the research question itself should be neutral.
Null vs alternative hypotheses:
- Null hypothesis (H0): means are equal across groups (or no effect, no difference).
- Alternative hypothesis (H1): there are differences or an effect; can be non-directional or directional in practice.
Moderation vs interaction:
- Moderation refers to a variable (e.g., gender) altering the strength/direction of the relationship between two variables (an interaction effect in ANOVA can represent moderation).
Example structure for a literacy study:
- Dependent variable: literacy development (defined as reading fluency, comprehension, and vocabulary).
- Independent variable: intervention vs no intervention.
- Time factor: measurements over 12 weeks (longitudinal data).
Multiple research questions can be used if exploring different aspects (e.g., the overall relation between stress and depression, and whether gender moderates that relation).

Analyzing and Reporting Results in ANOVA

Key concept: ANOVA tests whether group means are significantly different.
Degrees of freedom (df) come from the design; F-statistic:
- For a simple comparison: F(df1, df2) where df1 is between-group df and df2 is within-group df.
- Example reported result: $$ extit{F}(1, extit{df2}) = 41.146,\