Experimental Research Designs in Communication Science
Foundations of Experimental Research
Definition of a Social Experiment:
- A social experiment is formally defined as a procedure in which researchers take human subjects, do something to them, and observe the effects of what they did to them (Baxter & Babbie, 2004).
- Experimental research is frequently performed in social science and communication research without specialized laboratory paraphernalia such as lab coats or Bunsen burners.
- Communication scientists have utilized experimental procedures since the inception of the discipline to investigate how individuals process, utilize, and respond to information across interpersonal interactions, mediated messages, organizational dynamics, and human-computer interactions.
Variables in Experimental Research:
- A variable represents any entity, property, or event that can be measured or observed by a researcher.
- In general empirical research, variables can be measured through three primary modes:
- Self-report.
- Other-report.
- Direct behavioral observation.
- In experimental configurations, the independent variable can also be directly manipulated or introduced as an induction.
- Measurement techniques align with the nature of the variable being assessed:
- Beliefs, attitudes, opinions, and retrospective reports of behavior or meaning are assessed through survey instruments.
- The content of media messages is systematically evaluated using content analysis.
- The structural and interpersonal dynamics of conversational messages are evaluated via interaction analysis.
- Both surveys and interaction analyses can function as independent research methods or as embedded measurement instruments within experimental designs.
Independent Variables ():
- An independent variable is defined as any observation or condition that is directly controlled or manipulated by the researcher.
- Conceptualized as causes, independent variables are actively introduced to produce a specific outcome or observe differential effects across groups.
- Example: Testing the efficacy of an antismoking public service announcement (PSA) by designing two distinct versions—one utilizing a fear appeal to induce compliance and another omitting fear—and exposing separate participant groups to each ad. The type of message delivered represents the independent variable.
Dependent Variables ():
- A dependent variable represents an observed effect, outcome, or consequence that is not directly manipulated by the researcher.
- Variations in the dependent variable are presumed to result from changes or interventions implemented in the independent variable.
- Example: Following exposure to fear-based versus non-fear-based antismoking PSAs, participants' self-reported willingness to take up smoking represents the dependent variable.
Applicability of Experimental Designs:
- Experimental investigations are ideally suited for addressing micro-level research questions regarding how individuals respond to specific information stimuli.
- They are uniquely designed to isolate and verify relationships of cause and effect.
- Exemplary experimental questions include:
- Do video games cause people to respond more aggressively?
- Does physical attractiveness cause individuals to disclose more personal information?
- Experimental research is unsuited for purely descriptive goals due to its narrow situational focus, nor is it intended for identifying non-causal correlational associations.
- Experiments establish causality by introducing a pretest baseline, exposing subjects to a controlled stimulus, and measuring post-stimulus shifts to confirm that the intervention generated the change.
- Causal inferences require establishing non-spuriousness by demonstrating that the independent variable is the sole factor responsible for alterations in the dependent variable, thereby eliminating competing third-variable explanations.
- Applied research settings: Christine Davis utilized experimental pretest-posttest control group designs within commercial market research to evaluate alternative advertisement drafts and determine their impact on potential product usage.
Experimental Notation and Core Terminology
Standard Experimental Notational Elements:
- Observation (): Denotes an empirical measurement or assessment of spoken, written, or observed behavior or attitude; represents the fundamental building block of experimental observation sequences.
- Induction (): Denotes the experimental intervention, stimulus, or treatment administered to a subject (e.g., assigning a participant to play a video game, delivering a persuasive argument, or screening a video clip).
- Technical Distinction: An induction is what a researcher does to a participant; it is technically termed a manipulation only when the induction successfully alters the targeted internal or behavioral state.
- Synonymous terms: Interventions or treatments (prevalent in health communication research) and stimulus materials (prevalent in media research).
- Random Assignment (): Denotes the random distribution of participants across different experimental and control conditions, giving each subject an equal statistical probability of being assigned to any group.
- Purpose: Maximizes equivalence between groups prior to intervention, neutralizing spuriousness and distributing background variables evenly.
- Relevant baseline equivalence variables include health status and education in health literacy research, organizational tenure and training in workplace productivity research, or relationship history in studies comparing request politeness across email versus voicemail channels (Duthler, 2006).
- In non-randomized designs, dashed or solid lines separating group notation rows designate non-equivalent assignment.
Core Experimental Design Terminology:
- Experimental Group: The participant group exposed to the induction (). Studies with multiple induction variations contain multiple experimental groups.
- Control Group: A baseline comparison group that receives no experimental induction.
- Between-Subjects Design: An experimental design structure that assesses outcomes by making comparisons across two or more separate groups of participants.
- Within-Subjects Design: An experimental design structure that tracks and compares the exact same group of participants across multiple observation intervals or conditions.
- Pretest (Baseline Measurement): An observation () administered prior to the induction to document subjects' initial baseline standing.
- Posttest: An observation () administered following the administration of the induction to evaluate shifts in the dependent variable.
Internal Validity Threats in Experimental Design
- Primary Internal Validity Threats:
- History: Extraneous events occurring outside the experimental laboratory environment during the study that inadvertently alter participants' posttest scores.
- Maturation: Natural psychological, biological, or physiological changes occurring within participants over time (e.g., fatigue, aging, hunger, cognitive development) that influence responses independently of the induction.
- Testing: The phenomenon wherein participants demonstrate altered responses or improved performance on subsequent tests simply because repeated exposure to pretest instruments makes them familiar with the measures.
- Instrumentation: Inconsistencies or systematic errors arising from utilizing varying measurement instruments, alternating observers, or changing testing scales between baseline and posttest evaluations.
- Evaluator Apprehension: Participant anxiety or altered behavior triggered by awareness of being observed or evaluated by a researcher.
- Hawthorne Effect: The behavioral alteration exhibited by subjects merely as a consequence of knowing they are participating in a study.
- Selection Bias: Systematic baseline differences between non-randomized groups prior to the intervention, confounding subsequent posttest comparisons.
Preexperimental Designs
Overview of Preexperimental Designs:
- Classified by Campbell and Stanley (1963) as rudimentary design configurations that lack either a pretest baseline or a legitimate comparison control group.
- Highly vulnerable to internal validity threats; unable to confirm whether observed outcomes are attributable to the induction.
One Shot Case Study Design:
- Structural Notation:
- Procedure: A single group of participants is exposed to an experimental induction (), followed immediately by an observation ().
- Example: Participants enter a laboratory, view a blue jeans advertisement (), and answer questions evaluating their purchase intentions ().
- Methodological Weaknesses:
- Total absence of baseline pretest data makes it impossible to establish prior product attitudes.
- Does not control for history, maturation, selection, or instrumentation biases.
- Serves purely as a foundational baseline from which to analyze more sophisticated controls.
One Group Pretest Posttest Design:
- Structural Notation:
- Procedure: A single group is assessed on the dependent variable (), receives the induction (), and is reassessed on the identical measure ().
- Example: Participants rate their baseline purchase interest in blue jeans (), view the advertisement (), and rate their interest again ().
- Methodological Characteristics:
- Permits within-subject comparisons of pre- and post-intervention shifts.
- Lacks an unexposed control group, making it impossible to establish whether observed shifts resulted from the induction or extraneous factors such as history, maturation, or testing effects.
Static Group Comparison Design:
- Structural Notation:
- Procedure: Compares an experimental group exposed to an induction () and evaluated at posttest () against a static comparison group that receives no induction and completes only the posttest ().
- Example: Group 1 views a blue jeans advertisement and answers evaluation questions; Group 2 receives no ad exposure and answers the same questions.
- Methodological Characteristics:
- Introduces a comparison group to contrast exposed versus unexposed individuals.
- Fails to assess baseline equivalence between the two groups due to the absence of pretests.
- Cannot verify whether groups possessed divergent initial attitudes, nor does it control for testing fatigue or historical events occurring between non-synchronized test times.
Quasi-Experimental Designs
Overview of Quasi-Experimental Designs:
- Defined by Campbell and Stanley (1963) as designs that incorporate pretests and posttests in complex configurations but lack random assignment ().
- Represent a marked methodological improvement over preexperimental designs and are widely implemented in communication research when random assignment is logistically unfeasible.
Time-Series Design:
- Structural Notation:
- Procedure: A single group is evaluated across multiple consecutive pretest baseline periods (), exposed to an induction (), and subsequently evaluated across multiple consecutive posttest periods ().
- Methodological Capabilities:
- Designed primarily to detect and rule out history and maturation threats.
- Tracks baseline trend lines to determine if participant attitudes were already shifting prior to the induction.
- Identifies environmental interruptions outside the laboratory that affect the trajectory of posttest responses.
- Lacks a concurrent control group.
Nonequivalent Control Group Design:
- Structural Notation:
- Procedure: Involves two non-randomly assigned groups; both complete pretest () and posttest () assessments, but only the experimental group receives the induction ().
- Methodological Capabilities:
- Frequently utilized by communication scholars due to practical constraints surrounding participant assignment.
- Effectively controls against testing, instrumentation, and maturation threats because both groups experience identical testing procedures over time.
- Does not fully eliminate history threats and remains vulnerable to selection bias, as initial group equivalence is not guaranteed.
Multiple Time-Series Design:
- Structural Notation:
- Procedure: Synthesizes the Nonequivalent Control Group Design with the Time-Series Design by tracking both an experimental group and a comparison group across extended sequences of pre- and post-induction observations.
- Methodological Capabilities:
- Incorporates safeguards against instrumentation, testing, and maturation.
- Directly isolates historical threats through comparative longitudinal tracking between conditions.
- Remains constrained by the absence of random assignment.
True Experimental Designs
Overview of True Experimental Designs:
- Defined by Campbell and Stanley (1963) as designs that enforce the random assignment () of subjects to both experimental and control conditions.
- Distributes individual differences equally across conditions, effectively controlling subject-related threats such as evaluator apprehension, the Hawthorne effect, and selection bias.
Pretest Posttest Control Group Design:
- Structural Notation:
- Operational Mechanics:
- Structurally mirrors the Nonequivalent Control Group Design but incorporates random assignment ().
- Rules out nearly all major internal validity threats while maintaining operational economy.
- Research Study Application (Heimendinger et al., 2005):
- Examined four alternative health messages formulated to increase fruit and vegetable consumption among callers to the National Cancer Institute's (NCI) Cancer Information Service (CIS).
- Callers completed an initial baseline interview () upon contacting the CIS.
- Callers were randomly assigned () to one of three experimental message conditions promoting produce consumption () or to an unrelated control message condition.
- Follow-up survey interviews were conducted at two distinct posttest intervals: five months and twelve months later ().
- Message responses were quantified using multi-item survey interview scales, including Likert-type scales.
Posttest-Only Control Group Design:
- Structural Notation:
- Operational Mechanics:
- Employs random assignment () to equate groups across conditions without administering a baseline pretest, avoiding pretest sensitization or testing fatigue.
- Research Study Application (Lee & Guerrero, 2001):
- Investigated interpersonal interpretations of dyadic touch by having undergraduate students randomly assigned to watch a videotaped interaction ending in one of nine touch behaviors (stimulus materials):
- Handshake.
- Clasping hands.
- Soft touch on the forearm.
- Arm around the shoulder.
- Arm around the waist.
- Soft touch on the cheek.
- Tapping the shoulder condescendingly.
- Push against the shoulder.
- No touch (control condition).
- Dependent variables were assessed via Likert-scale questions in a survey interview format.
- Touch variations generated statistically significant differences in perceptions of affection, romantic attraction, flirtation, and love, but did not significantly alter perceived trust or happiness.
- Face touches were perceived as the most intimate category of touch.
- The design could not experimentally balance actor sex pairings or ethnic compositions within the video dyads, so these potential confounds were controlled statistically.
- Research Study Application (Triana, Kirkman, & Wagstaff, 2012):
- Investigated whether the sequence of communication channels influenced outcomes in diverse teams by comparing Computer-Mediated Communication (CMC) and Face-to-Face (FTF) interactions.
- Sample comprised total four-person teams (each consisting of three men and one woman):
- teams were randomly assigned to communicate via CMC first, followed by FTF.
- teams were randomly assigned to communicate via FTF first, followed by CMC.
Solomon Four-Group Design:
- Structural Notation:
- Operational Configuration:
- Regarded as the definitive gold standard of experimental methodology because it simultaneously controls for all recognized internal validity threats.
- Group 1 (): Pretested experimental group.
- Group 2 (): Pretested control group; controls for maturation and history.
- Group 3 (): Unpretested experimental group; evaluates testing and instrumentation interactions.
- Group 4 (): Unpretested control group; isolates outside history and maturation effects without testing exposure.
- Methodological and Practical Limitations:
- Highly resource-intensive and structurally uneconomical.
- Only Group 1 yields primary data for formal hypothesis testing; Groups 2, 3, and 4 exist exclusively to check against validity threats.
- Requires quadruple the sample size to test a target number of participants (e.g., assessing experimental outcomes requires recruiting subjects; evaluating outcomes requires recruiting subjects).
- Researchers frequently opt for the Pretest Posttest Control Group or Nonequivalent Control Group designs due to practical constraints involving laboratory access, student labor, and participant recruitment budgets.
Factorial Designs
Structural Principles of Factorial Designs:
- Analyses that incorporate two or more independent variables (factors) simultaneously to examine main effects and interaction effects.
- Enables researchers to evaluate independent variables that contain multiple distinct categories, levels, or qualitative types.
- The total number of required experimental groups equals the mathematical product of the levels across all factors.
Factorial Design Applications in Communication Research:
- Factorial Design (Lee & Guerrero, 2001):
- Factor 1: Type of touch gesture ( categorical levels).
- Factor 2: Biological sex of the observer ( levels: male, female).
- Requires a total of distinct experimental groups.
- Design Notation:
- Group assignment details:
- Condition (): Handshake viewed by a male observer.
- Condition (): Handshake viewed by a female observer.
- Condition (): Hand clasp viewed by a male observer.
- Condition (): Hand clasp viewed by a female observer, continuing systematically through condition ().
- Findings: Male observers consistently rated touch behaviors as more flirtatious and indicative of love than female observers across all nine touch conditions, demonstrating that perceptual differences were driven by observer sex rather than touch manipulation alone.
- Factorial Design (Haumer & Donsbach, 2009):
- Examined how nonverbal behaviors in a mock televised interview influenced viewer evaluations of a politician's public image.
- Factor 1: Politician's nonverbal style ( levels: active vs. passive).
- Factor 2: Television host's displayed reactions ( levels: negative, positive, neutral).
- Factor 3: Studio audience's displayed reactions ( levels: negative, positive, neutral).
- Experimental configuration: separate experimental video conditions.
- Sample allocation: Posttest-only framework using participants per group, totaling participants.
- Dependent variables (measured via Likert scales in survey interviews):
- Problem-solving competence.
- Leadership abilities.
- Integrity.
- Personal qualities.
Field and Natural Experiments
Comparative Characteristics of Experimental Environments:
- Laboratory Experiments: Maximize internal validity through rigorous environmental control and variable manipulation, but often suffer from reduced ecological validity.
- Field Experiments: Executed in naturalistic settings while retaining intentional researcher manipulation of independent variables; possess high ecological validity at the expense of strict environmental control.
- Natural Experiments: Conducted in naturalistic settings without researcher-initiated manipulation; researchers evaluate the causal influence of naturally occurring real-world interventions.
Empirical Studies in Naturalistic Settings:
- Field Experiment: Retail Attractiveness (Argo, Dahl, & Morales, 2008):
- Conducted in an operational university bookstore to evaluate consumer contagion.
- Independent Variable (): Physical attractiveness of an individual touching a retail item.
- Dependent Variable (): Consumer purchase intentions toward the touched merchandise.
- Operationalization: A confederate sales clerk and a confederate shopper were positioned in the retail environment; the confederate shopper touched a specific T-shirt under conditions of varied physical attractiveness, after which actual shoppers' purchase intentions were assessed.
- Natural Experiment: Broadcast Advertising Ban (Holak & Reddy, 1986):
- Evaluated the long-term impact of the federal ban prohibiting cigarette advertisements on television and radio.
- Independent Variable (): Implementation of the advertising prohibition (categorized as pre-ban vs. post-ban).
- Dependent Variables ():
- Cigarette retail pricing.
- Industry advertising expenditure levels.
- Consumer product demand.
- Findings: The naturally occurring policy intervention produced statistically significant shifts across all three dependent variables.
- Operational Challenges in Field Settings: Field researchers face ambiguous distinctions between independent and dependent variables, lack environmental controls, and rarely deploy complex factorial designs, though they retain superior ecological validity for evaluating causal dynamics.
Methodological Trade-offs and External Validity
- The External Validity Dilemma:
- While laboratory experiments effectively isolate linear causal relationships (), their artificial procedures limit the extent to which findings generalize to real-world communication behaviors.
- Experimental measurement procedures fail to mirror routine social interactions:
- In everyday life, individuals do not complete baseline and post-exposure surveys before and after viewing home television broadcasts.
- Normal interpersonal conversations with peers are never followed by formal survey rating forms assessing partner disclosiveness.
- Establishing micro-level causal associations often comes at the expense of generalizability, necessitating alternative methodological frameworks that capture communication experiences rather than isolated cause-and-effect relationships.
Complete Key Terms and Definitions
- Baseline: An examination of a participant conducted before exposure to a stimulus in order to argue, to a certain extent, that the experimental intervention caused any observed changes; also called a pretest.
- Between-Subjects Design: Experimental design structures that evaluate outcomes by comparing multiple distinct groups of participants.
- Control Group: A baseline comparison group in an experiment that does not receive the experimental induction.
- Experimental Group: The group of participants in an experiment exposed to the induction.
- Factorial Design: Experimental analytical designs that incorporate multiple independent variables.
- Induction: The experimental treatment, intervention, or task administered to a participant in an experiment.
- Posttest: An examination, assessment, or measurement conducted after the administration of an experimental induction.
- Preexperimental Designs: Rudimentary experimental designs that do not include a true control group, pretest baseline, or posttest comparison.
- Pretest: An examination of a participant conducted before exposure to a stimulus in order to argue, to a certain extent, that the experimental intervention caused any observed changes; also called a baseline measurement.
- Quasi-Experimental Designs: Experimental designs that implement pretests and posttests in complex configurations but lack random assignment.
- Random Assignment: The procedure of assigning participants to different groups within an experimental design entirely at random, ensuring equal selection probability.
- Social Experiment: A procedural methodology in which researchers take human subjects, do something to them, and observe the effects of what they did to them.
- True Experimental Designs: Experimental designs that randomly assign participants to both experimental and control groups.
- Within-Subjects Design: Experimental designs that evaluate outcomes by measuring the same participant group across multiple observation intervals.
Academic References
- Argo, J. J., Dahl, D. W., & Morales, A. C. (2008). Positive consumer contagion: Responses to attractive others in a retail context. Journal of Marketing Research, 45, 690–701.
- Baxter, L. A., & Babbie, E. (2004). The basics of communication research. Belmont, CA: Wadsworth.
- Campbell, D., & Stanley, J. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand-McNally.
- Duthler, K. W. (2006). The politeness of requests made via email and voicemail: Support for the hyperpersonal model. Journal of Computer-Mediated Communication, 11(2), article 6.
- Haumer, F., & Donsbach, W. (2009). The rivalry of nonverbal cues on the perception of politicians by television viewers. Journal of Broadcasting & Electronic Media, 53(2), 262–279.
- Heimendinger, J., O'Neill, C., Marcus, A. C., Wolfe, P., Julesburg, K., Morra, M., Allen, A., Davis, S., Mowad, L., Perocchia, R., Ward, J. D., Strecher, V., Warnecke, R., Nowak, M., Graf, I., Fairclough, D., Bryant, L., & Lipkus, I. (2005). Multiple tailored messages are effective in increasing fruit and vegetable consumption among callers to the Cancer Information Service. Journal of Health Communication, 10(Supplement 1), 65–82.
- Holak, S. L., & Reddy, S. K. (1986). Effects of a television and radio advertising ban: A study of the cigarette industry. Journal of Marketing, 50(4), 219–227.
- Lee, J. W., & Guerrero, L. K. (2001). Types of touch in cross-sex relationships between coworkers: Perceptions of relational and emotional messages, inappropriateness, and sexual harassment. Journal of Applied Communication Research, 29(3), 197–220.
- Triana, M. d. C., Kirkman, B. L., & Wagstaff, M. F. (2012). Does the order of face-to-face and computer-mediated communication matter in diverse project teams? An investigation of communication order effects on minority inclusion and participation. Journal of Business and Psychology, 27, 57–70.