COMG 102: Everyday Communication with Numbers - Experiments

Recap & Introduction to Experiments

Recap

  • Research Design: Reviewed different approaches to structuring research.

  • Cross-sectional vs. Longitudinal Research: Distinguished between studies that collect data at a single point in time versus those that collect data over an extended period.

  • Survey Research: Discussed the methodology of surveys, including:

    • Survey Question Formats: Different types of questions used in surveys.

    • Common Wording Problems: Potential issues in phrasing questions that can affect validity.

    • Different Survey Methods: Various ways to administer surveys.

    • Funnel vs. Inverted Funnel Structures: Approaches to organizing survey questions.

    • Question Scaling Techniques: Methods used to quantify responses, including Likert scales and semantic differential scales, which help in measuring attitudes and perceptions.

Experiments vs. Surveys

  • Surveys: Primarily focus on asking people what they think or feel ("Let's ask people and see what they think."). They gather opinions, attitudes, and reported behaviors.

  • Experiments: Involve actively doing something and observing its consequences ("Let's do something and see what happens."). They aim to establish cause-and-effect relationships.

Core Concept of Experimentation

  • Manipulation of Variables: Experimentation involves manipulating one variable, known as the Independent Variable (IV), to observe its effect on another variable, the Dependent Variable (DV).

  • Longitudinal Research: Experimental designs are inherently longitudinal, as they involve observations over time to measure change.

  • Purpose: The primary goal of experiments is to determine whether variables have causal relationships.

Cross-lagged Designs vs. Experimental Designs

  • Both are Longitudinal: Both design types allow researchers to investigate changes in study variables over time.

  • Pure Cross-lagged Designs:

    • Researchers do not manipulate an Independent Variable.

    • They simply measure study variables at two distinct points in time: "Time 1" and "Time 2."

  • Experimental Designs (AKA "Experimental Treatment"):

    • Researchers manipulate an Independent Variable.

    • They measure variables at "Time 1" and "Time 2" to specifically investigate changes in a Dependent Variable that occur as a result of the manipulation (i.e., changes in the IV).

    • Manipulation: This refers to the deliberate process by which researchers change or influence the independent variable.

Causality: The Main Reason for Experiments

Importance of Causality

  • Causality is the central focus and main reason why experiments are conducted.

  • It is a subject of intense interest across various fields, including:

    • Parents and Politicians: "Will exposure to violent video games cause some undesirable effects in children?"

    • Educators: "Will a new online teaching method lead to student performance improvement?"

    • Marketers: "Will a new marketing strategy cause an increase in sales?"

    • Communication Researchers: "Will heavy use of the internet cause a change in the frequency of interpersonal communication?"

Fundamental Causal Question

  • “Does A cause B?” (A —> B?)

    • “Does an IV cause a DV?

Three Criteria for Establishing Causality

To be certain that variable A truly causes a change in variable B, three essential criteria must be met:

  1. Time Order:

    • Principle: Variable A must precede variable B in time.

    • Explanation: The cause (A) must occur before the effect (B) is observed.

    • Example: For a new marketing strategy to cause an increase in sales, the strategy must be implemented before the sales increase is observed.

    • Measurement: To measure changes, variables must be assessed at an initial "Time 1" and then again later at "Time 2."

    • Limitation of Cross-sectional Research: A major weakness of cross-sectional studies (e.g., surveys) is that they measure variables only once (at "Time 1"), making it impossible to determine time order and thus causality.

  2. Covariance:

    • Principle: Variables A and B must vary together; there must be a discernible relationship between them.

    • Explanation: As the independent variable (A) changes, the dependent variable (B) should also change in a consistent manner.

    • Example: If heavy internet use causes a change in interpersonal communication frequency, then as internet use changes, so too should the frequency of interpersonal communication. There should be a demonstrable relationship where the variables covary.

    • Implication: If a change in the independent variable does not lead to any change in the dependent variable, then covariance is not present, and causality cannot be established.

  3. No Extraneous Factors:

    • Principle: Variable B must be caused by A, and nothing else.

    • Explanation: Even if time order and covariance are met, it is crucial to eliminate the possibility that other external or confounding variables are responsible for the observed effect.

    • Example (Marketing Strategy): If sales increase after a new marketing strategy, but this increase could also be attributed to an active market, a pandemic, or issues with a main rival company, then causality linking only the marketing strategy to the sales increase is questionable.

    • Example (Internet Use): If the frequency of interpersonal communication changes, but this change could be due to factors other than internet use (e.g., pandemic-related social distancing, relocation, cost of living, personal stress), then the causal link to internet use is weakened.

Basic Experimental Designs

One-Group Pretest-Posttest Design

  • Description: This design involves a single group that is observed, exposed to a manipulation, and then observed again.

  • Structure:

    • Step 1: Baseline Observation (O1): A baseline measure of the Dependent Variable (DV) is taken before the manipulation.

    • Step 2: Experimental Manipulation (X): The group is exposed to the Independent Variable (IV) or experimental treatment.

    • Step 3: Post-experimental Observation (O2): The Dependent Variable (DV) is measured again after the manipulation.

  • Observation: Researchers look for a difference between the "before" and “after” measures between O1 and O2.

  • Conclusion: With this design, it's possible to see changes that might occur as a result of the experimental condition.

Example: Hand-washing Campaign Video

  • Research Question: "Does exposure to a hand-washing campaign video lead to improved hand-washing?"

  • Variables:

    • IV: Hand-washing campaign video.

    • DV: Hand-washing behavior.

  • Conceptualization of DV: Washing hands with soap and water for at least 20 seconds (as per CDC, 2020).

Procedure:

  1. Observation 1 (O1): Participants' hand-washing behavior is initially observed (e.g., duration, use of soap).

  2. Manipulation (X): Participants are exposed to the hand-washing campaign video.

  3. Observation 2 (O2): After watching the video, participants are asked to wash their hands again, and their behavior is re-observed.

  • If a significant difference is found between O1 and O2, one might conclude the video caused the change.

Limitations of One-Group Pretest-Posttest Design

  • This design has significant weaknesses because it cannot effectively rule out two critical possibilities, which undermine causal claims:

    1. Unintentional Changes: Any observed change in the dependent variable might have occurred unintentionally due to factors internal to the participants (e.g., natural maturation, improved awareness from the first observation).

    2. External Influences/Extraneous Variables: The observed change might be due to some external influence or an unaccounted-for variable other than the manipulated IV.

      • Participant Characteristics: Factors like participants' pre-existing cleanliness levels, fear of germs, or motivation might influence results.

      • Environmental Factors: Participants might behave differently if they are aware they are being watched (Hawthorne effect).

      • Experimental Effects: The experiment itself (e.g., the act of being observed at O1) may have unintentional effects on O2.

  • In essence, many other variables not explicitly included in the design could be influencing the outcome, making it hard to confidently attribute change solely to the IV.

Control Groups

  • Definition: Control groups are groups of participants that are not exposed to any experimental manipulation or treatment.

  • Purpose: They serve as baselines. Any changes observed in groups exposed to experimental manipulations are measured against these control groups.

  • Addressing Limitations: Adding a control group helps address the limitations of the one-group design by providing a comparison point free from the specific experimental treatment.

Two-Group Pretest-Posttest Design

  • Description: This design improves upon the one-group design by incorporating a control group, allowing for a more robust assessment of causality.

  • Structure: It uses two groups of participants (a Treatment Group and a Control Group).

  • Steps (Using Hand-washing Example):

    1. Pretest (O1): Measure the hand-washing behavior of both groups.

    2. Manipulation (X): The hand-washing campaign video is played only to Group 1 (the "Treatment Group"). Group 2 (the "Control Group") is not exposed to the video.

    3. Posttest (O2): Measure the hand-washing behavior of both groups again after the manipulation (or lack thereof).

  • Expected Outcomes for Causality: For the IV to be considered causal, researchers would look for:

    • A significant difference between the "before" O1 and “after” O2 measures only in Group 1 (Treatment Group).

    • No significant difference between the "before" O1 and “after” O2 measures in Group 2 (Control Group).

    • Significantly greater improved hand-washing observed in Group 1 compared to Group 2, or Group 1 showing improvement while Group 2 shows no improvement or even a decline.

Experiment Pros and Cons

Advantages (Pros)

  • High Control: Experiments offer a high degree of control over variables and environmental conditions.

  • Confidence in Causality: Due to this control, researchers can be more confident that it is the Independent Variable that is leading to the observed change in the Dependent Variable.

Disadvantages (Cons)

  • Lack of Ecological Isomorphism (External Validity):

    • The highly controlled conditions of an experiment often do not perfectly replicate real-world situations.

    • This means that the findings, while internally valid (causation within the experiment is clear), may not generalize well to the outside world.

    • This limitation is also referred to as a lack of "external validity."

Summary of Key Concepts

  • Experimental Designs: These research methodologies are specifically employed to investigate and establish causal relationships between variables.

  • Experimental Manipulation of an IV: A hallmark of experiments is the deliberate intervention by researchers to change or influence the independent variable.

  • Three Criteria for Causality: To confidently conclude that A causes B, all three conditions must be met:

    1. Time Order: A must occur before B.

    2. Covariance: A and B must vary together.

    3. No Extraneous Factors: The change in B must be attributable solely to A, ruling out other influences.

  • Experimental Designs Reviewed:

    • One-Group (O-G) Pretest-Posttest Design: A single group observed, manipulated, then re-observed.

    • Two-Group (T-G) Pretest-Posttest Design: Incorporates a control group for comparison, significantly strengthening the ability to infer causality.