Chapter 3 Notes: Three Claims, Four Validities

Variables

  • Variable (2+ levels) vs. constant (1 level)
  • Distinctions: measured vs. manipulated; conceptualized vs. operationalized
  • L = number of levels (a variable with 2+ levels is a variable; a constant has 1 level)

Measured vs Manipulated Variables

  • A measured variable is observed and recorded (not controlled by the researcher)
  • A manipulated variable is controlled (the researcher assigns levels)
  • Some variables can only be measured (e.g., for ethical reasons like trauma)
  • Some variables can be either manipulated or measured depending on the study design

Describing Variables

  • Way of describing a variable (Table 3.1):
    • Construct, conceptual variable: conceptual definition; the name of the concept; a careful, theoretical definition of the construct
    • Operational definition, operationalization: how the construct is measured or manipulated in an actual study
  • Example 1:
    • Concept: "Satisfaction with life"
    • Conceptual definition: "A person’s cognitive evaluation of his or her life" (Diener et al., 1985)
    • Operationalization: Five questionnaire items on the Satisfaction with Life scale, answered on a scale of 1 (strongly disagree) to 5 (strongly agree). Example item: "All in all, I am satisfied with my life"
  • Example 2:
    • Concept: "Perseverance" (in young children)
    • Conceptual definition: "The ability to push through when confronted with obstacles" (White et al., 2017)
    • Operationalization: How long a child will choose to engage in a slow-paced, boring activity that involves pressing a button when they see a picture of cheese on a screen and not pressing it when they see a cat on the screen

Describing Variables to Measure: Semantic Interference (Stroop Task)

  • Variable: SEMANTIC INTERFERENCE (delay or increased errors when related information interferes with encoding of the target information)
  • Operationalization example: THE STROOP TASK
  • Reference: Stroop, John Ridley (1935) Studies of interference in serial verbal reactions. Journal of Experimental Psychology. 18 (6): 643–662. doi:10.1037/h0054651
  • Example stimuli colors: RED, GREEN, BLUE

From Conceptual Variables to Operational Definitions: Examples

  • Car ownership
    • Conceptual variable: Car ownership
    • Operational definition (one possibility): Circle "I own a car" or "I do not own a car" on a questionnaire
    • Levels: 2 levels (own vs. not own)
    • Measured
  • Expressing gratitude to a romantic partner
    • Operational definition: Items like "I tell my partner often that s/he is the best"
    • Levels: 7 levels from 1 (strongly disagree) to 7 (strongly agree)
    • Measured
  • Exposure to disinformation
    • Operational definition: Hearing false information either one time or two times
    • Levels: 2 levels (one time vs. twice)
    • Manipulated
  • What time children eat dinner
    • Operational definition: Use a daily food diary; divide children into groups: dinner between 2 P.M. and 8 P.M. vs after 8 P.M.
    • Levels: 2 (2–8 PM vs after 8 PM)
    • Manipulated

Operationalizing "School Achievement" (Figure 3.2)

  • Conceptual variable: School achievement
  • Operationalizations:
    • What grades do you get? (self-report)
    • All As
    • Mostly As and Bs
    • Mostly Bs
    • Mostly Bs and Cs
    • Measurement methods:
    • Self-report questionnaire
    • Checking records
    • Teachers’ observations

The Three Claims

  • Frequency claims: describe a particular level or degree of a single variable (involve only one measured variable)
  • Association claims: one level of a variable is likely associated with a particular level of another variable; supported by studies with at least two measured variables; associated variables are said to correlate
  • Causal claims: a causal claim argues that one variable causes changes in the level of another variable; must be supported by experiments (manipulated IV, measured DV)
  • Note: Not all claims are based on research

Frequency Claims

  • Describe a level or degree of a single variable; involve only one measured variable
  • Example context: COVID-19 vaccination in a region as of a date (population-level description)
  • Important: Internal validity is not typically relevant for frequency claims (they do not assert causality)

Association Claims

  • Definition: one level of a variable is likely associated with a level of another variable
  • Supported by correlational studies (at least two measured variables)
  • Term for related variables: they correlate
  • Example context: study links (e.g.,) exercise to higher pay
  • Internal validity: not asserting causality; avoid claiming causality from mere association

Visual Examples of Associations (Figures 3.3)

  • Positive association example: Higher income associated with more exercise vs lower income with less exercise
  • Negative association example: Higher depression associated with more coffee vs lower depression with less coffee
  • Zero association example: Relationship between childhood weight and time of dinner

Making Predictions Based on Associations

  • Stronger association yields more accurate predictions
  • Both positive and negative associations can help predictions
  • Zero association provides little predictive power

Causal Claims

  • A causal claim asserts that one variable causes changes in another
  • Supported by experiments (randomized controlled trials) where IV is manipulated and DV is measured

Verbs for Causal Claims (Table 3.4)

  • Association claims verbs:
    • is linked to, is related to, is associated with, is correlated with
  • Causal claims verbs:
    • causes, promotes, affects, reduces, prevents, exacerbates, worsens, increases, changes, leads to, improves, etc.

Interrogating the Three Claims Using the Four Big Validities

  • Four big validities:
    • Construct validity: how well variables are measured or manipulated; how well operational variables approximate conceptual variables
    • External validity: generalizability to larger populations, other times, other contexts
    • Internal validity: whether A (not some other variable C) causes changes in B when examining relationships
    • Statistical validity: how well the numbers support the claim; strength of effect, precision of estimate, replication
  • Core idea: validity = extent to which the tools measure what we intend to measure

The Four Big Validities (Definitions)

  • Construct validity: How well the variables in a study are measured or manipulated; the extent to which the operational variables are good approximations of the conceptual variables
  • External validity: The extent to which results generalize to a larger population, other times, or other contexts
  • Statistical validity: How well the study’s conclusions are supported by the data; includes effect size, precision, and replication
  • Internal validity: In a relationship between A and B, the extent to which A, rather than some other variable C, is responsible for changes in B

Interrogating Frequency Claims

  • Construct validity: Are the variables well operationalized?
  • External validity: To what populations, settings, and times can we generalize this estimate? How representative is the sample? Was the sample random?
  • Statistical validity: What is the confidence interval (CI) of the estimate? Are there other estimates of the same percentage? How strong is the estimate?
  • Internal validity: Not usually relevant for frequency claims since causality is not asserted
  • Note: The slide provides a sample table format for evaluating a frequency claim like "4 in 10 teens admit to texting while driving" with CI and sampling considerations

Interrogating Association Claims

  • Construct validity: How well are the two variables measured?
  • External validity: Generalization to populations, settings, and times; sample representativeness
  • Statistical validity: Estimated effect size; precision; confidence interval; replication across studies
  • Internal validity: Not inherently about causality; avoid implying causation from correlation
  • Example workflow: quantify two variables, report correlation, assess generalizability

Interrogating Causal Claims

  • Prerequisite: Association must be present; but association alone does not imply causation
  • External validity: Generalization to populations, settings, and times; representativeness of manipulations and measures
  • Internal validity: Temporal precedence, control for confounds via random assignment, and avoidance of internal validity threats (Chs. 10–11 references)
  • Temporal precedence: Ensure the cause precedes the effect in time
  • Internal validity threats: Confounding variables, design flaws, etc.

Experiments Can Support Causal Claims

  • Experimental design basics:
    • Independent variable (IV) is manipulated
    • Dependent variable (DV) is measured
    • Random assignment of participants to conditions
  • Example figure: The Batman Effect study (persistence/performance in children) illustrating manipulated perspective and subsequent DV
  • Key takeaway: Experiments are the primary method to establish causality due to control over confounds and temporal ordering

Interrogating the Three Types of Claims Using the Four Big Validities (Table 3.6)

  • Frequency claims (e.g., "4 in 10 teens admit to texting while driving")
    • Construct validity: How well was the variable measured?
    • Statistical validity: Confidence interval, margin of error, replication
    • Internal validity: Often not relevant for frequency claims (no causal claim)
    • External validity: Generalizability to populations, settings, times
  • Association claims (e.g., study links exercise to higher pay)
    • Construct validity: How well were both variables measured?
    • Statistical validity: Effect size, confidence interval, replication
    • Internal validity: Not inherently about causality; avoid causal language
    • External validity: Generalizability to populations, settings, times
  • Causal claims (e.g., a program reduces eating disorders via improved family meals)
    • Construct validity: How well were manipulated/measured variables?
    • Statistical validity: Effect size, precision, replication
    • Internal validity: Temporal precedence, random assignment, control for confounds
    • External validity: Generalizability to populations, settings, times

When Causal Claims Are a Mistake

  • Core idea: Correlation does not imply causation
  • Navigating causal claims (Family meals example):
    1) This was a correlational study; they found a relationship, but…
    2) Temporal precedence: Which came first — family meals or eating disorders?
    3) Could there be a common cause? Consider alternative explanations
    4) Stop: the study design does not support a causal claim
  • Takeaway: Always check temporal order and internal validity before inferring causality

Prioritizing Validities

  • Question: Which of the four validities is most important?
  • Answer: It depends on the type of claim being made and the researcher’s priorities