Generalizability I-II

What are the two main types of generalizability?

• Generalizability to other settings/contexts

• Generalizability to other people

What does generalizability to settings involve?

Whether results hold across different experimental environments, labs, procedures, and real-world contexts.

What are interaction effects, and what two types are there?

Interaction effects = social components of the experimenter–participant interaction that may influence results.

• Biosocial effects: Experimenter’s demographics (e.g., gender, age, race) influence participant behavior.

• Psychosocial effects: Experimenter’s attitude or personality influences behavior (e.g., enthusiasm increases positive responses).

Why are interaction effects a generalizability concern?

What are interaction effects in generalizability?

They make one lab’s results fail to generalize to another lab with different researchers.

Researcher characteristics or behaviour that influence participant responses.

How do researchers mitigate interaction effects?

• Standardize experimenter–participant interactions

• Use scripts

• Maintain consistent demeanor

What is ecological validity?

The extent to which the study environment simulates real-world conditions.

What is mundane realism?

How much the tasks and setting resemble real-life situations.

What is experimental realism?

How realistically engaging or impactful the study feels to participants, even if artificial.

What is the key question when thinking about real-world generalizability?

“Do in-lab results translate to real-world results?”

(Example: Do people actually stare at eyes/noses in real life like they do in lab eye-tracking tasks?)

What is the college sophomore problem?

Over-reliance on university students as participants, leading to low external validity.

Why are college students considered WEIRD?

What does WEIRD sampling threaten?

W — Western
E — Educated
I — Industrialized
R — Rich
D — Democratic
→ Not representative of global populations.

External validity — results may not apply to non-WEIRD populations.

Pros of using university students as participants?

• Convenience sampling → easy access

• Some diversity (international students, opinions)

Cons of using university students?

• Restricted age range

• More affluent

• Not representative of general population

Does the college sophomore problem invalidate studies?

No.
It means we need:

• More data

• Tests of boundary conditions

• Samples outside colleges

What are boundary conditions?

Knowing when a finding does or doesn’t apply, and what features of the college sample make results differ.

How should researchers address sex/gender differences in generalizability?

Include sex/gender as person variables and analyze differences in responding or interpretation.

What is a direct replication?

Repeating a study’s procedures exactly to see if the same results occur.

What is the purpose of direct replication?

Test whether results generalize to new samples under the same conditions.

Example of unsuccessful direct replication?

• Original: Rasinski et al. (2005) → honesty primes increased admitting problematic behaviors

• Replication: Pashler et al. (2013), two large samples → failed to replicate
  → Suggests original effect may not generalize.

What is a conceptual replication?

Testing the same theoretical relationship using different operationalizations of IV or DV.

Why do researchers prefer conceptual replications?

• More creative

• Allows new discoveries

• Tests if the relationship holds across different definitions

Example of successful conceptual replication?

• Original: Kassin & Kiechel (1996) → false confessions under accusation

• Conceptual: Horselenberg et al. (2001)

  • Added financial penalties & extra measures

  • Still found high false confession rates (82%)
    → Supports theory, rules out alternative explanations.

What is a meta-analysis?

A statistical technique that combines results across many studies to identify overall trends.

Why are meta-analyses important for generalizability?

They:

• Use multiple operationalizations

• Include different populations and settings

• Provide a robust, big-picture estimate of effects

• Detect reliable patterns across studies

What can meta-analyses reveal that single studies cannot?

• Whether results are consistent across labs

• Whether effects are small/large

• Whether certain variables moderate effects

• Broader generalizability of psychological phenomena

Direct vs. Conceptual Replication?

• Direct: Same procedure → checks if findings duplicate

• Conceptual: Different procedure → checks if theory holds

B) is correct

A) is correct