Generalization and Reporting Research

external validity

the extent to which the results of a study can be generalized to other people, settings, times, and measures

• The extent to which generalization is reasonable of justified

convenient samples - college students and volunteers

• Unlikely to be representative of the general population

• Different types of people volunteer for different types of studies

• Reduce external validity

convenient samples - online samples

• Amazon Mechanical Turk; Prolific

  • Likely to be a biased sample

  • Bots

in defense of convenient sampels

• Bigger issue in some types of research than others

• Should not be discarded unless there are good reasons

• Increasingly diverse and representative

• Replications

replication and robustness

• From lab to real-world context

  • Anderson, Lindsay & Bushman (1999). 38 pairs of studies. Lab studies generally produced psychological truths

• Conceptual replications

  • Use different procedures to replicate a research finding

  • Operationalize variables different

reviews

• synthetic lit review

• meta-analyses

• Lindberg et al. (2010)

synthetic lit review

• Read a large number of studies on a particular topic

• Summarize and integrate findings

• Evaluate the strengths and weaknesses of the findings and conclusions

• Point out inconsistencies or areas where findings are lacking

• Discuss future directions

meta-analyses

• Combines the findings of a number of studies using a set of statistical procedures

• Focus on effect sizes

• Examine moderators

• Correct for the file drawer problem

Lindberg et al. (2010)

• Gender differences in math performance

• 242 studies between 1990 and 2007

• Overall minimum effect size of gender difference

generalization - non-human animals

• Generalize to humans

• Basic processes: learning, memory, perception, motivation, etc.

• Biological mechanisms

reporting research

• The importance of honest and complete reporting

• Distinguishing statistical significance vs practical significance

• Books or book chapters

• Academic journals

• Dissertations and thesis

• Conferences

types of peer reviews

• Single blind

• Double blind

• Transparent

• Preprint repositories

  • PsyArXiv, bioArXiv, GitHub