Rad 2018

Authors & Publication Info

• Mostafa Salari Rad, Alison Jane Martingano, Jeremy Ginges

• Affiliation: Department of Psychology, New School for Social Research and Princeton University

• Edited by Douglas L. Medin, Northwestern University

• Approved Date: July 25, 2018

• Received for review: January 16, 2018

Introduction

• Core Goals of Psychological Science: The fundamental objectives of psychological science are two-fold: first, to identify and understand the universal aspects of human psychology that apply across all individuals and cultures; and second, to investigate and explain how specific contexts and cultural backgrounds contribute to the observed variability in psychological phenomena.

• Current Sampling Issues: A significant challenge in current psychological research is its overwhelming reliance on WEIRD populations (Western, Educated, Industrialized, Rich, and Democratic). Data indicates that approximately $68\%$ of psychological research participants are from the U.S., and a remarkable $96\%$ are drawn from Western nations.

  • Significance: This highly concentrated sample, representing only about $12\%$ of the world

’s population, raises serious concerns. Theories developed from such a narrow group are potentially unrepresentative of the vast diversity of human psychology, leading to conclusions that may not generalize to humans as a whole. This lack of representativeness can skew our understanding of fundamental human cognitive and behavioral processes.

• Review by Henrich et al. (2010): Landmark research previously highlighted that WEIRD populations frequently exhibit statistically anomalous or outlier characteristics across a wide range of psychological phenomena. These include, but are not limited to, fundamental aspects such as visual perception (e.g., susceptibility to optical illusions), spatial reasoning, concepts of fairness, moral reasoning, self-concept, and even basic cognitive processes like analytical vs. holistic thinking. This suggests that findings from WEIRD samples may not offer a reliable baseline for human universals.

Persistence of WEIRD Sampling

• Research Methodology: To assess the evolution of sampling practices, the authors conducted a rigorous examination using content analysis of research articles published in Psychological Science, a leading journal in the field, specifically focusing on publications from 2014 and 2017.

• Documented Findings: The analysis revealed a concerning persistence in the reliance on WEIRD samples across psychological studies. Crucially, there was minimal observable change in sampling practices even after the scientific community became increasingly aware of and critically discussed the problem of over-reliance on WEIRD populations. This indicates a systemic challenge that goes beyond mere awareness.

• Questions Explored: This research aimed to address several critical questions:

  1. Have psychological scientists proactively diversified their sampling efforts to include a broader range of human populations beyond the traditional WEIRD groups?

  2. Are scholars sufficiently aware of the profound influence that culture and context play in shaping psychological phenomena, and do they consider these factors when making broad generalizations about human behavior?

Analysis of Psychological Science Publications

• 2014 Study Analysis: The initial analysis in 2014 involved a comprehensive review of a total of $286$ articles. After meticulously excluding commentaries, reviews, and studies focusing on non-human subjects, $223$ original research articles remained for in-depth examination.

  • Samples Analyzed: From these articles, $450$ distinct samples, derived from $428$ individual studies, were thoroughly categorized. Classification was based on geographical origin, cultural context, and the extent to which they fit the WEIRD characteristics. This detailed categorization allowed for a nuanced understanding of where participants were sourced.

  • Demographic Reporting: A significant issue identified was the lack of transparency in reporting; $51$ studies, representing $11.41\%$ of the analyzed samples, failed to provide clear demographic identification of their participants. This omission hinders the ability of other researchers to assess representativeness and generalizability.

  • Diversity Reporting: The geographical breakdown of samples further underscored the bias: $57.76\%$ originated from the U.S., $71.25\%$ from English-speaking countries, and an overwhelming $94.15\%$ from Western countries. Concurrently, there was noted underrepresentation in terms of ethnic and gender diversity among participants, even within these predominantly Western samples.

• 2014 Findings: The analysis of the 2014 publications revealed several critical data points regarding reporting practices:

  • A substantial $72\%$ of the abstracts did not contain any information regarding the sample population used in the research, making it difficult for readers to immediately gauge the study

’s scope and generalizability.

- $83\%$ of the studies failed to analyze or discuss the potential effects of sample diversity on their findings, neglecting a key aspect of methodological rigor.

- An equally high $84\%$ of articles did not include recommendations for future research to study the investigated phenomena in other cultural contexts, indicating a missed opportunity for promoting broader scientific inquiry.

2017 Follow-Up Study

• Scope: To assess any shifts over time, a follow-up study was conducted analyzing $40$ articles and $94$ individual studies sampled from three distinct issues of Psychological Science published in 2017.

• Regional Sample Composition: The results showed a continuing pattern, with $50.8\%$ of the samples originating from the U.S. and a combined total of $70\%$ from North America and Europe. This indicates a consistent geographical bias over the period examined.

  • Underrepresentation: Alarmingly, the analysis revealed that no studies in the sampled issues from 2017 included populations from Africa or Latin America, highlighting massive gaps in geographical representation.

  • The follow-up analysis confirmed that consistent patterns were found in both 2014 and 2017 analyses across various metrics, including sample diversity and reporting practices, suggesting a persistent challenge rather than an improving trend.

Recommendations for Improvement

For Authors

• Sample Characteristics Reporting: Authors must diligently report comprehensive demographic data for their samples, including but not limited to age, socioeconomic status (SES), ethnicity, gender, and nationality. Providing these details is crucial for assessing internal and external validity and for allowing other researchers to evaluate the generalizability of findings.

• Findings Qualification: Researchers should explicitly tie their findings to the specific sampled populations, clearly detailing any generalizability constraints. This involves articulating what populations the results can reasonably be generalized to and, importantly, which populations they cannot, based on the sample

’s characteristics.

• Justifying Sample Choice: Authors are encouraged to provide clear and robust justifications for their selection of a particular demographic. This could involve theoretical reasons for studying a specific group, practical constraints, or the initial exploration of a phenomenon within a well-understood population, always with an eye toward future diversification.

• Implications of Cultural Context: It is imperative for authors to proactively analyze and discuss how the cultural background and specific context of their sample may have influenced the observed findings. This critical reflection helps to disentangle universal psychological processes from culturally specific manifestations.

For Editors and Reviewers

• Novelty and Importance of Non-WEIRD Samples: Editorial boards and reviewers should actively recognize and prioritize studies that engage with non-WEIRD populations, treating them as uniquely valuable and novel contributions to the psychological literature. This shift in perspective can encourage more diverse research.

• Diversity Badges: Journals should consider introducing a system of badges or labels to explicitly recognize and highlight studies that demonstrate robust efforts in diversifying their samples beyond WEIRD populations. This could encourage other researchers to adopt similar inclusive sampling practices.