Critical Evaluation and Thinking in Psychological Research

Seven Broad Questions for Critical Evaluation of Research

• General Maxim: The principle of ‘caveat emptor’ (let the buyer beware) applies to consuming research. One should not take "University studies indicate…" at face value as most studies have limitations.

• 1. Does the theoretical framework make sense?

  • Does the specific hypothesis flow logically from a broader theory?

  • Are terms defined logically and consistently throughout the study?

  • Example: If studying the relationship between social class and intelligence, the article must explain why such a relationship should exist and maintain consistent definitions for these terms.

• 2. Is the sample adequate and appropriate?

  • Representativeness: Does the sample represent the population of interest? If the goal is to generalize to the community, a sample of only undergraduates is insufficient.

  • Sample Size ($n$): The sample must be large enough to determine if results are meaningful or accidental.

  • Statistical Example: Rolling a dice $6$ times and getting ‘snake eyes’ twice is not enough evidence to conclude the dice are loaded, as this could happen by chance.

  • Qualitative Considerations: Researchers should consider if the strategy of inquiry and specific methods are appropriate for the research question.

• 3. Are the measures or methods and procedures adequate?

  • Quantitative Validity: Do the measures assess what they were designed to assess?

  • Control: Were proper control groups used to rule out alternative explanations?

  • Confounding Variables: Did investigators control for outside influences?

  • Example: In interviews, researchers must consider if the gender of the interviewer influenced participant responses.

  • Qualitative Suitability: The chosen approach must be suitable for the specific research questions being explored.

• 4. Are the data conclusive?

  • Do the data demonstrate the researcher’s claims?

  • Data is typically presented in a "Results" section via graphs, charts, or tables.

  • Readers must look for alternative interpretations that might explain the findings as well as or better than the researcher's explanation.

  • Qualitative Analysis: The choice of methodology determines how data is analyzed and interpreted.

• 5. Are the broader conclusions warranted?

  • Correlation vs. Causation: Researchers must not overstate findings.

  • Example: A finding that children who watch violent TV hit other children shows a correlation, but it does not prove causation. It may be that aggressive children prefer violent TV, or that such shows only trigger violence in predisposed children.

  • Epistemological Position: In qualitative research, the underlying philosophical position provides deep insight into psychological phenomena.

• 6. Does the study say anything meaningful?

  • This is known as the "so what?" test. It asks if the study tells us anything new or leads to future research.

  • Meaningfulness depends on the importance and adequacy of the theoretical perspective.

  • Important studies often produce surprising findings or help choose between opposing theories (Abelson, $1995$).

• 7. Is the study ethical?

  • Are human or animal participants treated humanely?

  • Do the "ends" (incremental knowledge) justify the "means"?

  • Governance: The Australian Psychological Society (APS) publishes guidelines (APS, $2002$, $2003$, $2007$).

  • Institutional Oversight: Universities and institutions use ethics committees and Institutional Review Boards (IRBs) to review proposals. These boards can reject studies or demand revisions to protect participant welfare, even for benign studies like those focusing on memory or math.

The Replicability Crisis in Psychology

• Definition: Replicability (or reproducibility) is a key attribute of empirical research, meaning that conducting the same study again should yield the same results. The "crisis" refers to the difficulty researchers face in reproducing results of earlier research.

• The Reproducibility Project (Nosek et al., 2015):

  • Collaboration of $270$ researchers.

  • Attempted to replicate $100$ studies published in $3$ psychology journals in $2008$.

  • Findings: Of the original studies, $97$ out of $100$ showed statistically significant results. Only $36$ of the replications achieved significant results.

• Perspectives on the Crisis:

  • Stroebe and Strack (2014): Argue that failure to replicate is not necessarily a crisis; repeating a study at a different time with a different population might not reflect the same theoretical construct.

  • Stangor and Lemay (2016): Suggest small-to-medium sample sizes lead to data misrepresentation and exaggerated effects.

  • Řwiątkowski and Dompnier (2017): Point to the recurrent use of low statistical power as a reason for low replicability.

• Systemic Issues and Ethics:

  • Publication Bias: There is a bias toward publishing new, surprising results over less exciting replications (Social Science LibreTexts, $2017$).

  • Questionable Research Practices (QRPs): To get published, researchers may use selective deletion of results, "cherry-picking" data, or manipulating participant numbers.

  • Outright Dishonesty: Diederik Stapel ($2011$) admitted to widespread data fabrication affecting over $30$ publications.

• Solutions and Actions:

  • Open Research: Making all data, results, and protocols available to improve transparency (Leyser et al., $2017$).

  • Pre-registration: Outlining the study layout and analytical strategies before data collection begins to prevent selective data use.

  • Reward Structures: Ottoline Leyser (Cambridge University) suggests the Reproducibility Crisis is actually a "publication bias crisis" driven by reward structures. Systems should reward confirmatory work and data sharing, not just "headline-grabbing" results.

Critical Thinking in Psychology

• Definition: Carefully examining and analyzing information to judge its value and considering alternative explanations before accepting information as true.

• The Purpose of Critical Thinking: It is not about simply criticizing; it is a logical and rational assessment of strengths and weaknesses. It is essential for interpreting scholarly journals and building research on sound foundations (Burton, $2020$).

• Three Key Principles:

  1. Scepticism: Questioning assumptions and analyzing if evidence supports results. One should not accept an assertion just because it is in print or from an authority figure.

  2. Objectivity: Taking an impartial approach based on logic and evidence, setting aside personal biases or subjective beliefs.

  3. Open-mindedness: Considering all sides of an issue and being willing to accept evidence that contradicts personal experience.

Fallacies in Arguments

• 1. Straw Man: Authors deliberately attack a weakened or "decoy" version of an opposing argument (the straw man) to make their own position seem stronger. Destroying a weak argument does not prove the validity of one's own; only evidence can do that.

• 2. Appeals to Popularity: The fallacy that an argument is true because it is popular or widespread.

  • Historical Example: The belief that the earth was flat was once popular but untrue.

• 3. Appeals to Authority: The fallacy that an argument must be true simply because of the status of the person making it. Evidence should be assessed independently of the author's reputation.

• 4. Arguments Directed to the Person (Ad Hominem): Attempting to strengthen a position by attacking the character or supposed failings of the people who hold alternative arguments, rather than addressing the arguments themselves.