Psychological Research Methods, Scientific Integrity, and Replication

Human Decision-Making and Memory Processes

Irrationality in Decision-Making:
- Human judgment and decision-making routinely diverge from purely rational models.
- Water Bottle Study:
  - In an experimental setup, half of the participants were given a water bottle and asked how much money they were willing to sell it for.
  - Being in physical possession of the water bottle conferred a greater sense of value to the object for those participants.
  - Endowment Effect: The phenomenon where simply owning an object causes it to hold greater subjective value to that person.
  - Demonstrates that humans are not rational in decision-making, highlighting the need to re-examine how choices are carried out.
- Colonoscopy Perception Study:
  - Investigated how individuals actually perceive and evaluate experiences over time.
  - Findings showed that people take cognitive "shortcuts," evaluating an overall experience based on its peak worst parts and the ultimate outcome, rather than integrating the experience in its entirety or evaluating overall duration.
  - When presented with a longer procedure that ended with a less demanding period versus a shorter, continuously uncomfortable procedure, patients chosen the longer experience over the shorter, less demanding one.
- The Trolley Problem:
  - A classic moral dilemma evaluating whether an individual should flip a track switch to kill $1$ person on an alternate track to save $5$ people on the main track.
  - Rational Option: Save the $5$ people and sacrifice the $1$ person.
  - Irrational/Emotional Barrier: The heavy emotional impact of actively causing harm creates an obstacle that makes individuals unable to carry out the rational action.
Practical Usefulness of Psychology:
- Approximately $50\%$ of individuals will experience a diagnosable mental illness during their lifetime.
- Almost all people will personally know someone who struggles or has struggled with mental illness.
The Generation Effect and Memory Retention:
- Experimental Design:
  - Participants were presented with $504$ nouns across $3$ sessions on $3$ successive days ( $168$ words per session).
  - Task: Generate associate words for each noun.
  - Conditions:
    - $1$ -associate condition: e.g., target car $\rightarrow$ associate drive.
    - $3$ -associate condition: e.g., target car $\rightarrow$ associates drive, fast, crash.
- Memory Testing Protocol:
  - An incidental memory test was administered immediately following session $3$ (participants were unaware that a memory test was coming).
  - Participants were provided with retrieval cues that were either self-generated or generated by someone else.
- Findings:
  - Self-generated cues resulted in a higher recall rate than cues generated by another person, particularly when participants were asked to associate $3$ words.
  - The number of cognitive connections positively correlates with memory retention.

Psychology as a Science and the Replication Crisis

The Replication Crisis:
- A widespread methodological crisis in psychological science stemming from non-replicable findings, questionable practices, and explicit fraud.
- Unbelievable Scientific Claims:
  - A peer-reviewed journal published evidence across $9$ different studies claiming that people could predict the future (precognition).
  - Task: Participants were shown $2$ curtains and had to select which curtain concealed a photo.
  - Results: Male participants predicted the photo location at a rate better than chance, specifically when the hidden image was a form of erotica.
- Academic Fraud Case Study (Diederik Stapel):
  - Social psychologist Diederik Stapel committed severe academic fraud by fabricating data across numerous published studies.
  - Data was never actually collected, meaning the theories published were never empirically tested.
  - Demonstrates how heavily the scientific enterprise relies on trust.
  - The fraud was identified by undergraduate students rather than peer reviewers.
  - Spurred the development of meta-science, where researchers recruit other researchers to systematically replicate published studies.
- Scope of Unreplicability: Empirical audits revealed that approximately $67\%$ of tested studies failed to replicate.
Rationale for Empirical Science:
- Science is highly esteemed, yet non-scientific services persist (e.g., eHarmony markets a dating compatibility application that was not grounded in scientific evidence).
- Science provides a superior framework compared to traditional alternatives, such as reliance on authority and intuition.
Cognitive Limitations and Biases:
- Hindsight Bias: The tendency to believe, after learning an outcome, that one would have foreseen it (commonly referred to as the "I knew it all along" phenomenon).
- Overconfidence: The systematic human tendency to think we know more than we actually do.
- The Wason Selection Task:
  - Participants inspect a series of cards, where each card has a triangle on one side and a circle on the other, colored red or black.
  - Goal: Determine which cards must be turned over to validate a conditional rule.
  - Takeaway: Scientific inquiry requires explicitly testing cases that could disconfirm or prove a rule wrong, rather than seeking only confirming evidence.
- Confirmation Bias:
  - The bias of seeking out confirming evidence while ignoring disconfirming evidence (e.g., believing oneself to be considerate and one's roommate to be a jerk by recalling $12$ instances of considerate behavior while ignoring counter-examples).
  - Remedy: Actively pay attention to disconfirming evidence; this mindset is critically important in understanding and addressing the replication crisis.

Statistical Hypothesis Testing and Problematic Research Practices

Null Hypothesis Significance Testing (NHST):
- Baseline Assumption: Research begins with the assumption that the null hypothesis ( $H_0$ ) is true.
- Decision Logic: Extremely unlikely outcomes under $H_0$ are interpreted as evidence against the null hypothesis.
- Significance Threshold ( $p$ -value): Defined as an outcome occurring less than $1$ in $20$ times ( $p < .05$ ).
- Coin Flip Illustration:
  - Flipping a fair coin $10$ times carries an expected outcome of $50\%$ heads/tails ( $5$ heads).
  - Extremely high or low outcomes ( $0$ , $1$ , $9$ , or $10$ heads) occur less than $5\%$ of the time under a fair coin assumption.
  - Obtaining $9$ heads out of $10$ flips yields a statistically significant result, leading to rejection of the null hypothesis.
  - Error Rate in Large Samples: Across a large number of trials, statistically significant results will occur purely due to random error at a rate equal to the alpha threshold ( $5\%$ ).
- Methodological Goal: Test an effect that is extremely unlikely under $H_0$ , but would be likely under the alternative hypothesis ( $H_1$ ); if observed, reject H_0$.\n\n* **Definitions and Misconceptions**:\n * **Significance Testing**: Tests whether it is likely that obtained results occurred by chance, assuming the null hypothesis is true.\n * **P $-Value**: The probability of obtaining a result at least as extreme as the observed result, assuming the null hypothesis is true. A result of$ p = .02 $means there is a$ 2\% $chance of obtaining a difference at least this big if$ H_0 were true.\n * **Common Misconceptions**:\n * The p-value is **NOT** the likelihood that the null hypothesis is true.\n * The p-value is **NOT** the likelihood that the alternative hypothesis is true.\n * Statistical significance does **NOT** equal real-world importance.\n\n* **Problematic Research Practices (QRPs)**:\n * Flawed applications that exploit the baseline statistical error rate (5\%). How surprising an event appears depends directly on how many total tests or observations were conducted.\n * **File Drawer Problem**:\n * Running multiple studies, reporting only the significant results, and hiding non-significant findings in the "file drawer".\n * Distorts scientific literature because a significant result occurs about 5\% of the time even when the null hypothesis is true.\n * **Multiple Comparisons**:\n * Running multiple statistical tests on the same dataset and selectively interpreting only the significant findings.\n * Powered by hindsight bias, which enables researchers to construct compelling post-hoc narratives around random noise.\n * *Example*: Evaluating quarters across 9 $trials versus$ 75 trials alters the background error rate compared to standard coin flips.\n * **P-Hacking**:\n * The act of altering statistical analyses (e.g., tweaking data inclusions, changing metrics) based on how those adjustments affect the obtained p $-value until$ p < .05 is reached.\n * Occurs easily because researchers do not feel like they are doing anything wrong while crossing ethical boundaries.\n * **Contributing Factors**: Poor understanding of p-values, overconfidence, hindsight bias, and a historical lack of replication studies.\n\n# Solutions to the Replication Crisis and Mertonian Norms\n\n* **Institutional Solutions and Scientific Norms**:\n * The core philosophy of science is fundamentally sound, but its integrity depends entirely on upholding scientific norms.\n * **Merton's Norms**:\n * **Universalism**: The validity of a claim must not depend on who is making the claim (evaluating merit regardless of social status; maintaining equality of opportunity).\n * **Communality**: Scientific findings are common property of the scientific community and general public; research must be shared openly rather than hidden.\n * **Disinterestedness**: Scientists should focus strictly on identifying objective truth rather than personal career advancement, status, or financial gain (blocking entitlement).\n * **Organized Skepticism**: Researchers must not accept claims at face value; proof and verification are mandatory. Researchers must critically seek flaws in both their own work and others' work.\n\n* **Practical Structural Solutions**:\n * **Open Science**:\n * Making study materials and data openly accessible to the public.\n * *Problems Solved*: Directly mitigates the file drawer problem, p-hacking, and unreported multiple comparisons.\n * *Norms Supported*: Universalism, Communality, Disinterestedness, Organized Skepticism.\n * **Replications**:\n * Repeating studies systematically, preferably conducted by independent research teams.\n * **Preregistration**:\n * Formally declaring all research decisions (hypotheses, methodologies, analysis plans) before conducting the study.\n * Posting plans publicly so external researchers can verify compliance and prevent post-hoc analytical changes.\n\n# Process and Methods of Psychological Research\n\n* **Comparison of Psychological History, Process, and Daily Life**:\n\n![Comparison table showing the alignment between the history of psychology, scientific process, and everyday logic.](https://assets.knowt.com/pdf-flow-prod/d5284e63-a78e-41a1-80c7-027a7b129a2a-figures/0.png)\n\n * *History of Psychology*: Moves from Intuition Thinking \rightarrow $Systematic Observation$ \rightarrow Psychological Research: Testing Hypotheses.\n * *Scientific Process*: Moves from Theory \rightarrow $Descriptive Research$ \rightarrow Hypothesis Testing: Correlational Studies & Experiments.\n * *Everyday Life / Same Process*: Moves from Intuition Thinking Logic \rightarrow $Systematic (?) Observation$ \rightarrow$$ Testing Hypotheses (?).
The Scientific Attitude:
- Intuition is not inherently bad, but it is frequently incorrect.
- Subjective confidence is a poor guide for objective accuracy.
- Researchers must transition beyond intuition and rely on systematic empirical observations.
Types of Descriptive Research:
- Case Study: An observation technique in which one person (or other unit) is studied in depth in the hope of revealing universal principles.
- Survey: A technique for ascertaining the self-reported attitudes or behaviors of people.
- Naturalistic Observation: Observing and recording behavior in naturally occurring situations without trying to manipulate and control the situation.
- Ecological Validity: The degree to which an effect has been obtained under conditions that are typical for what happens in everyday life.
Core Research Vocabulary:
- Theory: An explanation using an integrated set of principles that organizes and predicts observations.
- Hypothesis: A testable prediction that is often implied by a theory and sometimes unstated (agreed upon broadly in scientific practice).
- Operational Definition: A precise statement of procedures (operations) used to define research variables.
  - Example: Intelligence may be operationally defined as what a specific intelligence test measures.
- Illusory Correlation: Perceiving a correlation or relationship between variables when no actual correlation exists.