Experimental Psychology: The Foundations of Scientific Research and Literature
Course Materials and Orientation
Slides and Notes: Slides provided in class serve as summaries of content and are intended to be used as study notes for exams and quizzes. They cover virtually all required material.
Pedagogical Strategy: Students are encouraged not to transcribe slides verbatim during class but to use the pre-distributed versions to organize materials.
Replication as a Cornerstone of the Scientific Method
Core Principle: Replication is a fundamental constraint in science ensuring that no one has to take a scientist's word; any phenomenon can be reproduced in another lab given specific equipment and circumstances.
Operational Definitions: To make psychology a science, it requires a method involving operational definitions. These specify the procedures used to define a property or measure a variable.
Method Section Fidelity: Research reports include a method section specifying precise procedures so that other researchers can replicate the experiment exactly.
Laws of Nature: A phenomenon, as a law of nature, should present itself identically to different researchers if the specified procedural conditions are met.
The Replication Crisis in Social Sciences
2013 Peak: Around 2013, watchdogs attempted to reproduce experiments from prestigious psychological journals.
Failure Rates: Only slightly less than of experiments successfully replicated, meaning approximately of published knowledge was unreliable.
Sociological Drivers:
Economic pressure on scientists.
Individual ego and status concerns.
The "Publish or Perish" incentive structure of academia.
Modern Responses to the Replication Crisis
Registration of Studies: A request for researchers to register their study design and hypotheses before data collection begins.
Adversarial Collaboration: An experiment designed by Lab A is run by Lab B (often a researcher with a competing theory). This maximizes the power of confronting theory with reality and eliminates bias in data collection.
Publication of Negative Findings: Moving away from the bias of only publishing positive results. Pre-registration ensures a study is published regardless of whether the result is positive (phenomenon exists) or negative (no evidence of phenomenon).
Errors in Scientific Reasoning: Type I and Type II
Type I Error (True-Negative Miss): Asserting an effect exists when it does not. This is considered the more damaging error because it contaminates the body of knowledge with unreliable "facts."
Type II Error (Miss): Failing to claim an effect exists when it actually does. This often happens if the standard of proof is too high, leading to a slowdown in scientific progress.
Conservative Nature of Science: Science prioritizes reliability over speed; therefore, researchers generally worry more about Type I errors.
Malpractice: P-Hacking and Publication Bias
Definition of P-Hacking: Maneuvering statistical procedures to reach a desired level of significance.
Data Peeking: A common p-hacking method where researchers keep collecting data until they reach a significant p-value and then stop.
Best Practice: Researchers must decide on the sample size (e.g., ) systematically before starting and stick to that number regardless of the intermediate p-value.
Journal Bias: Academic journals are historically uninterested in negative results, creating a strong bias toward outcomes that show an effect.
Classroom Demonstration: Hindsight and Confirmation Biases
Detroit vs. Michigan Homicide Study: People consistently overestimate homicides in Detroit while underestimating them for the state of Michigan as a whole.
Confirmation Bias: Approximately of people tend to only look for evidence that confirms their theory (e.g., turning over a card that matches a rule) rather than seeking disconfirmation.
Case Study: Testing Zodiac Sign Personality Predictions
Theory Being Tested: Zodiac signs (based on birth date) can predict specific personality traits.
Methodology: Students were presented with personality profiles (A through L) and asked to select the one that fit them best without knowing the associated sign.
The Logic of Hits and Misses: People often overestimate "hits" (traits that apply) and ignore "misses." Providing all profiles forces a fair choice of which one represents the subject "the most."
Results: In a class of 5 students, there were 0 correct predictions ( hits).
Hypothesis Testing Logic and Mathematics
Alternative Hypothesis (): The theory being tested (e.g., zodiac signs can predict personality).
Strong version: The theory predicts correctly of the time ( in class).
Weak version: The theory predicts better than expected by chance.
Null Hypothesis (): The default position that there is no effect or the theory does not work.
Weak version: Successes are not reliably above chance level.
Probability Thresholds (-values):
Social Sciences: Results are considered significant if the probability they occurred by chance alone is lower than ().
Physical Sciences: Thresholds are much stricter, often reaching for experiments involving particles or masses.
Calculations for N=5:
Probability of at least 1 hit by chance: .
Probability of at least 2 hits by chance: (considered "marginally significant").
Probability of at least 3 hits by chance: Less than (strong evidence for theory).
Statistical Software Tools
R: A very powerful statistical programming language. It is the current industry standard but has a steep learning curve.
JASP: Developed by the University of Amsterdam. It uses R as its engine but provides a user-friendly graphical interface. It is the tool used in this course.
Variable Types: The zodiac experiment used a "nominal" variable (Correct vs. Incorrect).
Confidence Intervals: In JASP, if the confidence interval does not touch the chance level line ( or ), the result is significant.
Sources of Research Ideas
Non-Systematic Sources
Inspiration: Creative flashes regarding a problem.
Serendipity: Finding something useful while looking for something else (e.g., the discovery of penicillin by Fleming, though the transcript attributes the spirit of intuition to Pascal's context).
Everyday Life: Observing interactions, such as those on a New York subway, and applying a "scientist's hat" to observe human communication.
Systematic Sources
Past Research: Reading literature to find conflicting findings (e.g., does exercise help depression?).
Conflicting Interpretations: Designing a study to determine which of two different theories correctly explains a single result.
Failure to Replicate: Investigating why a previously accepted effect can no longer be found.
Extension and Qualification: Testing if known effects apply to new populations (e.g., grad students vs. undergrads).
Theory Testing: Testing specific sections or implications of a broad theory.
Research Frameworks: Basic vs. Applied
Basic Research:
Goal: Discovery of universal truths about the species (homo sapiens).
Settings: Academic labs/Universities.
Timescale: Flexible, sometimes decade-long.
Limitation: May have no immediate practical relevance.
Applied Research:
Goal: Solve specific, real-world problems.
Settings: Professional agencies, corporations (e.g., social work agencies).
Timescale: Tight schedule determined by stakeholders (CEOs, federal agencies).
Limitation: Knowledge may be limited to specific circumstances and not generalizable.
Qualitative vs. Quantitative Research
Qualitative Research:
Focus: Exploring ideas and formulating hypotheses.
Methods: Summarizing, categorizing, interpreting.
Data: Expressed in words; includes open-ended questions.
Participants: Few subjects.
Quantitative Research:
Focus: Testing specific hypotheses.
Methods: Mathematical and statistical analysis.
Data: Numbers, graphs, tables; sharp/minimalist language.
Participants: Many subjects to increase statistical power.
Levels of Scientific Knowledge
Exploratory: Focuses on whether a phenomenon exists at all (e.g., investigating the actual percentage of fraud in food stamps).
Descriptive: Focused on identifying the properties of a phenomenon (e.g., what substances are people with trauma most likely to abuse?).
Predictive: Focused on relationships between phenomena without claiming causation (e.g., age as a predictor of car accidents for insurance premiums).
Explanatory: Focused on identifying causes. This is the primary focus of the current course (e.g., heat as a cause of increased aggression). Requires experimental manipulation of variables.
The Scientific Literature: Types of Articles
Research Reports: Standard reports on a specific project (what students produce for the class).
Reviews:
Systematic Review: Uses pre-defined, transparent methods to gather all available empirical research on a specific question to minimize cherry-picking.
Nonsystematic Literature Review: Broad, descriptive, or historical overviews (e.g., a 20-year history of a specific theory).
Meta-Analysis: Statistical process of combining results from many studies to reach a precise measurement. It is used to settle controversies from conflicting claims across the literature.
Theoretical Papers: Large-scope works that establish major theories (e.g., attachment theory).
Position Papers: Propose new directions or challenge current methods at the cutting edge of a field.
Key Organizations and Journals in Psychology
American Psychological Association (APA): The oldest organization ( century); publishes over 50 journals; historically dominated by clinical psychology.
Key Journals: Psychological Review, Psychological Bulletin, Journal of Experimental Psychology.
Association for Psychological Science (APS): Founded in 1988 with a focus on scientific research.
Key Journals: Psychological Science, Current Directions in Psychological Science.
Psychonomic Society: Founded in 1959; focused on cognition and the mind; publishes Psychonomic Bulletin & Review.
Cognitive Science Society: Founded in 1979; focused on multidisciplinary mind research.
General Science Journals: Nature and Science are the most prestigious, publishing concise, groundbreaking reports, though they often lack methodological detail.
Practical Literature Search Tools
Literature Explosion Strategy: Find one good review article and use its references to find other papers, repeating the process to map the field.
Databases:
PsycINFO: Focus on psychology journals.
PsycTESTS: Specifically for finding measurement scales and surveys.
Google Scholar: A broad, efficient search engine.
Zotero: A free, cloud-based reference manager used to organize citations and automatically format papers in APA style.
AI (LLMs): Useful for brainstorming keywords and summarizing text, but unreliable for citations. AI tends to fabricate references or misrepresent findings, so it should only be a supplement to academic databases.
Questions & Discussion
Question from Harriet: Can the rigorous process of scientific constraints ever be taken too seriously, leading to a "Type II Error" (missing a real effect)?
Response: Yes, this happens, and it slows down the pace of science. However, science is a conservative effort that prefers reliability over speed. Proving a lie (Type I) is viewed as more damaging to the integrity of a discipline's textbook than missing a truth (Type II).
Question from Joshua: Overview of why we need a method in psychology to make it a science.
Response: Psychology uses operational definitions and structured data collection to confront theory with reality in a powerful, unbiased way, moving beyond subjective intuition.
Question from Abio: Should the brainstormed research question be quantitative and explanatory?
Response: Not necessarily. Any spark of curiosity is fine for the demonstration. While the class eventually focuses on quantitative methods, the goal of the next session is to practice the skill of searching the literature for any topic.