Comprehensive Study Guide: Psychological Research Methods, Falsifiability, and Statistical Significance

Course Administration, Logistics, and Academic Policies

• Weekly Required Readings:

  • Core Assigned Text Topics:

  • Why Science: Overview of scientific methodology and empirical inquiry.

  • Thinking Like a Psychological Scientist: Primer on the scientific mindset, leading into methodology and psychological research frameworks.

  • Outside D2L Reading Case Study:

  • Subject: A prominent Cornell University food scientist who served as a dietary and food policy adviser during the Barack Obama presidential administration (advising on national dietary strategies and public school lunch structures).

  • Incident: The researcher authored a self-reflective blog post intended to showcase his productivity and research workflow.

  • Outcome: The post inadvertently exposed severe, problematic research practices (including data dredging and selective reporting), revealing that his published findings were unreplicable and scientifically flawed.

  • Impact: Triggered widespread criticism across scientific social media (particularly scientific Twitter) and prompted an investigative journalism exposé by BuzzFeed, illustrating systemic vulnerabilities in research integrity.

• Course Management System (D2L) & iClicker Protocol:

  • Mandatory iClicker Participation:

  • Required during every class meeting as an automated attendance verification mechanism.

  • Scoring relies strictly on response submission; answers do not need to be correct to receive full credit.

  • Points are uploaded directly to D2L following the conclusion of each class session.

  • Technical Network Requirements for iClicker:

  • Students must connect mobile devices to designated campus Wi-Fi networks: redanyroom or MSU Net 3.0.

  • Connecting via cellular data alone or using the MSU Guest network causes synchronization failures and lost credit.

  • Excused Absence & Makeup Policy:

  • Students are permitted up to five ($5$) excused absences per semester.

  • Notification Window: Requests must be logged via D2L between 6:00 PM the evening prior to class and 8:00 AM immediately preceding the class session.

  • Makeup Procedure: Logging an absence opens a dedicated D2L portal where students submit original, iClicker-style multiple-choice questions based on course material to receive substitute attendance points.

  • Academic Integrity Rules: Submitting answers on behalf of an absent peer or bringing another student's device to class constitutes a severe academic integrity violation.

• Weekly Quizzes & Grading System:

  • Schedule: Weekly quizzes occur every week except for the first week of the term, final exam weeks, and midterm exam weeks.

  • Time Window: Quizzes open on Thursday mornings and close strictly on Sunday nights.

  • Quiz Environment Rules:

  • Format: Open-book, open-note, unlimited retakes allowed prior to the Sunday deadline to achieve maximum points.

  • Purpose: Low-stakes practice designed to reinforce note navigation, gauge comprehension, and simulate exam items.

  • Prohibition on Generative AI: Submitting quiz questions to Large Language Models (LLMs) or AI tools is explicitly prohibited, as automated retrieval impairs retention and degrades performance on closed-book examinations.

  • D2L Gradebook Mechanics:

  • Drop Policy: The three ($3$) lowest attendance/iClicker scores are dropped automatically at the conclusion of the semester.

  • D2L Display Error: Due to software limitations, D2L displays arbitrary "dropped" labels on current scores mid-semester. Students must disregard mid-semester "dropped" notifications and monitor raw point totals or utilize the official downloadable Grade Calculator spreadsheet.

Replicability in Psychological Science: Review & Empirical Findings

• Key Case Study: Bem (2011) Precognition Research:

  • Publication: Published in a premier peer-reviewed journal in social and personality psychology (Journal of Personality and Social Psychology).

  • Core Finding: Claimed empirical evidence that human beings possess extra-sensory perception (ESP) and can predict future random events (precognition).

  • ** experimental Protocol**: Participants selected which of two electronic curtains displayed on a screen would reveal a target picture. Crucially, the computer algorithm randomly assigned the target picture location *after* the participant registered their choice.

  • Statistical Outcome: Reported a statistically significant effect wherein participant choice accuracy exceeded chance expectations ($50\%$), purporting to demonstrate precognition.

• Open Science Collaboration: Reproducibility Project: Psychology:

  • Methodology: A collaborative effort by a vast network of international researchers to conduct rigorous direct replications of $100$ experimental and correlational studies published in top-tier psychology journals.

  • Primary Finding: Less than half (approximately $36\%$ to $39\%$) of the original research findings were successfully replicated when evaluated using standard criteria for scientific significance.

  • Implication: Highlighted widespread inflation of false-positive results in published literature, catalyzing the contemporary replication crisis.

Foundations of Scientific Inquiry and Alternative Decision-Making Models

• Comparative Decision-Making Frameworks:

  • Science vs. Authority: Relying on figures of authority is inherently vulnerable to bias, personal motives, and institutional agendas. Authority offers no logical guarantee of truth.

  • Science vs. Intuition: Intuition is an indispensable cognitive heuristic for daily survival but relies on unscrutinized cognitive shortcuts that routinely produce systematic, predictable errors.

  • Systematic Bias Example: Confirmation Bias:

  • Definition: The psychological tendency to selectively search for, recall, and favor information that confirms pre-existing hypotheses while ignoring or discounting disconfirming evidence.

  • Demonstration: Illustrated via the Wason Selection Task (card-selection task), where individuals consistently attempt to confirm rules rather than test conditions that could disprove them.

  • Mitigation Strategy: Rigorous decision-making requires deliberate searching for disconfirming evidence.

• Epistemic Esteem and Public Transparency:

  • Commercial entities (e.g., matching platforms like eHarmony, viral trends on TikTok or Instagram) label claims as "scientific" to leverage the high societal prestige and trust commanded by science.

  • Unlike authority or intuition—where underlying reasoning remains obscure—scientific inquiry is defined by methodological transparency. Experimental designs, raw data, statistical scripts, and analytic protocols are exposed for public and peer scrutiny.

• Demarcation Case Analysis: Evolution, ESP, and Ghosts:

  • Public Belief Distribution: High public belief in evolution, low belief in ESP, and moderate belief in paranormal phenomena/ghosts.

  • Barriers to Scientific Evaluation of Ghosts:

  • Lack of Operational Definitions: Absence of consensus regarding the foundational physical or spectral properties of a "ghost" (e.g., visible spectral humanoid vs. invisible energy field vs. localized temperature drop).

  • Inconsistent Evaluative Criteria: Paranormal investigations shift diagnostic criteria arbitrarily between cases (e.g., citing a cold spot in location A, a moving object in location B, and an auditory anomaly in location C), exemplifying confirmation bias.

  • Conditions for Scientific Feasibility: Any phenomenon—including ghosts or ESP—can be studied scientifically provided researchers establish operational definitions, standardized instrumentation (e.g., calibrated thermal sensors), uniform criteria, and controlled testing procedures.

Demarcation Problem and Theories of Philosophy of Science

• The Demarcation Problem:

  • Defined in philosophy of science as the formal challenge of establishing precise criteria to distinguish scientific disciplines and methods from non-science, pseudoscience, and informal reasoning (e.g., Chalmers, 2014, What Is This Thing Called Science?).

  • Science is defined entirely by its methodological approach, not by its subject matter, historical subject classifications, or physical testing location.

• Deconstructing the "Common Sense" View of Science:

  • Common Sense Assumptions:

  1. Science consists of gathering objective, unmediated, clearly observable facts.

  2. Observable facts directly aggregate to build scientific theories.

  3. Science primarily occurs within a physical laboratory.

  • Flaws in the Common Sense Model:

  • Sensory Fallibility: Raw observations are frequently erroneous (e.g., historical belief in a flat Earth based on localized sensory horizons; the optical moon illusion; geocentric solar system models based on visual sun tracking).

  • Theory-Laden Observation: Facts are not self-interpreting. Observations require theoretical context to possess scientific meaning.

• Theory-Laden Observation Principles:

  • Einstein’s General Theory of Relativity: Predicted that immense gravitational masses (such as the Sun) bend surrounding light paths. During a solar eclipse, light from a star situated behind the Sun shifted visually to the side. This observable fact was impossible to interpret—or even look for—without the overarching framework of general relativity.

  • Botanical Field Observation: An untrained observer in a rainforest perceives an undifferentiated mass of green foliage, whereas a botanist applies theoretical taxonomy to observe specific species adaptations, structural variations, and ecological interactions.

• Chalmers’ Criterion for Scientific Facts:

  • An observation statement constitutes a valid scientific fact if it can be straightforwardly evaluated by the senses and survives rigorous physical testing designed to challenge the observation (e.g., altering physical perspective to determine whether a visual artifact is an internal window reflection or an external object).

• The Real Role of Laboratory Research:

  • Laboratories possess no intrinsic scientific magic; their utility lies in enabling variable isolation, precise manipulation, and systemic control over extraneous influences.

  • In non-physical disciplines like psychology, "laboratory control" translates to methodological controls that isolate psychological processes and eliminate confounding explanations, whether conducted inside a physical laboratory or in naturalistic field settings.

Epistemological Reasoning: Inductive vs. Deductive Logic

• Inductive Reasoning (Induction):

  • Definition: A mode of logical reasoning wherein specific empirical observations or premises are accumulated to supply evidence for a generalized conclusion.

  • Direction of Inference: Moves from specific observations to broad generalizations.

  • Fundamental Logical Limitation: Inductive conclusions are never logically certain, regardless of the volume of supporting observations. A single future disconfirming observation refutes the generalization.

  • Developmental Psychology Example:

  • Premise: Baby Maya speaks her first word at $12\,m$ (12 months).

  • Premise: Subsequent observed infants speak their first words at $12\,m$.

  • Inductive Conclusion: All infants speak their first word at $12\,m$.

  • Logical Status: Invalid for absolute certainty; an infant observing a first word at $9\,m$ or $14\,m$ immediately invalidates the universal conclusion.

• Deductive Reasoning (Deduction):

  • Definition: A mode of logical reasoning that moves from one or more general premises to derive a logically necessary, certain conclusion.

  • Direction of Inference: Moves from general structural rules to specific conclusions.

  • Logical Guarantee: If all premises are true, the conclusion must logically be true.

  • Classical Philosophical Syllogism:

  • Premise 1: All humans are mortal.

  • Premise 2: I am human.

  • Deductive Conclusion: Therefore, I am mortal.

Scientific Methodologies and the Falsifiability Framework

• Popperian Falsifiability Framework:

  • Formulated to address the inherent logical limitations of inductive proof in science. Because no amount of positive evidence can inductively verify a theory as absolute truth, science operates deductively via falsification.

  • Operational Objective: Scientists do not attempt to "prove" theories true; instead, they design rigorous empirical tests specifically intended to prove their theories wrong (falsify them).

• Deductive Mechanics of Falsification:

  • Formal Theory specifies a prediction: Predicts Outcome $X$ and explicitly rules out Outcome $Y$.

  • An empirical test is executed:

  • If Outcome $Y$ occurs: The theory is deductively falsified (proven logically wrong) and must be revised or abandoned.

  • If Outcome $X$ occurs: The theory is not verified as absolute truth; rather, it has survived a critical test of falsification.

  • Current scientific knowledge represents the cumulative body of theories that have successfully resisted exhaustive attempts at empirical falsification.

Statistical Significance Testing, Null Hypotheses, and P-Values

• Conceptual Definition of Statistical Significance:

  • Statistical significance is the standard mathematical criterion for evaluating empirical research claims.

  • Core Function: Formally quantifies how surprising or unexpected an empirical result is, assuming that the baseline null hypothesis is correct.

• Probability and Coin-Flipping Model:

  • Baseline Assumptions: Flipping a fair coin has a theoretical probability $P(\text{Heads}) = 0.50$ and P(\text{Tails}) = 0.50$.\n * **Random Variance across Sample Sizes**:\n * 10$Flips ($N = 10):\n * 6$Heads /$4$Tails:$34\%$probability ($p = 0.34); highly expected, not surprising.\n * 7$Heads /$3$Tails:$34\% cumulative probability; not surprising.\n * 8$Heads /$2$Tails:$11\%$cumulative probability ($p = 0.11); moderately surprising, not statistically significant.\n * 9$Heads /$1$Tail:$2\%$cumulative probability ($p = 0.02$); highly surprising, passes significance criteria ($p < 0.05).\n * 1,000$Flips ($N = 1000):\n * 999$Tails /$1$Head: Infinitesimal probability under a fair-coin assumption ($p < 0.0001); leads to absolute rejection of the fair-coin hypothesis.\n\n* **Formal Statistical Definitions**:\n * **Null Hypothesis (H_0)**:\n * The foundational baseline assumption asserting **no effect**, **no relationship**, or **no difference** between experimental groups.\n * *Fair Coin Example*: H_0: P(\text{Heads}) = 0.50$.

  • Canine Breed Mass Study: $H_0: \mu_{\text{Mastiff}} = \mu_{\text{Mutt}}$ (English Mastiffs and mutts have identical average body weights).

  • Pharmaceutical Trial: $H_0: \mu_{\text{Drug}} = \mu_{\text{Placebo}}$ (Symptom reduction in the treatment group equals symptom reduction in the control group).

  • Alternative Hypothesis ($H_1$ or $H_a$):

  • The hypothesis of interest, asserting that an effect, relationship, or group difference truly exists.

  • Fair Coin Example: $H_1: P(\text{Heads}) \neq 0.50$ (The coin is weighted/biased).

  • Canine Breed Mass Study: $H_1: \mu_{\text{Mastiff}} \neq \mu_{\text{Mutt}}$ (English Mastiffs differ significantly in mass from mutts).

  • Pharmaceutical Trial: $H_1: \mu_{\text{Drug}} \neq \mu_{\text{Placebo}}$ (The active drug significantly alters symptom severity relative to placebo).

  • Significance Level ($\alpha$):

  • Verbatim Definition: The probability of rejecting the null hypothesis given that it were true.

  • Scientific Convention: Set by arbitrary consensus across disciplines at $\alpha = 0.05$ ($5\%$ threshold).

  • P-Value ($p$):

  • Verbatim Definition: The probability of obtaining a result at least as extreme if the null hypothesis were true.

  • Decision Rule:

    • If $p < 0.05$: The observed data are sufficiently improbable under $H_0$; researchers reject $H_0$ and declare the result statistically significant.

    • If $p \ge 0.05$: The observed data fail to exceed the threshold of surprisingness; researchers fail to reject $H_0$.

• Type I Error and Methodological Implications:

  • An $\alpha = 0.05$ convention establishes a baseline $5\%$ false-positive rate. In $1$ out of every $20$ statistical tests where the null hypothesis is completely true, sampling variation will yield a false statistically significant result ($p < 0.05$).

  • Widespread misinterpretation of $p$-values (such as mistaking $p < 0.05$ for proof that the alternative hypothesis is true) directly fueled problematic research practices and the broader replication crisis.

• Interactive Dynamics of Sample Size ($N$) and Effect Size:

  • Small Sample Sizes ($N$): Statistical significance requires massive deviations or huge effect sizes.

  • Example: With $N = 10$ coin flips, achieving significance requires a minimal deviation of $90\%$ heads ($9/10$ flips).

  • Large Sample Sizes ($N$): High statistical power allows minuscule deviations or minor effect sizes to achieve statistical significance.

  • Example: With $N = 100$ flips, $60\%$ heads ($60/100$) is significant. With $N = 1,000$ flips, a tiny deviation of $53\%$ heads ($530/1,000$) achieves $p < 0.05$.

  • Application to Bem's ESP Research:

  • Bem utilized very large sample sizes, allowing a minimal departure from chance accuracy ($52.0\%$ observed vs. $50.0\%$ chance) to cross the threshold of statistical significance ($p < 0.05$).

  • While statistically significant, the absolute effect size was extremely small, demonstrating how large samples convert minor, trivial noise into statistically significant findings.