CH. 14 Big Picture

Type 1 error

when researcher decide a result is statistically significant when there is actually no relationship in the population (reject a true null hypothesis)

Type 2 error

when researchers decide a result is not statistically significant when there actually is a relationship in the population (fail to reject a false null hypothesis)

Publication bias

bias among researchers and editors in favor of publishing statistically significant results and against publishing

file drawer problem

when statistically nonsignificant results are stashed away in researchers’ file drawers

readers have no idea how many similar studies have been conducted with

nonsignificant results

researcher degrees of freedom

certain decisions that researchers make during the research process to increase the chances they will find statistical significant results by chance

Researcher degrees of freedom include

-when to stop collecting data

-whether to exclude any participants from the analyses

-which analyses to do

-which dependent variables and conditions to report

researcher degrees of freedom is increasingly recognized as

poor research practice

replication crisis

idea that many results in psychology cannot be replicated and do not reflect real psychological phenomena

Open Science Collaboration conducted replications of 100 studies

slightly more than a third of results were successfully replicated

Many labs replication project replicated 13 studies in an online format with 36 different samples for each study

11 of the 13 results were successfully replicated

Meta-Analysis

set of techniques for finding all the studies that address a particular research question and then combing the results of those studies statistially to see the overall pattern

It is better to analyze all study results ____

statistically rather than individually

step 1 of Meta-Analysis

identify the research question

step 2 of Meta-Analysis

find all the research that addresses that question

step 3 of Meta-Analysis

find or compute the effect size in each study

step 4 of Meta-Analysis

combine all the effect sizes into a single overall effect size

step 5 of Meta-Analysis

test potential moderators of the effect size

strengths of a meta-analysis

-can provide a fairly definitive answer to a research question

-clan identify areas that need further study

limitations of a meta-analysis

-can only be done reliably when a sufficient number of primary studies have already been conducted

-does not necessarily get around the problem of publication bias

-will reflect the same flaw as the studies that go into a meta-analysis if they are flawed in some consistent way

open science

movement to make scientific practice more transparent to other scientist and the general public

in open science researchers should report

all their studies, conditions, dependent variables, and analyses (not just the one that worked)

Preregistration

researchers write a short description of the study and provide details about exactly how they will conduct it

preregistered reports

reports based solely on preregistered descriptions

preregistered reports completely

eliminates publication bias