PSYC 312: Advanced Research Methods (Exam 1)

Criticisms of the OSC

• Studies were picked based on a random sample, not importance in the field

• Sample sizes were small

• Some procedures were not recreated well (e.g. context/population differed)

• Many experimenters were not experts in the subfield

Many Labs Projects

ML1: Tested core findings and found a higher replication rate (11/13) ML2: Tested across different contexts and found context didn't matter ML3: Tested time of semester and found it did not matter ML4: Tested with collaboration from the original authors, the effect didn't replicate ML5: Tested with one team using the original study's methods and one team using OSC methods, found it didn't matter

Chamber's Sins of Bias

1. Publication bias and neophilia
2. Devaluing direct replication
3. Encouraging HARKing

3 Issues with Conceptual Replication

1. Subjectivity in what counts as testing the same effect
2. Previously replicated studies can become un-replicated
3. Fuels confirmation bias as a finding can never be disconfirmed

Ionnadis' Three Factors

1. Statistical power of the studies testing a finding
2. Number of studies testing a finding
3. Ratio of true and non-true findings in a field

Ionnadis' Six Corollaries

1. The smaller the studies in a field, the less likely a finding is to be true
2. The smaller the effect sizes in a field, the less likely a finding is to be true
3. The more a field publishes hypothesis-generating studies, and the less it publishes finding-confirming studies, the less likely a finding is to be true
4. The greater the flexibility in design and analysis options, the less likely a finding is to be true
5. The greater the financial incentives, the less likely a finding is to be true
6. The "hotter" a scientific field, the less likely a finding is to be true

Recommendations to Reduce QRPs for Authors

1. Authors must decide in advance when to stop data collection and report this rule
2. Authors must collect at least 20 observations per cell, or justify why they did not using a cost-of-collection analysis
3. Authors must report all variables collected
4. Authors must report all experimental conditions
5. If making data exclusions, authors must report what the results would be without those exclusions
6. If using covariate analysis, authors must report what the results would be without the covariate

Recommendations to Reduce QRPs for Reviewers

1. Ensure authors follow the rules
2. Be more tolerant of imperfections in research
3. Require authors to justify that results do not hinge on arbitrary decisions
4. If justifications are suspect, require a direct replication

Kerr (1998)

• Surveyed psychologists to ask whether they had observed or suspecting HARKing from those in their field
• 40% had witnessed direct HARKing, and 60% had suspected it

John et al. (2012)

• Surveyed psychologists to collect measures of self-admitted QRPs and estimate the actual prevalence
• 35% of respondents admitted to claiming to predict unexpected results
• Estimate of HARKing prevalence was 90%

Silberzahn et al. (2015)

• Gave 29 research teams real data of red cards given in soccer and asked them to determine if red cards were given more often to darker-skinned players
• 20 teams found a difference, 9 didn't
• 21 different covariate analyses were used
• Effect size estimates ranged wildly (odds ratio of approx 0.89-2.93)

Botnivik-Nezer et al. (2020)

• 70 research teams were given fMRI data to test 9 hypothesis
• No two teams used the same data workflow
• Conclusions varied wildly depending on analysis workflow

Four Problems with NHST

1. The null is usually absurd
2. It tells us nothing about what we actually care about
3. It treats understanding as absolute, rather than probablistic
4. The cutoff of p < .05 is absurd

3 Reasons NHST Lingers

1. Status quo bias
2. Aversion to perceived uncertainty of other approaches
3. It makes decisions easier

Alternatives to NHST

• Effect sizes and confidence intervals
• Power analysis
• Bayesian statistics

Statistical power

The probability of correctly rejecting a false null hypothesis, which is a function of study design, effect size, and sample size

Steps of Power Analysis

1. Calculate the critical t-value
2. Make an estimate of the effect size
3. Using the estimate, determine how many people to collect to control the Type II error such that the critical value for a 5% Type I error rate and 20% Type II error rate match up