Open Science

Questionable Research Practices (QRP)

Methods researchers use to increase the chance of getting a statistically significant result (p < 0.05), even if there is no real effect.

P-Value

P-value is the probability of getting your results (or more extreme results) if the null hypothesis is true (under 5% = not due to just chance)

• False Positive - mistake of concluding that an effect is there even though it isn't (false hope)

Specific QRPs

1. Measure the dependent variable (Y) in multiple ways, reporting the best outcome

2. Gradually add more observations

3. Add and drop covariates (variables)

4. Add and drop experimental conditions (groups)

Lady Macbeth Effect

When you feel you've done something morally wrong, you have the desire to cleanse yourself physically (washing hands)

Publication/file bias

When studies show null/no effects wind up in a file drawer instead of a journal, thus heavily biased toward studies that "worked".

What Helps?

1. Preregistration - write up a detailed plan before beginning the actual data collection (with date).

• Design, analyze, write up, collect data

2. Open materials and data - post it online, being transparent

3. Publishing null results - publish your paper before you collect your data, direct replication (conduct same study again even if it fails)

Evaluating Past Research

1. Research Question

• Clear aim, hypotheses, terms/variables clearly defined?

2. Sample

• Are the participants/samples representative, avoids selection bias?

3. Research Design

• Valid and reliable measures (methodology)

4. Discussion and Conclusion

• Findings justify the conclusions (is it accurate or biased), any strengths/weaknesses of the study?

Why Write a Literature Review

• To identify studies related to your topic, giving your reader background information

• To identify gaps related to the topic, helps you decide if your study fills the missing gap

• To identify methods used in other studies, help you develop your own methodology

• To show the value of your research, using existing sources lets others know that this topic is important

Organizing Literature Reviews

• Thematic: Organize by topics, themes, or variables.

• Methodological: Organize by research methods (e.g., surveys, experiments, qualitative) and compare them.

• Theoretical: Organize by different theories and argue which is best supported.

• Chronological: Organize by time, showing how the research developed over the years.

What is the Replication Crisis?

Many classic psychology studies fail when researchers repeat them

Why Does the Crisis Happen?

• Increasing false positive rate

• Small sample sizes

• Publication bias - journals like significant findings

• QRPs

Diederik Stapel

• Fabricated data

• Published many influential papers

• Fooled students, collaborators, and journals for years

Ego Depletion

The idea that self-control is like a limited resource. After using self-control on one task, people have less self-control available for the next task.

• Many studies could not consistently replicate the original findings.

Self Control and Self Regulation

Self control: resisting temptations/impulses, stopping yourself

Self regulation: regulating behaviour, mind, emotions, monitoring and adjusting your behaviour