Critical Perspectives

What is replication?

Repeatedly finding the same results

Why do studies replicate other studies?

1. Protects against false positives (e.g. sampling error)

2. Controls for artifacts

3. Addresses researcher fraud

4. Test whether findings generalise to different populations

5. Test the same hypothesis using a different procedure

What is direct replication?

A scientific attempt to recreate the critical elements (e.g., samples, procedures, and measures) of an original study

What is conceptual replication?

To test the same hypothesis using a different procedure

What are reasons for the inability to replicate findings?

the nine circles of scientific hell

What are the nine circles of scientific hell?

1. Limbo

2. Overselling

3. Post-Hoc Storytelling

4. P-value fishing

5. Creative outliers

6. Plagiarism

7. Non-publication

8. Partial publication

9. Inventing data

What is outcome switching?

Changing the outcomes of interest in the study depending on the observed results

What is p-hacking?

Removing demographics until the p value is significant

How common is ‘sloppy science’?

The percentage of respondents who have engaged in questionable practices was surprisingly high

What are moderators?

Variables that influence the nature (e.g., direction and / or size) of an effect

Why are identifying moderators good?

It improves our understanding.

What is publication bias?

Findings that are statistically significant are more likely to be published than those that are not.

What is open science?

the process of making the content and process of producing evidence and claims transparent and accessible to others

What is open methodology?

Documenting the methods and process by which those methods were developed / decided upon.

What is pre-registration?

Define the research questions, methods, and approach to analysis before observing the research outcomes

What is HARKing?

Hypothesizing After Results are Known

Does pre-registration improve replicability?

Replicated the expected effects in 86% of attempts (Protzko et al., 2023)

What are the stages of peer review?

Reviewers and editors assess a detailed protocol & Following favourable reviews (and probably revision to meet methodological standards), the journal offers acceptance in-principle

What is open data?

• Making the dataset freely available:

  • Allows other scientists to verify the (original) analyses

  • Facilitates research beyond the scope of the original research

  • Avoids duplication of data collection

What is FAIR?

Findable, Accessible, Interoperable, Reusable

What are the issues with open access?

Limits access to those who have funds to pay for articles / subscriptions

What are the types of open access publishing?

Gold & green open access

What is gold open access publishing?

The researchers (or more likely the funders or host institution) pay the journal to publish the article

What is green open access publishing?

An (unformatted) version of a manuscript into a repository.

What are the effects of open access publication?

Open access works are used more & facilitate meta-research