The Replication Crisis and Open Science Practices
Questionable Research Practices (QRPs)
The Roots of Unreplicability
The replication crisis, prominently highlighted by the Open Science Collaboration (2015), revealed that less than 40% of classic psychology studies replicated. This low rate is largely driven by QRPs such as p-hacking (running multiple analyses and only reporting significant ones), optional stopping, and HARKing.
The Registered Reports Publication Process
-
1
Stage 1 Submission: Authors submit an introduction and rigorous methodology/analysis plan before data collection.
-
2
In-Principle Acceptance (IPA): Peer review focuses solely on the theoretical importance and methodological rigor. If accepted, publication is guaranteed regardless of results.
-
3
Data Collection: Authors strictly follow the approved protocol.
-
4
Stage 2 Submission: Authors submit the final manuscript. Reviewers verify adherence to the Stage 1 protocol.
Optional Stopping
Optional stopping (peeking at the data and stopping collection once ) inflates the Type I error rate exponentially. If you must peek, you must use sequential analysis frameworks with alpha spending functions.
Which of the following best defines HARKing?
HARKing stands for Hypothesizing After Results are Known. It distorts the scientific literature by making exploratory analyses look like confirmatory tests.
Correct answer: Presenting a post hoc hypothesis as if it were an a priori hypothesis.
What open science practice involves time-stamping a study design, hypotheses, and analysis plan on a public repository before data collection begins?
Preregistration creates a permanent, read-only record of a study plan prior to data collection, effectively separating confirmatory hypothesis testing from exploratory analysis.
Correct answer: Preregistration
Traditional vs. Open Science Models
| Aspect | Traditional Model | Open Science Model |
|---|---|---|
| Data Availability | Available upon request (often lost) | Uploaded to public repositories (e.g., OSF) |
| Analysis Code | Rarely shared | Shared via GitHub or OSF |
| Publication Bias | High (null results rarely published) | Low (via Registered Reports) |