A Primer on the ‘Reproducibility Crisis’ and Ways to Fix It

• Date: 01 June 2018
• DOI: http://doi.org/10.1111/1467-8462.12262
• Published date: 01 June 2018
• Author: W. Robert Reed

For the Student

A Primer on the ‘Reproducibility Crisis’and Ways to Fix It

W. Robert Reed*

Abstract

This article uses the framework of Ioannidis

(2005) to organise a discussion of issues

related to the ‘reproducibility crisis’. It then

goes on to use that framework to evaluate

various proposals to f‌ix the problem. Of

particular interest is the ‘post-study probabil-

ity’, the probability that a reported research

f‌inding represents a true relationship. This

probability is inherently unknowable. How-

ever, a number of insightful results emerge if

we are willing to make some conjectures about

reasonable parameter values. Among other

things, this analysis demonstrates the impor-

tant role that replication can play in improving

the signal value of empirical research.

1. Introduction

The last two decades have seen increasing

doubt about the credibility of empirical

research in science. This has come to be known

as the ‘Reproducibility Crisis’, with the name

derived from the fact that many reported

empirical f‌indings cannot be reproduced/repli-

cated. While concerns have been raised about

all areas of science, medicine and the social

sciences, particularly psychology, have been

the subject of greatest concern. Economics has

been relatively slow to recognise the problem

and consider solutions.

A thorough discussion of the evidence for a

reproducibility crisis would require consider-

able space, and thus cannot be undertaken here.

Suff‌ice it to say that while an increasing

number of researchers are convinced that there

is a problem, others remain unpersuaded. The

2017 Papers and Proceedings issue of the

American Economic Review provides a sam-

pling of different perspectives.

1

This article will adopt the framework

developed by Ioannidis (2005) to organise a

discussion of issues related to the ‘reproduc-

ibility crisis’, and to analyse some of the

solutions that have been proposed to f‌ix it. The

title of Ioannidis’paper is ‘Why Most

Published Research Findings are False’.It

presents a simple mathematical model to argue

that it is unlikely that any published paper that

claims a statistically signif‌icant relationship is

actually true. It is among the most highly cited

papers in the social sciences.

Ioannidis’model is most appropriate for

experiments of the kind reported in Camerer

et al. (2016), though the issues have broader

applicability.Camerer et al. (2016) investigated

subjects such as the effect of expectational

reference points on work effort, the effect of

punishmenton performance when monitoring is

* Department of Economics and Finance, University of

Canterbury, NewZe aland; email: bob.reed@canterbury.ac.nz.

The author wishes to acknowledge comments and feedback

from Lucas Coffman, Tom Coup

e,Brian Haig,DaniëlLakens,

Blakely McShane, Andrea Menclova, E. J. Wagenmakers

and Benjamin D. K. Wood. Acknowledgement of their

feedback should not be interpreted as implying their agreement

with the views expressed in this article.

The Australian Economic Review, vol. 51, no. 2, pp. 286–300

°

C2018 The University of Melbourne, Melbourne Institute of Applied Economic and Social Research

Published by John Wiley & Sons Australia, Ltd