Alex Holcombe's blog

open science, open access, meta-science, perception, neuroscience, …

Are most scientific results replicated?

with 4 comments

To what extent should we have confidence in the average published scientific result? The news that Professor Marc Hauser of Harvard University may have been involved in scientific misconduct, yielding irreproducible results, prompts us to reflect on whether irreproducible results are usually caught, or not. Investigations of scientific misconduct are *extremely* rare, so I don’t think irreproducible results are usually caught through the official-investigation route. Irreproducible results might be flagged if people tried to replicate them, failed, and then published a paper describing the non-replication. Unfortunately, publishing a non-replication is usually difficult. Those that know this, shy away from the attempt (the “file-drawer problem
“). Those scientists naive enough to not know this, or idealistic enough to attempt to publish a non-replication anyway, sometimes succeed but the successful numbers will be low.

If irreproducible results are unlikely to be flagged as such in the scientific literature, things look very bad for science generally. However, some scientists completely disagree, and think that irreproducible results are usually corrected.

Over at Psychology Today, Professor Art Markam wrote a blog post entitled “Why science is self-correcting“, subtitled “There’s no point in scientific misconduct; it is always found”:

If a result piques the interest of other scientists, then their first step is usually to try to repeat the experiment, perhaps with a few changes to test alternative explanations for a finding. Because scientists are always repeating each other’s experiments, it is hard for a fictitious result to hang on for very long.

As should be clear from what I wrote above, I think this is a misleading picture of the vast majority of fields. I do believe Art is correct for some corners of science. An example is some subfields of vision science, especially those that revolve around visual illusions that can be replicated simply by looking at the display and asking yourself whether you see the illusion! Unfortunately, in most areas of science, experiments are not free and instantaneous. Instead, they are costly and time-consuming.

In such expensive and time-consuming areas of science, even results that attract lots of attempted replications may often end up with published replications when the original result was actually a statistical fluke rather than a true portrait of reality. There’s a vicious cycle: amazing results (which are sometimes reflect a mistake or a statistical fluke) often attract many scientists to pursue them further. If the original result was a mistake or a scientific fluke, most of these attempted replications will fail and never be published (the file-drawer problem). However, a few will succeed via a statistical fluke, and these are likely to be published as replications. John P.A. Joannides has done a fascinating analysis of where this and other sources of systematic distortion in the scientific literature may lead. His provocative conclusion provided the title of his paper: “Why Most Published Research Findings Are False”.

It is certainly appropriate to argue with whether Joannides is right that more than 50% of published findings are actually false, and I’d love to see an attempt to test the title of this blog post—to estimate the proportion of scientific results that are actually replicated. But even without these issues fully settled, I think it does science a disservice to say as Professor Markram did that science always self-corrects. For science to contribute as much as it should to solving society’s problems, society needs to have confidence in science. We need to face science’s problems head on and work towards resolving them. Public confidence in science, and the extent to which the public looks to science, has got to increase! Otherwise, decisions by elected officials may continue to be more faith-based than science-based.

A small part of the solution, I believe, is to lower the barriers to publication of negative results so that more non-replications escape from the file drawer and see the light of day. It’s true that this could create an unmanageable burden of papers that need to be reviewed, but this can be avoided by moving some fraction of the scientific literature towards post-publication review. Another important step will be to make publishing one’s original data the norm. There are objections to these suggestions which I won’t go in to here, for now suffice it to say that I think that on balance, a more prolific science would lead us closer to truth.

About these ads

Written by alexholcombe

August 19, 2010 at 10:44 pm

4 Responses

Subscribe to comments with RSS.

  1. I’m sure you’re aware the canonical Feynmann essay on the topic.

    Re-reading it, it’s amazing up to date and depressingly insightful about the flaws in how science done.

    Jonathan Hunt

    August 20, 2010 at 12:41 am

  2. Thanks Jonathan, I had heard of it but now you prompted me to read it. It’s excellent because not only does he also point out other problems with science, he somehow manages to do it entertainingly enough to make it a commencement address! Admittedly, it’s a (nerdy) *Caltech* commencement address, but still…


    August 20, 2010 at 6:01 am

  3. Hehe, Feynman is great. Very apt speech which has lost none of its relevance in the past few decades. And of course, his anecdotes and analogies are always entertaining in how they wouldn’t be socially acceptable any more today ;)

    Sam Schwarzkopf

    August 20, 2010 at 6:18 pm

  4. [...] Are most scientific results replicated? (lower the barriers to publication of negative results) (A.Holcombe) Heel belangrijk te weten wat niet werkt. [...]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.

Join 766 other followers