Everything’s fine with peer review- if there are any flaws, they’ll be taken care of by evolution?

Bradley Voytek spotted a disturbing question in an official “Responsible Conduct of Research” training program:

This defense of the status quo has no place in a “Responsible Conduct of Research” training program. It reads like the old guard self-interestedly maintaining the current system by foisting unjustified beliefs onto young researchers!

The part that bothers me the most is the sentence “It is likely that the peer review process will evolve to minimize bias and conflicts of interest”. What is the evidence for this?

Has the process been evolving to minimize bias and conflicts, or to increase them? I don’t think the answer is very clear. As counterweight to the official optimistic opinion, here are a few corrupting influences:

  • Pharmaceutical companies continue to buy influence with medical journals, by buying hundreds of copies of journal issues that run studies that support their products.
  • Pharmaceutical companies continue to ghostwrite journal articles for doctors, to plant their views in the medical literature.
  • Scientists of every stripe often fail to disclose their conflicts of interest.
  • Journals develop new revenue streams, like fast-tracking articles for a fee, that may open them to favoring the select authors who pay.
  • Many reviewers are, like most humans, biased towards their own self-interest. This can yield a bias to recommend rejection of papers by rivals. Because reviewers in most journals are anonymous, they are never held to account.
  • Journals don’t have the resources to investigate authors accused of fraud, and universities often try to avoid finding fault with the researchers they employ.

Many people have suggested partial remedies to these problems, but it’s an uphill battle to implement them, due to the slow pace of change in the journal system. We have to remember this and not be lulled into complacency by the propaganda seen in that training program. It was created by an organization of academics called CITI.

UPDATE: In the comments below, Jason Snyder pointed out an article from CITI in which CITI reports that over 6,000 researchers a month are taking this course — being subjected to this biased question. Some of us object not only to their characterization of the peer review process, but also to their suggestion that blogs are not a good place to do science. We don’t want thousands of researchers to continue to be forced to assent to the conservative opinion articulated by CITI, so we’re drafting a letter asking them to delete the question.


9 thoughts on “Everything’s fine with peer review- if there are any flaws, they’ll be taken care of by evolution?

  1. Thoroughly bizarre. “Correct answer: false”. No, the correct answer is: let’s discuss it.

  2. Exactly @mattoddchem! Alex, this is a great breakdown of how wrong this thinking is.

  3. Truly strange. Have any other Pis noticed this in a training session?

  4. We should let CITI know about our concerns.

  5. Well then, it’s ironic that CITI has published a study on how amazing they are:


    in a non-peer reviewed journal comprised solely of articles by companies:


    with the following disclaimer:


  6. I’d certainly sign a complaint. I support peer review, but agree with mattoddchem that this is not a question that can be answered true/false.

  7. […] But Cox and Foreshaw are far from isolated in their opinion. A link from that article leads to an interesting story about a question in a course about “Responsible Conduct of Research”. The question was […]

  8. I just came across this document that offers a much more measured approach to what young investigators should know about peer review, from Sense About Science: http://www.senseaboutscience.org/resources.php/99/peer-review-the-nuts-and-bolts

    • Thanks Todd, I guess it’s about time to follow up with CITI and see whether after assessing it, they decided to change the question.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s