The current system of publishing scientific papers hampers scientific advancement in particular ways. I’d like to diagnose the problems so that new publishing efforts (like PLoS ONE) can be shaped to remedy them.
I believe that a major flaw in the modern journal system is that it discourages complete treatments of the literature. A few factors work together to cause this:
- Most prestigious papers nowadays are very short- e.g. high impact, multidisciplinary journals mostly contain very short papers.
- To get their paper in, authors often are better off not mentioning problematic aspects of one’s paper or its relationship to the literature. After all they can always work on them if the reviewers force them to, but why undermine one’s own argument and raise red flags for the reviewers and reader if they don’t have to?
- Even in long papers, most authors avoid direct criticism of papers if the issue is not critical to their thesis.
These problems combine to create a large “back story” of information to every paper to which most readers will be forever oblivious. And there is even more that I won’t discuss now- we all know papers are largely a fairy tale, one more digestible than would be an accurate record of the twists and turns that really happened in the lab.
To factor 2, one might respond that in practically all writing of any kind, the authors have an incentive to sweep their problems under the rug. Perhaps there is little hope of remedying the problem. But the problem is worse than it need be. For one thing, knowledge of the problems with an article is available, but suppressed. During the review process, journals ask experts in the area to evaluate the science of a manuscript in depth. Reviewers make very extensive comments including links to other literature and concerns about the methods. Unfortunately, very few journals actually publish these comments (there are reasons for this, but they could be published more often). The authors incorporate many of the comments into their manuscript revisions but many are left out. Even those concerns that result in the manuscript being modified are usually hard to detect by all but the most expert of readers of the final manuscript.
Besides the reviewers, many readers immediately recognize problems with a new article and unmentioned links to other scientific results. Traditionally publishing these concerns is pretty difficult. Now that some journals allow anyone to post comments on an article, there is no real obstacle. Nevertheless the average scientist does not post comments, probably because he believes he is more likely to be punished for doing so (by the authors responding in kind with criticism) than rewarded (he cannot put the comments on his all-important academic CV). Many people (e.g. Cameron) have thought about how to reform the system to reward these comments, and I’ll set that aside for now.
Regarding point 3 above, in some instances of “selective citation”, the authors have omitted citing a paper because they have a reason not to believe its conclusion. They may have spotted a logical or technical methodological problem, or made an unreported attempt at replication that failed. I have done this many times myself.
Do you agree that these are major problems? How can we address them?
I’ll describe some ideas for partial remedies in subsequent posts.