As discussed in my last post, there’s a web-based course called “Responsible Conduct of Research” that many thousands of researchers are required to complete each year. Brad Voytek spotted this question that seems a bit hostile (although quite possibly unintentionally) to new forms of scientific communication outside traditional journals.
I suggested we should write to the organization responsible for the course, and a few people commented on my post to indicate that they agreed. A few tweets later, we had a draft letter going. It’s been really cool to see how social media was able to quickly get a bunch of like-minded scientists together to achieve a goal. This in and of itself undermines the question that we wanted to question :) Below see our letter- we emailed it to CITI and they responded promptly to thank us for the feedback and to say they’d consider the issues we raised.
Dear Professor Braunschweiger (CITI co-founder) and Professor Ed Prentice (CITI Executive Advisory Committee chair):
We write to challenge the answer to one of the questions in the “Responsible Conduct of Research” online course. The question reads “A good alternative to the current peer review process would be web logs (BLOGS) where papers would be posted and reviewed by those who have an interest in the work”. The answer deemed correct by your system is “False” and the explanation provided includes the assertion that “It is likely that the peer review process will evolve to minimize bias and conflicts of interest”.
We question these claims for two reasons. First, we see real examples of rigorous science happening outside of the traditional system of journal-based peer review. Second, we believe that the future path of scholarly communication is uncertain, and indicating to young researchers that such an important issue is closed is both inaccurate and unhelpful to informed debate.
As an example of science that does not fit the mold suggested by the phrase “the current peer review process”, consider the use of the arXiv preprint server in certain areas of astronomy and physics. In these areas, researchers usually begin by posting their manuscripts to the arXiv server. They then receive comments by those who have an interest in the work. Some of those manuscripts subsequently are submitted to journals and undergo traditional peer review, but many working scientists stay abreast of their field chiefly by reading manuscripts in the arXiv before they are accepted by journals.
Even in areas that are more tightly bound to traditional journals, there are recent examples where both effective peer review of science  and science itself  have occurred primarily via blogs and other online platforms. In these cases, the online activity appears to have resulted in more rapid progress than would have been possible through the traditional system. A growing body of research suggests that scholars use social media in ways that reflect and produce serious scholarship .
As for the future path of the current mainstream peer review model, we believe it is speculation to say that “It is likely that the peer review process will evolve to minimize bias and conflicts of interest”. The current peer review process may be under considerable strain  and unfortunately there is little evidence that it significantly improves the quality of manuscripts . This raises the possibility that big changes are required, not just modifications to reduce bias and conflicts of interest. Furthermore, the question presupposes that the future entity into which peer review will evolve does not involve blogging. No one can see the future clearly enough to make that assumption.
We encourage discussion of this important topic, and would be interested in the inclusion in your program of material that sparks such discussion. However, we believe a true/false question on this topic to be inappropriate, as it limits rather than promotes discussion. All of us wish to see the development and optimization of rigorous systems, both new and traditional, for scientific scholarship. Requiring young researchers to adopt a particular position on this controversial, multifaceted issue may hinder open discussion and future progress.
Bradley Voytek, PhD, University of California, San Francisco Department of Neurology
Jason Snyder, PhD, National Institutes of Health, USA
Alex O. Holcombe, PhD, School of Psychology, University of Sydney, Australia
William G. Gunn, PhD, Mendeley, USA/UK
Matthew Todd, PhD, School of Chemistry, University of Sydney, Australia
Daniel Mietchen, PhD, Open Knowledge Foundation Germany
Jason Priem, School of Library and Information Science, University of North Carolina at Chapel Hill
Heather Piwowar, PhD, DataONE/NESCent, Canada
Todd Vision, PhD, Department of Biology, University of North Carolina at Chapel Hill
Cameron Neylon, PhD, Science and Technology Facilities Council, UK, Editor in Chief, Open Research Computation
 Online experimental peer review of the “Arsenic Life” paper that recently appeared in Science: http://rrresearch.fieldofscience.com/2010/12/arsenic-associated-bacteria-nasas.html
 Open Science is a Research Accelerator, M. Woelfle, P. Olliaro and M. H. Todd, Nature Chemistry 2011, 3, 745-748. http://www.nature.com/nchem/journal/v3/n10/full/nchem.1149.html
 Groth, P., & Gurney, T. (2010). Studying Scientific Discourse on the Web using Bibliometrics: A Chemistry Blogging Case Study. Presented at the WebSci10: Extending the Frontiers of Society On-Line, Raleigh, NC: US. Retrieved from http://journal.webscience.org/308/
 Priem, J., & Costello, K. L. (2010). How and why scholars cite on Twitter. Proceedings of the 73rd ASIS&T Annual Meeting. Presented at the American Society for Information Science & Technology Annual Meeting, Pittsburgh PA, USA. doi:10.1002/meet.14504701201
 Weller, K., Dröge, E., & Puschmann, C. (2011). Citation Analysis in Twitter. Approaches for Defining and Measuring Information Flows within Tweets during Scientific Conferences. Proceedings of Making Sense of Microposts Workshop (# MSM2011). Co-located with Extended Semantic Web Conference, Crete, Greece.
 Smith R. Classical peer review: an empty gun. Breast Cancer Research 2010, 12(Suppl 4):S13 http://dx.doi.org/10.1186/bcr2742
 Jefferson T, Rudin M, Brodney Folse S, Davidoff F. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database of Systematic Reviews 2007, 2:MR000016. http://dx.doi.org/10.1002/14651858.MR000016.pub3