As discussed in my last post, there’s a web-based course called “Responsible Conduct of Research” that many thousands of researchers are required to complete each year. Brad Voytek spotted this question that seems a bit hostile (although quite possibly unintentionally) to new forms of scientific communication outside traditional journals.
I suggested we should write to the organization responsible for the course, and a few people commented on my post to indicate that they agreed. A few tweets later, we had a draft letter going. It’s been really cool to see how social media was able to quickly get a bunch of like-minded scientists together to achieve a goal. This in and of itself undermines the question that we wanted to question :) Below see our letter- we emailed it to CITI and they responded promptly to thank us for the feedback and to say they’d consider the issues we raised.
Dear Professor Braunschweiger (CITI co-founder) and Professor Ed Prentice (CITI Executive Advisory Committee chair):
We write to challenge the answer to one of the questions in the “Responsible Conduct of Research” online course. The question reads “A good alternative to the current peer review process would be web logs (BLOGS) where papers would be posted and reviewed by those who have an interest in the work”. The answer deemed correct by your system is “False” and the explanation provided includes the assertion that “It is likely that the peer review process will evolve to minimize bias and conflicts of interest”.
We question these claims for two reasons. First, we see real examples of rigorous science happening outside of the traditional system of journal-based peer review. Second, we believe that the future path of scholarly communication is uncertain, and indicating to young researchers that such an important issue is closed is both inaccurate and unhelpful to informed debate.
As an example of science that does not fit the mold suggested by the phrase “the current peer review process”, consider the use of the arXiv preprint server in certain areas of astronomy and physics. In these areas, researchers usually begin by posting their manuscripts to the arXiv server. They then receive comments by those who have an interest in the work. Some of those manuscripts subsequently are submitted to journals and undergo traditional peer review, but many working scientists stay abreast of their field chiefly by reading manuscripts in the arXiv before they are accepted by journals.
Even in areas that are more tightly bound to traditional journals, there are recent examples where both effective peer review of science  and science itself  have occurred primarily via blogs and other online platforms. In these cases, the online activity appears to have resulted in more rapid progress than would have been possible through the traditional system. A growing body of research suggests that scholars use social media in ways that reflect and produce serious scholarship .
As for the future path of the current mainstream peer review model, we believe it is speculation to say that “It is likely that the peer review process will evolve to minimize bias and conflicts of interest”. The current peer review process may be under considerable strain  and unfortunately there is little evidence that it significantly improves the quality of manuscripts . This raises the possibility that big changes are required, not just modifications to reduce bias and conflicts of interest. Furthermore, the question presupposes that the future entity into which peer review will evolve does not involve blogging. No one can see the future clearly enough to make that assumption.
We encourage discussion of this important topic, and would be interested in the inclusion in your program of material that sparks such discussion. However, we believe a true/false question on this topic to be inappropriate, as it limits rather than promotes discussion. All of us wish to see the development and optimization of rigorous systems, both new and traditional, for scientific scholarship. Requiring young researchers to adopt a particular position on this controversial, multifaceted issue may hinder open discussion and future progress.
Bradley Voytek, PhD, University of California, San Francisco Department of Neurology
Jason Snyder, PhD, National Institutes of Health, USA
Alex O. Holcombe, PhD, School of Psychology, University of Sydney, Australia
William G. Gunn, PhD, Mendeley, USA/UK
Matthew Todd, PhD, School of Chemistry, University of Sydney, Australia
Daniel Mietchen, PhD, Open Knowledge Foundation Germany
Jason Priem, School of Library and Information Science, University of North Carolina at Chapel Hill
Heather Piwowar, PhD, DataONE/NESCent, Canada
Todd Vision, PhD, Department of Biology, University of North Carolina at Chapel Hill
Cameron Neylon, PhD, Science and Technology Facilities Council, UK, Editor in Chief, Open Research Computation
 Online experimental peer review of the “Arsenic Life” paper that recently appeared in Science: http://rrresearch.fieldofscience.com/2010/12/arsenic-associated-bacteria-nasas.html
 Open Science is a Research Accelerator, M. Woelfle, P. Olliaro and M. H. Todd, Nature Chemistry 2011, 3, 745-748. http://www.nature.com/nchem/journal/v3/n10/full/nchem.1149.html
 Groth, P., & Gurney, T. (2010). Studying Scientific Discourse on the Web using Bibliometrics: A Chemistry Blogging Case Study. Presented at the WebSci10: Extending the Frontiers of Society On-Line, Raleigh, NC: US. Retrieved from http://journal.webscience.org/308/
 Priem, J., & Costello, K. L. (2010). How and why scholars cite on Twitter. Proceedings of the 73rd ASIS&T Annual Meeting. Presented at the American Society for Information Science & Technology Annual Meeting, Pittsburgh PA, USA. doi:10.1002/meet.14504701201
 Weller, K., Dröge, E., & Puschmann, C. (2011). Citation Analysis in Twitter. Approaches for Defining and Measuring Information Flows within Tweets during Scientific Conferences. Proceedings of Making Sense of Microposts Workshop (# MSM2011). Co-located with Extended Semantic Web Conference, Crete, Greece.
 Smith R. Classical peer review: an empty gun. Breast Cancer Research 2010, 12(Suppl 4):S13 http://dx.doi.org/10.1186/bcr2742
 Jefferson T, Rudin M, Brodney Folse S, Davidoff F. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database of Systematic Reviews 2007, 2:MR000016. http://dx.doi.org/10.1002/14651858.MR000016.pub3
Bradley Voytek spotted a disturbing question in an official “Responsible Conduct of Research” training program:
This defense of the status quo has no place in a “Responsible Conduct of Research” training program. It reads like the old guard self-interestedly maintaining the current system by foisting unjustified beliefs onto young researchers!
The part that bothers me the most is the sentence “It is likely that the peer review process will evolve to minimize bias and conflicts of interest”. What is the evidence for this?
Has the process been evolving to minimize bias and conflicts, or to increase them? I don’t think the answer is very clear. As counterweight to the official optimistic opinion, here are a few corrupting influences:
- Pharmaceutical companies continue to buy influence with medical journals, by buying hundreds of copies of journal issues that run studies that support their products.
- Pharmaceutical companies continue to ghostwrite journal articles for doctors, to plant their views in the medical literature.
- Scientists of every stripe often fail to disclose their conflicts of interest.
- Journals develop new revenue streams, like fast-tracking articles for a fee, that may open them to favoring the select authors who pay.
- Many reviewers are, like most humans, biased towards their own self-interest. This can yield a bias to recommend rejection of papers by rivals. Because reviewers in most journals are anonymous, they are never held to account.
- Journals don’t have the resources to investigate authors accused of fraud, and universities often try to avoid finding fault with the researchers they employ.
Many people have suggested partial remedies to these problems, but it’s an uphill battle to implement them, due to the slow pace of change in the journal system. We have to remember this and not be lulled into complacency by the propaganda seen in that training program. It was created by an organization of academics called CITI.
UPDATE: In the comments below, Jason Snyder pointed out an article from CITI in which CITI reports that over 6,000 researchers a month are taking this course — being subjected to this biased question. Some of us object not only to their characterization of the peer review process, but also to their suggestion that blogs are not a good place to do science. We don’t want thousands of researchers to continue to be forced to assent to the conservative opinion articulated by CITI, so we’re drafting a letter asking them to delete the question.
Academic publishing is stuck in an outmoded system. Most scientific research is paid for by government and non-profit university funds, but high-profit corporate publishers often control access to the results of the research. In this video, we showcased the absurdity of the situation and also pointed towards how to get ourselves un-stuck.
There are significant costs associated with what journal publishers do, so we need publishers in some form. But there’s no need for publishing to involve millions in profits and universities having to pay many thousands of dollars for a subscription to a journal.
In the video, the scientist mentions two of the ways we can move towards journal articles being available for free. First is supporting open-access journals. Most charge authors a fee, but one that is not too much higher than their costs, and the result is that anyone can download the article for free.
Another way a researcher can make an article freely available is by depositing the “post-print” in their university or institutional repository. A “post-print” is the draft of the article after it has been peer-reviewed. After a researcher revises their article in accordance with the comments of reviewers, they’ve got a file that may have the same content as that which the journal typesets and publishes. Although the journal usually owns the copyright to the journal version (after the author signs the copyright form), the researcher still in nearly all cases can take their own file, the post-print, and put it in an institution’s official web repository.
If enough of us supported open-access journals, and deposited our other manuscripts in repositories, then journals could no longer charge exorbitant subscription fees. The reason is that with a high percentage of manuscripts available from open-access journals and repositories, universities would cancel their subscriptions to particularly expensive journals.
It’s not just authors that provide free labor to the publishers. It’s also the academics that review each of the articles. So, as reviewers we can also push things towards open access, by saying yes more often to reviewing manuscripts that will be open access, and less often to those that won’t. If we can get a lot of people together to commit to this, it will make a direct impact as well as let others know how many of us support open access. To organize that, I’ve drafted a website called openaccesspledge.com. It also lists other pledges.
We (Hal Pashler, UCSD, hpashler at gmail and Alex Holcombe) are developing web-based software to help people interested in a scientific issue represent and inspect multiple competing hypotheses and the evidence which supports or fails to support each hypothesis.
One goal is to develop a tool that will help scientists wrap their minds around the state of a complex empirical debate more quickly and accurately than can be done by studying written review articles, commentaries, rebuttals, and so forth. Our software has been developed and tested in informal ways, and as our next step we want to get experts involved in real scientific debates to try it out and see how it works.
We are hoping to hire (probably in Southern California), part-time, a graduate student or similarly qualified person who is interested in scientific debate to oversee the implementation of some real debates. The person doesn’t have to be a software developer (we have one of those), but it would be good if they were generally tech-savvy and excited by internet tools. Clear communication and diplomatic skill will be essential in working with scientists trying out the software. Initial tasks will include writing documentation to guide real users in using the system, and helping to develop rules for structuring the use of the software by groups with very different views of controversial topics. We’d like to hire someone for six months at about 10-15 hours per week, to start. The project (currently funded mostly by NSF) could also potentially lead to publications, but our primary focus is on figuring out how to make the software maximally useful to people engaged in real debates. Anyone interested should email us.
3 CHAST (www.chast.org) lectures:
Why we cannot make life
Professor Bert Meijer, Molecular Sciences, Organic Chemistry, Eindhoven University of Technology and 2011 Cornforth Foundation Lecturer, University of Sydney
Where: Old Geology Lecture Theatre, Edgeworth David Bldg, University of Sydney, map: http://bit.ly/hmm5U8
When: Wednesday 31 August, 6:30-7:30pm. Free admission, no bookings. All welcome.
Abstract: “The origin of life on earth” is without doubt one of the most intriguing scientific topics, while the wish to create life in a laboratory is amongst its most difficult challenges. The enormous progress in science and technology over the past decades has provided many deep insights into the miraculous composition and functioning of living systems. Today, on the one hand, we can clone sheep, grow organs from stem cells, while cells, plants, animals and bacteria have been genetically modified. On the other hand, the synthesis of small and large molecules has become so sophisticated that almost every molecule that exists on earth can now be made in a laboratory, including long strands of DNA, proteins and complex drugs that can cure diseases. These many insights, however, also show the complexity of the molecular biology of living cells. As a result, the astonishment about how life could ever have originated has further increased. The lecture will illustrate the greatest challenges that are encountered while seeking to understand the origin of life, including an explanation of why it will take a very, very long time before a living cell can be made in a laboratory out of its individual components, if it is possible at all. Special attention will be paid to the self-organization of complex molecular systems as a critical step in the building process.
E.W. “Bert” Meijer is Distinguished University Professor in the Molecular Sciences, Professor of Organic Chemistry at the Eindhoven University of Technology and scientific director of the Institute for Complex Molecular Systems. After receiving his PhD degree at the University of Groningen, he worked for 10 years in industry (Philips and DSM). In 1991 he was appointed in Eindhoven, while in the meantime he has held part-time positions in Nijmegen and Santa Barbara, CA. Bert Meijer is a member of many editorial advisory boards, including Chemical Communications, Angewandte Chemie, and the Journal of the American Chemical Society. Bert Meijer has received numerous awards, including the 1999 Silver Medal of the Macro UK group, the Spinoza Award in 2001, the ACS Award for Polymer Chemistry in 2006, the AkzoNobel Science Award 2010. He is a member of the Royal Netherlands Academy of Science.
Announcing a CHAST (www.chast.org) Lecture:
Numbers: Their Human Aspects. Perspective from Indigenous Cultures
Dr. Helen Verran, History and Philosophy of Science, University of Melbourne
Where: Old Geology Lecture Theatre, Edgeworth David Bldg, University of Sydney, map: http://bit.ly/hmm5U8
When: Tuesday 8 November, 5:30-6:30pm. Free admission, no bookings. All welcome.
Abstract: Many people spend a lot of time looking at numbers, or more to the point, looking through numbers at something else. In this talk I take a look at numbers as such. How can we ‘see’ numbers? And why would we want to? I will tell of the experience of working with teachers in primary school classrooms in Nigeria. This had me recognizing that if we are going to understand how science might come to life as a significant cultural element in places like Nigeria we need a way to see the cultural lives that things like numbers have. Having done some preliminary thinking with the help of Nigerian primary school children I turn to my experiences of working with Yolngu Aboriginal Australians who own lands in northeast Arnhem Land. I will make a rather surprising analogy which I suggest can help us better understand the sorts of things numbers are.
Helen Verran is a Reader in History and Philosophy of Science at the University of Melbourne. She has a PhD in metabolic biochemistry. For most of the 1980s she worked as a science lecturer in the Institute for Education at Obafemi Awolowo University in Nigeria. Her book Science and an African Logic (2001) was published out of this experience. Since she returned to Australia she has worked with Yolngu Aboriginal communities in northeast Arnhem Land an early product of this work was the small book Singing the Land Signing the Land now available on-line. http://singing.indigenousknowledge.org/ which provides background for her CHAST Lecture.
The 2011 Templeton Lecture (www.chast.org):
The Emotional Brain
Professor Joseph LeDoux, Center for Neural Science, New York University
Where: Eastern Avenue Auditorium, University of Sydney, map: http://bit.ly/qHUKUd
When: Monday 17 October, 6:00-7:30pm. Free admission, no bookings. All welcome.
Abstract: The study of emotion has been hampered by a fixation on feelings. Feelings are important, but not all important. Problems arise when we use feelings, and their semantic labels, as guides to studying brain function in other animals. Rather than imposing concepts based on human introspective experience to the brains of other creatures, we should attempt to understand how the human brain is similar to the brains of other animals. This then becomes a foundation for understanding differences between humans and other animals. I propose that much of what is called emotion in studies of other animals is accounted for by the operation survival circuits, circuits involved in defense, energy/nutrition supplies, fluid balance, thermoregulation, and procreation. These circuits are highly conserved in mammals, including humans. While the behavioral expression of survival circuits can be species-specific, the circuits are species-general. Some other approaches also emphasize the adaptive function of emotions, but typically define emotions in terms of feelings. Survival functions are the real topic in most animal studies of emotion. By focusing on the adaptive function itself (rather than the behavioral expression or the conscious consequences) of survival circuits we have a way of characterizing phenomena that fall under the rubric of “emotion” in all mammals (perhaps all animals) without recourse to feelings. Feelings are what happens when consciousness witnesses the overall outcome (in the brain and body) of survival circuit activation. Feelings, which cannot be studied scientifically in non-human organisms, are neither necessary nor sufficient to understand survival circuits and their functions. By reorienting the comparative study of emotions around survival circuit functions, we have the opportunity to understand similarities and differences in emotional functions between humans and other animals.
The transmission of new scientific ideas and knowledge is needlessly slow:
|Journal subscription fees||Open access mandates|
|Competition to be first-to-publish motivates secrecy||Open Science mandates|
|Jargon||Increase science communication; science blogging|
|Pressure to publish high quantity means no time for learning from other areas||Reform of incentives in academia|
|Inefficient format of journal articles (e.g. prose)||Evidence charts, ?|
|Long lag time until things are published||Peer review post publication, not pre publication|
|Difficulty publishing fragmentary criticisms||Open peer review; incentivize post-publication commenting|
|Information contained in peer reviewers’ reviews is never published||Open peer review or publication of (possibly anonymous) reviews; incentivize online post-publication commenting|
|Difficulty publishing non-replications||Open Science|
UPDATE: Daniel Mietchen, in the true spirit of open science, has put up an editable version of this very incomplete table.
My talk for the Sydney University psychology department will be at 4pm in the Education building, room 424. Below, the abstract:
Localizing a single object relative to oneself is fairly easy—ever seen a plant reaching towards the sun? It’s a no-brainer. A less trivial task is determining the position of two objects relative to each other. Humans evolved brains that can do it, probably because it’s important for survival, but we don’t know how we do it. I asked some people to look at a stable scene and report which objects are adjacent. They said, “the red disc is next to the green disc” or “the red disc is next to the yellow disc”. Performance was essentially perfect. When the display started spinning, however, queer things started happening. These things suggest that apprehending the spatial relationship among objects requires a shift of attention from one object to the other. They also suggest that to perceive the spatial relationships among moving objects, the ability to follow an object with attention is critical. The ability to follow an object with attention was tested by many previous investigators, who found that people can keep track of about four objects at once. But previous investigators never moved their objects as fast as we move ours. Our findings with speedy objects dispel previous theories of tracking and suggest that the faster an object moves, the more attentional resource it consumes. Until nothing is left.
1. Holcombe, A., Linares, D., & Vaziri-Pashkam, M. (2011). Perceiving Spatial Relations via Attentional Tracking and Shifting. Current Biology, 21, 1-5. DOI: 10.1016/j.cub.2011.05.031
2. Holcombe, A.O., Chen, W.Y. (2011, submitted). Tracking a single fast-moving object exhausts attentional resources. (See the associated poster)
[Updated post with the time (4pm). Thanks Mat!]