Scholarly publishers and their high profits

I recently published the below chart to document the outrageous profit margins of scholarly publishers in the sciences.

Screen Shot 2013-01-09 at 12.35.26 PM

This post is to provide the sources for the numbers in the chart.

The Woolworths number comes from their website, where they write “As a group, Woolworths Limited makes less than seven cents in the dollar before we then pay interest and tax”.

The Rio Tinto figure of 23% is based on the operating profit they report divided by the consolidated sales revenue in their 2011 financial summary.

Apple’s profit of 35% is based on these numbers, dividing their operating income for the year ending September 2012 of 55.2 billion by the revenue for the same period of 156.5 billion.

The 34% number for Springer comes from Heather Morrison’s PhD thesis, in which she writes that “Springer’s Science + Business Media (2010) reported a return on sales (operating profit) of 33.9% or € 294 million on revenue of € 866 million, an increase of 4% over the profit of the previous year.”

For Elsevier, I used the figure reported by investment analyst Claudio Aspesi.

For Wiley, I again used Heather Morrison’s analysis in her thesis, based on $99 million in profit on $245 million in revenue.

Thanks to Nick Scott-Samuel and Mike Taylor.

Protect yourself during the replicability crisis of science

Scientists of all sorts increasingly recognize the existence of systemic problems in science, and that as a consequence of these problems we cannot trust the results we read in journal articles. One of the biggest problems is the file-drawer problem. Indeed, it is mostly as a consequence of the file-drawer problem that in many areas most published findings are false.

Consider cancer preclinical bench research, just as an example. The head of Amgen cancer research tried to replicate 53 landmark papers. He could not replicate 47 of the 53 findings.

In experimental psychology, a rash of articles has pointed out the causes of false findings, and a replication project that will dwarf Amgen’s is well underway. The drumbeat of bad news will only get louder.

What will be the consequences for you as an individual scientist? Field-wide reforms will certainly come, partly because of changes in journal and grant funder policies. Some of these reforms will be effective, but they will not arrive fast enough to halt the continued decline of the reputation of many areas.

In the interim, more and more results will be viewed with suspicion. This will affect individual scientists directly, including those without sin. There will be:

  • increased suspicion by reviewers and editors of results in submitted manuscripts (“Given the history of results in this area, shouldn’t we require an additional experiment?“)
  • lower evaluation of job applicants for faculty and postdoctoral positions (“I’ve just seen too many unreliable findings in that area“)
  • lower scores for grant applications (“I don’t think they should be building on that paper without more pilot data replicating it“)

These effects will be unevenly distributed. They will often manifest as exaggerations of existing biases. If a senior scientist already had a dim view of social psychology, for example, then the continuing replicability crisis will likely magnify his bias, whereas his view of other fields that the scientist “trusts” will not be as affected by the whiff of scandal, at least for awhile- people have a way of making excuses for themselves and their friends.

But there are some things you can do to protect yourself. These practices will eventually become widespread. But get a head start, and look good by comparison.

  • Preregister your study hypotheses, methods, and analysis plan. If you go on record with your plan before you do the study, this will allay the suspicion that your result is not robust, that you fished around with techniques and statistics until you got a statistically significant result. Journals will increasingly endorse a policy of favoring submitted manuscripts that have preregistered their plan in this way. Although websites set up to take these plans may not yet be available in your field, they are coming, and in the meantime you can post something on your own website, on FigShare perhaps, or in your university publicly accessible e-repository.
  • Post your raw data (where ethically possible), experiment code, and analysis code to the web. This says you’ve got nothing to hide. No dodgy analyses, and you welcome the contributions of others to improve your statistical practices.
  • Post all pilot data, interim results, and everything you do to the web, as the data come in. This is the ultimate in open science. You can link to your “electronic laboratory notebooks” in your grants and papers. Your reviewers will have no excuse to harbor dark thoughts about how your results came about, when they can go through the whole record.

The proponents of open science are sometimes accused of being naifs who don’t understand that secretive practices are necessary to avoid being scooped, or that sweeping inconvenient results under the rug is what you got to get your results into those high impact-factor journals. But the lay of the land has begun to change.

Make way for the cynics! We are about to see people practice open science not out of idealism, but rather out of self interest, as a defensive measure. All to the better of science.

Top 15 most popular laws in psychology journal abstracts

How many of these laws do you know? The top 15, listed below, are based on psychology journal articles 1900-1999, as calculated by Teigen (2002):

LAW  (REFERENCE)   NUMBER OF MENTIONS
1. Weber’s law (Weber 1834)  336
2. Stevens’ power law (Stevens 1957)  241
3. Matching law (Herrnstein 1961)  183
4. Law of effect (Thorndike 1911)  177
5. Fechner’s law (Fechner 1860) 100
6. Fitts’ Law (Fitts 1954) 82
7. Law of initial values (Wilder 1957) 82
8. Law of comparative judgment (Thurstone 1927) 72
9. Yerkes-Dodson law (Yerkes & Dodson 1908) 52
10. All-or-none law (Bowditch 1871) 45
11. Emmert’s law (Emmert 1881) 43
12. Bloch’s law (Bloch 1885) 41
13. Gestalt laws (Wertheimer 1923) 41
14. Hick’s law (Hick 1952) 31
15. Listing’s law (Listing 1870) 29

Although it’s no longer in fashion in psychology to suggest that empirical generalizations are “laws”, I think the perception ones have held up fairly well. In perhaps every case exceptions have been found, but most of the laws are still useful as generalizations over a lot of empirical territory.

Many people are generally skeptical of psychology as a science, and their voices have grown louder thanks to recent cases of fraud and to articles such as “Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant”, recently published in Psychological Science. So it’s nice to be reminded that psychological science has produced robust generalizations.

On the other hand, few question the validity of perception and psychophysics, which provides many of the laws above; the skeptics are thinking more of other areas, perhaps social psychology, clinical psychology, or developmental psychology. In those areas, effect sizes are smaller and data is harder to gather, so published results are more likely to be statistical flukes.

The “file drawer problem” is clearly one of the biggest reasons to mistrust psychological results, and I’d say it’s probably the biggest problem in all of science. The file drawer problem refers in part to the fact that when scientists can’t replicate a previously published effect, they are very likely to file the results away rather than try to publish them. So I’ve been helping create a website, psychfiledrawer.org (currently in beta), for people to report their failed replications.

Teigen, K. (2002). One Hundred Years of Laws in Psychology The American Journal of Psychology, 115 (1) DOI: 10.2307/1423676

Opening access with peer reviewing pledges

Most academics agree that most scientific articles should be freely available, but we’re stuck in a system where scientific articles still tend to be submitted to journals that one needs a subscription to read.

One way we researchers perpetuate this system is by donating our labor to provide “peer review” of manuscripts that will require others to pay hefty subscription fees to read.

Over the years, some researchers have pledged to no longer do this. It’s been only a trickle, and I thought getting more visibility for open access pledgers would help the cause. I made a web page listing those I could find and also created a site people could pledge with, but getting the pledge right is tough.

Some think we should refuse to do any reviews for journals that are not open access- journals whose articles are behind paywalls rather than free for anyone to download. Mike Taylor wrote an angry article advocating this, and several commenters agreed with him, such as Peter Murray-Rust, who frequently blogs about the issue. In the area of computer security conference proceedings and journals, there’s been enormous success with a strong pledge, thanks to an independently- and concurrently-created excellent pledge registration website by Stuart Schechter. Although it’s evident that in the computer security conference and journal domain, the field may be ready and has the infrastructure to transition almost immediately to full open access, I don’t think that’s true in most of the traditional sciences.

Unwillingness to sign on to a categorical pledge of no reviewing for closed journals is something I’ve heard from many colleagues, including several long-time open science advocates that I’ve been communicating with about the issues over the last few months. I’ve come to share two of their objections:

The hypocrite objection It looks hypocritical to refuse to review for closed journals unless one also stops submitting manuscripts to closed journals. Where one submits to is more constrained, by one’s co-authors and one’s career prospects, so it’s harder to stop submitting. Although I’m not sure there’s anything wrong with donating one’s reviewing time only to open access journals, I do know it looks hypocritical (I have been called a hypocrite for this reason), which hurts the cause. This also can lead to the perception that pledgers are finding a convenient way to shirk the extra work of reviewing. Maybe these arguments can be won with individual name-callers, but it takes a lot of time to win those fights.

The green road objection Most journals, including closed access journals, allow researchers to put their post-print (their final version of the manuscript, before the publisher type-sets it) up on the web in their university’s or institution’s digital repository. In other words, the only thing stopping all these articles from being freely available is the authors themselves. If everybody posted their articles in their institution’s repository, then the articles would all be free, publishers couldn’t charge exorbitant subscription fees (although they might still have a role) and we wouldn’t have to win any fights or topple any publishers. This is called the green road (as explained here by Stevan Harnad), as opposed to the gold road of paying open-access journals to publish our articles. If you believe this is the best way to achieve open access, then you may be more concerned with supporting this then to starving the closed journals.

A couple other, more straightforward objections I heard were that some people want to continue reviewing the best articles in their field (or not give up that opportunity if they are junior and aren’t asked often) and others are in fields that don’t have a good open-access outlet, meaning they would end up not reviewing any articles.

The feedback I got (thanks especially to Fabiana Kubke and Rochelle Tractenberg) was that the hypocrite and green road pledging problems could be solved by adding some clauses to the pledge. The pledge I’ve arrived at is this:

I pledge to devote most of my reviewing and editing efforts to manuscripts destined for open access. For other manuscripts, I will restrict myself to one review by me for each review obtained for me by an outlet that is not open access.

The hypocrite problem is solved by the second sentence- one agrees to return the favor for each review of one’s own work that one gets from a closed outlet.

The green road problem is solved by the phrasing “destined for open access”. I defined “destined for open access” as “those that the authors or journal post on institutional or university web repositories, or those that are made open access by the publisher within 12 months.”  (the reason that personal webpages aren’t mentioned is that those are notoriously transient and not always indexed appropriately by scholarly search engines) The tricky bit is that one usually doesn’t know whether the authors will be putting their post-print in a repository, in which case my plan is to put the onus on the journal editor, telling them I can’t review the manuscript unless the authors have promised to put it in a repository. If the authors are funded by the NIH or the Wellcome Trust or are at certain universities with strong open access mandates, it’s reasonable to assume the manuscripts will be posted (although that doesn’t always happen).

I set up openaccesspledge.com, but didn’t try to promote it much, as I thought (as suggested by Mike Taylor) we might instead do some sort of multiple-choice pledge, but that would require some php programming that’s beyond me (any volunteers?), and I’m not sure it’s the best course. In the meantime we’ve collected more than 14 signatures- if you want to take the pledge, please sign.

What do people think of this?  Should we peer-review only for gold open access journals, or also for manuscripts headed for repositories?

Can “Responsible Conduct of Research” include publishing science via blogs?

For Open Access Week 2011, which starts today, I’ve made a video, a draft pledging website, an inspirational website, am giving a talk, and co-written a group letter. This post is about the letter.

As discussed in my last post, there’s a web-based course called “Responsible Conduct of Research” that many thousands of researchers are required to complete each year.  Brad Voytek spotted this question that seems a bit hostile (although quite possibly unintentionally) to new forms of scientific communication outside traditional journals.

I suggested we should write to the organization responsible for the course, and a few people commented on my post to indicate that they agreed. A few tweets later, we had a draft letter going. It’s been really cool to see how social media was able to quickly get a bunch of like-minded scientists together to achieve a goal. This in and of itself undermines the question that we wanted to question :)  Below see our letter- we emailed it to CITI and they responded promptly to thank us for the feedback and to say they’d consider the issues we raised.
———————————————————————————-
Dear Professor Braunschweiger (CITI co-founder) and Professor Ed Prentice (CITI Executive Advisory Committee chair):

We write to challenge the answer to one of the questions in the “Responsible Conduct of Research” online course. The question reads “A good alternative to the current peer review process would be web logs (BLOGS) where papers would be posted and reviewed by those who have an interest in the work”. The answer deemed correct by your system is “False” and the explanation provided includes the assertion that “It is likely that the peer review process will evolve to minimize bias and conflicts of interest”.

We question these claims for two reasons. First, we see real examples of rigorous science happening outside of the traditional system of journal-based peer review. Second, we believe that the future path of scholarly communication is uncertain, and indicating to young researchers that such an important issue is closed is both inaccurate and unhelpful to informed debate.

As an example of science that does not fit the mold suggested by the phrase “the current peer review process”, consider the use of the arXiv preprint server in certain areas of astronomy and physics. In these areas, researchers usually begin by posting their manuscripts to the arXiv server. They then receive comments by those who have an interest in the work. Some of those manuscripts subsequently are submitted to journals and undergo traditional peer review, but many working scientists stay abreast of their field chiefly by reading manuscripts in the arXiv before they are accepted by journals.

Even in areas that are more tightly bound to traditional journals, there are recent examples where both effective peer review of science [1] and science itself [2] have occurred primarily via blogs and other online platforms. In these cases, the online activity appears to have resulted in more rapid progress than would have been possible through the traditional system. A growing body of research suggests that scholars use social media in ways that reflect and produce serious scholarship [3][4][5].

As for the future path of the current mainstream peer review model, we believe it is speculation to say that “It is likely that the peer review process will evolve to minimize bias and conflicts of interest”. The current peer review process may be under considerable strain [6] and unfortunately there is little evidence that it significantly improves the quality of manuscripts [7]. This raises the possibility that big changes are required, not just modifications to reduce bias and conflicts of interest. Furthermore, the question presupposes that the future entity into which peer review will evolve does not involve blogging. No one can see the future clearly enough to make that assumption.

We encourage discussion of this important topic, and would be interested in the inclusion in your program of material that sparks such discussion. However, we believe a true/false question on this topic to be inappropriate, as it limits rather than promotes discussion. All of us wish to see the development and optimization of rigorous systems, both new and traditional, for scientific scholarship. Requiring young researchers to adopt a particular position on this controversial, multifaceted issue may hinder open discussion and future progress.

Sincerely,

Bradley Voytek, PhD, University of California, San Francisco Department of Neurology
Jason Snyder, PhD, National Institutes of Health, USA
Alex O. Holcombe, PhD, School of Psychology, University of Sydney, Australia
William G. Gunn, PhD, Mendeley, USA/UK
Matthew Todd, PhD, School of Chemistry, University of Sydney, Australia
Daniel Mietchen, PhD, Open Knowledge Foundation Germany
Jason Priem, School of Library and Information Science, University of North Carolina at Chapel Hill
Heather Piwowar, PhD, DataONE/NESCent, Canada
Todd Vision, PhD, Department of Biology, University of North Carolina at Chapel Hill
Cameron Neylon, PhD, Science and Technology Facilities Council, UK, Editor in Chief, Open Research Computation

[1] Online experimental peer review of the “Arsenic Life” paper that recently appeared in Science: http://rrresearch.fieldofscience.com/2010/12/arsenic-associated-bacteria-nasas.html
[2] Open Science is a Research Accelerator, M. Woelfle, P. Olliaro and M. H. Todd, Nature Chemistry 2011, 3, 745-748. http://www.nature.com/nchem/journal/v3/n10/full/nchem.1149.html
[3] Groth, P., & Gurney, T. (2010). Studying Scientific Discourse on the Web using Bibliometrics: A Chemistry Blogging Case Study. Presented at the WebSci10: Extending the Frontiers of Society On-Line, Raleigh, NC: US. Retrieved from http://journal.webscience.org/308/
[4] Priem, J., & Costello, K. L. (2010). How and why scholars cite on Twitter. Proceedings of the 73rd ASIS&T Annual Meeting. Presented at the American Society for Information Science & Technology Annual Meeting, Pittsburgh PA, USA. doi:10.1002/meet.14504701201
[5] Weller, K., Dröge, E., & Puschmann, C. (2011). Citation Analysis in Twitter. Approaches for Defining and Measuring Information Flows within Tweets during Scientific Conferences. Proceedings of Making Sense of Microposts Workshop (# MSM2011). Co-located with Extended Semantic Web Conference, Crete, Greece.
[6] Smith R. Classical peer review: an empty gun. Breast Cancer Research 2010, 12(Suppl 4):S13 http://dx.doi.org/10.1186/bcr2742
[7] Jefferson T, Rudin M, Brodney Folse S, Davidoff F. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database of Systematic Reviews 2007, 2:MR000016. http://dx.doi.org/10.1002/14651858.MR000016.pub3

Everything’s fine with peer review- if there are any flaws, they’ll be taken care of by evolution?

Bradley Voytek spotted a disturbing question in an official “Responsible Conduct of Research” training program:

This defense of the status quo has no place in a “Responsible Conduct of Research” training program. It reads like the old guard self-interestedly maintaining the current system by foisting unjustified beliefs onto young researchers!

The part that bothers me the most is the sentence “It is likely that the peer review process will evolve to minimize bias and conflicts of interest”. What is the evidence for this?

Has the process been evolving to minimize bias and conflicts, or to increase them? I don’t think the answer is very clear. As counterweight to the official optimistic opinion, here are a few corrupting influences:

  • Pharmaceutical companies continue to buy influence with medical journals, by buying hundreds of copies of journal issues that run studies that support their products.
  • Pharmaceutical companies continue to ghostwrite journal articles for doctors, to plant their views in the medical literature.
  • Scientists of every stripe often fail to disclose their conflicts of interest.
  • Journals develop new revenue streams, like fast-tracking articles for a fee, that may open them to favoring the select authors who pay.
  • Many reviewers are, like most humans, biased towards their own self-interest. This can yield a bias to recommend rejection of papers by rivals. Because reviewers in most journals are anonymous, they are never held to account.
  • Journals don’t have the resources to investigate authors accused of fraud, and universities often try to avoid finding fault with the researchers they employ.

Many people have suggested partial remedies to these problems, but it’s an uphill battle to implement them, due to the slow pace of change in the journal system. We have to remember this and not be lulled into complacency by the propaganda seen in that training program. It was created by an organization of academics called CITI.

UPDATE: In the comments below, Jason Snyder pointed out an article from CITI in which CITI reports that over 6,000 researchers a month are taking this course — being subjected to this biased question. Some of us object not only to their characterization of the peer review process, but also to their suggestion that blogs are not a good place to do science. We don’t want thousands of researchers to continue to be forced to assent to the conservative opinion articulated by CITI, so we’re drafting a letter asking them to delete the question.

Scientist meets Publisher- Explaining the video

Academic publishing is stuck in an outmoded system. Most scientific research is paid for by government and non-profit university funds, but high-profit corporate publishers often control access to the results of the research. In this video, we showcased the absurdity of the situation and also pointed towards how to get ourselves un-stuck.

There are significant costs associated with what journal publishers do, so we need publishers in some form. But there’s no need for publishing to involve millions in profits and universities having to pay many thousands of dollars for a subscription to a journal.

In the video, the scientist mentions two of the ways we can move towards journal articles being available for free. First is supporting open-access journals. Most charge authors a fee, but one that is not too much higher than their costs, and the result is that anyone can download the article for free.

Another way a researcher can make an article freely available is by depositing the “post-print” in their university or institutional repository. A “post-print” is the draft of the article after it has been peer-reviewed. After a researcher revises their article in accordance with the comments of reviewers, they’ve got a file that may have the same content as that which the journal typesets and publishes. Although the journal usually owns the copyright to the journal version (after the author signs the copyright form), the researcher still in nearly all cases can take their own file, the post-print, and put it in an institution’s official web repository.

If enough of us supported open-access journals, and deposited our other manuscripts in repositories, then journals could no longer charge exorbitant subscription fees. The reason is that with a high percentage of manuscripts available from open-access journals and repositories, universities would cancel their subscriptions to particularly expensive journals.

It’s not just authors that provide free labor to the publishers. It’s also the academics that review each of the articles. So, as reviewers we can also push things towards open access, by saying yes more often to reviewing manuscripts that will be open access, and less often to those that won’t. If we can get a lot of people together to commit to this, it will make a direct impact as well as let others know how many of us support open access. To organize that, I’ve drafted a website called openaccesspledge.com. It also lists other pledges.