Archive for the ‘open science’ Category
Science is broken; let’s fix it. This has been my mantra for some years now, and today we are launching an initiative aimed squarely at one of science’s biggest problems. The problem is called publication bias or the file-drawer problem and it’s resulted in what some have called a replicability crisis.
When researchers do a study and get negative or inconclusive results, those results usually end up in file drawers rather than published. When this is true for studies attempting to replicate already-published findings, we end up with a replicability crisis where people don’t know which published findings can be trusted.
To address the problem, Dan Simons and I are introducing a new article format at the journal Perspectives on Psychological Science (PoPS). The new article format is called Registered Replication Reports (RRR). The process will begin with a psychological scientist interested in replicating an already-published finding. They will explain to we editors why they think replicating the study would be worthwhile (perhaps it has been widely influential but had few or no published replications). If we agree with them, they will be invited to submit a methods section and analysis plan and submit it to we editors. The submission will be sent to reviewers, preferably the authors of the original article that was proposed to be replicated. These reviewers will be asked to help the replicating authors ensure their method is nearly identical to the original study. The submission will at that point be accepted or rejected, and the authors will be told to report back when the data comes in. The methods will also be made public and other laboratories will be invited to join the replication attempt. All the results will be posted in the end, with a meta-analytic estimate of the effect size combining all the data sets (including the original study’s data if it is available). The Open Science Framework website will be used to post some of this. The press release is here, and the details can be found at the PoPS website.
Professor Daniel J. Simons (University of Illlinois) and I are co-editors for the RRRs. The chief editor of Perspectives on Psychological Science is Barbara A. Spellman (University of Virginia), and leadership and staff at the Association for Psychological Science, especially Eric Eich and Aime Ballard, have also played an important role (see their press release).
Three features make RRRs very different from the usual way that science gets published:
1. Preregistration of replication study design and analysis plan and statistics to be conducted BEFORE the data is collected.
- Normally researchers have a disincentive to do replication studies because they usually are difficult to publish. Here we circumvent the usual obstacles to replications by giving researchers a guarantee (provided they meet the conditions agreed during the review process) that their replication will be published, before they do the study.
- There will be no experimenter degrees of freedom to analyse the data in multiple ways until a significant but likely spurious result is found. This is particularly important for complex designs or multiple outcome variables, where those degrees of freedom allow one to always achieve a significant result. Not here.
2. Study is sent for review to the original author on the basis of the plan, BEFORE the data come in.
- Unlike standard replication attempts where the author of the published, replicated study sees it only after the results come in, we will catch the replicated author at an early stage. Many will provide constructive feedback to help perfect the planned protocol so it has the best chance of replicating the already-published target effect.
3. The results will not be presented as a “successful replication” or “failed replication”. Rarely is any one data set definitive by itself, so we will concentrate on making a cumulative estimate of the relevant effect’s size, together with a confidence interval or credibility interval.
- This will encourage people to make more quantitative theories aimed at predicting a certain effect size, rather than only worrying about whether the null hypothesis can be rejected (as we know, the null hypothesis is almost never true, so can almost always be rejected if one gets enough data).
This initiative is the latest in a long journey for me. Ten years ago, thinking that allowing the posting of comments on published papers would result in flaws and missed connections to come to light much earlier, David Eagleman and I published a letter to that effect in Nature and campaigned (unsuccessfully) for commenting to be allowed on PubMed abstracts.
Since then, we’ve seen that even where comments are allowed, few scientists make them, probably because there is little incentive to do so and doing it would risk antagonising their colleagues. In 2007 I became an academic editor and advisory board member for PLoS ONE, which poses fewer obstacles to publishing replication studies than do most journals. I’m lucky to have gone along on the ride as PLoS ONE rapidly became the largest journal in the world (I resigned my positions at PLoS ONE to make time for the gig at PoPS). But despite the general success of PLoS ONE, replication studies were still few and far between.
In 2011, Hal Pashler, Bobbie Spellman, Sean Kang and I started PsychFileDrawer, a website for researchers to post notices about replication studies. This has enjoyed some success, but it seems without the carrot of a published journal article, few researchers will upload results, or perhaps even conduct replication studies.
Finally with this Perspectives on Psychological Science initiative, a number of things have come together to overcome the main obstacles to publication studies: fear of antagonising other researchers and the uphill battle required to get the study published. Some other worthy efforts to encourage replication studies are happening at Cortex and BMC Psychology.
If you’re interested in proposing to conduct a replication study for eventual publication, check out the instructions and then drop us a line at replicationseditor @ psychologicalscience.org!
Scientists of all sorts increasingly recognize the existence of systemic problems in science, and that as a consequence of these problems we cannot trust the results we read in journal articles. One of the biggest problems is the file-drawer problem. Indeed, it is mostly as a consequence of the file-drawer problem that in many areas most published findings are false.
Consider cancer preclinical bench research, just as an example. The head of Amgen cancer research tried to replicate 53 landmark papers. He could not replicate 47 of the 53 findings.
In experimental psychology, a rash of articles has pointed out the causes of false findings, and a replication project that will dwarf Amgen’s is well underway. The drumbeat of bad news will only get louder.
What will be the consequences for you as an individual scientist? Field-wide reforms will certainly come, partly because of changes in journal and grant funder policies. Some of these reforms will be effective, but they will not arrive fast enough to halt the continued decline of the reputation of many areas.
In the interim, more and more results will be viewed with suspicion. This will affect individual scientists directly, including those without sin. There will be:
- increased suspicion by reviewers and editors of results in submitted manuscripts (“Given the history of results in this area, shouldn’t we require an additional experiment?“)
- lower evaluation of job applicants for faculty and postdoctoral positions (“I’ve just seen too many unreliable findings in that area“)
- lower scores for grant applications (“I don’t think they should be building on that paper without more pilot data replicating it“)
These effects will be unevenly distributed. They will often manifest as exaggerations of existing biases. If a senior scientist already had a dim view of social psychology, for example, then the continuing replicability crisis will likely magnify his bias, whereas his view of other fields that the scientist “trusts” will not be as affected by the whiff of scandal, at least for awhile- people have a way of making excuses for themselves and their friends.
But there are some things you can do to protect yourself. These practices will eventually become widespread. But get a head start, and look good by comparison.
- Preregister your study hypotheses, methods, and analysis plan. If you go on record with your plan before you do the study, this will allay the suspicion that your result is not robust, that you fished around with techniques and statistics until you got a statistically significant result. Journals will increasingly endorse a policy of favoring submitted manuscripts that have preregistered their plan in this way. Although websites set up to take these plans may not yet be available in your field, they are coming, and in the meantime you can post something on your own website, on FigShare perhaps, or in your university publicly accessible e-repository.
- Post your raw data (where ethically possible), experiment code, and analysis code to the web. This says you’ve got nothing to hide. No dodgy analyses, and you welcome the contributions of others to improve your statistical practices.
- Post all pilot data, interim results, and everything you do to the web, as the data come in. This is the ultimate in open science. You can link to your “electronic laboratory notebooks” in your grants and papers. Your reviewers will have no excuse to harbor dark thoughts about how your results came about, when they can go through the whole record.
The proponents of open science are sometimes accused of being naifs who don’t understand that secretive practices are necessary to avoid being scooped, or that sweeping inconvenient results under the rug is what you got to get your results into those high impact-factor journals. But the lay of the land has begun to change.
Make way for the cynics! We are about to see people practice open science not out of idealism, but rather out of self interest, as a defensive measure. All to the better of science.
Below are research presentations I’m involved in for Vision Sciences Society in May. If you’re attending VSS, don’t forget about the Publishing, Open Access, and Open Science satellite which will be Friday at 11am. Let us know your opinion on the issues and what should be discussed here
Splitting attention slows attention: poor temporal resolution in multiple object tracking
Alex O. Holcombe, Wei-Ying Chen
Session Name: Attention: Tracking (Talk session)
Session Date and Time: Sunday, May 13, 2012, 10:45 am – 12:30 pm
Location: Royal Ballroom 4-5
When attention is split into foci at disparate locations, the minimum size of the selection focus at each location is larger than if only one location is targeted (Franconeri, Alvarez, & Enns, 2007)- splitting attention reduces its spatial resolution. Here we tested temporal resolution and speed limits. STIMULUS. Three concentric circular arrays (separated by large distances to avoid spatial interactions between them) of identical discs were centered on fixation. Up to three discs (one from each ring) were designated as targets. The discs orbited fixation at a constant speed, occasionally reversing direction. After the discs stopped, participants were prompted to report the location of one of the targets. DESIGN. Across trials, the speed of the discs and the number in each array was varied, which jointly determined the temporal frequency. For instance, with 9 objects in the array, a speed of 1.1 rps would be 9.9 Hz. RESULTS. With only one target, tracking was not possible above about 9 Hz, far below the limits for perceiving the direction of the motion, and consistent with Verstraten, Cavanagh, & LaBianca (2000). The data additionally suggest a speed limit, with tracking impossible above 1.8 rps, even when temporal frequency was relatively low. Tracking two targets could only be done at lower speeds (1.4 rps) and lower temporal frequencies (6 Hz). This decrease is approximately that predicted if at high speeds and high temporal frequencies, only a single target could be tracked. Tracking three yielded still lower limits. Little impairment was seen at very slow speeds, suggesting these results were not caused by a reduction in spatial resolution. CONCLUSION. Splitting attention reduces the speed limits and the temporal frequency limits on tracking. We suggest a parallel processing resource is split among targets, with less resource on a target yielding poorer spatial and temporal precision and slower maximum speed.
A hemisphere-specific attentional resource supports tracking only one fast-moving object.
Wei-Ying Chen & Alex O. Holcombe
Session Name: Attention: Tracking (Talk session)
Session Date and Time: Sunday, May 13, 2012, 10:45 am – 12:30 pm
Location: Royal Ballroom 4-5
Playing a team sport or taking children to the beach involves tracking multiple moving targets. Resource theory asserts that a limited resource is divided among targets, and performance reflects the amount available per target. Holcombe and Chen (2011) validated this with evidence that tracking a fast-moving target depletes the resource. Using slow speeds Alvarez and Cavanagh (2005) found the resource consumed by additional targets is hemisphere-specific. They didn’t test the effect of speed, and here we tested whether speed also depletes a hemisphere-specific resource. To put any speed limit cost in perspective, we modeled a “total depletion” scenario- the speed limit cost if at high speeds one could not track the additional target at all and had to guess one target. Experiment 1 found that the speed limit for tracking two targets in one hemifield was similar to that predicted by total depletion, suggesting that the resource was totally depleted. If the second target was instead placed in the opposite hemifield, little decrement in speed limit occurred. Experiment 2 extended this comparison to tracking two vs. four targets. Compared to the speed limit for tracking two targets in a single hemifield, adding two more targets in the opposite hemifield left the speed limit largely unchanged. However starting with one target in both the left and right hemifields, adding another to each hemifield had a severe cost similar to that of the total depletion model. Both experiments support the theory that an object moving very fast exhausts a hemisphere-specific attentional tracking resource.
Attending to one green item while ignoring another: Costly, but with curious effects of stimulus arrangement
Shih-Yu Lo & Alex O. Holcombe
Session Name: Attention: Features I (Poster session)
Session Date and Time: Monday, May 14, 2012, 8:15 am – 12:15 pm
Location: Vista Ballroom
Splitting attention between targets of different colors is not costly by itself. As we found previously, however, monitoring a target of a particular color makes one more vulnerable to interference by distracters that share the target color. Participants monitored the changing spatial frequencies of two targets of either the same (e.g., red and red) or different colors (e.g., red and green). The changing stimuli disappeared without warning and participants reported the final spatial frequency of one of the targets. In the different-colors condition, a large cost occurs if a green distracter is superposed on the red target in the first location and a red distracter is superposed on the green target in the second location. This likely reflects a difficulty with attending to a color in one location while ignoring it in another. Here we focus on a subsidiary finding regarding perceptual lags. Participants reported spatial frequency values from the past rather than the correct final value, and such lags were greater in the different-colors condition. This “perceptual lag” cost was found when the two stimuli were horizontally arrayed but not, curiously, when they were vertically arrayed. Arrangement was confounded however with processing by separate brain hemispheres (opposite hemifields). In our new study, we unconfounded arrangement and presentation in separate hemifields with a diagonal condition- targets were not horizontally arrayed but were still presented to different hemifields. No significant different-colors lag cost was found in this diagonal arrangement (5 ms) or in the vertical arrangement (86 ms), but the cost (167 ms) was significant in the horizontal arrangement, as in previous experiments. Horizontal arrangement apparently has a special effect apart from the targets being processed by different hemispheres. To speculate, this may reflect sensitivity to bilateral symmetry and its violation when the target colors are different.
Dysmetric saccades to targets moving in predictable but nonlinear trajectories
Reza Azadi, Alex Holcombe, and Jay Edelman
A saccadic eye movement to a moving object requires taking both the object’s position and velocity into account. While recent studies have demonstrated that saccades can do this quite well for linear trajectories, its ability to do so for stimuli moving in more complex, yet predictable, trajectories is unknown. With objects moving in circular trajectories, we document failures of saccades not only to compensate for target motion, but even to saccade successfully to any location on the object trajectory. While maintaining central fixation, subjects viewed a target moving in a circular trajectory at an eccentricity of 6, 9, or 12 deg for 1-2 sec. The stimulus orbited fixation at a rate of 0.375, 0.75, or 1.5 revolutions/sec. The disappearance of the central fixation point cued the saccade. Quite unexpectedly, the circularly moving stimuli substantially compromised saccade generation. Compared with saccades to non-moving targets, saccades to circularly moving targets at all eccentricities had substantially lower amplitude gains, greater curvature, and longer reaction times. Gains decreased by 20% at 0.375 cycles/sec and more than 50% at 1.5 cycles/sec. Reaction times increased by over 100ms for 1.5 cycles/sec. In contrast, the relationship between peak velocity and amplitude was unchanged. Given the delayed nature of the saccade task, the system ought to have sufficient time to program a reasonable voluntary saccade to some particular location on the trajectory. But, the abnormal gain, curvature, and increased reaction time indicate that something else is going on. The successive visual transients along the target trajectory perhaps engage elements of the reflexive system continually, possibly engaging vector averaging processes and preventing anticipation. These results indicate that motor output can be inextricably bound to sensory input even during a highly voluntary motor act, and thus suggest that current understanding of reflexive vs. voluntary saccades is incomplete.
The transmission of new scientific ideas and knowledge is needlessly slow:
|Journal subscription fees||Open access mandates|
|Competition to be first-to-publish motivates secrecy||Open Science mandates|
|Jargon||Increase science communication; science blogging|
|Pressure to publish high quantity means no time for learning from other areas||Reform of incentives in academia|
|Inefficient format of journal articles (e.g. prose)||Evidence charts, ?|
|Long lag time until things are published||Peer review post publication, not pre publication|
|Difficulty publishing fragmentary criticisms||Open peer review; incentivize post-publication commenting|
|Information contained in peer reviewers’ reviews is never published||Open peer review or publication of (possibly anonymous) reviews; incentivize online post-publication commenting|
|Difficulty publishing non-replications||Open Science|
UPDATE: Daniel Mietchen, in the true spirit of open science, has put up an editable version of this very incomplete table.
I recently learned that a journal called Obesity Reviews has a “Fast Track Facility”:
A submission fee of $1,000 or £750 for articles up to 9000 words long, or $1500 for articles more than 9,000 words long guarantees peer-review within 10 working days
This is a terrible development for academia. It creates a two-tier system, wherein scientists who are well-funded such as those from rich countries now have an unfair advantage over those who don’t. Science traditionally has been a partial refuge from the injustice of rich vs. poor. Although of course it was never entirely insulated from it, scientific institutions and journals have in the past have tried to treat all authors similarly.
To some, this “Fast Track Facility” may seem similar to the system of open-access journals where authors pay a fee to have their article published if it passes peer review. At least in the case of PLoS ONE (the open-access journal I am an editor for), however, this is very different because PLoS ONE waives the fee for authors who cannot pay. Authors who cannot pay are treated the same as authors who can pay. The “Fast Track Facility” policy of Obesity Reviews violates this fundamental principle of fairness.
UPDATE: Help write and sign on to a protest letter
Previously, the Australian Research Council (ARC) expressly forbade use of grant funds to pay publication charges. This prevented many of us from publishing in open-access journals, as they generally charge a fee.
Fortunately, the newly-revised funding rules change that, and instead strongly encourage open access, via journals and via depositing one’s research in an institutional repository.
5.2.2 Publication and dissemination of Project outputs and outreach activity costs
may be supported at up to two (2) per cent of total ARC funding awarded to
the Project. The ARC strongly encourages publication in publicly accessible
outlets and the depositing of data and any publications arising from a
Project in an appropriate subject and/or institutional repository.
13.3.2 The Final Report must justify why any publications from a Project have
not been deposited in appropriate repositories within 12 months of
publication. The Final Report must outline how data arising from the
Project has been made publicly accessible where appropriate.
If you have ideas of how this should be made stronger for future years, let me know and maybe we can sign a letter to the ARC together. It would be good to move towards a mandate, as suggested by the people of open access week.
A new article in the New York Times regarding the allegations against Marc Hauser illustrate how difficult it is to determine whether one is guilty of scientific fraud. A main problem is that record-keeping standards are so lax.
This is another reason why open science is important. Open science involves releasing original data and analyses, which is much easier if you have been keeping good records along the way. So it pushes one to keep better records.
In a more ironic twist regarding open science (via my colleague Bart), the maker of Baby Einstein videos (maligned in some scientific papers claiming to show that his products provided no benefits to children) has filed a court complaint asking the university to release the original data. A very one-sided press release claims that the university has balked and stalled repeatedly, which if true is shameful. Norms need to shift in science to make release of original data a commonplace, not something that’s disputed.
This is open-access week.
Most already know they should be publishing in open-access journals and/or self-archiving their papers. But moving science towards open access has been… slow.
I was going to write that things have been moving at a glacial pace, thinking that would be an exaggeration; but nowadays I worry the world’s glaciers may be moving/melting fast enough to be gone before open access is the norm. To get science moving faster, scientists need to convince their institutions and funders to mandate that their science be available free online.
And what about the program code or script you use to power your experiment or analyze your data? Have you ever gotten all the way through your analysis, been adding what you thought were the final touches, and then found an error in your code? Coding errors can be hard to catch. We need to be publishing our code together with our paper and our data because:
- Scientists spend too much time programming things from scratch that others have already programmed.
- To verify published science, you ought to be able to examine the associated code. In science, coding errors do happen.
Publishing one’s code and data as well as the manuscript is part of open science.
When a scientific article is published, ideally the data behind the reported results should be made available. Anyone should be able to scrutinize the basis of scientific claims.
While this has long been the ideal, it has rarely been practiced. But this has been changing, and momentum is building to actually require the posting of data in circumstances where it’s feasible. See the list below for links regarding the movement of various scientific fields, science funders, and repositories towards requiring and enabling data sharing.
The PLoS multidisciplinary journals, including PLoS ONE, are considering nudging authors towards sharing by requiring a “data availability statement”. Thanks to the thoughtful people on FriendFeed who have commented on the idea.
Objections have been raised to this proposal. I don’t think any of the objections are show-stoppers, but some make some valid points. Here’s one- requiring posting of raw data would not prevent fraudulent behaviors as the evil-doers would simply start manipulating their raw data or fabricating it. I have two thoughts about that. First, fabricating raw data is often a much bigger task for an author than fudging a few data points on a plot, or changing a p-value to be better than it was. If one is to fabricate an entire dataset, one has to self-consciously think like a criminal for an extended period, all along realizing that what one is doing is wrong. Out-and-out cheaters will certainly do this, but many will stop short of fabricating the raw data. It is not a behavior that can be self-justified as some kind of shortcut or ‘making up for how the instrument wasn’t working well that day’ or other justifications for fudging. And I think most current problems with reported results fall into these latter categories. A second point is that despite the seeming ease of fabricating raw data, when humans do this they often leave tell-tale signs that indicate the data were tampered with. See Benford’s Law for one example. I know, I know, perhaps only the stupid scientists wouldn’t be able to randomize their fake data properly but nevertheless many frauds are detected based on these kinds of mistakes.
Below are some examples of the growing movement towards more sharing of raw data. This is an exciting moment for science!
- A Trends in Cognitive Sciences article by neuroscience bigwigs explains why we must share more of the data
- Proteomics journals decide to mandate data sharing
- new NSF requirement to explain how grantees will share their data
- Biomed Central moves towards requiring that authors make their data available
- The Panton Principles for open data
- Dryad is a repository of data, initially focusing on evolution, ecology, and related fields
- Climategate scientists were cleared but “had not shown sufficient openness” regarding the data
More and more researchers agree that more access is needed to the original data behind published research articles. Of course, the more general point is that not just the original data, but all materials needed to scrutinize the claims of a manuscript should be available.
The policy of the journal Science is:
After publication, all data necessary to understand, assess, and extend the conclusions of the manuscript must be available to any reader of Science… Large data sets with no appropriate approved repository must be housed as supporting online material at Science, or only when this is not possible, on an archived institutional Web site, provided a copy of the data is held in escrow at Science to ensure availability to readers.
There are many cases where aspects of the data cannot be made available, for example due to patient privacy requirements, and Science of course makes allowances for that.
But In contrast to Science‘s enlightened policy, the Journal of Neuroscience looks to have just taken a step backward. They announced that they will no longer allow any supplemental material to be submitted along with the main text of authors’ articles.
Supplemental materials were being used to include a broad array of things that help to interpret the content of an article. These things are really needed to increase the transparency of science. Still, I do sympathize a bit with the desire of J Neurosci to do away with them. Often they are used in a very annoying fashion. Authors sometimes put critical data analyses and experiments in the supplemental material, and as a reader it becomes really difficult to read an article when one has to switch between the often overly-concise main text and the sometimes poorly organized supplemental material. J Neurosci points out that evaluating these materials was often a significant burden on the reviewers. However, I don’t think that simply eliminating supplemental materials is an appropriate response.
Elimination of supplemental materials should be accompanied by a clear policy on how the information otherwise therein will be made available to readers, to ensure the integrity of science. The announcement does make a gesture in that direction, saying that authors should provide a weblink to information on their own site, and that perhaps the elimination of supplemental material will “motivate more scientific communities to create repositories for specific types of structured data, which are vastly superior to supplemental material as a mechanism for disseminating data.” However, overall the announcement gives the impression that if anything, the recommendation that authors provide supporting data and material has been relaxed. Heather Piwowar has more.
J Neurosci should have accompanied this change with a statement that the expectation that authors fully back up their claims with public information is increasing, not decreasing!
In the longer term, I believe the solution to all this may come from more fundamental reform of how science is communicated. Ideally, the workings of a scientific enterprise and the progress being made should be visible even before formal publication. This is called Open science.
The development of better open science tools will obviate some of the concerns of the Journal of Neuroscience. Formal publication of an article will not be as big a step as it is now, where suddenly all of the materials associated with a scientific claim appear out of nowhere. Instead, there will be an electronic paper trail linking back to the data records created as the data were collected and the analyses were done. There are many reasons why currently scientists do not or cannot do such things, but those who can and are making the effort to do so should be applauded and supported.