Correcting the literature: an uphill battle

Although a search of recent literature will turn up periodic examples in which current research rejects previously published ideas, the difficulty that arises in attempting to do so, particularly on a large scale, is greatly ignored. Extreme difficulties in correcting the literature have been experienced by many, but discussed openly by few. Consultant cardiologist Dr Peter Wilmshurst was particularly proactive in discussing this topic and his own experiences. Wilmshurst dedicated himself for two decades to disseminating his negative findings to reverse the aftermath of a drug trial regarding the heart-contractility drug amrinone. The original article, published in The New England Journal of Medicine (Benotti et al., 1978), stated that amrinone increased heart contractility in a small clinical trial. Wilmshurst’s subsequent studies, by contrast, repeatedly showed that “although amrinone increased the strength of contraction of normal heart muscle, it did not affect contractility in patients with heart failure” (Wilmshurst, 2003). Interestingly, in spite of Wilmshurst’s efforts to disseminate these findings and push for article retraction, he remains unsuccessful.

Australian cell biologist Professor David Vaux recently published an essay regarding his own attempts to refute research and retract his work from arguably the most influential scientific journal, Nature (Vaux, 2013). In 1995, Vaux was invited to peer-review a paper for the journal and was extremely excited by the results that proposed a mechanism to overcome the rejection of transplanted tissue (Bellgrau et al., 1995). This paradigm-changing data influenced him to both write a ‘News and Views’ piece, published in the same issue of Nature (Vaux, 1995), as well as directed his own laboratory’s future research. However, over the next years, the Vaux lab repeatedly failed to replicate these experimental outcomes. Despite Nature’s policies to publish work that refutes data in their publications, Vaux’s findings were ultimately rejected, and later received the same fate from Nature Medicine. Vaux writes, “Little did we know that instead of providing an answer to transplant rejection, these experiments would teach us a great deal about editorial practices and the difficulty of correcting errors once they appear in the literature” (Vaux, 2013). After 2 years, Vaux finally succeeded in publishing his negative findings in PNAS (Allison et al., 1997) and retracted his Nature ‘News and Views’ piece.

Science is, by its nature, a collaborative discipline, and one of the principal reasons why we should report negative results is so our colleagues do not waste their time and resources repeating our findings. Should Vaux not have so forcefully pushed for his negative findings to be published and for retraction of his commentary, there is no doubt that scientists would still be pursuing research in this direction, wasting valuable time and resources. Interestingly, shortly after Vaux was granted the retraction, Nature Medicine published a paper in support of the negative results despite originally rejecting Vaux’s negative findings (Kang et al., 1997). Now, all the findings – positive and negative – have been disseminated and all in the field are entirely informed. Unfortunately, this is not always the case.

Examples from psychiatric research

At a recent conference, two colleagues discovered that they had both unsuccessfully attempted to alter depression-like behaviour in the CD1 mouse strain (a widely used animal model for toxicology studies) with a variety of classical antipsychotics. These findings were surprising in light of the many studies demonstrating efficacy of antipsychotic drugs in different experimental models. Realising that this had occurred in two separate labs, they considered that this might not have been a lack of experiments performed using CD1 mice, but rather a lack of publications on negative findings. They have since corroborated their findings with others. Because the results were unpublished, research groups had continued to follow the same lines of thought and the same paths of investigation, only to all fail in the same way, ultimately wasting time and resources. To our knowledge, these results remain unpublished.

Fundamentally, not publishing the inefficiency of antipsychotic drugs in this animal model could be detrimental to the progression of science; however, one can understand why it was not pursued. The time and effort required to construct the paper and survive the peer-review process is not outweighed by the benefits. Other than ‘for the greater good of science’, this information is not considered high-impact knowledge, and will not result in a highly cited paper. When time is money, and our research output is judged based on impact and citations, why waste the time? In our view, negative results are just as useful as positive findings, but, unfortunately, they do not attract the same citations (Fanelli, 2010).

Pessimism surrounding negative results is obviously a problem, but how are we going to reverse the anti-negative-finding culture? As proposed by Lieberman and Cunningham (Lieberman and Cunningham, 2009), “Perhaps a lab should have to correct for the total number of published results in a given year”. The authors suggest that researchers should be obligated to retract their previous works throughout the progression of their career as they “…[find] that [their] previous tests in old papers are no longer significant in light of their success and, ironically, [their] contribution to the field” (Lieberman and Cunningham, 2009). The view of these authors is that the pressure for positive outcomes is simply putting the scientific community under unnecessary burden and decreases the overall research quality. Negative findings are fundamental to science: they encourage good scientific practice, teach us to critically analyse our pre-existing thoughts and direct new avenues of research. However, while the current scientific culture continues to favour positive findings, negative results will continue to face criticism.

This was the case within our own group when we attempted to disseminate ‘negative’ results. We focus on metabotropic glutamate receptors (mGluRs), which have been implicated in the pathology of schizophrenia and are proposed to be targets of antipsychotic intervention. Despite the large amount of interest and money being poured into the development of mGluR-based therapeutics to treat schizophrenia, the status of these receptors in the pathological state is largely unknown. Before our group embarked on this investigative path, it was unclear whether mGluR protein expression was affected in the pathological states that the novel therapeutics aimed to treat, so examination of mGluR protein expression was a crucial and linear line of investigation. Our findings showed that mGluR2/3 (Frank et al., 2011) and mGluR5 (Matosin et al., 2013) expression was not affected in the pathophysiology of schizophrenia in one of the largest post-mortem human cohorts to date. In the follow-up study, we showed that total mGluR2/3 and mGluR5 expression was unchanged in the cortex of 52 patients with schizophrenia across three independent cohorts compared with controls (our unpublished observations). If you approach these results from the perspective of uncovering the underlying cause of schizophrenia, then yes, these results have little impact. However, in light of the convincing preclinical studies that show promising therapeutic benefits of mGluR-based drugs in schizophrenia-relevant paradigms, our results have interesting translational consequences: if mGluR expression was overly reduced in the pathological state, then one would anticipate problems with the efficacy of these novel drugs in vivo. Although our findings were published in journals that recognise the value in reporting negative findings, we have found that members of the wider scientific community are less accepting. Indeed, we particularly faced resistance at scientific conferences when disseminating our evidence as we were criticised for overstating our findings. It makes one wonder, why is a negative finding viewed as such a bad thing? The notion of a negative finding does seem more philosophical than practical. A negative result is in response to a positive question. If you rephrased to a negative question, does that mean you have a positive finding? These negative findings were integral to our group, positively directing our group’s research and encouraging us to look beneath the surface, where we have since uncovered crucially relevant mechanisms.

Negating the negativity

The revolt against publication bias has sparked a movement in which some have attempted to reverse the pessimism towards negative results. For example, a group of journals have been specifically created to publish the ‘rejects’ (Journal of Negative Results in Biomedicine, The All Results Journals, Journal of Articles in Support Of the Null Hypothesis…and there are more), and new ways to provide access to negative data have emerged (e.g. https://pubpeer.com/). Negative findings are also increasingly represented in broad-scope journals such as Disease Models & Mechanisms and PLoS ONE. Even so, new journals launched with the specific scope of publishing negative findings often do not attract as many papers, demonstrating that it is the underlying scientific culture that requires change and not only the journal policies.

Publication bias is a common theme in the history of science, and it still remains an issue. This is encapsulated in a piece of commentary published in Nature: “…negative findings are still a low priority for publication, so we need to find ways to make publishing them more attractive” (O’Hara, 2011). Negative findings can have positive outcomes, and positive results do not equate to productive science. A reader commented online in response to the points raised by O’Hara: “Imagine a meticulously edited, online-only journal publishing negative results of the highest quality with controversial or paradigm-shifting impact. Nature Negatives” (O’Hara, 2011). Negative results are considered to be taboo, but they can still have extensive implications that are worthy of publication and, as such, real clinical relevance that can be translated to other related research fields.

So, although the current scientific culture assumes that negative results are not worthy of attention, here we present another perspective. Sharing negative results does not mean making a story out of dust, that the results are less significant (excuse the pun), or that the results should go unpublished. It means that the direction of scientific research should not be determined by the pressure to win the ‘significance lottery’, but rather systematic, hypothesis-driven attempts to fill holes in our knowledge. At the core, it is our duty as scientists to both: (1) publish all data, no matter what the outcome, because a negative finding is still an important finding; and (2) have a hypothesis to explain the finding. If the experiment has been performed to plan, the data has not been manipulated or pulled out of context and there is compiled evidence of a negative result, then it is our duty to provide an explanation as to why we are seeing what we are seeing. Only by truly rethinking the current scientific culture, which clearly favours positive findings, will negative results be esteemed for their entire value. Only then can we work towards an improved scientific paradigm.