New APPS

Art, Politics, Philosophy, Science. Mostly Philosophy.

recent posts

about

    When should a journal article be retracted?

    The editor of the journal Food and Chemical Toxicology (FCT), Dr A. Wallace Hayes, has decided to retract the study by the team of Prof Gilles-Eric Séralini, which found that rats fed a Monsanto genetically modified (GM) maize NK603 and tiny amounts of the Roundup herbicide it is grown with suffered severe toxic effects, including kidney and liver damage and increased rates of tumours and mortality.

    But as this article points out, the retraction goes beyond the guidelines of the Committee on Publication Ethics (COPE), of which FCT is a member.  The guidelines state that the a journal should consider retracting a paper if:

    • they have clear evidence that the findings are unreliable, either as a result of misconduct (e.g. data fabri- cation) or honest error (e.g. miscalculation or experimental error)
    • the findings have previously been published elsewhere without proper crossreferencing, permission or justification (i.e. cases of redundant publication)
    • it constitutes plagiarism
    • it reports unethical research

    But none of these applied to the paper by Séralini et al. The journal found "no evidence of fraud or intentional misrepresentation of the data" but that there " is a legitimate cause for concern regarding both the number of animals in each study group and the particular strain selected."



    Suppose, then, that the biggest flaw of this paper is that its results are inconclusive.  Does that mean it should be retracted?  Does it mean that it never should have been published?

    This paper has been out for over a year and has been hotly debated. I don't think it serves science or the general public to retract it.

    Morever, it is on a topic that involves human and other animal health and the actions of major corporations like Monsanto.  Even if inconclusive, it can serve as a spur for other research to either confirm or disconfirm.  Interested citizens can use this information to decide for themselves whether to take action in various campaigns concerning GMOs.

    What is this retraction really about?  The first article I linked to offers some interesting speculations, which I leave as an exercise for the reader.

    Posted in ,

    26 responses to “When should a journal article be retracted?”

    1. Eric Winsberg Avatar
      Eric Winsberg

      Small sample size is clearly not a reason to retract an article. It’s certainly reason to remain agnostic about its conclusions. But a small-sample-sized study can still usefully contribute to a meta-analysis, which is increasingly the way we get reliable information out such studies.
      As for strain selection, this is a standard inductive risk issue. But again, a growing body of evidence combined with good meta-analysis can eventually mitigate this problem. But not if papers are retracted.

      Like

      Reply
    2. Joshua A. Miller Avatar
      Joshua A. Miller

      A piece of research claims to find a result, but on more careful statistical analysis, it is shown by detractors that it doesn’t. There needn’t be fraud to justify retraction, there can be an “honest error.” You second article claims there was no honest error in the raw data, which is supported by the Editor’s investigation. But there was an “honest error” in the data analysis. So the Editor retracts the article on the basis of that error: the “honest error” is in publishing inconclusive results and claiming that they are conclusive. The other possibility is to issue a correction: but here a correction would utterly eliminate the article; it is not correcting peripheral issues, but the heart, the substance of the research. Without that finding, the article isn’t worth publishing under contemporary scientific research norms.
      Certainly Monsanto will have encouraged close attention to this research, since it has a profit motive to refute the research if it can. But that doesn’t make inconclusive data conclusive, and it doesn’t make an honest error not an error. What’s more, Seralini has been taken to task on misuse of statistical methods repeatedly, which makes what at first appears to be an honest statistical error appear less honest in retrospect.
      Another solution to this is to allow for the publication of null results. It actually is interesting that a study of Roundup maize found no conclusive results: that suggests that the possible effect size of any toxicity is more limited than many GMO researchers have worried that it might be. As a consumer of GMO maize, I can take some comfort in this result, even while wishing for further research. But this demand for the publication of data supporting the null hypothesis is not an argument based on COPE’s publication ethics guidelines; it focuses on how those ethical guidelines ought to be changed. Right now the scientific practice prefers positive results, and arguing for an exception here looks like special pleading.

      Like

      Reply
    3. Roberta L. Millstein Avatar
      Roberta L. Millstein

      Good points. Agreed on all counts.

      Like

      Reply
    4. Roberta L. Millstein Avatar
      Roberta L. Millstein

      Thank you for a reasoned presentation of an alternative point of view.
      I meant to say this in my original piece, but I forgot: I think the bar for a retraction should be higher than the bar for not publishing. That is, I think that, once published, even a piece which you might not have otherwise published should remain unless there is a serious reason to retract it; this is especially true for a piece that has stirred as much public debate as this one. I think that they do more harm than good by retracting it (perhaps ironically to the cause of GMOs themselves, as the GM Watch article suggests), and I don’t think that inconclusive results are sufficiently serious to warrant retraction.
      I am sympathetic to your suggestion that we allow publication of null results. But I actually think they they showed more than a null result, and I disagree with you that they made an “honest error” in stating their results as conclusive — they did not say their results were conclusive. Rather, the fourth (“Discussion”) section of the paper makes a number of qualifications and calls for future research.
      It seems to me that there is something in between “conclusive” and “null result”: suggestive. I find their results suggestive, and thus worthy of publication in the first place and worthy of future research. And as Eric Winsberg states above, such suggestive results can be combined with other results to produce more substantive conclusions.

      Like

      Reply
    5. Charles Pence Avatar
      Charles Pence

      There were also concerns about the health and welfare of the animals — namely, that the animals were measured based on time to mortality, rather than time to cancer incidence, which, in a strain of rats that is quick to develop cancer, means that they spent much of their lives dying slowly of large amounts of cancer, far from the normal research standard of care for laboratory animals. I’m surprised that the journal’s own press release doesn’t mention that, as it’s been covered in several other news outlets (a quick search finds a quote here <“>http://www.sciencemediacentre.co.nz/2013/11/29/gm-maize-and-cancer-study-retracted-experts-respond/&gt; from a fellow researcher; I’ve seen the worry raised elsewhere as well).
      Not that that necessarily should rise to the level of retraction, but it does mean that the last of the four guidelines was likely reached.

      Like

      Reply
    6. Eric Winsberg Avatar
      Eric Winsberg

      That does seem like a serious breach of research ethics, but I’m curious whether that sort of thing is typically the cause of a publication retraction. Anyone know about this?

      Like

      Reply
    7. Roberta L. Millstein Avatar
      Roberta L. Millstein

      Good point, Charles…

      Like

      Reply
    8. Roberta L. Millstein Avatar
      Roberta L. Millstein

      …and good question, Eric. I don’t know the answer either.

      Like

      Reply
    9. Duncan Law Avatar
      Duncan Law

      In addition to the excellent points raised above by Joshua A. Miller and Charles Pence (which are more than sufficient to meet the COPE guidelines and warrant retraction, in my view), it’s worth noting that the researchers also gave copies of the paper to journalists

      provided they signed confidentiality agreements that prevented them from consulting other experts about the research before publication.

      [See this post at Embargo Watch.]
      This is also very poor practice, and does not increase one’s confidence in the ethics of the project, or in the researchers’ own confidence in the reliability of their findings.

      Like

      Reply
    10. Joshua Miller Avatar
      Joshua Miller

      I don’t understand how results that are indistinguishable from normal variance can be “suggestive.” Don’t they suggest normal variance?

      Like

      Reply
    11. Roberta L. Millstein Avatar
      Roberta L. Millstein

      sug·ges·tive
      adj.
      1.
      a. Tending to suggest; evocative: artifacts suggestive of an ancient society.
      b. Stimulating further thought: “Suggestive here is the Southern, often Western and rural locus of these tales” (Mark Muro).
      c. Conveying a hint or suggestion: conveyed the message with a few suggestive words.
      (American Heritage Dictionary)
      Look at the journal that Séralini et al. published in: a toxicity journal. Look at the protocol that they say that they are following: OECD 452, which is for chronic toxicity studies. In other words, even though the pictures of rats with tumors got all of the press, cancer was not the original point of the study — toxicity was. And it is not unusual to use small sample sizes for toxicity studies. Séralini et al. reported, “In treated males, liver congestions and necrosis were 2.5–5.5 times higher. This pathology was confirmed by optic and transmission electron microscopy. Marked and severe kidney nephropathies were also generally 1.3–2.3 greater” — findings that were not disputed by the journal. Note the reason for retraction: “A more in-depth look at the raw data revealed that no definitive conclusions can be reached with this small sample size regarding the role of either NK603 or glyphosate in regards to overall mortality or tumor incidence” (emphasis added). No mention of the kidney or liver effects. I find that strange.
      As for the tumors, they reported the numbers, and made no claim for statistical significance. Should they not have reported the tumors? That would seem to suppress potentially important, or “suggestive” evidence.
      Yes, this study should be repeated with a larger sample size. But it was wrong to retract it.

      Like

      Reply
    12. Joshua A. Miller Avatar
      Joshua A. Miller

      Here’s what the French report said about the liver morphology issue you mention:

      “As for the biological parameters, the variations described cannot be interpreted as linked to severe liver or kidney damage. Liver damage is described only morphologically, and variations in the blood parameters are either not biologically significant (for example, two parameters that ought to vary along the same lines (ALAT and ASAT) vary diametrically, Fig. 5A) or are not reported by the authors. The variations in the kidney parameters are modest and not always correlated with each other (Fig. 5 and Table 3). Kidney failure is defined by a reduction in the glomerular filtration rate (GFR). Creatinine is the parameter most widely used to estimate GFR, which is calculated by measuring the urinary and serum concentration of the creatinine. Kidney damage results in a rise in serum creatinine and serum urea, a drop in urinary creatinine, a drop in creatinine clearance, and abnormal quantities of protein in urine. In this study, consistency between biological parameters is lacking: in Figure 5A (females fed the 33% GM maize diet) serum creatinine is reduced and urea is normal, urinary creatinine is 15 reduced and creatinine clearance (termed ‘U.Clearance’20) is reduced. If reduced clearance is a sign of kidney failure, a reduction in the creatinine present in the blood is abnormal here. With such biological inconsistency, the hypothesis of kidney failure or kidney damage cannot be verified.“

      That the Editor did not mention this issue doesn’t mean it doesn’t also figure in her decision; a long list of the study’s failings just makes the Editor look bad for publishing it, so it’s enough to mention the sufficient conditions of retraction and leave the rest to others. The French report also indicates that there are enough irregularities in methods and reporting that including this study in a metastudy would be fruitless. That actually undermines my call for null hypothesis publication, sadly.
      Perhaps a new study will establish the link, but we should be careful about “suggestive,” and we shouldn’t allow suggestions based in inadequate evidence to sway our priors at all. To do otherwise is to allow data that has been shown to be massaged and cherry-picked to confirm our suspicions, which we would never do if the data opposed our preferred outcomes. For instance, rats that ate a larger percentage of Roundup maize actually had 66% lower mortality rates than the control group (1 dead from the 22% and 33% groups v 3 dead from the control group.) Yet I don’t think that “suggests” that Roundup maize is a panacea that saves rats’ lives, I think it means that inconclusive data was reported in such a way as to make it “suggestive” by ignoring all the disconfirming bits.

      Like

      Reply
    13. Rebecca Kukla Avatar
      Rebecca Kukla

      I don’t know hard data on what is ‘typically’ the reason for retraction, but in the biomed world, all the retractions that I have seen have been because of either academic dishonesty (often undisclosed and major conflicts of interest that have come out) or because of straight-ahead error. I agree that this is very fishy business, here.

      Like

      Reply
    14. Eric Winsberg Avatar
      Eric Winsberg

      At the very least, retracting an article because the data were collected unethically (but lets stipulate: in an epistemologically sound manner) strikes me as something for extreme cases. We can argue about whether data from Nazi hypothermia experiments ought to be published and used or not. And we can all agree that scientists who mistreat animals (by, say, letting them die a slow death of cancer unnecessarily) should be sanctioned/disciplined/whatever. But retracting their published data or conclusions on those grounds seems to be missing the point.

      Like

      Reply
    15. Charles Pence Avatar
      Charles Pence

      Eric–
      I completely agree. I’d assume that unethical treatment of animals is only even on the COPE guidelines precisely because of extreme cases of animal neglect and abuse, to give journal editors some way to punish authors who dramatically violate protocol. But I could be wrong; I don’t actually know what the usual response is is to (though it feels somehow wrong to phrase it this way) “moderate” violations of research ethics.

      Like

      Reply
    16. Roberta L. Millstein Avatar
      Roberta L. Millstein

      I have to admit that the discussion of liver and kidney damage goes beyond my area of expertise. However, I don’t find your explanation of why the Editor might leave off mention of it very plausible. Hayes had to know that this would be a highly scrutinized decision and that it would behoove him to make his case for retraction as strong as possible. He’s already taken a year’s full of heat for publishing the article in the first place; I doubt that there is anything new that could be said now about that decision. Furthermore, he has to know that he is breaking new ground with this retraction (if anyone knows of any similar cases, please mention them), and for that reason as well, needs to have it be fully justified. It seems unlikely to me that he’d leave off these reasons if they were so obviously definitive, again, especially since Séralini et al have made it clear (in the original paper and since) that studying such effects were the original and primary goal of the study.

      Like

      Reply
    17. Roberta L. Millstein Avatar
      Roberta L. Millstein

      Rebecca, Charles, and Eric: I too am having trouble parsing degrees of ethical violation here, but I suspect my standards are different from COPE’s.

      Like

      Reply
    18. Duncan Law Avatar
      Duncan Law

      One other reason why the Editor’s statement may not articulate the reasons for retraction as strongly as possible, is Séralini’s threat of a lawsuit against the journal.

      Like

      Reply
    19. Roberta L. Millstein Avatar
      Roberta L. Millstein

      I’m not a lawyer (or a journal editor), but that seems backwards to me. I would think that the weaker you make your retraction case, the more you invite a lawsuit. It also seems likely to me that the lawsuit would have happened regardless, especially if (as I suspect) there are no similar retraction precedents.

      Like

      Reply
    20. Duncan Law Avatar
      Duncan Law

      I’m not a lawyer or an editor either – but I imagine a lawsuit would be over reputational damage to Séralini’s team – and in such a circumstance it would seem prudent of an editor who had decided there was reason to retract the paper, to aim to minimise this reputational damage to the greatest extent compatible with providing warrant for the retraction. This goal seems to be compatible with the phrasing of Hayes’ statement.
      I’m not sure what you mean when you say there are no similar retraction precedents – presumably not that no journal has ever previously retracted a paper on the basis of flawed data analysis. In what respect is this retraction unique?

      Like

      Reply
    21. Joshua Miller Avatar
      Joshua Miller

      I am not a lawyer, either, but I do think that caution is warranted when dealing with lawsuits, especially when the cause of action is likely to be libel. Saying as little as possible in that circumstance is the best defense, because you won’t be liable for the retraction, you’ll be liable for the public justification you give.
      Notably, Seralini has sued for libel before: he was awarded 1 euro in damages, plus legal fees. So a small amount of real damage can be quite expensive if the legal fees mount. In the US we call this a SLAPP suit and judge it harshly, but I suppose norms (like laws) differ.

      Like

      Reply
    22. Roberta L. Millstein Avatar
      Roberta L. Millstein

      Again, not a lawyer, but I disagree. I’d guess that the lawsuit is for the retraction and purported harms to Séralini and his group, and would likely come regardless. The less basis there is for a retraction, the stronger their case against the journal, it seems to me. (But I repeat myself). I don’t see how adding anything about the purported liver and kidney effects would increase or decrease purported libel.
      I did not say that this retraction was unique. I said I was unaware of any similar cases, and asked if anyone else was. Again, I suspect that there aren’t, but of course I could be wrong. I think retraction is overwhelming reserved for misconduct or error, not “inconclusive results.”

      Like

      Reply
    23. Duncan Law Avatar
      Duncan Law

      This is a case of error – error in data analysis that led the paper to draw conclusions unsupported by the data.

      Like

      Reply
    24. Duncan Law Avatar
      Duncan Law

      For example, here’s the penultimate sentence of the paper: “Altogether, the significant biochemical disturbances and physiological failures documented in this work confirm the pathological effects of these GMO and R treatments in both sexes, with different amplitudes.” But the paper confirms no such thing. That’s an error.

      Like

      Reply
    25. Roberta L. Millstein Avatar
      Roberta L. Millstein

      And as we’ve already been down that path and failed to convince the other, I am not going to respond on this point any further.

      Like

      Reply
    26. Duncan Law Avatar
      Duncan Law

      Ok. I would urge interested readers to look at the French High Council of Biotechnology Scientific Committee Opinion, for an articulation of the reasons why the findings of this study are unreliable.

      Like

      Reply

    Leave a comment