Fake news, zombie papers and fabricated evidence

Abstract

'To protect participants and maintain public confidence in research, it is important that all research is conducted lawfully, with honesty and integrity, and in accordance with good practice.'1 The global pandemic caused by COVID-19 has released a tidal wave of information upon both the medical profession and the general public. There has been an unprecedented unity and response among researchers to the crisis2 and the ability to conduct research at short notice during the global pandemic should be commended. In addition, it is praiseworthy that publishers have made COVID-19 articles open access rather than hiding them behind paywalls to help inform clinicians and other researchers. However at the same time, there has been a general dumping of information with the rise of a phenomenon known as medicine by press release and preprint release of articles, none of which are peer reviewed, and this has led to questioning of the validity of the evidence provided.3,4 Compared with randomised controlled trials (RCTs), nonrandomised and observational studies cannot be scrutinised, as such, for discrepancies in random sampling assumptions. They are more likely to contain differences in baseline variables and to overestimate the magnitude of the true effect size.5 Therefore, they are often excluded from systematic review and meta-analysis (SRMA). Trust has been bestowed upon the beloved RCT as the gold standard of research and these trials, understandably, make up the backbone of SRMA. The idiom of not being able to make a silk purse out of a sow's ear seems particularly apt when looking at the validity of evidence provided by a SRMA when it is tainted with the stench of fabricated or erroneous data. In addition, the common conclusion of the SRMA is often for more research,6 which may further perpetuate the false data paper mills, as well as generate a plethora of yet more inadequate reviews. The questioning of the integrities of research and data are not new,7 nor is it limited to the medical profession. Just as fact-checking has arisen to deal with the claims of fake news, statistical and data checking have come to the fore to assess the veracity of data in research. The retraction of the work linking the Measles, Mumps and Rubella (MMR) vaccine to autism by Andrew Wakefield following investigation by Brian Deer is possibly one of the most infamous cases of fraudulent research.8 Analysis of psychology research data has been put into practice by Michèle Nuijten helping to create Statcheck,9 whilst Nick Brown and James Heathers have created the Granularity-Related Inconsistency of Means (GRIM) test to evaluate published research10 and sterling work is being carried out in anaesthesia by John Carlisle.11 It is with good reason that research within anaesthesia is being scoured so thoroughly: the small patient numbers involved in trials and ability to work alone has led to a veritable rogues gallery of fictitious research. Scott Reuben submitted 21 articles of entirely fictitious data on the subject of peri-operative analgesia that then snowballed, as all the articles had been included in multiple SRMAs and served to form evidence-based medicine.12 Similarly, the prolific fabrications of Joachim Boldt on the subject of colloids spanned 91 articles and altered clinical practice for years, as his underpowered but clinically significant studies found their way into multiple reviews.13 Fujii, the most prolific of the offenders, again published multiple small-scale trials for postoperative nausea and vomiting (PONV). Even when editors and reviewers of anaesthesia journals began to catch whiff of his duplicity, the falsehoods continued in surgical and pharmacology subspecialty publications.11 Ultimately, it was work by Carlisle that led to the exposure of fabricated data by Fujii, with the resultant retraction of 183 articles.14,15 Furthermore, subsequent probing of the work of a collaborator of Fujii - Yuhji Saitoh, resulted in a further 32 retractions,16,17 and this lead Anaesthesia to start screening all RCTs submitted to the journal.18 The prevalence of falsified data from RCTs within the medical sphere and papers with such incredulous data are such that they have now been termed 'zombie' papers.14,19 The prevalence of zombie papers has increased rapidly in recent years with a jump from 3% of analysed papers to 17%, with the vast majority coming from five countries in particular (China, India, Iran, South Korea and Japan). The likelihood is that this is an underestimate of the amount of false data that is being used to guide current medical practice given that the analysis came from manuscripts submitted to a single journal.19 The extent of the problem should not be dismissed, nor should the bias towards suspect countries be disregarded. In analysing articles submitted from 2019 to 2020 to Anaesthesia, Carlisle was able to show that the prevalence of zombie trials was astonishingly high: 100% of trials from Egypt (7/7), 75% from Iran (3/4), 54% from India (7/13), 46% from China (22/48), 40% from Turkey (2/5), 25% from South Korea (5/20) and 18% from Japan.11 If this analysis of individual data sets from a single journal were to be extrapolated to the wider network of trials registered with the WHO international clinical trials registry, then it may be that there as many as 90 000 trials with false data of which up to 50 000 are fabricated.11,18 For this reason that Carlisle began to request the actual data of RCTs for analysis, a practice that has since been taken up by BMJ and PLoS Medicine. This is an area in which there is much debate as to whether the entire data-set is to be considered suspect in deciding between honest error or manipulation. Clinical research is frequently error prone due to its complexity, and as Carlisle notes, we rely on the subjective scrutiny of expert reviewers that question data as unbelievable and meriting further interrogation.11 In addition, it should be noted that data manipulators may not just originate from the five countries already named, it may well transpire that more sophisticated data fabrication may be being practiced in the USA, UK and Europe19 and much like the World Anti-Doping Agency in identifying doping of athletes, we are always playing catch-up. It should be noted that some distinction should be made between simple statistical error and more sophisticated fabrication of data. At present, we often resort to the blunt tool that is retraction or expression of concern until investigation is concluded.20 Retraction does not necessarily distinguish between honest mistakes and fraud. This lack of subtlety or distinction may also be a reason to deter editors such that few articles have been retracted. The increased awareness of erroneous data, zombie papers and paper mills spewing fictitious data created by the work of Carlisle has led to more publications and specialities taking concerns of reviewers and readers more seriously. Recently, Atherosclerosis has launched an investigation into a series of papers published by Cheo-Ke Tang that contain concerning data.21 Similarly with the rush to publish research into COVID-19, there has been a spate of retractions, not least, nor more high profile than the clutch of papers sharing the SurgiSphere dataset.22,23 False research and erroneous claims are nothing new in medicine. It is the nature of astute clinicians, expert reviewers and editors to question and rigorously evaluate data being presented to prevent harm befalling our patients. Given the possible extent of the problem, researchers should be prepared for more requests and be expected to supply actual data as part of the process of research publication, given the inexorable trend to Open Data.

Ähnliche Arbeiten