Retractions in orthopaedic research

Search strategy

On September 9, 2015, we searched three online databases (EMBASE, MEDLINE and Cochrane), as well as the website Retraction Watch (www.retractionwatch.com), for retractions in orthopaedic research. Within the databases we used the following key words in our search: “orthopaedic”; “bone”; “musculoskeletal”; “cartilage”; “sports medicine”; “joints”; “spine”; “rheumatology”; “rehabilitation”; and “peri-operative”. Our search was limited to studies published between 1995 and 2015. We used the Boolean operator “or” between each orthopaedic search term. In MEDLINE, we selected the ‘retraction’ filter to limit our search strategy to retracted articles. In EMBASE, which does not have this filter, we also used the following keywords to indicate that we were narrowing our search to retracted studies: “retraction of study”; “retracted study”; “notice of retraction”; “retraction notice”; “retracted publication”; and “retraction of publication”, connected by the Boolean operator “or”. We then also used the Boolean operator “and” between the combination of orthopaedic search terms and the combination of the retraction search terms.

Study screening

Two reviewers (JY, AM) independently screened titles, abstracts, and full texts of the retrieved studies. We used the following inclusion criteria: studies involving any musculoskeletal science or musculoskeletal conditions; all levels of evidence; studies on humans and animals, including cadaver studies; and basic science studies. We used the following exclusion criteria: topics unrelated to orthopaedic research; duplicate papers; full article unavailable; and publication not available in English. Two reviewers noted any discrepancies during this search and discussed their findings to settle disagreements with respect to inclusion. When unresolved conflicts arose, a third senior reviewer (NE) was brought in to mediate discussion until a consensus was reached.

Data extraction

Two reviewers (AM, L-PB) independently extracted relevant study data from the final pool of included articles and recorded the data in an electronic database. The first 10% of studies underwent data extraction in duplicate in order to calibrate agreement between the two reviewers. The reviewers then divided the remaining studies equally to undergo data extraction. Extracted data included author names, year of publication, year of retraction, country of origin, journal name, journal impact factor, number of citations, reasons for retraction, availability of retracted papers online, retraction watermark, study design, and musculoskeletal subspecialty. The Web of Science database was used to determine journal impact factor.²⁰

Statistical analysis

We reported discrete variables as counts or proportions, normally distributed continuous variables as means with standard deviations (sd), and skewed continuous variables as medians with interquartile ranges (IQRs). In order to assess inter-reviewer agreement, we calculated a weighted κ (kappa) for the article screening at the title-abstract screening stage. Agreement was categorised as follows: κ > 0.61 to indicate substantial agreement; 0.21 < κ < 0.60 to indicate moderate agreement; and κ < 0.20 to indicate slight agreement.²¹ A 95% interval was calculated for the kappa statistic. All analyses were performed using Microsoft Excel (Microsoft, Santa Rosa, California).

Retractions identified and dates of retraction

We identified 1192 potentially eligible studies from our search strategy, of which we excluded 1082 (Fig. 1), for a total of 110 studies for data extraction. The kappa statistic for inter-reviewer agreement for study inclusion was 0.95 (95% CI 0.93 to 0.98) which indicates substantial agreement. In total, 25 papers were retracted in the first ten years (1995 to 2004), while 85 papers were retracted between the years 2005 and 2015 (Fig. 2). The mean time from article publication to retraction was 2.8 years (sd 3.3).

Fig. 1

Search strategy process.

Fig. 2

Number of orthopaedic retraction notifications issued per year in five-year intervals between 1995 and 2015.

Journal and author characteristics

The retracted studies originated from institutions based in 22 different countries. The 110 retracted papers were found across 67 different journals. Two of these journals are no longer in print. The highest 2014 impact factor of a journal with a retracted study was 13.26. The most highly cited retracted publication has been cited 165 times,²² however, the median number of times a retracted paper has been cited was 6 (IQR 2 to 19).

We identified several authors who were named on multiple retractions. One author is listed on 17 retractions, and another on 11 retractions. In addition, one author is listed on four retractions, and another on three retractions. There are two authors with two retracted papers. In total, 39 of the total 110 retracted studies involved these six authors.

Reasons for retraction

The most common reasons for retraction were fraudulent data (29), plagiarism (25), duplicate publication (20), and data errors (14) (Table I). A total of 28 studies were retracted for other reasons such as authorship misattribution, falsified peer reviewer details, or lack of ethics approval. Two studies were removed on the basis of committing both plagiarism and scientific fraud. One study misreported ethics approval. One study was retracted for both plagiarism and duplication. Occasionally, retraction notices were vague, with phrases such as “…the article contains several examples of incorrect presentation of scientific data…”,²³ or “…account of concerns over the validity of the data reported therein based on further consideration by the journal’s Editor…”²⁴ obscuring whether an article was retracted due to error or an act of misconduct.

Method of retraction notification

Nine of the studies were unavailable online from the publisher’s website (indicating that they had been removed), and 34 were available online without any indication that they had been retracted. Among the remaining 67 studies, 37 were labelled as retracted with translucent watermarks, 22 with opaque watermarks, five with headers, footers, edited titles, and/or forewords, and three with both a watermark (one opaque, two translucent) and a header (Table II).

Limitations

The main limitation of our study is that, without standard search methodologies for identifying retractions, some retractions may have been missed in our investigation. The nature of our electronic searches accounted only for papers that were actually retracted, and ignores any flawed studies that have not been retracted. Additionally, there is a time delay between when an article is formally retracted and when the retraction is updated on a searchable database. One study found this latency period to be up to six months²⁵ and indicates that some retractions in our search window may have been missed. The relatively small number of relevant retractions identified also limits our investigation. However, we examined three large and commonly used databases in our search strategy, as well as a popular website well known for leading the documentation of retracted articles.⁸

A final limitation of this study is that we were only able to assess the reason for retraction based on the information available from the retraction notice. Other more detailed sources unavailable to us include archived reports from the Office of Research Integrity and direct interviews with the pertinent institutions.^8,12 Nonetheless, the retraction notices used were published according to COPE guidelines.⁹

Relation to previous work

Our findings are consistent with a number of other studies outside the orthopaedic literature that have identified an increasing incidence of retracted studies in the general scientific literature.^12-14 Overall, we found that 82.7% of orthopaedic retractions were due to academic misconduct, which included plagiarism, fraudulent data, duplicate publishing, failure to obtain ethics approval, peer review process manipulation, and copyright violation. This compares to 77.4% of retractions in the overall scientific literature determined to be due to misconduct, defined by Fang et al¹² as fraud or suspected fraud, duplicate publication, and plagiarism. Van Noorden¹⁵ found 44% of studies retracted due to misconduct, defined as fabrication or falsification, self-plagiarism, and plagiarism. Redman et al¹⁷ stated that only 17% of studies were retracted due to research misconduct, but had identified plagiarism as a separate entity which accounted for 17% of retractions. “Inability to reproduce” was the reason for another 20% of retractions and the authors speculated that these retracted works were actually due to misconduct. Therefore, some discrepancy in the literature may be attributed to the varying definitions or categorisations of research misconduct.²⁶

Several earlier studies on retractions found that the majority of retractions were due to published errors.^10,11 However, more recent studies have applied greater scrutiny and reclassified many of the studies labelled ‘retracted due to error’ as ‘misconduct’.^12-17 Van Noorden et al¹⁵ showed that only 28% of retractions could be attributed to honest mistakes, while Fang et al¹² found that only 21.3% of retractions are due to error. The current study found that error was the cause for retraction in only 12.7% of the orthopaedic-related studies.

Implications

In total, 85 of the 110 identified orthopaedic studies (77.3%) were retracted in the most recent ten years of our search. It is possible that this increase in retractions is due to increased awareness and availability of retraction notices online.^15,18,27,28 In addition, advancement in detection software has enhanced methods of screening for plagiarism.¹⁵ Improved information and communications technology allows for easier reporting and widespread online communication of errors and misconduct. The internet has allowed the post-publication peer review process to develop into a more open and transparent format. The development and successful growth of sites such as Retraction Watch and PubPeer²⁹ has increased the extent to which a published paper can be scrutinised. The format of PubPeer, which provides anonymity to commenters, can potentially provide protection for whistleblowers pointing out errors or misconduct in a paper.^29-31 As more vigilance is applied to publication standards, we can expect the number of retractions reported in the future to increase until further measures are put in place to discourage research misconduct. These measures may include a collective database for retractions, greater transparency in the reasons for retractions, tighter and carefully constructed research practice guidelines in conjunction with more stringent and enforced reporting guidelines, and greater resources and training dedicated towards replication and reproduction of studies.^4,13,15,27

The method by which the journals handled the original retracted article varied considerably. Most of the journals included a watermark across the text that presented in some form the word “retraction”, however, these watermarks could either be translucent so that a reader could still read the original text, or opaque which obscured sections of the text. Moreover, there were many original articles that gave no indication that they were retracted nor notice that they contained errors or evidence of misconduct. This lack of consistency regarding the handling of original articles can have serious consequences as it can allow misinformation to perpetuate.³ If these retracted articles continue to be cited by other authors or transition into clinical practice, it is possible that their erroneous conclusions could lead to detrimental medical care or misinformed public knowledge.¹⁷

The lack of consistency in issuing retraction notices and in the handling of retracted articles may reflect the fact that academic journals and editors, as a community, have not yet developed retraction standards. COPE provides guidelines for handling research misconduct, but the guidelines do not specify how a retraction is to be issued.^9,28,32 Greater awareness of, and adherence to, the guidelines set by COPE would help improve overall standards of quality among retractions.⁹ In 2006, Sox and Rennie outlined, as designated by the Office of Research Integrity, the responsibilities of academic institutions, journal editors, and authors on taking action against fraudulent articles.⁵

The author who was involved in 17 retracted articles is trained as an anaesthesiologist and all of the associated retracted publications were published in anaesthesia journals.³³ We included these retractions as they studied anaesthetic techniques used for orthopaedic procedures and, therefore, would apply to the orthopaedic patient population.³⁴ The majority of the authors who had multiple retractions had their publications retracted due to misconduct, which is consistent with the findings by Fang et al,¹² whose study found that these cases are among the most notorious examples of misconduct in the field of research ethics.¹² Similarly, Steen et al¹⁴ found that “repeat offenders” were the first named author in the majority of fraudulent papers. Retractions from “repeat offenders” skews the data for both specific journal and year, as detailed investigations often expose a large clump of such cases within a narrow time frame.^35,36

Future directions

We noted in our findings that there was inconsistency in how journals and editors release and phrase retraction notices. Additionally, we came across instances where the phrasing in the notices was vague and these findings are consistent with those of previous studies.^6,16,18 It is difficult to assess why these vague statements are made and perhaps journals are hesitant to publish statements that could tarnish an author’s reputation.⁶ However, the editors of Retraction Watch have found that authors who openly self-report honest mistakes (as opposed to attempting to obfuscate fraudulent actions), do not suffer negative consequences in their research careers.¹⁸ It would be worthwhile as a future study to survey orthopaedic journals that have issued retraction notices to understand the processes behind the retractions and the reasons for the choice of words used in the retraction notices, in particular with respect to the effect of the retraction on the reputation of the journal itself and the authors involved.

The pace of retractions in the musculoskeletal literature is increasing, with the majority of retractions occurring due to academic misconduct and fraud. Orthopaedic retractions originate from numerous journals and countries, indicating that misconduct issues are widespread. The results of this study raise awareness of the need to address academic integrity when training the next generation of orthopaedic investigators.