Improving the peer review process in orthopaedic journals

Introduction

Peer review has evolved over hundreds of years into the current system, which, in most instances, involves some form of blinding within the process. The most common form of peer review in medical and scientific journals is single-blind review, in which the authors are known to the reviewers, but the reviewer’s comments are anonymised before release to the authors. An alternative form is the double-blind review through which the identity of the authors and reviewers are both anonymised from one another. The effectiveness and fairness of blinding continues to be debated by the general medical community, with some journals moving to a more open or unblinded system, claiming that this makes the review process more effective and transparent. Such is the importance of the peer review process that a comprehensive parliamentary review by the Science and Technology Committee has been undertaken in an attempt to better understand the process and identify avenues for improvement.¹ This article addresses the peer review process in orthopaedics and indicates avenues for maximising publication success.

The peer review process in orthopaedic journals

According to the Journal Citation Reports website there are 61 journals worldwide that currently publish on orthopaedic topics.² The potential for effective high-level research in orthopaedics is considerable and the importance of generating good quality evidence to inform practice cannot be overemphasised. Articles published in these journals are relied upon to inform and guide evidence-based practice and while some of these provide higher levels of evidence, the majority remain of insufficient quality to significantly impact day-to-day practice.³ As such there is a risk that an article with low levels of evidence may change practice, particularly if written by a well-known and respected author. One reason for the dearth in good quality publications has been poor levels of investment in fundamental orthopaedic research. Orthopaedic surgeons remain busy clinicians with little or no protected time for structured research. Studies conducted in this environment tend to be poorly structured or retrospective in nature, frequently providing only partial insight into the condition being investigated. Orthopaedic specialists who fail to publish their work often feel discouraged by the experience and this may, at least in part, be due to the opaque process of peer review. Most orthopaedic journals employ a system of single- or double-blind review and this system may be defended on the grounds that there is evidence that blinded reviewers will provide less biased responses⁴ and better quality reviews.⁵ However, the blinded system used by most journals may tempt reviewers to express criticisms in a harsh manner due to the simple fact that they know that they will not be identified. The use of such a system has historically led to the initial rejection of important papers describing the safety and appropriateness of using modern treatment techniques. A classic example of this was the initial rejection of a paper demonstrating that lumpectomy and radiotherapy rather than total mastectomy is a safe option for selected breast cancer patients.⁶

Most journals rely on hundreds of reviewers each year and therefore formal training is hard to achieve. TheBone & Joint Journal holds a specific annual reviewers’ training day in order to ensure that good standards of peer review are upheld, but most journals do not offer this. Scoring systems are seldom used and therefore objective quality control of the reviews is difficult to implement. A lack of training and feedback to reviewers may contribute to differing outcomes of papers submitted to the same journal. One study highlighting this effect involved the resubmission (with only changes to the titles of the submitting institutions) to various psychology journals of 12 manuscripts that had already been published in the same journals. From the 12 resubmitted articles only three were correctly identified as previous submissions, eight articles were rejected and the remaining paper re-accepted.⁷ This study highlighted a serious drawback of the peer review system, with different outcomes arising from resubmission of manuscripts to the same journals. In order to remain current and effective, journals rely on a large pool of reviewers. Annals of the Rheumatic Diseases has a high impact factor with good resources and relied on no less than 777 reviewers in 2010,⁸ while Osteoarthritis and Cartilage, with a slightly lower impact factor but also well resourced, relied on 559 reviewers in the same year.⁹ Given that large numbers of reviewers are required for each journal, questions arise as to how they are selected. Obviously all journals have an editorial board that selects the majority of reviewers who are usually experienced in their field. However, there are concerns that there may be further delegation from this point to other less experienced specialists or even trainees, whose input into the review may not be recognised. While this may not result in an unfair review, the issues surrounding these practices require further scrutiny. A more transparent review system may allow the assistance from delegated trainees and specialists to be acknowledged.

Maximising publication success

A fundamental point to maximise publication success is to ensure that the manuscript complies as closely as possible with the author guidelines and is submitted without basic errors. Experienced reviewers will usually reject a document that has not been constructed with attention to detail, either in its scientific content or in clarity of expression. While some journals may correct spelling mistakes and grammatical errors that are not excessive in the final editing process, others may reject the manuscript outright. It is therefore imperative that the submission is written with a good understanding of the peer review process and should be proof-read by all authors and ideally a clinician or scientist not directly involved in the work. If a manuscript is rejected, it is important to sift through reviewers comments, no matter how negative, to find the constructive elements that may be useful for the next submission. Moreover, learning the attitudes of the reviewers and gaining familiarity with each peer-review system builds the ability to write in a manner that takes these factors in to account. Traditionally, once a study is completed, suitable journals that are likely to accept the manuscript are considered. Some journals, such as Clinical Orthopaedics and RelatedResearch, publish each issue on a particular topic and will call for articles to suit the theme. Awareness of these factors allows researchers to choose an appropriate journal for their submission and improve the likelihood of its publication. Other issues to consider were highlighted in a recent comprehensive survey of the editors of 30 orthopaedic journals.¹⁰ This revealed that 59% used a review proforma, 52% were double-blind and that correct justification of the study conclusions was the most important feature for study design in 80% of the journals. Importantly, 73% responded that an understandable manuscript was the most important factor influencing acceptance.¹⁰

Which is the best peer review process?

Few would question the requirement for peer review. The debate however, surrounds the varying levels of transparency employed by journals. Theoretically, the purpose of anonymising the authors prevents reviewers from allowing any of their own preconceptions of said authors to bias their opinion on the submission. However, reviewers may identify the authors either from the subject matter or writing style. Indeed the authors may intentionally or unavoidably include features in the document that may make their identity obvious. Moreover, a randomised controlled trial assessing the effect on manuscript reviews in a variety of journals of masking versus unmasking author identity showed no difference in review quality.¹¹ The main motive for anonymising reviewers is to avoid a backlash from the authors should the comments be critical, but this may allow reviewers to become particularly negative with their responses and may have the overall effect of dissuading emerging authors from attempting further submissions to either the same journal or elsewhere. The British Medical Journal has moved to an open review system following a randomised controlled trial of an open versus closed review system.¹² The outcome of their trial demonstrated no difference in the quality of reviews from either system. However, there was a higher rate of refusal to review in the open group (35% versus 23%), which just reached statistical significance (p = 0.0499).¹² The principle of open review was given as a reason for refusal in only 19% of cases but at the same time they reported few problems regarding author complaints. The potential for increased reviewer refusals has been one of the main reasons why few journals have pursued an open review system. Reviewers of journals with fewer resources and lower impact factors may consider their role relatively less prestigious and may be concerned that reviewer availability would dwindle if an open review process was instigated for all publications. There is evidence to the contrary however, with a randomised controlled trial conducted by the British Journal of Psychiatry demonstrating that 76% of reviewers agreed to sign their names for open review.¹³ The articles assessed by open review received more courteous comments and were more likely to be recommended for publication, but took longer to complete than unsigned reviews.¹³

Opening up peer review should offer important benefits. The main positive impact would be to sway reviewer criticism from negative comments to constructive feedback, which will encourage authors to revise their manuscripts for resubmission. In order to formalise open peer review, guidelines for reviewers should be more comprehensive and available publicly. This will allow reviewers to follow a pathway that results in more positive comments and enables authors to have a clearer understanding of how the reviewers will assess and score their submission. We suggest that reviewer refusals would decrease over time and that reviewer training may help overcome this. Additionally, opportunities for authors to score the quality of the reviews that they receive using standardised tools should be considered.¹⁴ These changes would allow authors to feel more involved in the process and therefore encourage future submissions. Such tools could be employed by editors to ensure standards are upheld, to measure the effectiveness of their review systems and indicate areas for improvement. It is unrealistic to expect all journals to switch to open review systems quickly. Arrangements for transition to an open system can be made, for example by offering reviewers the option of signing their comments and therefore identifying themselves to the authors. This measure, along with allowing comments to be shared between the fellow reviewers of a manuscript, has received positive feedback in one large American journal.¹⁵Making the reviewer guidelines available to authors is an interim step towards transparency in the review process. The Journal of Bone & Joint Surgery (American volume) displays their guidelines on the manuscript submission section of the website. These guidelines are obviously helpful not just for the reviewers, but enable submitting authors to understand the process their submission will go through. Increasing the opportunities for authors to propose and/or request exclusion of reviewers is already in use by some journals. It confers the benefit of including authors in the review process, although there are concerns that this could facilitate favourable but less objective reviews. A further step would be to offer an author appeals system but most journals however, would struggle to find the resources to support this.

Conclusions

The debate surrounding quality of peer review in medical journals is ongoing. Given the need for good quality research in trauma and orthopaedic surgery, it is important to refine the peer review systems of our journals. Transparency, unmasking and engagement of reviewers and authors in the peer review process may improve the quality of research conducted and submitted to orthopaedic journals. Not all of these steps need to be taken by all journals simultaneously, but incremental changes with close monitoring of author and reviewer reactions may allow refinement of our peer review system for the better.