Abstract
The Bone & Joint Journal has published several consensus statements in recent years, many of which have positively influenced clinical practice and policy.1-13 However, even the most valued consensus statements have limitations, and all ultimately represent Level V evidence. Consensus studies add greatest value where higher-order evidence to aid decision making is ambiguous or lacking. In all settings, care must be taken to critically appraise standards of methodology, with particular attention to potential biases that may influence the conclusions which are drawn.
Cite this article: Bone Joint J 2023;105-B(4):343–346.
Take home message
Consensus statements can add value where there is limited higher-order evidence to guide practice.
Care must be taken to critically appraise standards of consensus methodology, with particular attention to potential biases that may influence conclusions.
The Bone & Joint Journal has introduced guidelines for authors of consensus statements that aim to support them in the design of high-quality studies, while providing the readership with a framework to allow critical appraisal of these studies.
While methods of obtaining consensus, based on the original Delphi approach,14 are now used extensively in orthopaedics, other techniques such as the RAND-UCLA Appropriateness Method (RAM)15 and the Nominal Group Technique,16-18 may be more appropriate in certain settings and should be considered (Table I). The Delphi approach, which was first described in the 1960s, is named after the ancient Greek oracle of Delphi who could predict the future, and aimed to promote anonymity and avoid direct confrontation between experts. This process was thought to be more conducive to independent thought and the gradual formation of a considered opinion when compared with traditional ‘consensus by committee’, in which individuals with strong opinions may disproportionately influence the outcome.14 Since then, a number of modified Delphi methods have been described and all consensus strategies have some methodological overlap.19-21 Factors that may influence the choice of a method of obtaining consensus include the user at whom the statement is aimed, the purpose of the statement (whether to aid diagnosis, management, research, policy, or to inform operative or investigative decision-making), and whether the process will be performed online or in person.
Table I.
Widely used methods of obtaining consensus, including a brief description and relative advantages and disadvantages of each.
| Approach | Outline | Advantages | Disadvantages |
|---|---|---|---|
| Delphi14 | Series of well-defined increasingly closed questionnaires, iterative, and based on repeated survey and the anonymized feedback of an expert group; frequently used incorporating ‘modifications’ | Ideal for larger numbers of participants; anonymized answers minimizes the risk of bias or ‘herd effect’ of prominent individuals disproportionately influencing the consensus; can cover many topics related to one field | Direction and quality of consensus directed by quality of statements and questionnaire; feedback provided can influence individuals to change opinion in order to achieve agreement; no personal contact between experts may limit the scope of discussion |
| RAND-UCLA Appropriateness15 | Combination of the synthesis of formal evidence and expert opinion; core and expert panel; panel provided with a series of clinical scenarios and asked to rate agreement to whether a particular intervention or investigation is suitable | Mandatory use of the synthesis of current literature; minimizes the risk of bias through anonymized ratings and the combination of group discussion and individual ratings; permits group discussion and individual ratings | Many case scenarios can be exhausting and time-consuming; narrow applicability |
| Nominal Group Technique16-18 | Meeting of experts led by an experienced moderator; presents a single question to which answers are suggested as ideas; ‘round-robin’ listing of ideas and serial discussion led by moderator, with independent ranking of ideas by all experts; summary discussion of the highest-ranked solutions | Potential for discussion; all participants have an equal opportunity to present their solution | The technique limits the number of areas in which consensus can be achieved; can be dominated by one or a few prominent individuals |
-
UCLA, University of California, Los Angeles.
Table II.
The Bone & Joint Journal 2023 guidelines for the submission of consensus statements (modified from The British Journal of Sports Medicine Author Guidelines for Consensus Statements).22
| Authorship |
| Please see The Bone & Joint Journal manuscript guidelines regarding authorship. While the number of named authors listed in the header will be limited to 8, the name of the consensus group name can be listed on the header and the list of authors included at the end of the manuscript. These authors will be linked to the article through PubMed. |
| Structure and core requirements |
|
Abstract
Introduction Methods This should include specific paragraphs pertaining to: - Panel and group selection methodology - Evidence review - Consensus process Results or recommendations This should include a specific sentence or paragraph pertaining to - Areas of disagreement Discussion - Implications of research - Limitations Conclusion |
| Detail and considerations of the requirements for each component |
| Abstract |
| The abstract should be no longer than 250 words under the headings Aims, Methods, Results, and Conclusion |
| Introduction |
| A relevant background and rationale for the consensus should be succinctly described in two paragraphs The final paragraph should describe the objective and scope of the consensus statement The following should be clearly stated as part of the aim: - The target patient population for which the consensus should apply (including age, sex, ethnicity if relevant, functional status, precise nature of pathology and its stage of severity) - The target end user of the statement - The purpose of the statement (whether to aid diagnosis, management, research, policy or to inform operative or investigative decision-making) |
| Methods: panel selection |
| This is a critical area which is often not reported or significantly under-reported in consensus statements and requires particular attention The composition and selection of the panel requires careful thought and an explicit rationale and methodology, in the same way as for the consensus process itself A well described methodology for the composition of the Delphi panel ensures balanced perspectives and accounts for any potential bias The panel should attempt to bring distinct perspectives through a diverse but appropriate range of expertise, academic backgrounds, and experience. Authors should describe how the members of the panel were selected and the rationale for this, considering diversity in expertise and ensuring balanced perspectives Are patients, or other stakeholders included who may provide distinct and holistic viewpoints to the consensus process? There should be an explicit description of how the methodology accounted for inclusivity, diversity and equity in panel selection. In particular, the methodology should describe how the selection accounted for inclusion of members from equity-deserving groups (including but not limited to women, ethnic minorities, people with disabilities, LGBTQIA+) Any perceived lack of diversity should be explained and accounted for. How were the views and opinions of different stakeholders on the panel were used to gain consensus. Were all stakeholders granted equal votes for all aspects of the consensus? Was a policy of proportionate or selected votes used based on expertise in different aspects of the consensus? If a steering committee was involved in the recruitment and selection of the panel, it should be noted whether members of the steering committee were also experts on the panel, were facilitators in the consensus process, or were entirely impartial to the process |
| Methods: evidence review |
| A formal review of the evidence and its synthesis is strongly recommended. This should ideally be published in advance of the start of the consensus process. A reference to the published review should be included If the review of the evidence is not published, an outline of the search strategy, search terms and criteria used to select evidence should be presented A robust justification should be presented if a review of the evidence is not provided It should be made clear whether the review of the evidence was used to formulate recommendations in addition to the consensus process, or to provide context to the members of the panel when developing the consensus Describe the reporting guideline which was followed. This journal recommends CREDES (Delphi methods) and AGREE II (clinical guidelines) |
| Methods: consensus process |
| Describe how the group was provided with the evidence synthesis and any feedback given Describe the specific technique which was used to obtain consensus (such as Delphi, RAND-UCLA), including any modifications and a justification for this. Provide a definition of an acceptable level of agreement a priori, and whether this was applied to all aspects of the consensus equally, or whether different levels of agreement were applied to specific statements Was voting or rating anonymized? If not, how was herding bias and/or power imbalances accounted for and mitigated to minimize bias? There should be comment on the nature of meetings, and the opportunities for discussion, including the number of meetings, whether these were “in-person” or “virtual”. Was consensus unanimous? If not, ensure that dissenting opinions or areas of disagreement are specifically recognized When evidence was found to be inconclusive or consensus was not achieved, divergent opinions should be presented in the statement to allow the reader to consider the differing sides of the debate and make their own decisions |
| Results or recommendations |
| Report the number of participants, and the stakeholder groups involved, at each stage of the process Report the level of agreement for each recommendation and statement Report the strengths and limitations of the review of the evidence, including a summary of the level of evidence, its quality, and risk of bias in the studies which were considered. Highlight a specific section dedicated to 'areas of disagreement' or 'dissenting viewpoints'. Here, the contentious points of the process should be discussed, with the proportion of participants that disagreed, and the professional background of these participants, being stated. The reasons for disagreement should also be discussed |
| Discussion |
| Within the Discussion, a specific sub-section entitled 'clinical implications' or 'research implications' should be included: - The potential benefits and harms of the recommendations which are made should be discussed. - Potential facilitators and barriers to the application of these recommendations, and methods to account for barriers, should be discussed. A separate section entitled 'limitations' should be included, with particular attention given to the following points: - Recognize and acknowledge the sources of bias in the methodology, panel and group selection process - Acknowledge and report the limitations of the synthesis of the evidence - Describe limitations in the range of stakeholders considered in the consensus process, including whether the patient’s voice was appropriately included Describe which other stakeholders would be affected by this statement Ensure that there is a timepoint or indication for updating the consensus statement |
| Conclusion |
| Provide a succinct summary of the key messages and clinical implications of the statement |
-
AGREE, Appraisal of Guidelines for REsearch & Evaluation; CREDES, Guidance on Conducting and REporting DElphi Studies.
Regardless of the approach, the methods used to select panel members, review and synthesize the existing evidence, and undertake the consensus process are critical. The selection of participants for a consensus panel requires careful consideration, and explicit rationale and methodology, to ensure balanced perspectives and to account for biases. Not all panels need to be made up of ‘ultra-experts’, and the value of having diverse perspectives should be considered. The inclusion of surgeons in academic fields as well as those in private practice, those with less experience, members of the wider clinical team, patients, and surgeons from countries with fewer resources may all provide valuable perspectives and allow the development of a consensus that is widely applicable. As an example, it may be easy to recommend diagnosis and management strategies in tertiary referral centres in resource-rich countries, but it can be much more difficult to strike a balance on how to conduct effective care where clinicians have less experience or fewer resources. Consensus panels should also involve members from equity-deserving groups, including but not limited to women, people of colour, people with disabilities, and members of the LGBTQIA+ community. There should be an explicit description of how the methodology accounted for the expertise, inclusivity, diversity, and equity when selecting the members of the panel. While increasing the heterogeneity of the panel may lead to a wider applicability of the statement, it can lead to difficulties, including different degrees of agreement and dissenting opinions. As long as the dissenting opinions and areas of disagreement are recognized, the impact of the process is unlikely to be weakened. Clearly a balance needs to be struck between heterogeneity to ensure equity, diversity, and inclusion, in order to maintain an appropriate level of knowledge.
Consensus statements are most valuable in areas where a higher-order evidence is ambiguous or lacking. Nonetheless, the process should be seen in context and informed by a review of the evidence. The methods of reviewing the evidence and its synthesis should be clearly set out, including the search strategy and criteria used to select the evidence that is deemed to be relevant, and should ideally be published in advance of the start of the consensus process, for example as a systematic or scoping review. How the synthesis of evidence is presented to the members of the panel must also be clearly described. It should be made clear whether the review was used to formulate recomendations in addition to the consensus process, or simply to provide context to the members when developing the consensus. It has been reported that when reviews of the literature are provided to panellists, the evidence presented may be used not only in the initial discussion, but throughout the consensus process,23 and that approaches which are based on well-conducted synthesis of the evidence result in guidelines of higher quality.24
It is self-evident that the process used to reach consensus may influence the makeup and content of the final statement or guidelines. Any deviations from the expected methodology should be described and justified and the criteria for achieving consensus set before the start of the process. Unfortunately, existing consensus statements are frequently limited by the inadequate reporting of factors that may critically influence the conclusions and recommendations. Specific reporting guidelines such as CREDES (for Delphi methods)25 or AGREE II (for clinical guidelines)26 can guide authors who are designing studies and facilitate the interpretation of results. In addition, a number of specialist journals, including The British Journal of Sports Medicine (BJSM), have produced their own guidelines for authors.22 We have recently introduced guidelines for authors of consensus statements for The Bone & Joint Journal, building on those produced by the BJSM, that outline core requirements for consensus statements, position statements, and clinical guidelines submitted to our journal, recognizing that that each statement will have unique features (Table II).
The generation of ‘consensus statements’ are currently in vogue in orthopaedics. While they can play a valuable role in guiding clinicians where a higher order of evidence is lacking, it must be recognized that many have considerable flaws and the methodology should be critically reviewed for each study. The Bone & Joint Journal guidelines for authors of consensus statements aim to support them in the design of high-quality studies, while providing our readership with a framework to allow critical appraisal of these studies.
References
1. Perry DC , Wright JG , Cooke S , et al. A consensus exercise identifying priorities for research into clinical effectiveness among children’s orthopaedic surgeons in the United Kingdom . Bone Joint J . 2018 ; 100-B ( 5 ): 680 – 684 . Google Scholar
2. Kalson NS , Borthwick LA , Mann DA , et al. International consensus on the definition and classification of fibrosis of the knee joint . Bone Joint J . 2016 ; 98-B ( 11 ): 1479 – 1488 . Crossref PubMed Google Scholar
3. Morgan-Jones R , Haddad FS . Is this the era of consensus? Bone Joint J . 2013 ; 95-B ( 11 ): 1441 – 1442 . Google Scholar
4. Gelfer Y , Davis N , Blanco J , et al. Attaining a British consensus on managing idiopathic congenital talipes equinovarus up to walking age . Bone Joint J . 2022 ; 104-B ( 6 ): 758 – 764 . Crossref PubMed Google Scholar
5. Johansen A , Ojeda-Thies C , Poacher AT , et al. Developing a minimum common dataset for hip fracture audit to help countries set up national audits that can support international comparisons . Bone Joint J . 2022 ; 104-B ( 6 ): 721 – 728 . Crossref PubMed Google Scholar
6. Marson BA , Manning JC , James M , et al. Development of the CORE-Kids CORE set of outcome domains for studies of childhood limb fractures . Bone Joint J . 2021 ; 103-B ( 12 ): 1821 – 1830 . Crossref PubMed Google Scholar
7. McNally M , Sousa R , Wouthuyzen-Bakker M , et al. The EBJIS definition of periprosthetic joint infection . Bone Joint J . 2021 ; 103-B ( 1 ): 18 – 25 . Crossref PubMed Google Scholar
8. Mathews JA , Kalson NS , Tarrant PM , Toms AD , Revision Knee Replacement Priority Setting Partnership steering group . Top ten research priorities for problematic knee arthroplasty . Bone Joint J . 2020 ; 102-B ( 9 ): 1176 – 1182 . Crossref PubMed Google Scholar
9. Leo DG , Jones H , Murphy R , et al. The outcomes of Perthes’ disease . Bone Joint J . 2020 ; 102-B ( 5 ): 611 – 617 . Google Scholar
10. Preston N , McHugh GA , Hensor EMA , et al. Developing a standardized approach to virtual clinic follow-up of hip and knee arthroplasty . Bone Joint J . 2019 ; 101-B ( 8 ): 951 – 959 . Crossref PubMed Google Scholar
11. Eftekhary N , Shimmin A , Lazennec JY , et al. A systematic approach to the hip-spine relationship and its applications to total hip arthroplasty . Bone Joint J . 2019 ; 101-B ( 7 ): 808 – 816 . Crossref PubMed Google Scholar
12. Abram SGF , Beard DJ , Price AJ , BASK Meniscal Working Group . Arthroscopic meniscal surgery: a national society treatment guideline and consensus statement . Bone Joint J . 2019 ; 101-B ( 6 ): 652 – 659 . Crossref PubMed Google Scholar
13. Haddad FS , Oussedik S , Meek RMD , Konan S , Stockley I , Gant V . Orthopaedic infection: is consensus the answer . Bone Joint J . 2018 ; 100-B ( 11 ): 1405 – 1406 . Crossref PubMed Google Scholar
14. Dalkey N , Helmer O . An experimental application of the DELPHI method to the use of experts . Management Science . 1963 ; 9 ( 3 ): 458 – 467 . Google Scholar
15. Fitch K , Bernstein SJ , Aguilar MD , et al. The RAND/UCLA appropriateness method user’s manual . Santa Monica, California, USA : RAND Corp , 2001 . Google Scholar
16. Van de Ven AH , Delbecq AL . The nominal group as a research instrument for exploratory health studies . Am J Public Health . 1972 ; 62 ( 3 ): 337 – 342 . Crossref PubMed Google Scholar
17. Murphy MK , Black NA , Lamping DL , et al. Consensus development methods, and their use in clinical guideline development . Health Technol Assess . 1998 ; 2 ( 3 ): 1 – 88 . PubMed Google Scholar
18. Gallagher M , Hares T , Spencer J , Bradshaw C , Webb I . The nominal group technique: a research tool for general practice? Fam Pract . 1993 ; 10 ( 1 ): 76 – 81 . Crossref PubMed Google Scholar
19. Jünger S , Payne S , Brearley S , Ploenes V , Radbruch L . Consensus building in palliative care: a Europe-wide delphi study on common understandings and conceptual differences . J Pain Symptom Manage . 2012 ; 44 ( 2 ): 192 – 205 . Crossref PubMed Google Scholar
20. Diamond IR , Grant RC , Feldman BM , et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies . J Clin Epidemiol . 2014 ; 67 ( 4 ): 401 – 409 . Crossref PubMed Google Scholar
21. Singer M , Deutschman CS , Seymour CW , et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) . JAMA . 2016 ; 315 ( 8 ): 801 – 810 . Crossref PubMed Google Scholar
22. Blazey P , Crossley KM , Ardern CL , van Middelkoop M , Scott A , Khan KM . It is time for consensus on “consensus statements.” Br J Sports Med . 2022 ; 56 ( 6 ): 306 – 307 . Google Scholar
23. Jacoby I . Evidence and consensus . JAMA . 1988 ; 259 ( 20 ): 3039 . Google Scholar
24. Cruse H , Winiarek M , Marshburn J , Clark O , Djulbegovic B . Quality and methods of developing practice guidelines . BMC Health Serv Res . 2002 ; 2 ( 1 ): 1 . Crossref PubMed Google Scholar
25. Jünger S , Payne SA , Brine J , Radbruch L , Brearley SG . Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review . Palliat Med . 2017 ; 31 ( 8 ): 684 – 706 . Crossref PubMed Google Scholar
26. Brouwers MC , Kho ME , Browman GP , et al. AGREE II: advancing guideline development, reporting and evaluation in health care . CMAJ . 2010 ; 182 ( 18 ): E839 – 42 . Crossref PubMed Google Scholar
Author contributions
I. R. Murray: Writing – original draft, Writing – review & editing.
N. S. Makaram: Writing – original draft, Writing – review & editing.
R. F. LaPrade: Writing – review & editing.
F. S. Haddad: Writing – review & editing.
Funding statement
The authors received no financial or material support for the research, authorship, and/or publication of this article.
ICMJE COI statement
I. R. Murray reports consulting fees from Stryker and Arthrex, unrelated to this study. R. F. LaPrade reports research grants, royalties and licenses, patents planned, issued, or pending, and consulting fees from Ossur and Smith & Nephew, as well as royalties and licenses from Arthrex and Elsevier, all of which are unrelated to this study. F. S. Haddad reports board membership of The Bone & Joint Journal and the Annals of the Royal College of Surgeons; consultancy for Smith & Nephew, Corin, MatOrtho, and Stryker; payment for lectures, including service on speakers’ bureaus, for Smith & Nephew and Stryker; and royalties paid by Smith & Nephew, MatOrtho, Corin, and Stryker, all of which are also unrelated to this article.
Follow I. R. Murray @MurraySportOrth
Follow N. S. Makaram @MakOrtho
Follow R. F. LaPrade @thekneedoc
Follow F. S. Haddad @bjjeditor
This article was primary edited by J. Scott.
