Abstract
Aims
To develop a multidisciplinary health research agenda (HRA) utilizing expertise from various disciplines to identify and prioritize evidence uncertainties in orthopaedics, thereby reducing research waste.
Methods
We employed a novel, structured framework to develop a HRA. We started by systematically collecting all evidence uncertainties from stakeholders with an interest in orthopaedic care, categorizing them into 13 sub-themes defined by the Dutch Orthopaedic Association (NOV). Subsequently, a modified two-phased Delphi study (two rounds per phase), adhering to the Conducting and REporting DElphi Studies (CREDES) guideline, was conducted. In Phase 1, board members assessed the collected evidence uncertainties on a three-point Likert scale to confirm knowledge gaps. In Phase 2, diverse stakeholders, including orthopaedic surgeons, rated the confirmed knowledge gaps on a seven-point Likert scale. Panel members rated one self-selected sub-theme and two randomly assigned sub-themes. The results from Phase 2 were ranked based on the overall average score for each uncertainty. Finally, a focus group discussion with patient associations’ representatives identified their top-ranked uncertainty from a predefined consensus process, leading to the final HRA. An advisory board, the Federation of Medical Specialists, and the NOV research coordinator oversaw the process.
Results
Of the 687 collected evidence uncertainties, 160 (zero to 33 per theme) were confirmed by 41 panel members (three to five per theme). In Phase 2, 124 panel members prioritized 41 evidence uncertainties (zero to five per theme). The focus group members identified 12 key evidence uncertainties leading to the final HRA. The remaining 29 evidence uncertainties will be addressed after research on the HRA’s prioritized evidence uncertainty is completed.
Conclusion
Our framework resulted in a multidisciplinary HRA, enabling an inclusive approach to consensus-building among healthcare professionals and patients on future research priorities within orthopaedic care. We anticipate this innovative framework will enhance inclusivity and transparency, leading to broader acceptance and optimized resource allocation, ultimately reducing research waste.
Cite this article: Bone Jt Open 2025;6(2):206–214.
Take home message
We expect that this new framework will enable a more informed, transparent, and inclusive decision-making process in prioritizing collected evidence uncertainties across various themes within a medical discipline.
Introduction
To avoid research waste, medical associations are becoming more active in establishing consensus on research priorities (evidence uncertainties or research gaps) by developing health research agendas (HRAs).1-6 A HRA helps to address the perceived evidence uncertainties by clinicians, patients, researchers, and other stakeholders. It is a valuable instrument for improving patient outcomes by stimulating evidence-based clinical practice. It enables clinicians, patients, carers, policymakers, and funding agencies to collaborate to identify and prioritize evidence uncertainties, and guides consensus on areas where increased research effort, collaboration, coordination, and investment would benefit society.7-9 The concept of HRAs first emerged internationally around the year 2000.10-12 The American Academy of Orthopaedic Surgeons (AAOS) examined current and future research needs of the musculoskeletal system during 1980 to set the first HRA,13 and launched the second HRA in 2003,14 followed by an update in 2014. 15 The recommendations and conclusions were set by expert panel members (orthopaedic surgeons and/or PhD-level researchers) and an internal committee of the AAOS. The Dutch Orthopaedic Association (NOV) took the initiative to set national research priorities in orthopaedics by creating a national HRA in 2015. 16 This agenda was subsequently updated in 2019.17 Both HRAs were set following a national protocol developed by the Dutch Federation of Medical Specialists (FMS). The FMS assisted the Advisory Board in identifying evidence uncertainties and organizing a prioritization meeting, including meetings with stakeholders with interest in orthopaedic care. The Advisory Board formulated a top-ten list of evidence uncertainties, which was approved by the NOV medical specialists board.17 In 2023, our research group reviewed the methods used by the FMS,18 and analyzed recent HRA development processes.19,20 In that study, we highlighted the importance of a transparent prioritization process for consensus building, increased inclusivity, and participation of all stakeholders with an interest in orthopaedic care. The results highlighted the need for a multidisciplinary HRA to address the field’s diverse perspectives.
This study aimed to establish a HRA by using an innovative framework for consensus-building on research priorities. This multistaged framework involved systematically identifying evidence uncertainties, followed by a two-phased (including two rounds) modified Delphi process to confirm and assess these uncertainties by rating them. Finally, a focus group gathered input from patient associations’ representatives. This article presents the development process and summarizes the key findings.
Methods
This study received an exemption from the non-Medical Research Ethics Committee; it was concluded that the Medical Research Involving Human Subjects Act did not apply to this study. All panel members gave informed consent before starting the Delphi questionnaire.
Design
We created a new framework by systematically collecting all evidence uncertainties, followed by a modified two-stage Delphi (including two rounds per stage) and a focus group (Supplementary Material).
We used the Conducting and Reporting of Delphi Studies (CREDES)21 guideline and the process template published by Belton et al22 to modify the Delphi study and ensure it is closely aligned with the study’s primary objective.23-25
The objective of Phase 1 was to assess whether the collected evidence uncertainties confirmed knowledge gaps within orthopaedic and affiliated care. The objective of Phase 2 was to rate the selected evidence uncertainties in Phase 1. Each phase consisted of two Delphi rounds. In the first round, panel members rated the evidence uncertainties. In the second round, panel members received personal and group feedback, allowing them to change their opinions based on these results.
Subsequently, a focus group of four patient group representatives discussed the result of the second Delphi study, resulting in the final HRA.
Advisory board
The first step of our framework involved establishing an advisory board to guide the process. This advisory board consisted of 15 members, with as chairman a professor specialized in healthcare evaluation and members with various backgrounds (five orthopaedic surgeons, one researcher, one member of the Patient Federation of the Netherlands, two physiotherapists, one rheumatologist, one sports medicine physician, one occupational health/company physician, one trauma surgeon, one nurse practitioner orthopaedic surgery, and one professor in clinical physiotherapy). The NOV’s research coordinator and two Federation of Medical Specialists advisors assisted the board. Stakeholders were invited by purposive sampling, a request to all NOV working groups, and the distribution of a newsletter of the NOV. One of the board members represented the national, interdisciplinary national patient association. The advisory board members’ backgrounds and work settings were documented.
Collection of evidence uncertainties
As a next step in our framework, the NOV’s research coordinator, assisted by the Federation of Medical Specialists, collected all existing and experienced evidence uncertainties in orthopaedics and related topics in daily practice, including underlying motivations. They reviewed the literature, existing guidelines, and trial registries. All interested stakeholders were approached via email and a newsletter distributed by the NOV. Duplicate evidence uncertainties needed to be clarified or reformulated. A thorough evaluation of the identified evidence uncertainties was conducted by examining existing guidelines and ongoing research. Finally, the advisory board members assessed whether the collected evidence uncertainties focused on human movement, healthcare evaluation, innovation, or organization. These findings were categorized into 13 sub-themes following the corresponding NOV working groups (Hip; Knee; Foot/ankle; Hand/wrist; Shoulder/elbow; Spine; Children’s orthopaedics; Trauma; Sports orthopaedics; Bone and soft-tissue tumours; Artificial intelligence; Impairment and disability medicine; Orthopaedic infections).
Panel members of the Delphi Study
Our modified Delphi study consisted of two phases, with one panel each to confirm and assess the collected evidence uncertainties.
Phase 1 (round 1 and 2)
Despite the recommendation for expert panel heterogeneity in a Delphi study,26 we opted for a more homogenous group with expert knowledge in the field in Phase 1 to align with its primary objective (consensus on evidence uncertainties). “Expert” refers to relevant knowledge and expertise in the field.27-29 Panel members were contacted by current or past board members in the NOV working groups: experts in orthopaedic care and experience in research. All respondents who expressed their willingness to participate in the prioritization process were included in the study. To increase response rates, panel members could participate in questionnaires on multiple themes.
Phase 2 (round 1 and 2)
For the second phase, to achieve a broadly supported HRA, we contacted panel members from several disciplinary backgrounds with interest in the final HRA (members’ orthopaedic associations, patient associations, orthopaedic residents, rheumatologists, sports physicians, occupational health/company physician, general practitioners, radiologists, occupational physicians, physiotherapists, researchers, nurse practitioners, nurses, podiatrists, infectiologists, health insurers). Recruitment involved sending emails to potential panel members with pre-announcements, requesting participation, and indicating their preferred theme for expressing opinions. All panel members who responded affirmatively and were willing to participate were included.
Focus group
For the focus group, we invited representatives of several patient associations relevant to the themes in the final HRA. The process was led by the advisory board’s chair, assisted by one advisory board member and the Federation of Medical Specialists’ advisor.
Sample size
We calculated the sample size recommended for each Delphi panel using the information from published literature.30-33 The sample size depended on the aim per phase. In Phase 1, we aimed at three panel members per NOV theme; in Phase 2, we aimed at 13. We aimed at four to eight representatives from the relevant patient associations for the focus group.
Prioritization of evidence uncertainties
Questionnaires
To gather quantitative feedback, we used structured, close-ended questionnaires in both phases, with personal and group feedback in between rounds, using percentages. We did not allow qualitative feedback to ensure clarity, having already collected an extensive list of evidence uncertainties with explanations/motivations beforehand. The questionnaire’s layout, content, and user-friendliness were pilot-tested by advisory board members.
Panel members
We contacted panel members in an email introducing the study, specifying the return deadline, and attaching a link with a short introduction to the questionnaire. The purpose, anticipated duration, and contact details of the research team were included in the introduction. We explained the Delphi study, including the terminology, instructions, and an example for questionnaire completion. All submissions were processed via a central coordinator (ARI). We allowed a maximum of two weeks for panel members to return their questionnaires and sent email reminders after seven and ten days.
Phase 1 (round 1 and 2)
In the first phase, panel members were asked to express their opinion on whether the selected evidence uncertainties confirmed knowledge gaps within orthopaedic surgery using a three-point Likert scale ranging from “disagree” to “agree”. The following panel members’ characteristics were recorded: background, occupation, and work setting. We incorporated non-responders of the first round in the subsequent round to ensure diverse opinions and minimize non-random loss of perspectives and alignment of opinions. This approach resulted in a comprehensive representation of the entire panel’s viewpoints.34
Phase 2 (round 1 and 2)
In Phase 2, panel members gave their opinion on one theme of their choice, plus two randomly assigned additional themes. Randomization was performed after panel members’ informed consent was obtained and their preferred theme was chosen. Welphi, the internet-based online Delphi tool, used a dynamic, adaptive allocation. The algorithm used a dynamic method to calculate allocation probabilities: the allocation probability to each group was not fixed (e.g. 0.5), but was recalculated for every participant based on panel members already allocated. This method protects the trial process by ensuring the allocation ratio is consistently maintained at a 1:1 ratio within each stratification variable and throughout the trial.35
Each participant’s opinion regarding each evidence uncertainty was measured on a seven-point Likert scale, ranging from “strongly disagree” to “strongly agree”. The following panel members’ characteristics were recorded: background, occupation, and work setting.
In both phases, individual scores remained anonymous to other respondents. All collected individual data were treated confidentially and, for transparency, only accessible to the central coordinator.
Consensus
Before starting the study in October 2022, the advisory board pre-approved the agreement level, which was less strict in Phase 1 than in Phase 2. Throughout the process, we quantitatively measured the agreement level. In Phase 1, we set a consensus threshold of 40% without restrictions on the selected items or minimum deviation from the central tendency. We selected the 40% consensus threshold to filter the extensive evidence uncertainties efficiently, ensuring that only confirmed knowledge gaps within research progressed to prioritization. This threshold balanced inclusivity and focus, acknowledging the limited sample size of panel members. After Phase 1, to enhance comprehension in Phase 2, the advisory board reformulated evidence uncertainties and their underlying reasons for concise and effective communication. To be part of the HRA, an evidence uncertainty (up to five per theme) had to have a median score of at least five on the seven-point Likert scale or higher based on central tendency and dispersion. The top five ranked evidence uncertainties of each theme were selected for discussion by the focus group. In previous HRAs, only the highest ranked evidence uncertainty was selected. In this particular framework, we chose to broaden our perspective.
Focus group
After prioritizing the five highest ranked evidence uncertainties per theme, patient associations were invited to share their input and opinions during a focus group discussion.
The focus group was led by the chair of the advisory board, assisted by an advisory board member and an advisor of the Federation of Medical Specialists. The purpose of the focus group was briefly explained to the panel members. The results of the Delphi study were discussed per theme, including a brief explanation of the evidence uncertainties and how consensus was reached in the Delphi study.
Final health research agenda
Before publishing the final HRA, the advisory board meticulously examined the findings of the focus group.
Analysis strategy
The XML file containing response data was obtained from Welphi and processed using a Python code v. 3.11.3 (Python Software Foundation, USA) to convert it into an SPSS file format. It was analyzed using SPSS Statistics v. 25 (IBM, USA). We used descriptive statistical analysis, measurement of central tendency (median, mean), and level of dispersion (variance, IQR, SD, and range). We also calculated the percentage of respondents rating the median or higher.
Results
Collection of evidence uncertainties
Of the 687 collected evidence uncertainties (Table I), 422 were excluded based on duplicates. The remaining list (n = 265) of evidence uncertainties was correctly classified among the NOV themes. The theme “impairment and disability medicine” had no remaining evidence uncertainties, which led to 265 evidence uncertainties categorized according to 12 corresponding NOV themes.
Table I.
Flow collected, selected and prioritized evidence uncertainties for the third health research agenda of the Dutch Orthopaedic Association.
| Subspecialty | Prioritization | |||||
|---|---|---|---|---|---|---|
| Collection (n = 687) * | Selection (n = 265) † | Phase 1 Delphi (n = 160) ‡ | Selection (n = 119) § | Phase 2 Delphi (n = 41) ¶ | Focus group (n = 12) ** | |
| Artificial intelligence | 6 | 6 | 6 | 4 | 2 | 1 |
| Bone and soft-tissue tumours | 8 | 3 | 2 | 1 | 0 | 0 |
| Impairment and disability | 2 | 0 | 0 | 0 | 0 | 0 |
| Foot and ankle | 59 | 19 | 7 | 7 | 3 | 1 |
| Hand and wrist | 26 | 14 | 6 | 5 | 3 | 1 |
| Hip | 38 | 27 | 14 | 10 | 5 | 1 |
| Knee | 53 | 30 | 14 | 11 | 5 | 1 |
| Orthopaedic infections | 18 | 10 | 5 | 4 | 3 | 1 |
| Paediatric orthopaedics | 115 | 40 | 33 | 27 | 5 | 1 |
| Shoulder and elbow | 33 | 22 | 17 | 16 | 3 | 1 |
| Spine | 27 | 26 | 15 | 12 | 4 | 1 |
| Sports orthopaedics | 23 | 11 | 9 | 8 | 3 | 1 |
| Trauma | 151 | 57 | 32 | 14 | 5 | 1 |
| Other | 128 | 0 | 0 | 0 | 0 | 1 |
-
*
Collection: literature review, existing guidelines, trial registries, and all experienced evidence uncertainties by stakeholders.
-
†
Selection: Advisory Board Health Research Agenda, assisted by the NOV’s research coordinator, the Federation of Medical Specialists, and the NOV Working Group Orthopedics and Science.
-
‡
Delphi - phase 1: the current or past board members of the NOV working groups.
-
§
Selection: The HRA Advisory Board assisted by the NOV Working Group Orthopedics and Science.
-
¶
Delphi - phase 2: orthopaedic surgeons, orthopaedic residents, rheumatologists, sports physicians, rehabilitation physicians, general practitioners, radiologists, occupational physicians, physiotherapists, researchers, nurse practitioners, nurses, podiatrists, infectiologists, health insurers.
-
**
Focus group: Patient associations, led by the Advisory Board’s chair, assisted by an Advisory Board member and the Federation of Medical Specialists advisor.
Panel members
In Phase 1, 41 panel members were included, leading to three to five panel members per subspecialty for the 12 orthopaedic themes. The response was 87% (n = 37) in round 1 and 73% (n = 33) in round 2. In the end, the opinion of 95% of the panel members was incorporated in the results of Phase 1, as four non-responding panel members in round 1 did respond in round 2. Characteristics of participating panel members can be found in Table II.
Table II.
Characteristics of panel members in phase 1 and 2 Delphi and participants focus group.
| Characteristic | Phase 1 Delphi (n = 41) | Phase 2 Delphi (n = 124) | Focus group (n = 6) |
|---|---|---|---|
| Background, n (%) | |||
| Orthopaedic surgeon | 33 (80.5) | 75 (60.5) | 1 (16.7) |
| Orthopaedic resident | 3 (7.3) | 4 (3.2) | 0 |
| Researcher | 1 (2.4) | 18 (14.5) | 1 (16.7) |
| Patient association | 0 | 0 | 0 |
| Sports medicine physician | 0 | 1 (0.8) | 0 |
| Nurse practitioner orthopaedic surgery | 0 | 3 (2.4) | 0 |
| Physiotherapist | 0 | 4 (3.2) | 0 |
| Podiatrist | 0 | 1 (0.8) | 0 |
| Patient association | 0 | 0 | 4 (66.7) |
| Other | 0 | 3 (2.4) | 0 |
| Unknown | 4 (9.6) | 15 (12.1) | 0 |
| Type of hospital | |||
| University hospital | 17 (41.5) | 34 (27.4) | 0 |
| Top-clinical teaching hospital | 16 (39.0) | 38 (30.6) | 1 (16.7) |
| General hospital | 3 (7.3) | 18 (14.5) | 1 (16.7) |
| Private hospital | 0 | 8 (6.5) | 0 |
| Rehabilitation centre | 0 | 6 (4.8) | 0 |
| Primary care centre | 0 | 2 (1.6) | 0 |
| Other | 1 (2.4) | 9 (7.3) | 0 |
| Not applicable | 0 | 0 | 4 (66.7) |
| Unknown | 4 (9.6) | 9 (7.3) | 0 |
In Phase 2, 170 panel members were recruited and invited to participate. The response was 67% (n = 115) in Round 1 and 50% (n = 86) in round 2, with nine new panel members in round 2. Hence, the opinion of 73% of the panellists was reported. Due to technical problems during round 1, 36 panel members (31%) were not directed to their first-choice theme and could not rate their preference. In total, 13 panel members rated all the themes. The technical issue was discovered and resolved during round 2, resulting in 16 panel members not rating the preferred theme and eight panel members rating all themes. Patients could not participate in Phase 2 due to the complexity and ambiguous terminology of the evidence uncertainties, including their background information.
Prioritization of evidence uncertainties
Phase 1 of the Delphi study started with 265 selected evidence uncertainties, ranging from 57 in trauma to three in bone and soft-tissue tumours (Table I). After Phase 1, 160 evidence uncertainties were selected based on the predetermined level of consensus, varying from 33 in children’s orthopaedics to two in bone and soft-tissue tumours. Of these 160 evidence uncertainties, 41 were removed due to issues with the research question’s quality or an unclear background/explanation. Phase 2 started with 119 evidence uncertainties across 12 themes, varying between 27 in children’s orthopaedics and one in bone and soft-tissue tumours. After Phase 2, a maximum of five evidence uncertainties (ranging between one to ) were selected per theme. No evidence uncertainties were prioritized in soft-tissue tumours since this evidence uncertainty did not meet the inclusion criteria for prioritization.
Focus group
The focus group members identified 12 key evidence uncertainties as the most important after discussions, which led to the development of the final HRA (Figure 1). The remaining evidence uncertainties not selected by the focus group will be examined in order of importance based on the selection criteria used in Phase 2 of the Delphi study (Table I).
Fig. 1
Methodological framework process of reaching a consensus on research priorities by applying a Delphi study and a focus group. HRA, health research agenda.
Final health research agenda
The advisory board examined the findings of the focus group, which led to the omission of one of the prioritized evidence uncertainties, which had already been investigated. The resulting HRA was communicated to stakeholders and the broader public at the national NOV conference on 5 October 2023. See Figure 2 for a flowchart of the collected, selected, and prioritized evidence uncertainties to set the final HRA.
Fig. 2
Flowchart collection, selection, and prioritization the evidence uncertainties to set the health research agenda.
Discussion
Our framework resulted in an orthopaedic multidisciplinary HRA. The prioritization process enhanced transparency, inclusivity, and participation by engaging a diverse group of healthcare professionals and patients, achieving widely accepted consensus. This represents an improvement compared to previous orthopaedic HRAs.
The study offered several notable advantages. First, by employing an e-Delphi study, panel members conveniently shared perspectives, leading to increased participation compared to previous NOV HRAs.17 Second, we minimized excessive individual influence within the prioritization process. This promoted a safe environment and consensus-oriented approach to expressing opinions, ultimately leading to the establishment of widely accepted research priorities. Third, introducing randomization in Phase 2 and the focus group enhanced inclusivity by gaining opinions from various stakeholder groups, including patients. This prevented the dominance of individuals and homogenous groups, and resulted in widespread support. Fourth, the study enabled transparent prioritization, with responses known only to the researcher and preset levels of agreement, ensuring a more data-driven approach in which panel members expressed unbiased opinions. Furthermore, the iterative nature of the technique incorporated feedback, allowing panel members to adjust opinions for maximum support and consensus.
The first limitation was the lack of specific guidance on an acceptable response rate. Generally, an approximate response rate of 80% for each stakeholder group is considered satisfactory. While the response rate in Phase 1 was deemed adequate, we observed a decline in participant engagement during Phase 2 of our study (67% in round 1 and 50% in round 2). This may have introduced attrition bias, as non-responding panel members could have had different views from non-participating peers within the stakeholder group.36 Previous research indicates that several factors could explain the decreased participation in the second round, including the time-consuming nature of the questionnaire, participant fatigue, technical issues encountered in the first round, and the lack of open-ended feedback options. 28 The second limitation relates to the size and homogeneity of the focus group. A focus group is an easy-to-administer informal meeting to reach a consensus with four to eight panel members, generally with a homogenous composition.37 The size and homogeneity of the group (four patient representatives) and their limited influence on the previous prioritization process might have influenced the result. Another limitation was using close-ended questions in the Delphi study, using all evidence uncertainties collected and selected earlier. Panel members could not comment on or add additional evidence uncertainties they deemed important. Although predetermining content may introduce bias by limiting considered topics,38 preliminary content generated from a literature review or other methods is widely accepted as a modification to the traditional open-ended questionnaire.39-44 This approach enhanced accessibility and saved time for panel members. Another limitation is that we did not ask participants for a conflict of interest before participating. We acknowledge the importance of transparency regarding potential conflicts of interest. However, we believe the inherent nature of the Delphi process, coupled with our modifications, effectively minimized bias risks. The diverse range of participants also contributed to this mitigation despite non-random selection based on expertise requirements. An additional limitation is the relative scarcity of research questions concerning tumours and impairment and disability medicine. With regard to the tumours, the scarcity likely stems from their rare occurrence. Furthermore, for both themes there might also be limited awareness among orthopaedic professionals.
With the new framework, we aim to develop a more transparent HRA. However, the influence of the Advisory Board throughout the process was significant. Following Phase 1, the advisory board reassessed the result of the Delphi study, leading to the removal of 41 of the initially confirmed 160 uncertainties. Additionally, their reassessment of the focus group’s findings might have significantly influenced the final HRA. To enhance transparency and minimize their impact on the outcome, we suggest restricting their role to providing guidance and facilitating the process, excluding them from voting privileges and assessing or reformulating the evidence uncertainties.
The second recommendation is related to biased patient influence. In the second phase of the Delphi study, the quality of collected evidence uncertainties with clear and concise explanations is crucial for patients’ opinions. The Delphi study excluded patients due to the complexity of the evidence uncertainties and background information, which hindered their understanding. Some questions were too technically detailed and challenging for an audience without extensive medical or statistical knowledge; using plain language can be complicated and time-consuming. We recommend implementing meticulous guidelines for formulating research questions, including background information, to comprehensively identify and gather evidence uncertainties. Further research is recommended to address increased patient participation, using clear, concise, and correct language and avoiding technical jargon.
Our final recommendation concerns setting up a focus group to actively involve patients by clarifying and explaining the evidence uncertainties. In our framework, the patients’ perspective was limited because they were only included at the end of the process. We suggest involving them earlier, allowing a more significant influence on their perspective.
The use of a Delphi study to reach a consensus on health research priorities based on evidence uncertainties within healthcare is widely recognized. Our research introduces an innovative methodological framework for consensus-building in determining research priorities to establish an HRA. As far as we know, this is the first time a medical association used this framework to set up a national HRA. We expect this new framework will enable a more informed, transparent, and inclusive decision-making process in prioritizing collected evidence uncertainties across various themes within a medical discipline. An equitable and comprehensive HRA might enhance the effectiveness and viability of a national research programme. This, in turn, might improve the allocation of resources, reduce research waste, and ultimately contribute to incorporating evidence-based, cost-effective medical treatments in daily patient care. However, this is a complex and challenging topic. Therefore, we recommend future research in this area.
References
1. The James Lind Alliance . Research Activity Following Asthma Treatment Uncertainty Priority Setting Exercise in 2007 – Paper Updated January 2010 , 2010 . https://www.jla.nihr.ac.uk/media/9376/download ( date last accessed 29 January 2025 ). Google Scholar
2. Cowan K , Oliver S . Chapter 12 - Toolbox of key priority setting partnership documents . In : The James Lind Alliance Guidebook . Southampton, UK : James Lind Alliance , 2021 : 96 . https://www.jla.nihr.ac.uk/jla-guidebook Google Scholar
3. Chalmers I , Bracken MB , Djulbegovic B , et al. How to increase value and reduce waste when research priorities are set . Lancet . 2014 ; 383 ( 9912 ): 156 – 165 . Crossref PubMed Google Scholar
4. Andronis L . Analytic approaches for research priority-setting: issues, challenges and the way forward . Expert Rev Pharmacoecon Outcomes Res . 2015 ; 15 ( 5 ): 745 – 754 . Crossref PubMed Google Scholar
5. Menon D , Stafinski T , Martin D . Priority-setting for healthcare: who, how, and is it fair? Health Policy . 2007 ; 84 ( 2–3 ): 220 – 233 . Crossref PubMed Google Scholar
6. Ahmed I , Metcalfe A . Research priorities of members of the British Association for Surgery of the Knee . Bone Joint J . 2024 ; 106-B ( 7 ): 662 – 668 . Crossref PubMed Google Scholar
7. No authors listed . Priority setting for health research: lessons from developing countries. The Working Group on Priority Setting . Health Policy Plan . 2000 ; 15 ( 2 ): 130 – 136 . Crossref PubMed Google Scholar
8. Abma TA , Pittens C , Visse M , Elberse JE , Broerse JEW . Patient involvement in research programming and implementation . Health Expect . 2015 ; 18 ( 6 ): 2449 – 2464 . Crossref PubMed Google Scholar
9. Bryant J , Sanson-Fisher R , Walsh J , Stewart J . Health research priority setting in selected high income countries: a narrative review of methods used and recommendations for future practice . Cost Eff Resour Alloc . 2014 ; 12 ( 23 ): 23 . Crossref PubMed Google Scholar
10. Caron-Flinterman JF , Broerse JEW , Teerling J , et al. Stakeholder participation in health research agenda setting: the case of asthma and COPD research in the Netherlands . Science and Public Policy . 2006 ; 33 ( 4 ): 291 – 304 . Crossref Google Scholar
11. Jeste DV , Alexopoulos GS , Bartels SJ , et al. Consensus statement on the upcoming crisis in geriatric mental health . Arch Gen Psychiatry . 1999 ; 56 ( 9 ): 848 . Crossref PubMed Google Scholar
12. Vernon H . The Development of a Research Agenda for the Canadian Chiropractic Profession: Report of the Consortium of Canadian Chiropractic Research Centres . J Can Chiropr Assoc . 2000 ; 2 : 86 – 92 . Google Scholar
13. Gartland JJ . Orthopaedic clinical research. Deficiencies in experimental design and determinations of outcome . J Bone Joint Surg Am . 1988 ; 70-A ( 9 ): 1357 – 1364 . PubMed Google Scholar
14. Rogers C , Jacobs JJ , Weisberg M . AAOS introduces new Unified Research Agenda . 2003 . https://web.archive.org/web/20060420043755/http:/www2.aaos.org/aaos/archives/bulletin/aug03/acdnws6.htm ( date last accessed 21 March 2024 ). Google Scholar
15. No authors listed . American Academy of Orthopaedic Surgeons . Research Priorities for the Unified Orthopaedic Research Agenda , 2014 . https://www.ors.org/wp-content/uploads/2016/10/URA_Internal.pdf ( date last accessed 17 February 2025 ). Google Scholar
16. Nelissen R . Onderzoeksagenda othopedie 2015 . In : Vereniging NO . ed . Nederlandse orthopaedische vereniging:’s-Hertogenbosch . 2015 : 43 . Google Scholar
17. Poolman RW . Onderzoeksagenda othopedie 2019 . In : Vereniging NO . ed . Nederlandse orthopaedische vereniging:’s-Hertogenbosch . 2019 : 31 . Google Scholar
18. Iken AR , Poolman RW , Nelissen RGHH , Gademan MGJ . Challenges in developing national orthopedic health research agendas in the Netherlands: process overview and recommendations . Acta Orthop . 2023 ; 94 : 230 – 235 . Crossref PubMed Google Scholar
19. Burgers JS , Wittenberg J , Keuken DG , et al. Development of a research agenda for general practice based on knowledge gaps identified in Dutch guidelines and input from 48 stakeholders . Eur J Gen Pract . 2019 ; 25 ( 1 ): 19 – 24 . Crossref PubMed Google Scholar
20. Wijma J , van Benthem PGG , Poolman RW , Verstraete E . Advisory Report. Health Care Evaluation: From Project to Process . Federatie Medisch Specialisten , 2016 . https://demedischspecialist.nl/sites/default/files/Adviesrapport_Zorgevaluatie_ENGELSE%20VERSIE.pdf ( date last accessed 17 February 2025 ). Google Scholar
21. Jünger S , Payne SA , Brine J , Radbruch L , Brearley SG . Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: recommendations based on a methodological systematic review . Palliat Med . 2017 ; 31 ( 8 ): 684 – 706 . Crossref PubMed Google Scholar
22. Belton I , MacDonald A , Wright G , Hamlin I . Improving the practical application of the Delphi method in group-based judgment: a six-step prescription for a well-founded and defensible process . Technol Forecast Soc Change . 2019 ; 147 : 72 – 82 . Crossref Google Scholar
23. Williamson PR , Altman DG , Bagley H , et al. The COMET handbook: version 1.0 . Trials . 2017 ; 18 ( Suppl 3 ): 280 . Crossref PubMed Google Scholar
24. Niederberger M , Spranger J . Delphi technique in health sciences: a map . Front Public Health . 2020 ; 8 : 457 . Crossref PubMed Google Scholar
25. Bolger F , Wright G . Improving the Delphi process: lessons from social psychological research . Technol Forecast Soc Change . 2011 ; 78 ( 9 ): 1500 – 1513 . Crossref Google Scholar
26. Geist MR . Using the Delphi method to engage stakeholders: a comparison of two studies . Eval Program Plann . 2010 ; 33 ( 2 ): 147 – 154 . Crossref PubMed Google Scholar
27. Hohmann E , Cote MP , Brand JC . Research pearls: expert consensus based evidence using the Delphi method . Arthroscopy J Arthrosc Relat Surg . 2018 ; 34 ( 12 ): 3278 – 3282 . Crossref PubMed Google Scholar
28. Hsu C-C , Sandford BA . The delphi technique: making sense of consensus . Pract Assess Res Eval . 2019 ; 12 ( 1 ): 10 . Crossref Google Scholar
29. Verweij LPE , Sierevelt IN , Baden DN , et al. A modified Delphi study to identify which items should be evaluated in shoulder instability research: a first step in developing a core outcome set . JSES Int . 2023 ; 7 ( 6 ): 2304 – 2310 . Crossref PubMed Google Scholar
30. Ogbeifun E , Agwa-Ejon J , Mbohwa C , Pretorius JHC . The Delphi technique: a credible research methodology . Proceedings of the 2016 International Conference on Industrial Engineering and Operations Management ; 2016 , http://ieomsociety.org/ieom_2016/pdfs/589.pdf ( date last accessed 29 January 2025 ). Google Scholar
31. Yousuf MI . Using experts' opinions through Delphi technique . Pract Assess Res Eval . 2007 ; 12 . Crossref Google Scholar
32. Hasson F , Keeney S , McKenna H . Research guidelines for the Delphi survey technique . J Adv Nurs . 2000 ; 32 ( 4 ): 1008 – 1015 . PubMed Google Scholar
33. Linstone HA , Turoff M . The Delphi Method: Techniques and Applications . Boston, Massachusetts, USA : Addison-Wesley , 1975 . Crossref Google Scholar
34. Boel A , Navarro-Compán V , Landewé R , van der Heijde D . Two different invitation approaches for consecutive rounds of a Delphi survey led to comparable final outcome . J Clin Epidemiol . 2021 ; 129 : 31 – 39 . Crossref PubMed Google Scholar
35. Russell D , Hoare ZSJ , Whitaker R , Whitaker CJ , Russell IT . Generalized method for adaptive randomization in clinical trials . Stat Med . 2011 ; 30 ( 9 ): 922 – 934 . Crossref PubMed Google Scholar
36. Lehoux P , Williams-Jones B , Miller F , Urbach D , Tailliez S . What leads to better health care innovation? Arguments for an integrated policy-oriented research agenda . J Health Serv Res Policy . 2008 ; 13 ( 4 ): 251 – 254 . Crossref PubMed Google Scholar
37. Cortini M , Galanti T , Fantinelli S . Focus group discussion: how many participants in a group? Encyclopaideia . 2019 ; 23 ( 54 ): 29 – 43 . Crossref Google Scholar
38. Keeney S , Hasson F , McKenna H . Consulting the oracle: ten lessons from using the Delphi technique in nursing research . J Adv Nurs . 2006 ; 53 ( 2 ): 205 – 212 . Crossref PubMed Google Scholar
39. Boulkedid R , Abdoul H , Loustau M , Sibony O , Alberti C . Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review . PLoS One . 2011 ; 6 ( 6 ): e20476 . Crossref PubMed Google Scholar
40. Powell C . The Delphi technique: myths and realities . J Adv Nurs . 2003 ; 41 ( 4 ): 376 – 382 . Crossref PubMed Google Scholar
41. Banayan J , Blood A , Park YS , Shahul S , Scavone BM . A modified Delphi method to create a scoring system for assessing team performance during maternal cardiopulmonary arrest . Hypertens Pregnancy . 2015 ; 34 ( 3 ): 314 – 331 . Crossref PubMed Google Scholar
42. Cantrill JA , Sibbald B , Buetow S . Indicators of the appropriateness of long-term prescribing in general practice in the United Kingdom: consensus development, face and content validity, feasibility, and reliability . Qual Health Care . 1998 ; 7 ( 3 ): 130 – 135 . Crossref PubMed Google Scholar
43. Eubank BH , Mohtadi NG , Lafave MR , et al. Using the modified Delphi method to establish clinical consensus for the diagnosis and treatment of patients with rotator cuff pathology . BMC Med Res Methodol . 2016 ; 16 ( 1 ): 56 . Crossref PubMed Google Scholar
44. Gattrell WT , Hungin AP , Price A , et al. ACCORD guideline for reporting consensus-based methods in biomedical research and clinical practice: a study protocol . Res Integr Peer Rev . 2022 ; 7 ( 1 ): 3 . Crossref PubMed Google Scholar
Author contributions
A. R. Iken: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Software, Visualization, Writing – original draft
M. G. J. Gademan: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Resources, Supervision, Validation, Writing – review & editing
B. A. M. Snoeker: Conceptualization, Funding acquisition, Resources, Writing – review & editing
T. P. M. Vliet Vlieland: Data curation, Formal analysis, Methodology, Writing – review & editing
R. W. Poolman: Conceptualization, Funding acquisition, Methodology, Resources, Supervision, Writing – review & editing
Funding statement
The authors disclose receipt of the following financial or material support for the research, authorship, and/or publication of this article: LROI Foundation (project title: Addendum PhD special chair; project number LUMC: 32269-8205).
ICMJE COI statement
M. G. J. Gademan, A. R. Iken, and R. W. Poolman report institutional funding from the LROI for this study. R. W. Poolman reports grants or contracts from ZonMW, and support for attending meetings and/or travel from Waldemar Link and OTC Foundation, unrelated to this study. Until August 2023, B. A. M. Snoeker worked for the Dutch Orthopaedic Association and was part of the advisory board. T. P. M. V. Vlieland was Vice President of Health Professionals in Rheumatology from 2020 to 2022, and holds an unpaid role on the EULAR Advocacy Committee.
Data sharing
The data that support the findings for this study are available to other researchers from the corresponding author upon reasonable request.
Acknowledgements
We would like to acknowledge the Advisory Board members of the working group of the third NOV Health Research Agenda for their contributions: T. Gosens, St. Elisabeth Hospital, Tilburg, The Netherlands; S.A.W. van de Groes, Radboud University Medical Center, Nijmegen, The Netherlands; M. C. van der Steen, Catharina hospital and Máxima Medical Center, Eindhoven, The Netherlands; P. C. Jutte University Medical Center Groningen, The Netherlands; Y. V. Kleinlugtenbelt, Deventer Hospital, The Netherlands; M. E. Major, Amsterdam University of Applied Sciences, The Netherlands; B. J. E. de Lange-Brokaar, University Medical Center Utrecht, The Netherlands; A. F. Lenssen, Maastricht University Medical Center, The Netherlands; W. O. Zimmermann, Department of Military Sports Medicine, Royal Netherlands Army, The Netherlands; G. A. W. Bruijn, Tergooi Hospital, Hilversum, The Netherlands; J. Zwerver, University Medical Center Groningen, The Netherlands; R. J. Derksen, VU University Medical Center, The Netherlands; E. G. M. Pels, Healthcare evaluation and appropriate use, Diemen, The Netherlands.
Open access funding
The open access fee for this article was funded by the LROI Foundation (Project title: Addendum PhD special chair; project number LUMC: 32269-8205).
Supplementary material
Final health research agenda of Dutch Orthopaedic Association - The Netherlands
© 2025 Iken et al. This is an open-access article distributed under the terms of the Creative Commons Attribution Non-Commercial No Derivatives (CC BY-NC-ND 4.0) licence, which permits the copying and redistribution of the work only, and provided the original author and source are credited. See https://creativecommons.org/licenses/by-nc-nd/4.0/
