An international survey to identify the intrinsic and extrinsic factors of research studies most likely to change orthopaedic practice

Survey development

A multi-stage development strategy was used to create our survey. An initial detailed review of the available literature was performed to formulate a template questionnaire in keeping with previously established guidelines for academic survey development.^16,17 Following template questionnaire formulation, a four-person focus group, which included a senior orthopaedic surgeon with clinical research experience, two senior orthopaedic residents and a biostatistics graduate student, was established. The focus group provided assistance with increasing survey readability and usability, while emphasising high-priority content and minimising survey bias.

We formulated an eight-question demographic section and 21-question respondent preference section (garnering opinions on both extrinsic and intrinsic features of research studies likely to influence a change in practice). To maximise usability of the survey and minimise respondent burden, we opted to use a web-based survey interface, administering the survey via SurveyMonkey (www.surveymonkey.com, Palo Alto, California). A cover letter introducing the study to potential survey respondents was created by the study team to highlight the study objective, explain passive consent via survey completion, emphasise the confidential nature of the questionnaire, and approximate the time expected to be required to complete the survey in its entirety.

Next, a two-stage piloting process was employed. The questionnaire was first piloted via the online interface to eight independent orthopaedic surgeons (four surgeons with academic practices, four surgeons in community practice). Pilot respondents were asked to provide feedback on survey usability, respondent burden as well as utility of the survey to effectively assess respondent opinions on our topic of inquiry. The identification of any technical challenges experienced with the online survey interface was requested.^16,17 From this feedback, the question stems were further revised and a status bar added to the online user interface to increase potential responses.¹⁶ A large pilot trial was then administered to active members of the COA (841 members). From this pilot study, we received a response total of 95 completed questionnaires.⁹ Feedback from this pilot trial was incorporated to decrease the timeline for accepting survey respondents and adjust the rollout period of the survey to avoid coinciding with major conferences.⁹ The proposed survey was reviewed with the communications department of the Journal of Bone and Joint Surgery (JBJS) (Am) and further revisions were made to adjust question stem wording to a more global sample population with an unbiased tone. Additionally, country of practice was added to the demographic section of the questionnaire.

The final survey was divided into an eight-question demographics section and a 21-question respondent preference section. The full survey is available online in supplementary material. Questions within the demographics section assessed surgeon age, gender, country of practice and formal academic research training. The respondent preference section sought to assess extrinsic factors of research studies, such as the role of residency and fellowship training on informing practice change as well as the importance of industry and patient requests in guiding clinical practice. Those factors pertaining to study methodology and conduct (intrinsic factors) included study design, sample size and methods used to present study outcomes.

Questionnaire administration

Our population of interest included a broad range of globally practicing orthopaedic surgeons from a convenience-sampling frame of the readership of JBJS (Am). Requests for participation were sent via two mass email communications from JBJS (Am)’s headquarters. The first email sent to members of JBJS (Am) included the study cover letter on departmental letterhead as well as a secure link to our SurveyMonkey hosted survey, which invited participants to complete the questionnaire. Passive consent was outlined in the cover letter and was implied as participants independently volunteered to complete the survey and all responses were completely anonymous. No monetary incentives or pre-notification of the study were provided before initial email contact from JBJS (Am). A reminder email was sent four weeks later. Data collection concluded eight weeks post-launch. This study and the questionnaire were approved by the Hamilton Integrated Research Ethics Board (Project Number: 14–169).

Statistical analysis

Completed responses were entered into a study-specific database and the data was analysed descriptively. Incomplete responses were discarded. Categorical variables are reported as counts and percentages, and continuous variables are summarised with means and standard deviations (sd). Composite combination of positive Likert scale responses (i.e. ‘strongly agree’ and ‘agree’) was performed to demonstrate trends among like-opinioned respondents. Response rate was defined as the number of individuals who completed the survey across those who received the email invitation. The analyses were performed using IBM SPSS version 21 (IBM, Chicago, Illinois) and Microsoft Excel (Microsoft, Santa Rosa, California).

Demographics

Respondent characteristics are shown in Table I. In total 92% of surgeons (325/353) are male, while 218/353 (62%) of respondents are > 50 years of age, with a mean respondent age of 52 years (sd 10.1). Hip and knee arthroplasty, 68/353 (19%) and sports medicine 67/353 (19%) composed the bulk of fellowship-trained respondents, with 103/353 (29%) of respondents not listing a subspecialty (fellowship-trained) practice focus area. Approximately one-third, 114/353 (32%) of respondents are in academic practice and 189/353 (54%) in private practice. Just over half, 197/353 (56%), responded that they do not supervise residents in training and 319/353 (90%) responded that they do not have a graduate degree in clinical research. Of 348 respondents who disclosed current continent of practice, 294/348 (84%) practice in North America, with the fewest respondents 3/348 (0.9%) practicing in Africa.

Extrinsic factors of research studies affecting clinical decision-making

88% of respondents (297/337) ‘strongly agree’ or ‘agree’ that the judicious integration of best-available research with patient values and clinical expertise is an important part of their clinical decision-making (Table II). Responses demonstrated that 244/337 (72%) of respondents ‘strongly agree’ or ‘agree’ that research published in a highly prestigious journal is likely to influence their clinical decision-making, while 227/335 (68%) ‘strongly agree’ or ‘agree’ that research published by highly skilled subspecialist surgeons is likely to influence their clinical decision-making. Only 21/336 (6.3%) of respondents ‘agreed’ or ‘strongly agreed’ that they would continue to practice based on concepts learned in training, independent of the current literature. Similarly, only 14/334 (4.2%) of respondents indicated they would implement a proposed intervention, which appeared to cause no harm to patients, independent of the literature showing benefit.

Intrinsic factors of research studies affecting clinical decision-making

A total of 77% of respondents (250/325) reported that the study design was an ‘important’ or ‘very important’ factor likely to influence their clinical decision-making (Table III). Respondents indicated that the top three study designs with the greatest potential to influence a change in current clinical practice were: randomised controlled trials 307/327 (94%), meta-analysis 244/327 (75%) and systematic review 215/327 (66%) (Table III). Randomised controlled trials 270/327 (83%) and meta-analyses 192/327 (59%) were the study designs felt by respondents to have had the most significant impact on their practice in the last five years, with case reports 16/327 (4.9%) and narrative reviews 17/327 (5.2%) having the lowest impact on practice over the same interval. Sample size was reported as ‘important’ or ‘very important’ for 254/323 (79%) of respondents, with the greatest proportion of respondents, 190/327 (58%) stating that a minimum sample size of 101 to 500 participants would be required from a study in order for a surgeon to consider such a study to influence their practice. In terms of outcomes reporting, 235/327 (72%) and 221/327 (68%) of respondents agreed that a study should report both p-values and 95% confidence intervals, respectively.

Key findings

High-quality health care implies practice that is consistent with best evidence.¹⁸ In keeping with the advancing implementation of evidence-based practice in orthopaedics, 84% of survey respondents agreed that judicious integration of best-available research with patient values and clinical expertise is vital to their clinical decision-making. Concurrently, journal reputation remains central to individual surgeon influence, as surgeons continue to place emphasis on research published in highly prestigious journals in addition to valuing research published by highly skilled subspecialist surgeons. Orthopaedic surgeon respondents are more likely to apply evidence to their practice if a study design is amongst the highest quality (i.e. meta-analyses, systematic reviews, randomised controlled trials), if sample sizes range from 101 to 500 subjects, and if p-values with associated 95% confidence intervals are reported.

Our response rate of 3.9% is the greatest limitation of this survey and is in keeping with the declining response rates seen among clinician surveys.^19-22 While clinicians, in particular specialists, are less likely to respond to electronic-based surveys, our primary goal in targeting our convenience sample was minimising respondent burden by using an accessible format.^20,22 Our low survey response rate may introduce selection bias and impact ‘generalisability’. However, Table I demonstrates that the respondents have a sufficient breadth in terms of surgeon age, length of practice, and fellowship training. Among respondents, a relatively equal proportion identified themselves as practicing in academic and community/private practice, thus supporting some generalisability of our results.²⁴ Importantly, a common trend has been shown from the responses received and while these responses may not be representative of the entire sample population, strong methodology in the survey development process in spite of a low response rate does help reduce the effect of non-responder bias in our study.²⁴

One strategy that may have improved response rates would have been to develop and administer the survey in the five most commonly spoken first languages of the JBJS (Am) readership. We are unable to quantify the number of potential respondents who may have participated had multiple survey languages been available to participants. Additionally, the authorship was not made privy to the number of potential participants who have office policies, which prohibit participation in online surveys, which likely further reduced our response rate.²³ Distribution of our survey through the main communication department of JBJS (Am) as opposed to personalised invitations to participate in the survey likely depleted our response rate. It is probable that potential participants may not have viewed the invitation as directly applicable to them and thus ignored the two invitations to participate in the survey, without reading the invitation.²¹

Another limitation, one common to all surveys, is the possibility that responses to our questionnaire may differ from one’s true practice. Efforts were made to emphasise the confidentiality and anonymity of our survey and therefore, to minimise response bias. While efforts were employed to determine the importance of individual study factors on informing a surgeon’s practice, not all extrinsic and intrinsic factors surveyed are of equal importance.⁹ The bulk of respondents were from North America and Europe. As such, it is possible that factors identified to be important for informing change of practice may not be applicable outside of these jurisdictions.

One consideration not addressed by our survey is the impact of cost and financial reimbursement on a surgeon’s clinical decision-making process. However, Hageman et al¹³ have previously shown that reimbursement is relatively unimportant for orthopaedic surgeons in informing clinical decisions in both Europe and North America, when evidence is inconclusive. However, it is possible that reimbursement could have a subconscious effect on decision-making not accounted for by our survey.²⁵

A strength of this study is the methodological rigour employed in the survey development. We developed a user-friendly survey interface with emphasis on reducing judgment and bias in the question stems, with increased usability of the survey via our online platform. A two-stage piloting strategy aided in establishing face and content validity, while strengthening the confidence in our findings. In keeping with previously established guidelines to maximise the number of respondents, we were able to ensure respondent anonymity, secure data collection, and provide appropriate time estimates for completion.¹⁶ In keeping with the recommendations for academic survey design by Burns et al,¹⁶ we designed our questionnaire to be applicable to all orthopaedic surgeons, incorporating questions relevant to general orthopaedics as well as the subspecialty population. To our knowledge, this is the largest sample size of orthopaedic surgeon respondents questioned in regards to features of research study designs important for informing practice changes.

Emerging trends from our survey

The results of our pilot Canadian survey and our current global survey indicate that orthopaedic surgeons recognise and respect the role of high-quality study types in practicing evidence-based orthopaedics.⁹ RCTs were the most important study types for potentially changing practice among our respondent population, consistent with previously reported surveys of orthopaedic surgeons.^9,10 Most published literature in orthopaedics is of a low level of evidence (retrospective non-comparative studies).⁷ In fact, Chaudhry et al¹² demonstrated that RCTs comprise only 3% of the orthopaedic literature, with only 11% of published orthopaedic literature classified as Level I evidence. Bhandari et al²⁶ further demonstrated there is a need for improved reporting of orthopaedic RCTs and emphasised that findings from RCTs cannot be taken at face value without critical appraisal of the literature. Poolman et al²⁷ have also shown that studies labeled as Level I evidence should not be automatically assumed to have high reporting quality.^26,27 The role of observational studies, particularly in surgery when many trials would be unethical or unfeasible, must not be overlooked. Given that observational studies currently represent the vast majority of the current literature, a surprisingly low number of respondents indicated that observational studies have had a significant impact on their practice in the last five years.²⁸

Recent high-quality RCT evidence has demonstrated an impact on clinical practice related to indications for knee arthroscopy.¹¹ However, in most instances, best evidence in the literature has seen a significant lag in clinical application.¹ Morris, Wooding and Grant²⁹ reviewed 23 papers that examined the lag of translational research to clinical practice and found that the mean delay from research evidence to widespread clinical practice change was 17 years. Similarly, Fitzpatrick³⁰ indicated that it could take anywhere from ten to 30 years for research to be implemented into clinical practice.

Only 10% of respondents hold a graduate degree in clinical research. Atesok et al¹⁵ recently stated that surgeon scientists will drive future developments of evidence-based orthopaedics. The incorporation of research methodology into the training of orthopaedic residents has the potential to increase scientific inquiry and improve musculoskeletal care.^31,32 It has been shown that greater pursuit of academic careers among orthopaedic surgeons can be promoted by encouraging research at the resident level.^33-35 Furthermore, a focus on research training during residency correlates with a greater likelihood of publication in peer-reviewed journals and higher impact journals.^33,35 Increasingly, active forms of continuing medical education will be crucial to advance evidence-based practice in orthopaedics and in medicine in general.^36,37

In conclusion, our results demonstrate that the vast majority of orthopaedic surgeon respondents recognise the role of high-level evidence in guiding their clinical practices. High-quality studies in the form of RCTs, meta-analysis and systematic reviews are most valued among orthopaedic surgeons for informing clinical practice changes. However, it is clear that some elements of literature appraisal among surgeons require further development. The majority of respondents continue to emphasise the reputation of the authors and the journal of publication when considering clinical practice changes. Continued educational emphasis on the generation of large, methodologically sound clinical trials and the tenets of evidence-based orthopaedics remain paramount to translating high-quality research findings into clinical practice.