Design and execution of clinical trials in orthopaedic surgery

Introduction

Surgical innovations have conventionally been accepted through the endorsements of renowned experts or through the findings of weak scientific evidence.^1,2 Transitioning from eminence-based medicine to evidence-based medicine (EBM), however, has become the widely accepted new paradigm. Not only does EBM advocate the use of evidence in clinical decision-making, it emphasises the particular need for high-quality evidence. Accordingly, randomised controlled trials (RCTs) remain at the cornerstone of EBM as the highest level for a therapeutic intervention in the hierarchy of evidence (Fig. 1), however, has become the widely accepted new paradigm. Not only does EBM advocate the use of evidence in clinical decision-making, it emphasises the particular need for high-quality evidence. Accordingly, randomised controlled trials (RCTs) remain at the cornerstone of EBM as the highest level for a therapeutic intervention in the hierarchy of evidence (Fig. 1) remain at the cornerstone of EBM as the highest level for a therapeutic intervention in the hierarchy of evidence (Fig. 1).

Fig. 1

Image demonstrating the hierarchy of evidence

Many of the interventions in orthopaedic surgery are aimed at improving patient function and quality of life.¹ Randomising alternative approaches, such as surgical techniques or implants, provides a robust assessment of the superiority of one treatment over another. While randomised trials are well respected and often quoted in orthopaedic surgery, there is a relative paucity of RCTs in orthopaedic journals³ representing approximately 3% of the orthopaedic literature.^4,5

Surgical trials proportionally are two-fold less common than trials in internal medicine.⁶ This scarcity may in large part be attributable to several perceived barriers associated with the design and execution of randomised trials of surgical interventions.

We highlight several challenges in the conduct of surgical trials. Our discussion extends to both methodological issues in relation to study design and to pragmatic issues relating to the execution of RCTs.

Design of modern-day RCTs

Randomised trials in orthopaedic surgery have evolved considerably over the last decade. However, several challenges remain. Among these, four common issues prevail: i)Â identifying the most robust controls and the ethics of sham (or ‘placebo’) surgery, ii)Â achieving large enough sample sizes to be meaningful, iii) using patient important outcomes in lieu of ‘convenient’ outcomes, and iv) managing the inherent differential expertise biases that exist in all surgical trials.

Using robust controls: the ethics of sham surgery

The double-blinded, placebo-controlled, randomised trial remains the reference standard in drug trial designs.¹ The importance of blinding in randomised trials has been well documented in the medical literature.^5,7,8 In a comprehensive review of meta-epidemiologic Studies (reviews of meta-analyses), Savovićet al⁷ studied the effect of blinding on treatment outcomes in 1057 randomised trials identified from 104 meta-analyses. They demonstrated that lack of, or unclear, double-blinding was associated with a statistically significant exaggeration in treatment effect (13% increase in odds ratio, CI 4% to 21%) when compared with studies that were double-blinded.

In a review of RCTs published in TheJournal of Bone & Joint Surgery[American Volume] between 1988 and 2000,⁴ it was demonstrated that drug trials more often reported being double-blinded in comparison with trials of surgical intervention. Practically, blinding surgeons in a surgical intervention trial is impossible. However, outcomes assessors, data analysts and patients, may be blinded. Among the 30 surgical RCTs in this study, only 17% made a clear statement about blinding of patients, and only 33% made a statement regarding blinding of outcome assessors.⁴ These findings are concerning given the above mentioned influence lack of blinding has on inflating treatment effect size.^5,7,8

Furthermore, blinding works to quell the placebo effect of an intervention—the positive change in symptoms that can be attributed to a patient’s expectations of their treatment as opposed to the treatment itself.^9,10 Blinding warrants even further attention in orthopaedic research, as there is often a heavy emphasis on subjective outcome measures that are vulnerable to placebo effects, such as pain, function, and quality of life.¹¹

Blinding of surgeons is always impractical (if not impossible) and blinding of patients is often difficult.⁶ To achieve the latter, the issue of sham surgery has garnered much attention in the conduct of surgical RCTs. Although a sham surgery control group can offer a clear benefit to a trial’s methodological rigour, there are ethical considerations.¹¹ The main concern is that sham surgeries offer no therapeutic benefit, while exposing patients to unnecessary risks—a threat to the ethical principles of beneficence and non-maleficence.^1,9 Furthermore, sham surgeries compromise the integrity of the physician-patient relationship, as patients must be kept unaware of their sham surgery in follow-up visits to preserve blinding.³ However, universally labelling all sham surgeries as unethical may be an unfair argument. From a theoretical standpoint, it has been argued that the ethics of clinical research are not tantamount to the ethics of daily clinical practice, as the objective of the former is to answer a research question while minimising bias.^1,9 Sham surgeries may be considered akin to blood draws, lumbar punctures, and biopsies routinely performed in medical trials, as investigators are ethically bound to minimise patient risk, but not completely eliminate it.^1,9,11 From a pragmatic standpoint, sham surgeries may be justified when there is a true lack of consensus as to whether surgical management is superior to no treatment, referred to as clinical equipoise.^1,9,10 Moreover, both the risks and potential benefits of sham surgery must be critically evaluated. Sham surgery may be suitable if it is minimally invasive, low risk, and may offer research findings that significantly benefit society.^1,11 Obtaining informed consent is a matter of critical importance in a trial with sham surgery, as patients must acknowledge the risks and benefits of participating in a trial where they may potentially receive no therapeutic benefit.^9,11 Finally, an additional surgeon independent of the trial may be recruited to carry out all follow-up visits, in order to to avoid the deception otherwise required by the operating surgeon.¹⁰

Moseley et al¹² randomised 180 patients with painful osteoarthritis of the knee to treatment with arthroscopic debridement, arthroscopic lavage, or sham surgery. The study found no significant differences in pain or function between the treatment groups compared with the sham surgery group.¹² This study serves as a suitable prototype for when sham surgery may be appropriate in a clinical trial. There was clinical equipoise around the arthroscopic management of knee osteoarthritis, the placebo surgery was low risk as it entailed intravenous sedation and three small incisions, and the consent process ensured patient understanding with regard to the trial.¹²

Given the ethical concerns and the highly specific circumstances that are amenable to sham surgery, however, the use of placebo-controlled, randomised trials remain rare in orthopaedic surgery. As future surgical advancements are likely to progress in modest steps from current non-placebo interventions, the more suitable control groups may be gold-standard interventions and the need for sham surgeries may become a less pressing concern.⁶

How big is big enough? – the challenge of sample size

Regardless of the methodological safeguards used to limit bias in orthopaedic trials, a small study with an inadequate sample size may be misleading and insufficient to guide clinical practice.^13,14 There is perhaps no better example in the orthopaedic forum of the importance of adequate sample-size recruitment than the findings of the Study to Prospectively evaluate Reamed Intramedullary Nails in Patients with Tibial fractures (SPRINT).¹⁵ In this multicentre trial, over 1200 patients with both closed and open tibial shaft fractures were randomised to reamed or unreamed intramedullary nailing. After the recruitment of 50 patients, the study’s findings suggested that reamed nailing leads to a higher risk of re-operation (RR 1.85, 0.64 to 5.35). However, as a greater number of patients were recruited, the findings of this study shifted. In the final analysis of 1226 patients, reamed nailing showed a trend towards lower re-operation rates in comparison with unreamed nailing (RR 0.89, 0.70 to 1.14).¹⁶ The importance of full patient recruitment has been further demonstrated in a comprehensive review by Bassler et al,¹⁷ in which RCTs that had been stopped early for apparent ‘benefit’ were compared with non-truncated RCTs evaluating the same research questions. In their analysis of 91 truncated RCTs and 424 matching non-truncated RCTs, it was found that truncated RCTs were significantly more likely to overestimate treatment effects (ratio of relative risks = 0.71, p < 0.001).

Performing sample size calculations prior to a study and ensuring adequate patient recruitment, is a crucial component of preserving a trial’s methodological rigour. An insufficient sample size directly decreases the statistical power of a study—i.e. the ability to detect a statistically significant difference when a true difference indeed exists between treatment arms. Alternatively stated, there is an increased risk of a false-negative study finding (a type-II or beta error). However, simply recruiting more patients may indeed identify small statistical differences that are clinically unimportant.^13,14 Investigators must establish the primary outcome of interest a priori and the minimally important clinical difference associated with that outcome measure. Once established, a sample-size calculation can be performed prior to commencing a trial to ensure that investigators can correlate statistical significance to clinical significance and appropriately power their study.

Given the challenges and resources required to perform a surgical RCT, it would be unfavourable to forego a sample size calculation at the risk of under-powering a trial and reaching an equivocal conclusion.¹⁸ Freedman et al¹⁸ have demonstrated that the statistical power of orthopaedic RCTs is severely compromised.¹⁸ In a review of 33 orthopaedic RCTs published in premier orthopaedic journals (The Journal of Bone & Joint Surgery American Volume (JBJS [Am]), The Journal of Bone & Joint Surgery British Volume (JBJS [Br]), Clinical Orthopaedics and Related Research (CORR)), they found that only 9% of studies performed statistical sample size calculations and that nearly half of all studies with insignificant findings were too underpowered to detect even a large treatment effect.¹⁸ Similar findings were reported by Bhandari et al⁴ in the review of 72 orthopaedic randomised trials.⁴ Specifically, they demonstrated that 94% of trials failed to calculate the effective sample size needed to provide acceptable study power.⁴

Despite the need for sufficient patient recruitment, most RCTs often overestimate their ability to recruit participants and are also hindered by patient dropout.³ In a review of the prospective and retrospective screening studies used for the SPRINT trial, it was demonstrated that both methods grossly overestimated actual patient recruitment in the definitive trial. Despite the similar estimates provided by both approaches, they both lacked validity as they overestimated recruitment by nearly 70%.¹⁹

Proposed strategies to improve patient recruitment have included providing patients with detailed information about the benefits and risks of trial participation, altering study design to cater to patient preferences (i.e. no placebo arm, two surgical arms), improving the consent process, and offering incentives.³ Ultimately, however, establishing multicentre trials will afford investigators an effective and efficient means of conducting large trials that are adequately powered.

There are several benchmark trials to suggest that multicentre trials are not only feasible, but should be the current norm. For example, The Heart Outcomes Prevention Evaluation (HOPE) study was a multinational, randomised trial that recruited 9297 patients across 276Â centres from 19 different countries. Patients with cardiovascular risk factors were randomised to receive either ramipril (angiotensin-converting enzyme inhibitor) or placebo medication. At a mean follow-up of five years, patients treated with ramipril had significantly lower rates of myocardial infarction, stroke, and cardiovascular death.²⁰ Such landmark trials in cardiology have had a substantial clinical impact, as the mortality associated with coronary disease has fallen by nearly 40% in the past several decades and approximately half of this reduction is directly due to evidence-based medical therapies.²¹ Large, multicentre RCTs are by no means exclusive to nonsurgical trials. The SPRINT trial recruited 1339Â patients from 29 different sites in Canada, USA, and The Netherlands.¹⁵ Currently underway, the Fluid Lavage of Open Wounds (FLOW) trial is a multicentre randomised trial evaluating both irrigation solution (soap versus saline) and pressure (gravity, low, high) on the re-operation rates for open fracture management. With a recruitment goal of 2280 patients, the study currently has already crossed the 2000 patient mark, with participating sites from Canada, Australia, USA, Norway, and India.^22,23 The establishment of large orthopaedic research networks, by means of orthopaedic associations and specialist societies, can foster a culture in which these large, multicentre trials are feasible. Such networks can allow surgeons to collectively identify important research topics, endorse RCTs, educate surgeons about trial methodology, assist in the co-ordination of large trials, and ultimately, help disseminate research findings.²⁴

Patient-important outcomes

Sample size is clearly tied to an orthopaedic trial’s primary outcome measure. The decision to use a scale like the SF-36 or a dichotomous (yes/no) measure like re-operation will have significant implications on the number of patients required for a trial to show incremental benefits.¹⁴

The World Health Organization defines health as a state of complete physical, mental, and social well-being.^25,26 This definition of health is of direct relevance to orthopaedic surgeons and investigators, as patients afflicted with orthopaedic conditions often suffer limitations in function and quality of life, which inevitably compromise their state of health. Accordingly, outcome measures used in a trial should be patient-important and as objective as possible.²⁷ In medical trials, it is frequently possible to couple objective evaluation with patient importance, through measures such as death, thromboembolic events, and bleeding rates. This remains a greater challenge in orthopaedic trials, as patient-important outcomes must include more complex subjective evaluations to appreciate how interventions influence day-to-day activities, participation in social activities, and fulfilment of societal expectations. Further underpinning the need for such subjective measures is the fact that physiological outcomes do not consistently correlate with self-reported health.²⁵ The FLOW trial serves as a notable example, with its inclusive focus on important outcomes. As mentioned, the primary outcome measure of this trial is re-operation for infection, managing wound healing and promoting bone healing. To supplement re-operation rates, the study is also assessing function and quality of life with the Short-Form 12 and Euro-Qol-5 Dimensions.²²

Health Related Quality of Life (HRQOL) measures are an important tool in the armamentarium of outcome measures available to orthopaedic investigators and are becoming more commonplace in orthopaedic trials.¹⁴ There are two types of HRQOL measures—generic and disease-specific. Generic HRQOL instruments measure a patient’s general health status by assessing the physical, functional and emotional aspects of health.²⁵ These measures are typically useful for comparing the health of patients with different diseases, severity of disease or undergoing different interventions. Due to their broad scope, however, they may be less sensitive in detecting small but meaningful changes in the health of patients. Disease-specific HRQOL instruments assess specific aspects of physical, mental and social health that are relevant to the condition of interest. Given the narrower focus, these instruments are able to detect smaller differences in health status, but are often not comparable across different disease states.^14,25 Ideally, both a generic and disease-specific instrument should be used in a trial.¹⁴ The challenge for investigators is ensuring that optimal instruments are selected. In doing so, a clear understanding of the trial’s objectives is needed such that outcome instruments that run parallel to these goals, can be selected. Furthermore, HRQOL measures must be valid (measure what they are intended to measure), reliable (consistent when applied repeatedly in a stable population) and responsive (detect important changes in health state).²⁵

Limiting differential expertise bias: novel designs

If significant discrepancy exists in a surgeon’s skill or expertise between the various treatment arms, there is a threat of introducing differential expertise bias into the trial.²⁸ The second challenge facing surgical RCTs is that of clinical equipoise. Although community equipoise may exist for a given condition among surgeons as a whole, individual surgeons often favour a particular intervention.² This lack of clinical equipoise, which unfortunately is often based on weak scientific findings, serves as a barrier and ethical challenge to surgeon participation in RCTs.^3,29

In light of these challenges, there has been increasing interest in the expertise-based RCT. In this design, a surgeon with expertise for one intervention is paired with a surgeon who holds expertise in the other intervention. Patients are then randomised to each surgeon directly, who is solely responsible for performing their procedure of expertise (Fig. 2).²⁹

Fig. 2

Flow diagrams showing conventional randomisation a) versus b) expertise-based randomisation.

The Hip Fracture Evaluation with Alternatives of Total Hip Arthroplasty versus Hemi-arthroplasty (HEALTH) trial is an international, multicentre randomised trial currently underway that has employed an expertise-based design (ClinicalTrials.gov identifier: NCT00556842). This design is fitting given that a recent international survey demonstrated that surgeons vary widely in their preferred choice of arthroplasty when managing patients between 60 and 80 years old with completely displaced fractures of the femoral neck.³⁰

Ideally, this study design will prevent differential expertise bias, avoid ethical challenges faced by surgeons, foster surgeon participation, and reduce procedural cross-over rates.²⁸ The concern surrounding expertise-based RCTs extends to the potential loss of generalisability, as community surgeons may not be as skilled as trial experts.² Nevertheless, the expertise-based RCT offers a pragmatic solution to these unavoidable challenges of surgeon experience and equipoise.

Clinical trial execution

Despite the fact that a clinical trial is well-designed, the integrity of collected data and the ensuing conclusions may be seriously undermined if the trial is not executed with assiduous attention to ethical and logistical issues. In our experience, these issues become especially pronounced in large multicentre clinical trials, owing to the volume of sites, personnel, and data. In this section, we will highlight three prominent themes that are central to the successful execution of clinical trials: i) ethical and logistical issues associated with institutional ethics review boards (IRBs); ii) importance of an appropriate organisational framework; and iii) securing sufficient funding.

Ethical research conduct and ethics review boards

Founded on the principles of the Declaration of Helsinki, the Good Clinical Practice (GCP) Guidelines produced by the International Conference on Harmonisation (ICH) were designed to guide the ethical conduct of clinical trials.³¹ IRBs are bestowed the responsibility to uphold this mandate through the review of clinical trial protocols and ensuring that they are ethically and methodologically sound. The latter is of paramount importance, as methodologically poor research is unethical research, offering not only misleading conclusions, but also compromising patients’ time and draining societal resources. We advise researchers to view each IRB application as an opportunity to refine study design and conduct.

The final IRB-approved protocol must be adhered-to meticulously throughout the execution of a clinical trial. Deviations from the protocol should be justifiable and approved by the IRB. This will ensure that any protocol deviation mid-trial does not unintentionally (or intentionally) bias the final outcome data. The importance of this principle is such that many granting agencies and journals now require trial pre-registration.^32,33 Various clinical registries are available for this purpose, including ISRCTN.org and ClinicalTrials.gov. Publication of protocols in the scientific literature, where they are available for widespread peer review, has also been advocated.^32,33

From a logistical perspective, IRB applications can result in a significant delay to the start-up of a trial, especially in multinational, multicentre designs.³⁴ Identifying these delays early can help investigators circumvent or, at the very least, anticipate them. Firstly, there is the inherent delay in waiting for IRB approval, which is unavoidable and certainly necessary. Often, there is at least one, but often more, revisions. There is also variability among legislations, protocols, and efficiencies between different geographical regions. For instance, in the Fixation using Alternative Implants for the Treatment of Hip Fractures (FAITH) trial, there were significant differences seen between the Netherlands and both USA and Canada in terms of median time to receive IRB approval (104 days versus 53 days and 104 versus 55 days, respectively).³⁴

After receiving full IRB approval, there are always further delays in getting a trial moving forward, as certain organisational aspects of a trial can only be organised once IRB approval is obtained. Data from the FAITH trial again reinforces this point; median time between receiving ethics approval and start-up ranged from 41 days (the Netherlands) to 232 days (Canada).³⁴ In that instance, it was determined in retrospect that a central regional co-ordinator could potentially reduce the logistical component of the delay by circumventing the need for contract negotiations with individual study sites.³⁴ Ultimately, anticipating logistical delays to start-up and a dedicated team that can help study sites traverse these challenges, is fundamental to successful and efficient trial execution.

Multicentre trial organisation

There are several organisational committees that must exist to ensure the ethical and efficient execution of a clinical trial (Fig. 3).

Fig. 3

Diagram showing the multicentre trial committee organisational framework

Conclusion

Although challenging, orthopaedic investigators must continue to pursue high-quality research in the form of randomised trials. Foregoing design considerations—such as sample size calculations, patient-important outcomes, and surgeon expertise—can ultimately threaten the validity of a surgical trial. In the setting of multicentre trials, neglecting the execution strategies of ethical conduct, trial organisation and funding procurement poses both ethical and pragmatic barriers to the successful conduct of a RCT. If given generous consideration, orthopaedic surgeon researchers will ensure that high quality evidence forms the basis for further evolution in orthopaedic surgery.