External validation of machine learning predictive models is achieved through evaluation of model performance on different groups of patients than were used for algorithm development. This important step is uncommonly performed, inhibiting clinical translation of newly developed models. Recently, machine learning was used to develop a tool that can quantify revision risk for a patient undergoing primary anterior cruciate ligament (ACL) reconstruction (https://swastvedt.shinyapps.io/calculator_rev/). The source of data included nearly 25,000 patients with primary ACL reconstruction recorded in the Norwegian Knee Ligament Register (NKLR). The result was a well-calibrated tool capable of predicting revision risk one, two, and five years after primary ACL reconstruction with moderate accuracy. The purpose of this study was to determine the external validity of the NKLR model by assessing algorithm performance when applied to patients from the Danish Knee Ligament Registry (DKLR).

The primary outcome measure of the NKLR model was probability of revision ACL reconstruction within 1, 2, and/or 5 years. For the index study, 24 total predictor variables in the NKLR were included and the models eliminated variables which did not significantly improve prediction ability - without sacrificing accuracy. The result was a well calibrated algorithm developed using the Cox Lasso model that only required five variables (out of the original 24) for outcome prediction. For this external validation study, all DKLR patients with complete data for the five variables required for NKLR prediction were included. The five variables were: graft choice, femur fixation device, Knee Injury and Osteoarthritis Outcome Score (KOOS) Quality of Life subscale score at surgery, years from injury to surgery, and age at surgery. Predicted revision probabilities were calculated for all DKLR patients. The model performance was assessed using the same metrics as the NKLR study: concordance and calibration.

In total, 10,922 DKLR patients were included for analysis. Average follow-up time or time-to-revision was 8.4 (±4.3) years and overall revision rate was 6.9%. Surgical technique trends (i.e., graft choice and fixation devices) and injury characteristics (i.e., concomitant meniscus and cartilage pathology) were dissimilar between registries. The model produced similar concordance when applied to the DKLR population compared to the original NKLR test data (DKLR: 0.68; NKLR: 0.68-0.69). Calibration was poorer for the DKLR population at one and five years post primary surgery but similar to the NKLR at two years.

The NKLR machine learning algorithm demonstrated similar performance when applied to patients from the DKLR, suggesting that it is valid for application outside of the initial patient population. This represents the first machine learning model for predicting revision ACL reconstruction that has been externally validated. Clinicians can use this in-clinic calculator to estimate revision risk at a patient specific level when discussing outcome expectations pre-operatively. While encouraging, it should be noted that the performance of the model on patients undergoing ACL reconstruction outside of Scandinavia remains unknown.

Surgery performed in low-volume centres has been associated with longer operating time, longer hospital stays, lower functional outcomes, and higher rates of revision surgery, complications, and mortality. This has been reported consistently in the arthroplasty literature, but there is a paucity of data regarding the relationship between surgical volume and outcome following anterior cruciate ligament (ACL) reconstruction. The purpose of this study was to compare the ACL reconstruction failure rate between hospitals performing different annual surgical volumes. The hypothesis was that ACL reconstructions performed at low-volume hospitals would be associated with higher failure rates than those performed at high-volume centres.

This level-II cohort study included all patients from the Norwegian Knee Ligament Registry that underwent isolated primary autograft ACL reconstruction between 2004 and 2016. Hospital volume was divided into quintiles based on the number of ACL reconstructions performed annually, defined arbitrarily as: 1–12 (V1), 13–24 (V2), 25–49 (V3), 50–99 (V4), and ≥100 (V5) annual procedures. Kaplan-Meier estimated survival curves and survival percentages were calculated with revision ACL reconstruction as the end point. Mean change in Knee Injury and Osteoarthritis Outcome Score (KOOS) Quality of Life and Sport subsections from pre-operative to two-year follow-up were compared using t-test.

19,204 patients met the inclusion criteria and 1,103 (5.7%) underwent subsequent revision ACL reconstruction over the study period. Patients in the lower volume categories (V1-3) were more often male (58–59% vs. 54–55% p=<0.001) and older (27 years vs. 24–25 years, p=<0.001) compared to the higher volume hospitals (V4-5). Concomitant meniscal injuries (52% vs. 40%) and participation in pivoting sports (63% vs. 56%) were most common in V5 compared with V1 (p=<0.001). Median operative time decreased as hospital volume increased, ranging from 90 minutes at V1 hospitals to 56 minutes at V5 hospitals (p=<0.005). Complications occurred at a rate of 3.8% at low-volume (V1) hospitals versus 1.9% at high-volume (V5) hospitals (p=<0.001). Unadjusted 10-year survival with 95% confidence intervals for each hospital volume category were: V1 – 95.1% (93.7–96.5%), V2 – 94.1% (93.1–95.1%), V3 – 94.2% (93.6–94.8%), V4 – 92.6% (91.8–93.4%), and V5 – 91.9% (90.9–92.9%). There was no difference in improvement between pre-operative and two-year follow-up KOOS scores between hospital volume categories.

Patients having ACL reconstruction at lower volume hospitals did not have inferior clinical or patient reported outcomes, and actually demonstrated a lower revision rate. Complications occurred more frequently however, and operative duration was longer. The decreased revision rate is an interesting finding that may be partly explained by the fact that patients being treated in these small, often rural hospitals, may be of lower demand as suggested by the increased age and decreased participation in pivoting sports. In addition, patients with more complicated pathology such as meniscal tears were more commonly treated in the larger volume hospitals. The most significant limitation of this study is that provider volume was not assessed, and the number of surgeons dividing up the surgical volume at each hospital is not known.