Purpose: The Instability Severity Index Score (ISIS) is a 6-item questionnaire that has been reported to predict failure of arthroscopic Bankart repair among patients treated for recurrent anterior glenohumeral instability. Two of the ISIS items pertain to radiographic features (presence of a Hill-Sachs lesion, loss of glenoid contour). These, however have yet to be validated. The goal of this study was to examine the inter – and intra-rater agreement and corresponding reliability of the radiographic aspects of the ISIS.

Method: Fifty-two plain, randomly selected, true antero-posterior radiographs in 45° gleno-humeral external rotation were evaluated by five assessors (three upper extremity orthopaedic surgeons, one senior orthopaedic resident and one musculoskeletal radiologist). Radiographs were retrieved for patients with documented recurrent shoulder instability requiring surgical stabilization and placed in a blinded Microsoft Powerpoint presentation for evaluation. Assessors were asked to determine the presence/absence of a Hill-Sachs lesion and if a loss of glenoid contour was present. Radiographs were evaluated in random order on two occasions, separated by a one-week time interval. Intra – and inter-rater reliability was assessed using percentage of agreement and kappa statistics.

Results: For session one, the number of Hill-Sachs lesions observed among raters ranged from 11 to 32. This resulted in inter-rater agreement ranging from 48% to 78% (k = 0.07 to 0.42), indicating poor to fair reliability. Cases with loss of glenoid contour ranged from seven to 14 with inter-rater agreement of 66% and 90% (k = 0.01 to 0.61), suggesting poor to moderate reliability. Session two led to modest increases in inter-rater agreement. The number of Hill-Sachs lesions observed ranged from four to 30 (agreement of 48% to 84%; k = 0.11 to 0.60) indicating fair to moderate reliability and the number of cases with loss of glenoid contour ranged from three to 14 (agreement of 66% to 94%; k = – 0.04 to 0.69), ranging from poor to moderate reliability. With respect to intra-rater reliability, agreement ranged from 71% to 94% (k = 0.41 to 0.86) for Hill-Sachs lesions, indicating fair to good reliability and 76% to 94% (k= 0.20 to 0.74) for loss of glenoid contour, ranging from fair to good reliability. Intra-rater agreement and corresponding kappa values were highest among the upper extremity surgeons and the musculoskeletal radiologist, particularly for loss of glenoid contour (85% to 94%; k = 0.56 to 0.74) suggesting there is moderate to good reliability in this measurement.

Conclusion: Our results indicate that the intra-rater reliability of the ISIS radiographic features was highest among upper extremity specialists and the musculoskeletal radiologist, suggesting that the ISIS may have utility in an experienced clinician’s individual practice. As the inter-rater reliability appears low, particularly for Hill-Sachs lesions, its wide-spread use across surgeons should be examined in further research.