Objective evaluations of resident performance can be difficult to simulate. A novel competency based surgical OSCE was developed to evaluate surgical skill. The goal of this study was to test the construct validity comparing previously validated Ottawa scores (O-scores) and Orthopaedic in-training evaluation scores (OITE).

An OSCE designed to simulate typical general orthopaedic surgical cases was developed to evaluate resident surgical performance. Post-graduate year (PGY) 3–5 trainees have an encounter (interview and physical exam) with a standardized patient and perform a correlating surgery on a cadaver. Examiners evaluate all components of the treatment plan and provide an overall score on the OSCE and also provide an O-score on overall surgical performance. Convergent and divergent validity was assessed comparing OSCE scores to O-scores and OITE scores. SPSS was used for statistical analysis. ANOVA was used to compare PGY averages and Pearson correlation coefficients were calculated to compare OSCE versus O-score and OITE scores.

A total of 96 simulated surgical cases were evaluated over a 3 year period for 24 trainees. There was a significant difference in OSCE scores based on year of training. (PGY3 − 6.06/15, PGY4 − 8.16/15 and PGY5 − 11.14/15, p < 0 .001). OSCE and O-scores demonstrated a strong positive correlation of +0.89 while OSCE and OITE scores demonstrated a moderate positive correlation of 0.68.

OSCE scores demonstrated strong convergent and moderate divergent correlation. A positive trajectory based on level of training and stronger correlations with established, validated scores supports the construct validity of the novel surgical OSCE.