Background: Clinical trials of surgery increasingly use disability and quality of life scales as their primary outcome measures. As such, they are the central dependent variables on which treatment decisions are based. It is therefore essential that these scales provide clinically meaningful and scientifically sound (reliable and valid) measurements.

Aim: The aim of this study was to determine if three scales used widely for spinal surgery (the Short form 36 item health survey – SF-36, Oswestry and Neck Disability Indices – ODI, NDI) satisfied basic requirements for reliable and valid measurement, and if they were suitable to detect clinically significant change.

Method: We analysed data from 147 people undergoing cervical (SF-36, NDI), and 233 people undergoing lumbar (SF-36, ODI) spine surgery. We tested the full range of measurement properties of these scales. These included the assumption that adding up items generates meaningful scores and, if that test was passed, scale targeting to study samples, reliability, validity and responsiveness.

Results: In both samples, the SF-36 had problems. Some scales had notable floor and ceiling effects. As a consequence they were unable to detect change. Other scales failed validity tests. Importantly, there was no support for using SF-36 summary scores in either cervical or lumbar surgery. With the ODI and NDI, there were problems with the individual questions. Specifically, the item response options were not working as anticipated. This compromises the reliability and validity of both scales.

Conclusions: This study, whose aim was to assess three scales used to evaluate surgery, not the surgery itself, demonstrates that all three have important limitations and questions their suitability for this crucial role. Essentially, all three scales give inaccurate estimates of treatment effectiveness. The result is that the benefits of spinal surgery are almost certainly being under-evaluated and spinal surgeons are selling themselves short.