Abstract
Introduction
Manual chart review is labor-intensive and requires specialized knowledge possessed by highly-trained medical professionals. The cost and infrastructure challenges required to implement this is prohibitive for most hospitals. Natural language processing (NLP) tools are distinctive in their ability to extract critical information from raw text in the electronic health records (EHR). As a simple proof-of-concept, for the potential application of this technology, we examined its ability to discriminate between a binary classification (periprosthetic fracture [PPFFx] vs. no PPFFx) followed by a more complex classification of the same problem (Vancouver).
Methods
PPFFx were identified among all THAs performed at a single academic institution between 1977 and 2015. A training cohort (n = 90 PPFFx) selected randomly by an electronic program was utilized to develop a prototype NLP algorithm and an additional randomly-selected 86 PPFFx were used to further validate the algorithm. Keywords to identify, and subsequently classify, Vancouver type PPFFx about THA were defined. The algorithm was applied to consult and operative notes to evaluate language used by surgeons as a means to predict the correct pathology in the absence of a listed, precise diagnosis (e.g. Vancouver B2). Validation statistics were calculated using manual chart review as the gold standard.
Results
In distinguishing between 2983 cases of PPFFx, 2898 cases of no PPFFx, and 85 cases of index THA performed for fracture, the NLP algorithm demonstrated an accuracy of 99.8%. Among 73 PPFFx test cases, the algorithm demonstrated a sensitivity of 87.1%, specificity of 78.6%, PPV of 75.0%, and NPV of 89.1% in determining the correct Vancouver classification. Overall Vancouver classification accuracy was moderate at 82.2%.
Conclusion
NLP-enabled algorithms are a promising alternative to the current gold standard of manual chart review for evaluating outcomes of large data sets in orthopedics. Despite their immaturity with respect to orthopedic applications, NLP algorithms applied to surgeon notes demonstrated excellent accuracy (99.8%) in delineating a simple binary outcome, in this case the presence or absence of PPFFx. However, accuracy of the algorithm was attenuated when trying to predict a Vancouver classification subtype given the wide variability in surgeon dictation styles and precision of language. Nevertheless, this study provides a proof-of-concept for use of this technology in clinical research and registry development endeavors as it can reliably extract certain select data of interest in an expeditious and cost-effective manner.
Summary
NLP-enabled algorithms are a promising alternative to the current gold standard of manual chart review for the extraction and evaluation of large data sets in orthopedics.
