Files and links (2)
Published VersionCC BY V4.0, Open Access
Journal article
Validation and algorithmic audit of a deep learning system for the detection of proximal femoral fractures in patients in the emergency department: a diagnostic accuracy study
The Lancet Digital Health, Vol.4(5), pp.E351-E358
2022
PMID: 35396184
Abstract
Background:
Proximal femoral fractures are an important clinical and public health issue associated with substantial morbidity and early mortality. Artificial intelligence might offer improved diagnostic accuracy for these fractures, but typical approaches to testing of artificial intelligence models can underestimate the risks of artificial intelligence-based diagnostic systems.
Methods:
We present a preclinical evaluation of a deep learning model intended to detect proximal femoral fractures in frontal x-ray films in emergency department patients, trained on films from the Royal Adelaide Hospital (Adelaide, SA, Australia). This evaluation included a reader study comparing the performance of the model against five radiologists (three musculoskeletal specialists and two general radiologists) on a dataset of 200 fracture cases and 200 non-fractures (also from the Royal Adelaide Hospital), an external validation study using a dataset obtained from Stanford University Medical Center, CA, USA, and an algorithmic audit to detect any unusual or unexpected model behaviour.
Findings:
In the reader study, the area under the receiver operating characteristic curve (AUC) for the performance of the deep learning model was 0·994 (95% CI 0·988–0·999) compared with an AUC of 0·969 (0·960–0·978) for the five radiologists. This strong model performance was maintained on external validation, with an AUC of 0·980 (0·931–1·000). However, the preclinical evaluation identified barriers to safe deployment, including a substantial shift in the model operating point on external validation and an increased error rate on cases with abnormal bones (eg, Paget's disease).
Interpretation:
The model outperformed the radiologists tested and maintained performance on external validation, but showed several unexpected limitations during further testing. Thorough preclinical evaluation of artificial intelligence models, including algorithmic auditing, can reveal unexpected and potentially harmful behaviour even in high-performance artificial intelligence systems, which can inform future clinical testing and deployment decisions.
Details
- Title
- Validation and algorithmic audit of a deep learning system for the detection of proximal femoral fractures in patients in the emergency department: a diagnostic accuracy study
- Authors
- Lauren Oakden-Rayner (Corresponding Author) - The University of AdelaideWilliam Gale (Author) - The University of AdelaideThomas A. Bonham (Author) - Stanford UniversityMatthew P. Lungren (Author) - Stanford UniversityGustavo Carneiro (Author) - The University of AdelaideAndrew Bradley (Author) - Queensland University of TechnologyLyle J. Palmer (Author) - The University of Adelaide
- Publication details
- The Lancet Digital Health, Vol.4(5), pp.E351-E358
- Publisher
- The Lancet Publishing Group
- Date published
- 2022
- DOI
- 10.1016/S2589-7500(22)00004-8
- ISSN
- 2589-7500
- PMID
- 35396184
- Copyright note
- © 2022 The Author(s). Published by Elsevier Ltd. This is an Open Access article under the CC BY 4.0 license.
- Organisation Unit
- School of Science, Technology and Engineering
- Language
- English
- Record Identifier
- 99704998302621
- Output Type
- Journal article
Metrics
146 File views/ downloads
127 Record Views
InCites Highlights
These are selected metrics from InCites Benchmarking & Analytics tool, related to this output
- Collaboration types
- Domestic collaboration
- International collaboration
- Web Of Science research areas
- Medical Informatics