Conference paper
Never mind the metrics - what about the uncertainty? Visualising binary confusion matrix metric distributions to put performance in perspective
Proceedings of Machine Learning Research (PMLR), Vol.202, pp.22702-22757
International Conference on Machine Learning, 40th (Hawaii, United States, 23-Jul-2023 - 29-Jul-2023)
Science Publishing Group
2023
Abstract
There are strong incentives to build classification systems that show outstanding performance on various datasets and benchmarks. This can encourage a narrow focus on models and the performance metrics used to evaluate and compare them—resulting in a growing body of literature to evaluate and compare metrics. This paper strives for a more balanced perspective on binary classifier performance metrics by showing how uncertainty in these metrics can easily eclipse differences in empirical performance. We emphasise the discrete nature of confusion matrices and show how they can be well represented in a 3D lattice whose cross-sections form the space of receiver operating characteristic (ROC) curves. We develop novel interactive visualisations of performance metric contours within (and beyond) ROC space, showing the discrete probability mass functions of true and false positive rates and how these relate to performance metric distributions. We aim to raise awareness of the substantial uncertainty in performance metric estimates that can arise when classifiers are evaluated on empirical datasets and benchmarks, and that performance claims should be tempered by this understanding.
Details
- Title
- Never mind the metrics - what about the uncertainty? Visualising binary confusion matrix metric distributions to put performance in perspective
- Authors
- David Lovell (Corresponding Author) - Queensland University of TechnologyDimity Miller (Author) - Queensland University of TechnologyJaiden Capra (Author) - Queensland University of TechnologyAndrew Bradley (Author) - University of the Sunshine Coast, Queensland, School of Science, Technology and Engineering
- Publication details
- Proceedings of Machine Learning Research (PMLR), Vol.202, pp.22702-22757
- Conference details
- International Conference on Machine Learning, 40th (Hawaii, United States, 23-Jul-2023 - 29-Jul-2023)
- Publisher
- Science Publishing Group
- Organisation Unit
- School of Science, Technology and Engineering; University of the Sunshine Coast, Queensland
- Language
- English
- Record Identifier
- 99971186302621
- Output Type
- Conference paper
Metrics
14 Record Views