Pointwise Reliability Assessment

Abstract

In critical areas with significant user impact, the effective application of machine learning models to support decision making faces the important challenge of lack of trust. Despite their impressive performance, machine learning models are often perceived as “black boxes,“ making their adoption difficult. This is the case of health care area, where the lack of trust is undoubtedly a barrier to the application of those models in daily clinical practice. While metrics like accuracy, sensitivity, specificity are valuable to assess the global performance of a model, they provide little insight at the individual instance level (single patient). This highlights the importance of pointwise reliability assessment, evaluating whether a model can be reliable to classify a specific instance. This work aims to develop a systematic method for comparing a machine learning model pointwise reliability provided by different methods. Besides error rate quantification, this framework comprises t-Distributed Stochastic Neighbor Embedding visualizations. Thus, different data driven methods for assessing pointwise reliability are compared. These methods address the problem based on two different perspectives: i) density principle; ii) local fit principle. The validation was performed in the context of cardiovascular secondary prevention supported by a real patient dataset (N=1544). The findings show that approaches integrating both density and local fit principles tend to surpass single principle methods, associating lower error rates to more reliable predictions.

Ähnliche Arbeiten