Quantifying AI Model Trust via Sureness Measure by Iterative Supervised Learning and Visual Knowledge Discovery

Abstract

Trust in machine learning models by domain experts and end users is essential for model deployment by them, particularly in high-stakes fields such as healthcare diagnosis. However, consistently quantifying model trust across diverse types of ML models is a difficult challenge in machine learning. Existing trust concepts often are narrow in scope and are not clearly defined for computing a trust score. This paper introduces a new concept of model sureness, which is a quantifiable and generalizable measure of one of the aspects of trust in ML models. To measure this model sureness a process that combines iterative supervised learning and visual knowledge discovery is proposed. It reduces required training data while preserving model accuracy as conducted case studies demonstrate. The method iteratively varies the training dataset and retrains models until a predefined efficiency criterion is met. The measure of the model sureness is defined as a ratio of the number of successful iterations to the total number of iterations. The models with higher ratio of eliminated cases are defined as having higher sureness measures. Case studies across three standard datasets from biology, medicine, and handwriting recognition are conducted. These demonstrate that the method can preserve model accuracy and eliminate 20% to 80% of noisy or redundant instances, with an average reduction of around 50%.

Ähnliche Arbeiten