Evaluation of the performance of an artificial intelligence model in recognizing the habitual mixed language in Taiwan for generating periodontal charting text reports

Abstract

Background/purpose: "Periodontal charting" in Taiwan typically involves a two-person collaboration. However, while advances in voice technology has made it feasible to use voice recognition to reduce these manpower demands in English-speaking countries, these artificial intelligent (AI) voice recognition tools have lacked localized features to make them feasible in our context of Taiwan. Materials and methods: To fill this research gap, we integrated GPT-4o-transcribe and GPT-4.1-mini into a two-stage system for this study, where stage 1 worked on speech-to-text and stage 2 focused on text-to-report. To evaluate this AI model's performance, five representative periodontal charting scenarios were constructed, and 15 voice recordings were collected. The evaluation focused on transcription accuracy and examined factors influencing recognition quality. Results: The system achieved an overall accuracy of 67.40 % in extracting and formatting data fields related to tooth identification, clinical parameters, and periodontal findings. Furcation involvement, mobility assessment, and missing tooth identification exceeded 90 % accuracy, while bleeding on probing reached 85.76 %. The plaque index and keratinized gingiva width had intermediate accuracies of 68.47 % and 70.83 %, respectively. Probing depth and gingival recession were lower at 53.33 % and 52.01 %. The main factors affecting accuracy included unstable speech speed, prolonged pronunciation leading to repeated numbers, and single positional errors causing chain errors. Conclusion: Our AI model showed strong speech recognition abilities without requiring extensive training in professional knowledge. It effectively identified dental terms and their related examinations and numbers in mixed language contexts. Future optimizations will focus on addressing errors from uneven speech speed and individual examiner habits.

Ähnliche Arbeiten