Evaluating Completeness of Large Language Model Generated Cesarean Birth Operative Reports: ChatGPT-4.0 Versus ChatGPT-3.5 [ID 1560]

Abstract

INTRODUCTION: Large language models, including ChatGPT, are a form of artificial intelligence that offer promise in generating human-like texts. ChatGPT-4.0, released in March 2023, offers improvements over ChatGPT-3.5 including accuracy, context, and coherence. However, no studies to date have examined whether ChatGPT-4.0 demonstrates improvements in generating obstetrical operative reports. This study compares completeness of cesarean birth operative reports generated by ChatGPT-3.5 and ChatGPT-4.0. METHODS: Twenty cesarean birth operative reports were generated by both ChatGPT-3.5 and ChatGPT-4.0. Each note was evaluated for inclusion and completeness of history of present illness, operative findings, technique of resection, limits of resection, technique of reconstruction, and closure technique using a Likert scale. Median completeness of the operative reports by each ChatGPT model were compared. RESULTS: Overall, cesarean birth operative reports generated by ChatGPT-4.0 demonstrated significant improvement in median Likert score compared to ChatGPT-3.5 in completeness of brief history of present illness ( P <.001), operative findings ( P =.035), and closure technique ( P =.013). There was no significant improvement in technique of resection ( P =.465), limits of resection ( P =.147), and technique of reconstruction ( P =.058). CONCLUSIONS/IMPLICATIONS: Although ChatGPT-4.0-generated cesarean birth operative reports demonstrated improved documentation completeness of history of present illness, operative findings, and closure technique when compared to those generated by ChatGPT-3.5, no improvement was demonstrated in the remaining variables. These findings highlight the need for further improvements in ChatGPT prior to its utilization by obstetricians for generating cesarean birth reports.

Ähnliche Arbeiten