Compare the features of several mainstream AI software programmes: explore their accuracy and repeatability in solving queries related to pancreatic cysts

Abstract

The prevalence of pancreatic cysts, a rare disease of the pancreas, has been increasing annually with advances in medical technology. However, health management education for such rare diseases is often inadequate, resulting in risky medical activities, patient confusion and concern. The development of large-scale language models (LLM) has provided a new platform for addressing these issues, with ChatGPT-4.0 and Deepseek-V3.1 being the most commonly used. In recent years, Libre Chat-v0.7.2 and Metaso AI-0.99 have gradually attracted widespread attention from scholars. Assessing the accuracy and comprehensiveness of such tools in addressing issues related to pancreatic cysts is crucial for their dissemination and application. Nineteen questions related to pancreatic cysts were screened for final inclusion and categorized into four domains: basic, diagnostic, treatment and prevention. The questions were also submitted to ChatGPT-4.0, Deepseek-V3.1, Metaso AI-0.99, and Libre Chat-v0.7.2 for answers. The prompts included: specific inquiries, a word limit of 300 characters, and a requirement for detailed bullet points to present the results more clearly. We simultaneously submitted the questions and requirements to Glass Health, a specialized medical payment AI, and invited eleven pancreatic disease specialists to independently score the AI’s results based on their expertise. The analysis was conducted based on these evaluations. These four artificial intelligence platforms generally provide accurate responses to questions concerning pancreatic cysts, though some answers may be incomplete or lack authoritative backing. When combined with assessments from eleven specialists, these large language model platforms score higher in addressing preventive measures for pancreatic cysts. Among these, Deepseek-V3.1 achieved a higher overall score than the other three platforms, demonstrating superior performance in pancreatic disease knowledge and patient comprehension, while also integrating cross-disciplinary insights for comprehensive responses. Similar to Glass Health, Metaso AI-0.99 conducts thorough online resource searches for each query and provides links to original literature. Users can explore these sources for further understanding. However, AI platform responses rely heavily on user-provided prompts, and users exhibit significant variations in medical comprehension. Future development requires updating big data models to achieve disease specificity by classifying user backgrounds, thereby delivering more appropriate responses. New large language model LLM tools may provide patients and healthcare professionals with basic and accurate information for the health management of pancreatic cysts. Further optimization and updating of training models will be required in the future to promote the application of the internet in the health management of rare and uncommon diseases and to enhance the practicality of LLM tools in clinical practice.

Ähnliche Arbeiten