Accuracy, Reliability, and Comprehensibility of ChatGPT-Generated Medical Responses for Patients With Nonalcoholic Fatty Liver Disease

Abstract

Nonalcoholic fatty liver disease (NAFLD) is an increasing global health problem and is expected to become the leading indication for liver transplantation.1Younossi Z.M. Hepatology. 2023; 77: 1335-1347Google Scholar There are no approved NAFLD-specific pharmacotherapies, and lifestyle modification is the primary recommended therapy.2Rinella M.E. et al.Hepatology. 2023; 77: 1797-1835Google Scholar Innovative approaches to facilitate the implementation and long-term maintenance of lifestyle changes are needed to address the challenging and complex nature of the management of NAFLD, which recently was renamed as metabolic dysfunction–associated steatotic liver disease, to overcome the limitations and stigma of the previous name.3Pugliese N. et al.Eur J Clin Invest. 2022; 52e13659Google Scholar,4Rinella M.E. et al.Hepatology. 2023; https://doi.org/10.1097/HEP.0000000000000520Google Scholar Artificial intelligence (AI)-powered chatbots have been shown to provide effective personalized support and education to patients, with the potential to complement health care resources. The OpenAI Foundation’s AI chatbot, Chat Generative Pretrained Transformer (ChatGPT), has attracted worldwide attention for its remarkable performance in question–answer tasks.5OpenAI.https://openai.com/blog/chatgptDate accessed: November 30, 2022Google Scholar, 6Lee P. et al.N Engl J Med. 2023; 388: 1233-1239Google Scholar, 7Lee T.C. et al.Gastroenterology. 2023; 165: 509-511Google Scholar This study evaluated the accuracy, completeness, and comprehensibility of chatGPT’s responses to NAFLD-related questions, with the aim of assessing its performance in addressing patients’ queries about the disease and lifestyle behaviors. This study was conducted in March 2023 and was Institutional Review Board exempt because no patient-level data were used. Two expert physicians (N.P. and A.A.) selected 15 questions commonly asked by NAFLD patients, taking into account their clinical expertise and guidelines (Table 1).2Rinella M.E. et al.Hepatology. 2023; 77: 1797-1835Google Scholar,3Pugliese N. et al.Eur J Clin Invest. 2022; 52e13659Google Scholar The questions can be grouped into 3 domains: (1) questions 1, 14, and 15 focused on the need for specialist referral and the ability to detect worsening/improvement of NAFLD; (2) questions 2 to 9 and 13 focused on diet composition, weight loss, medications, coffee consumption, and alcohol intake; and (3) questions 10 to 12 focused on the type of physical activity.Table 1Questions Posed to ChatGPTQuestion1I have been diagnosed with NAFLD. Should I be referred to a hepatologist? Should I undergo additional blood tests or radiologic examinations?2How much weight should I lose if I have NAFLD?3Which diet should I follow if I have NAFLD?4Which foods should I avoid if I have NAFLD?5Can I drink alcohol if I have NAFLD?6Which foods have a positive effect on NAFLD?7Can I use herbal remedies if I have NAFLD?8Can I drink coffee if I have NAFLD?9Can I smoke if I have NAFLD?10How much exercise should I do if I have NAFLD?11Which type of exercise is better for NAFLD?12Should I do cardio or lift weights for NAFLD?13Which drugs should I take if I have NAFLD?14How do I understand if NAFLD is worsening?15How do I understand if NAFLD is improving?ChatGPT, Chat Generative Pretrained Transformer; NAFLD, nonalcoholic fatty liver disease. Open table in a new tab ChatGPT, Chat Generative Pretrained Transformer; NAFLD, nonalcoholic fatty liver disease. On March 25, 2023, 3 users from Milan, Italy (N.P., C.H., and A.A.) separately and independently entered the questions into ChatGPT using the “New Chat” function.8ChatGPT.https://chat.openai.comDate accessed: March 25, 2023Google Scholar The responses generated by ChatGPT were recorded and distributed to 10 key opinion leaders (KOLs) in NAFLD and 1 nonphysician with expertise in patient advocacy in liver disease (Supplementary Table 1). Each KOL was asked to independently rate the AI-generated responses using the Likert scale for accuracy, completeness, and comprehensibility (Supplementary Table 2).9Likert R. Arch Psychol. 1932; 22: 1-55Google Scholar The nonphysician KOL was asked to rate the responses using the Likert scale for comprehensibility only. The results of the survey were analyzed using descriptive statistics and concordance measures (Supplementary Methods). The mean accuracy score was 4.84 ± 0.74 on a Likert scale ranging from 1 to 6, with questions 10 and 11 receiving the highest scores (5.6 ± 0.51 and 5.6 ± 0.69, respectively). Question 7 had the lowest mean score (3.3 ± 2). Questions 2 and 10 were the only 2 questions (13%) in which all KOLs scored the answer as 5 (nearly all correct) and 6 (correct). The physical activity domain had the highest mean score (5.56 ± 0.56), while the specialist referral domain had the lowest mean score (3.9 ± 1.44). Kendall’s coefficients of concordance ranged from 0.167 to 0.568 with a mean of 0.477, indicating a moderate level of agreement among KOLs (Figure 1A). The average completeness score was 2.08 ± 0.3 on a Likert scale of 1 to 3. Question 10 had the highest score of 2.6 ± 0.51, while question 1 had the lowest score of 1.5 ± 0.71. Only 5 responses (questions 6, 8, 10, 11, and 12) received unanimous evaluation scores of 2 and 3. Looking at the domains, the physical activity domain had the highest mean score (2.46 ± 0.5), while the specialist referral domain had the lowest score (1.73 ± 0.82). Kendall’s coefficients of concordance ranged from 0.371 to 0.616 with a mean of 0.471, indicating a moderate agreement among KOLs (Figure 1B). The results of the survey showed that the overall comprehensibility rating of ChatGPT's responses to the 15 NAFLD questions was high, with an average score of 2.87 ± 0.14 on a Likert scale of 1 to 3. Seven questions (questions 2, 3, 5, 8, 9, 10, and 12) received a unanimous score of 3 (easy to understand). The mean Kendall's coefficient of concordance across all 15 questions was 0.822, indicating a high level of agreement among the KOLs. Comprehensibility also was assessed by a nonphysician who rated 13 questions as 3, while questions 11 and 12 were rated as partly difficult to understand (Figure 1C). AI is expected to transform everyday clinical practice and chatbots provide an opportunity for real-time patient counseling. We conducted an evaluation of ChatGPT's performance in answering 15 NAFLD-related questions. The results showed that the answers were accurate (mean score, 4.84 on a 6-point scale), complete (mean score, 2.08 on a 3-point scale), and easy to understand (mean score, 2.87 on a 3-point scale). It is important to recognize that responses generated by ChatGPT may vary owing to training data, context, and language nuances. Questions asked in different contexts or at different times may yield different answers, potentially affecting the accuracy and completeness of the information provided. The language used also may affect the answers generated. Therefore, potential variability in ChatGPT responses must be taken into account when interpreting study results. Caution also should be exercised when relying solely on ChatGPT for patient counseling. The study highlights a case in which the answer generated by ChatGPT regarding the use of herbal remedies in NAFLD was considered potentially harmful and incorrect by the majority of the KOLs.10Li X. et al.Liver Int. 2022; 42: 1999-2014Google Scholar This example highlights the importance of validation and verification by health care professionals to ensure the accuracy and safety of information provided by the chatbot. We acknowledge the limitations of the study, such as a small cohort of physicians providing subjective ratings and the specific focus on ChatGPT 3.5, which may not represent the performance of other AI models, particularly those with medical-specific training. It also emphasizes that although ChatGPT cannot replace medical consultations for patients with NAFLD, efforts should be made to improve the reliability of AI-provided medical information through collaboration between health care professionals, patient associations, and medical literature databases. The aim is to improve the accuracy and completeness of AI models while maintaining their accessibility and ease of understanding for patients. In conclusion, although ChatGPT shows promise in providing information for NAFLD management, its use as a universal patient counseling tool should be approached with caution. Future research should address the identified limitations and work toward improving the reliability and usefulness of AI models in providing accurate and comprehensive medical information to patients. Contributors as part of the NAFLD Expert Chatbot Working Group: Laurent Castera,8,9 Cesare Hassan,10,11 Pinelopi Manousou,12,13 Luca Miele,14,15 Raquel Peck,16 Salvatore Petta,17 Luca Valenti,18,19 and Zobair M. Younossi20,21 8Université Paris Cité, UMR1149 (CRI), INSERM, Paris, France; 9Service d'Hépatologie, Assistance Publique-Hôpitaux de Paris, Hôpital Beaujon, Clichy-la-Garenne, France; 10Division of Gastroenterology and Digestive Endoscopy, Humanitas Research Hospital–Istituto di Ricovero e Cura a Carattere Scientifico, Rozzano (Milan), Italy; 11Department of Biomedical Sciences, Humanitas University, Pieve Emanuele, Milan, Italy; 12Imperial College Healthcare NHS Trust, London, United Kingdom; 13UCL Institute for Liver and Digestive Health, Division of Medicine, London, United Kingdom; 14Department of Scienze Mediche e Chirurgiche, Fondazione Policlinico Gemelli Istituto di Ricovero e Cura a Carattere Scientifico, Rome, Italy; 15Department of Medicina e Chirurgia Traslazionale, Università Cattolica Del Sacro Cuore, Rome, Italy; 16Clarion I, London, United Kingdom; 17Section of Gastroenterology and Hepatology, PROMISE, University of Palermo, Palermo, Italy; 18Precision Medicine Laboratory, Biological Resource Center, Department of Transfusion Medicine, Fondazione Istituto di Ricovero e Cura a Carattere Scientifico Ca’ Granda Ospedale Maggiore Policlinico, Milan, Italy; 19Department of Pathophysiology and Transplantation, Università degli Studi di Milano, Milan, Italy; 20Department of Medicine, Center for Liver Diseases, Inova Fairfax Medical Campus, Falls Church, Virginia; and 21Betty and Guy Beatty Center for Integrated Research, Inova Health System, Falls Church, Virginia. Descriptive statistics were used to calculate the mean and SD of the responses for each question. Measures of concordance, specifically Kendall's coefficients of concordance, were used to assess the level of agreement among the KOL responses. Kendall's coefficient of concordance is a nonparametric measure of agreement that accounts for both the magnitude and direction of the differences among raters. A coefficient of 1 indicates perfect agreement, while a coefficient of 0 indicates no agreement beyond chance. The SD was used to assess the variability of responses across the KOLs for each question. All statistical analyses were performed using NCSS 10 statistical software (NCSS).Supplementary Table 1Questions Posed to ChatGPTINPUT: I have been diagnosed with NAFLD. Should I be referred to a hepatologist? Should I undergo additional blood tests or radiologic examinations?OUTPUT: If you have been diagnosed with NAFLD, it is recommended that you see a liver specialist, such as a hepatologist, especially if you have advanced liver disease, such as nonalcoholic steatohepatitis (NASH) or cirrhosis. The hepatologist can help determine the severity of your condition and develop a personalized treatment plan. In terms of blood tests, your doctor may recommend liver function tests to assess the level of liver enzymes in your blood. These tests can help evaluate the degree of liver damage and monitor the progression of the disease. Additionally, your doctor may check your cholesterol and triglyceride levels because these are often increased in people with NAFLD. In terms of imaging tests, your doctor may recommend an abdominal ultrasound, which uses sound waves to create images of the liver. This test can help evaluate the amount of in your liver and detect such as liver or on the severity of your NAFLD, your doctor may also recommend a liver which taking a small of liver for a This test can help determine the degree of liver damage and the of or the specific tests and recommended for your NAFLD on your and the severity of your It is important to follow your and to monitor your condition and to your treatment much weight should I lose if I have NAFLD is a condition in which is an of in the which can to liver damage and other health often is recommended as a treatment for NAFLD because it can help liver and improve liver The amount of weight that you should aim to lose if you have NAFLD on such as your composition, and overall In a weight of of your weight can improve liver health and the severity of NAFLD. if you a weight of may be it is important to that the amount of weight to improve NAFLD may vary from to and it is to with a health care or a who can provide personalized on your specific also can help you develop a and effective weight that into account health or you may be diet should I follow if I have If you have NAFLD, a diet is an part of your treatment plan. are to (1) your if you are or weight can help improve NAFLD. should aim to lose weight 1 to 2 by your (2) replace and in your diet with such as fatty in and and in and (3) your foods such as and can help improve NAFLD. your of and much and such as and can to NAFLD. if you do not have liver disease, it is important to avoid alcohol if you have NAFLD. a a can help you develop a personalized diet that your and NAFLD. to with your health care should I avoid if I have If you have NAFLD, it is important to avoid foods that may your are foods to (1) and foods high in and such as fatty and can to the and progression of NAFLD. (2) and much and such as and can to and the of NAFLD. (3) small of alcohol can damage the liver and NAFLD. foods such as and often high of and and can to NAFLD. such as and can to NAFLD because of their foods that have a high such as and can and to the of NAFLD. that are and it is important to with your health care or a to develop a personalized that your and I drink alcohol if I have it is not recommended to drink alcohol if you have NAFLD. small of alcohol can damage the liver and NAFLD. is by the and alcohol can to and of the which can NAFLD. In people with NAFLD are at a of liver disease if drink in moderate liver disease can liver damage and such as liver if not Therefore, it is recommended that with NAFLD avoid alcohol to damage to the liver. If you are with alcohol or it is important to help from a health care or has positive on There are foods that have positive on NAFLD and may help improve liver are and a of and can help improve liver health because of their high and to a of and to the of and such as and complex and which can help improve and the of NAFLD. of such as and such as and These provide and help to of into your diet such as fatty in fatty and and in and have been shown to have a positive effect on liver have shown that coffee may help the of NAFLD, as as improve liver I use herbal remedies for herbal remedies may promise in improving liver function and with NAFLD, it is important to with your health care using herbal remedies because may with other or you are are of herbal remedies that have been for their potential in (1) is a that has been used to liver It a which has and have shown that may help improve liver function in people with NAFLD. (2) has been used as a and liver It has been shown to have and and that it may help improve liver function in people with NAFLD. (3) is a that a which has and have shown that may help improve liver function in people with NAFLD. which have and have that may help liver and in people with NAFLD. it is important to with your health care using herbal remedies or to NAFLD because may not be or effective for It is important to work with a health care to develop a comprehensive treatment that your I drink coffee if I have NAFLD is a condition in which in the liver. If you have been diagnosed with NAFLD, it is important to take to improve your liver In moderate coffee has been with a of liver disease, NAFLD. This may be owing to the of in such as and that have been shown to have on liver it is important to that responses to coffee may and also can have on the liver. Additionally, if you or to your it can your of and to weight which can NAFLD. Therefore, if you have NAFLD and you it is to in it is a to with your health care about your specific condition and or may I smoke if I have It is not recommended to smoke if you have NAFLD. can liver damage by increasing and in the liver. also can which is a for NAFLD and can to the progression of the disease. In is a for other health that can liver such as disease and can improve overall health and the of these If you are to it is important to help from a health care or can provide and support to help you and improve your liver much should I exercise if I have can be an effective to improve liver health and NAFLD. The College of Gastroenterology at of exercise for people with NAFLD. This can be into of exercise on of the exercise such as or it is important to and the and of exercise to or If you are new to exercise or have medical that may your ability to it is important to with your health care an exercise training such as or also can be in improving liver health and NAFLD. It is recommended to at a physical activity should be part of a comprehensive treatment that weight and management of other health such as or high It is important to work with your health care a to develop an exercise that your and physical exercise is better for exercise and training are both for NAFLD. is a of each type of exercise and it can help improve liver (1) also as that rate and and exercise has been shown to improve and liver and in people with NAFLD. (2) also as training or using weights or to and training has been shown to improve and liver in people with NAFLD. A of both of exercise is recommended for results. The College of Gastroenterology at of exercise and at 2 of training for people with NAFLD. It is important to and the and of exercise to or If you have medical or about it is important to with your health care an exercise Working with a also can help you develop a and effective exercise that your and Should I do cardio or lift weights for cardio and can be for NAFLD, it is recommended to both of exercise into your such as or can help improve and liver and in people with NAFLD. It is recommended to aim for at of exercise such as or can help improve and liver in people with NAFLD. It is recommended to aim for at 2 of training It is important to and the and of exercise to or If you have medical or about it is important to with your health care an exercise Working with a also can help you develop a and effective exercise that your and drugs should I take if I have an AI language I cannot provide medical or NAFLD is a medical condition that and treatment from a health care a or for NAFLD lifestyle changes such as weight loss, and a In also may be by a health care to such as high or high blood It is to with your health care taking because be to recommend the treatment for your and medical do I understand if NAFLD is NAFLD can to a condition nonalcoholic steatohepatitis (NASH) or advanced or liver in Therefore, it is important to monitor the condition and medical attention if are of are to understand if NAFLD is (1) NAFLD may not in the as the disease may such as abdominal and If you of these it is important to your (2) liver function tests can measure levels of liver enzymes that are increased in NAFLD. If these levels are increasing it that the disease is (3) ultrasound, and imaging can provide images of the liver to assess its and If the liver or has a it that the disease is a liver can the of or advanced It a small of liver for a If the shows liver damage it indicates that the disease is changes to your such as a and can help improve NAFLD. If you are not with these changes or your condition these it is important to with your If you have NAFLD, it is important to work with your doctor to monitor your condition and detect of doctor can recommend the tests and to help the disease and it from do I understand if NAFLD is NAFLD is a condition in which is an of in the liver that is not by alcohol NAFLD can to nonalcoholic steatohepatitis (NASH) and can to liver and liver if understand if NAFLD is can be (1) blood tests such as liver function tests, and blood can help monitor the of NAFLD. If these tests an in liver a in triglyceride and better blood it indicates that the liver is (2) imaging such as ultrasound, or can provide information about the degree of liver and liver If is a in liver and it that the liver is (3) is a that measures the of the liver. in liver that the liver is and lifestyle changes such as a increasing physical and alcohol can help improve NAFLD. If these changes in weight and better blood it indicates that NAFLD is It is important to that NAFLD can take to and the rate of can vary from to evaluation with a health care is to monitor and treatment as Chat Generative Pretrained Transformer; NAFLD, nonalcoholic fatty liver disease. Open table in a new tab Table rating by a 6-point Likert incorrect and all rating by a 3-point Likert of the are or all of the question and the amount of information to be considered all of the question and additional information or beyond was rating by a 3-point Likert to difficult to to understand Open table in a new tab ChatGPT, Chat Generative Pretrained Transformer; NAFLD, nonalcoholic fatty liver disease. ChatGPT in Medical The to and Gastroenterology and have with the and of Medical for Nonalcoholic Liver performance in addressing questions to nonalcoholic fatty liver disease as recently metabolic dysfunction–associated steatotic liver disease. the study is the increased of language models for clinical are regarding the used to the accuracy and the of variability for ChatGPT responses. Artificial for and for Gastroenterology and with the study by et that evaluated the performance of Chat Generative Transformer in addressing queries about nonalcoholic fatty liver disease, renamed as metabolic steatotic liver disease. The are indicating that intelligence chatbots patient We to 2 regarding patient education that

Ähnliche Arbeiten