Comparing the accuracy andreferencing of ChatGPT’s responses toherbal medicine queries: A zero-shotversus roleplay prompting approach

Abstract

Objectives: This study aims to assess the accuracy of answers generated by ChatGPT-3.5 using zero-shot and roleplay prompts in response to questions about herbal medicine from clinical pharmacists. In addition, we aim to evaluate the quality of text responses when ChatGPT-3.5 is prompted to provide references for the given answers. Materials and Methods: ChatGPT-3.5, developed by OpenAI in San Francisco, is an advanced artificial intelligence chatbot that utilizes a large language model to generate text resembling human-like responses. In this study, a total of 90 questions were posed to ChatGPT-3.5, and the accuracy of its responses was compared using both zero-shot and roleplay prompts. These questions were evenly distributed among fourcategories related to herbal medicine clinical practice. To evaluate the correctness of the responses, they were cross-referenced with peer-reviewed and trusted references. The authenticity of the references provided by ChatGPT-3.5 was also assessed. Results: Using the roleplay prompt with ChatGPT-3.5 led to an 86.67% correct answer rate for herbal medicine questions, compared to 77.78% with zero-shot prompts. When references were requested, 77.48% (zero-shot) and 75.69% (roleplay) were identified as “true” references, but ChatGPT-3.5 itself generated many references. Only 38.37% (zero-shot) and 50.46% (Roleplay) of these “true’ references, provided a sufficient background for accurate answers. Conclusion: In this pilot study, ChatGPT-3.5 with the roleplay prompt shows promise as an alternative tool for addressing herbal medicine questions. However, improvements are needed to enhance reference accuracy and the availability of comprehensive background information.

Ähnliche Arbeiten