Measuring accuracy of AI generated definitions using cosine similarity index across select GPT models

Abstract

Generative Pre-trained Transformers (GPT) has ushered in a transformative shift in the landscape of automated text generation. These models use artificial neural networks to produce responses as per human prompts. These GPTs are now widely used in many NLP tools such as ChatGPT, Gemini, Pi, and Perplexity. As these systems become more common in academic and knowledge-based work, it is important to carefully examine the authenticity and reliability of their outputs. This study investigates the text-generation capabilities of these GPT-based tools by employing the cosine similarity index to compare their outputs against ground truth definitions sourced from the Britannica Encyclopedia. Definitions of selected concepts in science and social science were analyzed to ascertain the alignment of GPT-generated texts with established knowledge. The results show clear differences in how closely the four GPT-based systems matched Britannica definitions. Perplexity achieved the highest similarity with the least variability, followed closely by Pi and ChatGPT, while Gemini consistently produced the lowest similarity scores. The analysis of variance confirmed the statistical significance of these differences. The post hoc testing showed that Gemini formed a separate group due to its lower alignment with ground-truth definitions. These findings highlight that although GPT tools provide reasonably accurate information, their outputs still vary from ground truths. Overall, Perplexity Pi, and ChatGPT were closer to ground truth than Gemini, indicating the need for further improvement in Gemini’s performance.

Ähnliche Arbeiten