Evaluation of an AI-Based Semantic Chatbot for Quran and Hadith Retrieval Using Expert Validation and NLP Performance Metrics

Abstract

This study presents the design and evaluation of an artificial intelligence (AI)-based semantic chatbot developed to retrieve Quranic verses and Hadith references in response to thematic user queries. The system integrates a natural language processing (NLP) pipeline to interpret user intent and match it with a curated corpus of authenticated Islamic sources using a hybrid semantic retrieval approach based on TF-IDF and BERT embeddings. Evaluation was conducted through two complementary approaches: expert validation and technical performance measurement. Expert validation involved two domain specialists in Al-Qur'an and Hadith sciences who rated 30 thematic queries using a 4-point Likert scale across three criteria: authenticity, thematic relevance, and contextual accuracy. The technical evaluation used standard NLP metrics, including Precision, Recall, and F1-Score. The dataset consisted of 6,236 Quranic verses and 14,838 Hadiths from Sahih Bukhari and Sahih Muslim in Bahasa Indonesia. Results demonstrated a mean Precision of 0.78, Recall of 0.98, and F1-Score of 0.84, indicating the chatbot's high accuracy and retrieval comprehensiveness. Expert assessments further confirmed strong reliability, with an average validation score exceeding 85% across all criteria. These findings highlight the potential of semantic AI systems in supporting digital Islamic learning environments and automated religious information services. Future work will extend semantic understanding to multilingual queries and incorporate user-centered evaluation and broader tafsir integration to enhance contextual reasoning.

Ähnliche Arbeiten