Facilitating Student Engagement and Learning Outcomes with GPT-based Chatbots: Evidence from a Randomized Controlled Trial in Afghanistan

Abstract

The integration of Generative AI (GenAI) technologies, such as ChatGPT, into online education is accelerating; however, their effectiveness in under-resourced contexts remains insufficiently studied. This paper investigates the impact of a Large Language Model (LLM)-based conversational agent on student engagement and learning outcomes in Afghanistan, where access to formal education—particularly for women—is severely restricted or banned. We conducted an experimental study involving 80 undergraduate computer science students (40 male, 40 female) in Afghanistan, randomly assigned to control and treatment groups. All participants attended identical 50-minute online lectures followed by 40-minute post-lecture discussions moderated by a human instructor ob both Day 1 and Day 2. A follow-up self-report questionnaire was completed by participants at the end of Day 2. The treatment group additionally engaged in AI-facilitated discussions using a GPT-4-based chatbot during post-lecture discussion. Analysis of discussion logs and post-intervention surveys revealed that the treatment group demonstrated impressively higher participation rates, with more posts and replies, during post-lecture discussion and reported greater confidence in their understanding of the course material. These findings underscore the potential of GPT-based chatbots to enhance interactive learning and foster educational inclusion, particularly for marginalized populations in low-resource environments.

Ähnliche Arbeiten