Evaluating ChatGPT for Automated Creation and Grading of Essay Questions in Higher Education

Abstract

Background As generative artificial intelligence (AI) tools such as ChatGPT become increasingly prevalent in higher education, key questions emerge regarding their effectiveness in supporting classroom learning. Objective This study examined whether an AI system could generate psychology course-specific essay questions, grade responses, and provide feedback, as well as how students perceived the process. Method We developed a custom web app using ChatGPT with retrieval-augmented generation to create and grade essay questions. Students ( N = 37) from an upper-division course at a regional public university completed four AI-graded assessments over six weeks, followed by an exit survey. We compared AI and instructor scores using intraclass correlation coefficients and Bland-Altman analysis. Results Agreement between AI and instructor grades was moderate, with intraclass correlations ranging from .21 to .81, and AI tended to avoid assigning extreme scores. Students valued the instant, personalized AI feedback, but only 35% perceived the AI grading as fair. Conclusion AI tools such as ChatGPT show promise for supporting assessment and feedback, but limitations in perceived fairness may reduce student acceptance.

Ähnliche Arbeiten