Is ChatGPT a Competent Teacher? Systematic Evaluation of Large Language Models on the Competency Model

Abstract

The capabilities of large language models (LLMs) in language comprehension, conversational interaction, and content generation have led to their widespread adoption across various educational stages and contexts. Given the fundamental role of education, concerns are rising about whether LLMs can serve as competent teachers. To address the challenge of comprehensively evaluating the competencies of LLMs as teachers, a systematic quantitative evaluation based on the competency model has emerged as a valuable approach. Our study, grounded in the teacher competency model and drawing from 14 existing scales, constructed an evaluation framework called TeacherComp. Based on TeacherComp, we evaluated six LLMs from OpenAI across four dimensions: knowledge, skills, values, and traits. Through comparisons between LLMs' responses and human norms, we found: 1) With each successive update, LLMs have shown overall improvements in knowledge, while their skills dimension scores have increasingly aligned with human norms. 2) There are both commonalities and differences in the performance of various LLMs regarding values and traits. For instance, while they all tend to exhibit more negative traits than humans, their morals can vary. 3) LLMs with reduced security, constructed using jailbreak techniques, exhibit values and traits more closely aligned with human norms. Building on these findings, we provided interpretations and suggestions for the application of LLMs in various educational contexts. Overall, this study helps teachers and students use LLMs in appropriate contexts and provides developers with guidance for future iterations, thereby advancing the role of LLMs in empowering education.

Ähnliche Arbeiten