Assessing the Maturity of Generative AI Systems: A Framework for Education and Public Engagement

Abstract

The integration of generative AI tools in educational settings, exemplified by chatbots, presents novel opportunities for personalized learning while simultaneously posing challenges related to trust, ethics, and technical preparedness. This study introduces ALES (Academic Learning Engagement System), a university chatbot, alongside a comprehensive framework for its design, implementation, and evaluation. The framework incorporates international standards, including ISO/IEC 25010, TRL, CMMI, and GQM, to ensure quality, scalability, and alignment with institutional requirements. ALES integrates a fine-tuned transformer model, retrieval-augmented generation, and a modular infrastructure. Its evaluation encompasses six critical dimensions: accuracy, usability, performance, reliability, ethics, and educational impact. A five-level maturity model guides the development process from prototype to full deployment. Empirical findings indicate 90% accuracy, high user satisfaction (SUS > 80), and positive learning outcomes. The ALES case study underscores the value of a structured, iterative approach to the responsible implementation of AI in education and offers a transferable model for broader adoption.

Ähnliche Arbeiten