Chinese Stylistic Competence: Evaluation Method and Datasets of Large Language Model's Performance

Abstract

Stylistic competence is an important pragmatic competence, and adequate stylistic competence is required for large language model (LLM) to land in language life. In this paper, stylistic competence is defined as the ability to use appropriate style for communication in a specific register, and based on this, three tasks of stylistic classification, stylistic generation, and stylistic transformation are designed to evaluate the Chinese stylistic competence of LLMs represented by ChatGPT. It is found that LLMs have their own advantages and limitations in different tasks and styles. GPT-4 demonstrates the most comprehensive and excellent Chinese stylistic competence, ChatGPT3.5 and ERNIE Bot have better performance, ChatGLM-6B and SparkDesk have unstable performance and notable shortcomings, with their overall abilities being somewhat lackluster. In addition, the degree of informality of the texts generated by each model is relatively limited, the literary grace is ordinary, and problems such as consistency errors, normative errors, factual errors, illogicality, insufficient sentence fluency, and obvious traces of machine translation still exist. For LLM, it should be viewed from an instrumental perspective, expanding its stylistic competence with rich and diversified stylistic data resources and technological advances, and at the same time reasonably utilizing and giving full play to its stylistic resources attribute, to make it better serve the language life.

Ähnliche Arbeiten