Enhancing ChatGPT-Based Writing Research Through Effective Prompt Use

Abstract

An increasing number of studies have investigated how ChatGPT can aid in written assessment and feedback provision. However, many studies overlook its conversational design and underlying architecture, raising concerns about the reliability and validity of their analytical outputs. Therefore, applying first principles thinking to prompt use, and through a case study approach, we explore some of the challenges in using ChatGPT to identify logical fallacies in student essays. Specifically, we examine the outputs from two kinds of interactions: (1) multiple iterations of a carefully designed prompt across new conversation windows, and (2) a single-shot application of the designed prompt followed by two rounds of follow-up prompts. Results show that combining outputs from multiple iterations of the same prompt can enhance precision and accuracy, and that follow-up prompts can generate a more exhaustive analysis, but there is also a risk of diminishing validity with excessive follow-ups. Therefore, to increase the validity and reliability of generative-AI outputs, we argue that in addition to rigorous prompt design, researchers and teachers need to systematically pilot and tailor the use of prompts to their specific research contexts or task needs. With this in mind, we provide some preliminary guidelines for prompt use.

Ähnliche Arbeiten