Can Chatbots Build CTFs? A Preliminary Assessment of LLMs in Jeopardy-Style Challenge Generation

Abstract

Jeopardy-style Capture- The-Flag (CTF) challenges play a central role in cybersecurity education and training; however, their creation remains a resource-intensive and tech-nically demanding process. This paper investigates the capability of general-purpose large language models (LLMs)—specifically ChatGPT, Gemini, Copilot, Claude and DeepSeek—to automate the generation of CTF challenges. We prompt each model to create full tasks, including titles, descriptions, artifacts, and so-lution write-ups, across core categories such as web exploitation, cryptography, and reverse engineering. The generated challenges are evaluated using criteria including technical correctness, solv-ability, clarity, creativity, and write-up quality. Our results high-light significant variability in the outputs, reflecting differences in how well each model interprets prompts, handles technical nuance, and structures complete scenarios. This preliminary study demonstrates both the potential and current limitations of using LLMs in CTF design, providing a foundation for more specialized tools aimed at automated cybersecurity challenge generation.

Ähnliche Arbeiten