Security and Quality in LLM-Generated Code: a Multi-Language, Multi-Model Analysis

Abstract

Artificial Intelligence (AI) driven code generation tools are increasingly used throughout the software development lifecycle to accelerate coding tasks. However, the security of AI-generated code using large language models (LLMs) remains underexplored, and recent studies have revealed various risks and weaknesses. This paper presents a measurement study of LLM-generated code across four programming languages (Python, Java, C++, and C) and five widely used LLM families. We construct a manually curated dataset of 200 programming tasks, grouped into seven functional and security-relevant categories, each with language-neutral specifications. For every combination of task, language, and model, we generate code and evaluate it along three axes: syntactic validity and compilation success, semantic correctness using 4,000 per program unit test files, and software quality and security using SonarQube and CodeQL, complemented by manual review of key static analysis findings. Our results show clear language effects: Python and Java achieve higher compilation and semantic correctness rates and produce fewer security findings than C and C++, where we observe more memory safety issues, hard-coded secrets, and cryptographic misuses. We also find that many models fail to make use of modern security features available in recent compiler and toolkit updates (i.e., in Java 17), and that outdated methods remain common, particularly in C++. These findings highlight the need to advance LLMs so that they better align with emerging secure coding practices and language-specific best practices. All code and data are available at GitHub.

Ähnliche Arbeiten