Comparative evaluation of ChatGPT, Claude, DeepSeek, and Gemini for text-to-style vector generation using RAG-enhanced BERT pipelines

Abstract

This study presents a comprehensive comparative analysis of four major Large Language Models (LLMs) - ChatGPT-4, Claude Sonnet 4, DeepSeek-7B, and Gemini 2.0 Flash - for text-to-style vector generation in design applications. The research evaluates these models within a unified RAG-enhanced pipeline incorporating BERT for semantic vector extraction. Using ROUGE-L F1, Cosine Similarity, BERT Token Count, and Generation Time as evaluation metrics, we assess each model’s capability to process design-oriented prompts and generate semantically coherent style vectors. Our findings reveal that DeepSeek-7B achieved superior performance in both structural alignment (0.27 ROUGE-L F1) and semantic preservation (0.83 cosine similarity), while ChatGPT-4 demonstrated optimal efficiency with the fastest processing time (1.94 seconds) and excellent token utilization (69 tokens). The study emphasizes the critical importance of complete pipeline validation and provides practical guidance for selecting appropriate LLMs based on specific application requirements in creative AI systems.

Ähnliche Arbeiten