Towards Robust Retrieval-Augmented Generation Based on Knowledge Graph: A Comparative Analysis

Abstract

Retrieval-Augmented Generation (RAG) was first introduced to enhance the capabilities of Large Language Models (LLMs) beyond their encoded-prior knowledge. This is achieved by providing LLMs with an external source of knowledge, which helps to reduce factual hallucinations and enables the access to new information, typically not available during their pretraining phase. Despite its benefits, there is an increasing concern with the impact of inconsistent retrieved information towards LLMs’ responses. Hence, the Retrieval-Augmented Generation Benchmark (RGB) was introduced as a new testbed for RAG evaluation, meant to assess the robustness of LLMs towards inconsistency in the retrieved information. In this work, we use the RGB corpus to evaluate LLMs in four scenarios: (1) noise robustness; (2) information integration; (3) negative rejection; and (4) counterfactual robustness. We perform a comparative analysis between the RAG baseline defined by the RGB and variations of GraphRAG, which is a RAG system based on a Knowledge Graph (KG) and developed to retrieve relevant information from large documents. We tested GraphRAG under three customization to improve its robustness. Our approach demonstrates improvements compared to the RGB baseline, providing insights on how to design more reliable RAG systems, tailored for real-world scenarios.

Ähnliche Arbeiten