Graph rag vs sql rag

I Both a graph database and an SQL database, and use various linguistic models (LLMS) to answer questions about the data in a way to get the money back (Rag). Using the same dataset and queries in both programs, I tested which database parabagm delivers the most accurate and insightful results.

Generation Retrieval (Rag) Is an AI framework that leverages large scale language models (LLMS) by enabling them bring it back relevant external information before generating an answer. Instead of depending on what the model was trained on, rabag is dynamically updated by the source of information (in this article SQL or SQL database or Graph information) and combines those results with its response. An introduction to the rag can be found here.

SQL Details Edit data in tables made up of rows and columns. Each row represents a record, and each column represents an attribute. Relationships between tables are defined using the key and joinsand all details follow fixed type of clauses. SQL Databases are ideal for structured, transactional data where flexibility and accuracy are important – for example, financial, financial, inventory, or patient records.

Graph Databases data data as nodes (businesses) and edges (Relationship) optional properties attached to both. Instead of joining tables, they directly represent relationships, allowing fast tracking of linked data. Graph Databases are ideal for modeling networks and relationships – Like social graphs, knowledge graphs, or molecular communication maps – where connections are as important as the objects themselves.

Text

The dataset I used to compare the performance of rags contains the results of Formula 1 from 1950 to 2024. The standings of drivers and constructors' championships after all races are also included.

Sql schema

This data is already organized into tables with keys so that the SQL database can be easily edited. Database schema is shown below:

Bumps Central table connected to all types of results and other additional information such as Season and Circuits. Results tables are also linked Drivers and They are not builders tables to record their result in each race. The standings of the championship after each race are kept in the Driving_agreement and Umebi_unatiza tables.

Graph Schema

The data schema of the graph is shown below:

Since a data graph can store information about locations and relationships it only needs six nodes compared to the 14 tables of an SQL database. This page A car A Node is an intermediate point used to indicate that a driver has driven a component car in a particular race. Since the driver – the construction pilings change over time, this relationship needs to be defined for each race. Race results are stored in relationships e.g : Race inside A car and The race. While : Smood_After The partnership consists of driving competitions and formations after each race.

Entering a database query

I used Langchain to build a rag chain for both types of data that generates a query based on the user's query, runs the query, and changes the result of the response to the user. The code can be found in this repo. I have described a generic System program that can be used to generate queries for any SQL or graph database. Only specific details of the data are entered by entering the auto-generated schema in the template. System Prompts can be found here.

Here is an example of how you can see the decat chain and ask this question: “Which driver won the 92 Grand Prix in Belgium?”

from langchain_community.utilities import SQLDatabase
from langchain_openai import ChatOpenAI
from qa_chain import GraphQAChain
from config import DATABASE_PATH

# connect to database
connection_string = f"sqlite:///{DATABASE_PATH}"
db = SQLDatabase.from_uri(connection_string)

# initialize LLM
llm = ChatOpenAI(temperature=0, model="gpt-5")

# initialize qa chain
chain = GraphQAChain(llm, db, db_type='SQL', verbose=True)

# ask a question
chain.invoke("What driver won the 92 Grand Prix in Belgium?")

Returns:

{'write_query': {'query': "SELECT d.forename, d.surname
FROM results r
JOIN races ra ON ra.raceId = r.raceId
JOIN drivers d ON d.driverId = r.driverId
WHERE ra.year = 1992
AND ra.name = 'Belgian Grand Prix'
AND r.positionOrder = 1
LIMIT 10;"}} 
{'execute_query': {'result': "[('Michael', 'Schumacher')]"}}
 {'generate_answer': {'answer': 'Michael Schumacher'}}

SQL query joins i The result, Bumpsagain Drivers The tables, you choose the race with the WELGIAN Grand Prix and the driver who finishes first. The LLM was changed in the year 92 to 1992 and the name of the race comes from “Grand Prix in Belgium” to “Belgian Grand Prix”. Found these changes from a data schema that includes three sample rows for each table. The query result is “Michael Schumacher” which is LLM returned as the answer.

Being able to be tested

Now the question I want to answer is if LLM is better in opening SQL or Graph Database. I have defined three levels of difficulty (easy, medium) where easy questions can be answered with one table or the necessary space between the tables or difficult spaces required. For each level of difficulty I have defined five questions. In addition, I defined five questions that can be answered with data from the database.

I answered each question with three LLM models (GPT-5, GPT-4, and GPT-3.5-turbo) to analyze if more advanced models are needed or old and cheap models can create satisfactory results. If the model gave the correct answer, it received 1 point, if it answered that it could not answer the question 0 points, and if it gave the wrong answer. All questions and answers are listed here. Below are scores for all models and data types:

It's surprising how much the advanced models outperformed the simple models: GPT-3-turbo got about half the number of questions wrong, GPT-4 got 2 to 3 questions wrong, and GPT-5 got all but one question wrong. Simpler models seem to perform better with SQL than with graph databases while GPT-5 gets similar scores with databases.

The only question that GPT-5 is wrong using the SQL database is “Which driver has won the most world championships?”. The answer “Lewis Hamilton, and 7 World Championships” is incorrect because Lewis Hamilton and Michael Schumacher won 7 World Championships. The generated SQL query aggregated the number of Championships by the driver, sorted them in order and selected only the first row while the driver in the second row had the same number of circles.

Using the Graphu Database, the only GPT-5 question that was incorrect was “Who won the formula 2 championship in 2017?” Answered with “Lewis Hamilton” (Lewis Hamilton Won formula 1 But not Formula 2 Championship that year). This is a tricky question because the database only contains formula 1 but not the results of formula 2. The expected answer would have been to answer that this question cannot be answered based on the data provided. However, considering that the program did not immediately provide specific information about the database, it is understandable that this question is not well answered.

It is interesting to use the SQL-5 database for SQL-5 gave the correct answer “Charles Leclerc”. The generated SQL query will only search the drivers table for the name “Charles Leclerc”. Here the LLM must have realized that the database does not have the results of formula 2 and answered this question from his general knowledge. Although this led to the correct answer in this case it could be dangerous when the LLM does not use the data provided to answer the questions. One way to reduce this risk would be to state clearly in the system immediately that the database must be the only source for answering queries.

Lasting

This comparison of rag performance using the formula 1 results dataset shows that the latest llms are more efficient, producing more accurate and realistic responses with accurate responses without additional engineering. While simple, new models like the GPT-5 handle complex questions with near accuracy. Importantly, there was no significant difference in performance between graph and SQL Database Methods – Users can simply choose the database paradigm that best fits their data structure.

The dataset used here serves only as an illustrative example; Results may vary when using other datasets, especially those that require domain-specific knowledge or access to non-public data sources. Overall, these findings highlight that far-reaching LLMs have advanced in combining structured data with natural language reasoning.

Unless otherwise stated, all images were created by the author.