RAG vs Fine-Tuning Explained: What They Really Do and When to Use Each

I've written a lot about RAG, starting with a Hitchhiker's guide to RAG with the ChatGPT API and LangChain, then exploring various topics related to RAG and AI, such as chunking, hybrid search, reordering, context retrieval, and a three-part series of retrieval quality tests. In other words, we covered a lot of things on the RAG side of things.

What we haven't talked about is clearly another major approach that people reach for when they want to develop an LLM application for a particular domain. That one fine tuning. And especially, we haven't talked about what happens when you put them both aside and try to figure out which one you really need.

If you search for “RAG vs fine tuning” on the Internet, you will find a lot of content that treats this as a competition with a winner. For some, RAG wins because it is cheaper to set up, for others, fine tuning wins because it produces better results, and so on. The problem with this framework is that it is very misleading, since RAG and fine tuning are not a competition that exists to solve different application problems under each I. what they do is actually a need to make a good decision.

So, let's take a look!

🍨 DataCream a newsletter about AI, data, and technology. If you are interested in these topics, register here!

What is RAG and what does it actually do?

If you've followed this thread, you already have a solid idea of ​​RAG. But let's put it again, because the precise definition is important when compared to the next good planning.

Therefore, RAG, or Retrieval-Augmented Generation, is a technique that improves the response of LLM by obtaining relevant external information during prediction and incorporating it into knowledge. The model itself is not changed in any way. What changes is what it sees as input.

The pipeline looks like this:

• First, the external documents (the knowledge base we want to use) are processed into embeds and stored in the vector database.
• When a user submits a query, the query is again converted to an embed, and fragments of the most similar document are retrieved from the database.
• Those components are then passed to the LLM along with the user's query, so the model can generate an answer based on that particular context returned.

And that's it.

Here is a small RAG example using the OpenAI API:

from openai import OpenAI
import numpy as np

client = OpenAI(api_key="your_api_key")

# our tiny knowledge base
documents = [
    "pialgorithms is an AI-powered document management platform.",
    "pialgorithms allows teams to search, extract, and automate document workflows.",
    "pialgorithms was founded in Athens, Greece.",
]

# embed the knowledge base
def embed(texts):
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=texts
    )
    return [r.embedding for r in response.data]

doc_embeddings = embed(documents)

# embed the user query and retrieve the most relevant chunk
query = "Where is pialgorithms based?"
query_embedding = embed([query])[0]

# cosine similarity
def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

similarities = [cosine_similarity(query_embedding, doc_emb) for doc_emb in doc_embeddings]
best_match = documents[np.argmax(similarities)]

# inject retrieved context into the prompt
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {
            "role": "system",
            "content": f"Answer the user's question using only the following context:nn{best_match}"
        },
        {
            "role": "user",
            "content": query
        }
    ]
)

print(response.choices[0].message.content)
# pialgorithms is based in Athens, Greece.

Let's take a moment to understand what's really going on here. Of course, the model doesn't know what the pialgorithms are in its training, but because it retrieved the right piece of document and injected it quickly, the model is able to respond accurately. Information appears outside the model, at query time, and the model itself is not affected.

This is the core of what RAG does: it gives the model access to external information it has not been trained on, dynamically, during forecasting.

And based on how it works, RAG excels in certain types of work, for example:

• Answering questions about documents, databases, or data that the model has never seen
• Staying up-to-date without retraining, as the knowledge base can be updated independently at any time
• It provides trackable, traceable responses, as you know exactly which part of the document was returned
• Managing confidential or proprietary information securely, without including that information in the model

On the other hand, here's what RAG won't do: it won't change the model's behavior, tone, mindset, or performance. If your model tends to be verbose, RAG won't make it any shorter. If it struggles with a certain output format, RAG won't fix it.

What is good planning and what does it actually do?

Fine-tuning is the process of taking a previously trained model and continuing to train it on a new, task-specific dataset, updating its weights in the process. To put it differently, while RAG changes the input of the model, optimization changes the model itself.

Specifically, a basic model such as GPT-4o-mini is pre-trained on a large dataset. Fine-tuning takes that model and applies an additional, shorter training loop to specific examples that fit our specific use case. Those examples are usually in the form of input-output pairs. In this way, the model weights are adjusted to produce outputs similar to these two examples.

Here's what a good programming job would look like using the OpenAI API:

from openai import OpenAI
import json

client = OpenAI(api_key="your_api_key")

# Step 1: prepare training data as a JSONL file
# each example is a conversation with a desired output
training_examples = [
    {
        "messages": [
            {"role": "system", "content": "You are a concise technical assistant. Always respond in one sentence."},
            {"role": "user", "content": "What is a vector database?"},
            {"role": "assistant", "content": "A vector database stores and retrieves data as high-dimensional numerical vectors, enabling fast semantic similarity search."}
        ]
    },
    {
        "messages": [
            {"role": "system", "content": "You are a concise technical assistant. Always respond in one sentence."},
            {"role": "user", "content": "What is chunking in RAG?"},
            {"role": "assistant", "content": "Chunking is the process of splitting large documents into smaller pieces before embedding them, so they fit within model context limits and improve retrieval precision."}
        ]
    },
    # in practice you would want at least 50-100 examples
]

# save as JSONL
with open("training_data.jsonl", "w") as f:
    for example in training_examples:
        f.write(json.dumps(example) + "n")

# upload the training file
with open("training_data.jsonl", "rb") as f:
    training_file = client.files.create(file=f, purpose="fine-tune")

# create the fine-tuning job
fine_tune_job = client.fine_tuning.jobs.create(
    training_file=training_file.id,
    model="gpt-4o-mini-2024-07-18"
)
print(fine_tune_job.id)

When the fine-tuning operation is complete, OpenAI returns a unique model identifier of your new fine-tuned model, in format. ft:base-model:your-org:your-suffix:unique-id. This is now a separate model that resides in your OpenAI account, separate from the base one gpt-4o-mini.

Opened printwe will come back an id in that properly configured model, it looks like this:

ft:gpt-4o-mini-2024-07-18:your-org:your-suffix:abc123

Then we can call it like any other model, by simply passing that identifier to model parameter:

# once the job is complete, use the fine-tuned model
response = client.chat.completions.create(
    model="ft:gpt-4o-mini-2024-07-18:your-org:your-suffix:abc123",
    messages=[
        {"role": "user", "content": "What is prompt caching?"}
    ]
)
print(response.choices[0].message.content)

The difference is that this model has already internalized the behavior we trained it on: in our example, it will now respond consistently with one short sentence, without us instructing it to do so for all system prompts. That's the kind of thing that fine-tuning is good for: consistent formatting, a certain tone, sticking to a certain output structure, or improved performance in a special type of work. And that, in fact, is what good repair is all about.

Notice how fine-tuning has no effect on entering specific information into the model. Contrary to what one might intuitively think, fine-tuning the model in your company's documentation will not make the model “read” that information and answer questions about it reliably. Indeed it may result in the model memorizing some facts from training examples here and there, but this memorization is empty and unreliable. The most likely result will be a model that is misleading about topics from the training examples, rather than a model that accurately remembers specific details from those examples. So, if data recovery is what you need, RAG is the right tool, not a fix.

Specifically, optimization actually does well in the following:

• “Teaching” models a consistent output format, tone, or style
• Improving the performance of a specific type of narrow task (eg always generating valid JSON, always truncation with three dots, etc.)
• Reduce the need for long, repetitive system commands by integrating those commands into the model
• Adapting the model to a specific language or domain terminology, so that it understands and uses the correct vocabulary.

However, fine tuning doesn't do that well in:

• In addition to the reliable information of the truth, the model can accurately recall
• Keeping the model up to date with changing information
• It provides trackable, available answers

So, when do we use each and when do we use both?

Now that we understand what each process actually does, “RAG vs fine tuning” question becomes much easier to answer, because in most cases it is not a “vs” type of question at all.

RAG and fine-tuning work at different layers of the AI ​​system. RAG works at the information layer, which means it controls what information the model has access to. On the other hand, fine-tuning works in the behavioral layer, which means that it describes how the model processes the given information and generates responses. These two layers are independent of each other, meaning you can use RAG, fine tuning, or both, depending on what you're trying to achieve.

So, here's a practical decision framework for deciding what to use:

The situation where we use both RAG and optimization is very common in real production systems. The easiest way to keep the two straight is this: fine-tune the behavior, use RAG for information.

Imagine, for example, that we are building a customer support assistant for a software product, and we need it to:

1. Always respond in a specific tone and format, consistent with our software product
2. Have accurate, up-to-date information on our product documentation

For such a task, we will have to use both RAG and fine tuning. In particular, optimization will address the first requirement by allowing the model to learn from examples of appropriate customer support responses, teaching it the right tone, the right level of detail, and the right output format. The second requirement will be included by the RAG: at a fixed time, the most relevant information from the product documentation is retrieved and included in the notification, allowing the model to provide reliable answers based on the documentation.

So, in practice, we can combine both fine-tuning and RAG by calling the fine-tuned model in the same way we would call any other model, but also injecting the returned context into the system notification, just as we would in a standard RAG pipeline.

# combining fine-tuned model with RAG
response = client.chat.completions.create(
    model="ft:gpt-4o-mini-2024-07-18:your-org:support-style:abc123",  # fine-tuned for tone/format
    messages=[
        {
            "role": "system",
            "content": f"You are a helpful support assistant for pialgorithms. "
                       f"Use only the following documentation to answer:nn{retrieved_context}"  # RAG context
        },
        {
            "role": "user",
            "content": user_question
        }
    ]
)

Fine tuning lets the model know how to respond, and RAG tells it what to say. So, this is not a “fine-tuning vs RAG” question, but rather fine-tuning and RAG complement each other and do different things.

In my mind

What I find most interesting about the RAG vs optimization debate is how often it is framed as a question about which method is better, when the more useful question is what problem you are trying to solve.

RAG and optimization address different failure modes of the underlying LLM. If the basic model fails because it doesn't know something, that's an information problem, and RAG solves it. If the underlying model fails because it behaves differently or produces output in the wrong format, that's a behavior problem, and fine-tuning solves it. If your model fails for both reasons at the same time, you may actually need both.

✨ Thanks for reading! ✨

If you've made it this far, you may find pialgorithms useful: a platform we've been building that helps teams securely manage organizational information in one place.

Did you like this post? Join me on 💌 A small stake and 💼 LinkedIn

All photos by the author, unless otherwise stated.