Building Reliable RAG Production Systems Through Continuous Evaluation

then you've already gone through this situation where it seems like your RAG system is working fine but it's still not working. Retrieval fetches certain components, passes them to the generating model, and writes a smooth response. There is nothing wrong with it, however the answer may be based on the wrong document, or missing half the information it needed, or based on something technically correct but three versions out of date. The only way to know and reduce this is to check regularly.

This article walks you through how to build that test pipeline yourself, starting with the parts that feel most important to write about, through automated testing with RAGAS, past the point where RAGAS stops being enough, and what it takes to do this as a real process rather than a notebook you open before a stakeholder meeting. The steps are written so that you can follow them in your RAG application, not just read them as theory.

This is part 4 RAG for Enterprise seriesand if you missed the previous parts I would strongly recommend you to check out part 3 here: Hybrid Search and Repositioning in Production RAG

What is in this article

1. Building a Golden Dataset
2. Easy pass check before automation
3. Make goals automatically with RAGAS
4. Add a custom LLM judge to what RAGAS can check
5. Using a person in a loop
6. View the drift after deploying the system
7. Running it like a pipe
8. The conclusion

Building a Golden Dataset

Before you contact any test library, you need a set of questions with known correct answers. This is called a gold data setand to avoid a common mistake that teams make while working on the RAG system. Jumping straight into using RAGAS on random questions with nothing to compare it to doesn't tell you anything about whether the system is right or wrong.

A good golden dataset entry has three parts: a question, an appropriate answer written by someone who knows the domain, and any text that contains the answer. The third field is the one that allows you to tell the retrieval failure (wrong episode downloaded) without the generation failure (correct episode, wrong answer written to it). These are different bugs with different fixes, and a dataset that doesn't have this field will leave you unaware that you're fixing a bug.

golden_set = [
    {
        "question": "What is the maximum file size for uploads?",
        "ground_truth": "25 MB per file on the free plan, 200 MB on paid plans.",
        "source_doc": "upload_limits.md",
        "category": "single_fact",
    },
    {
        "question": "Can I cancel my subscription mid-cycle and get a refund?",
        "ground_truth": "No, cancellations take effect at the end of the billing cycle. No partial refunds.",
        "source_doc": "billing_policy.md",
        "category": "single_fact",
    },
]

Twenty to thirty questions are enough to get a real signal at first. The stage of the division is more important than the number of entries, and this is often where teams have invested the least. A dataset created by simple fact-checking will pass everything and tell you nothing, because simple fact-checking is one failure mode that RAG programs rarely have. The sections that should be added, in the order they usually bite into production:

• Multi-hop – the answer requires two or more pieces combined, which is where retrieval slows down
• No response is expected – the correct behavior is to refuse to answer, not to guess from the nearest logical passage
• Conflicting or outdated documents – the old and current version of the same policy exists in the corpus, and only one is correct
• A negative expression – the same question asked in different words than those used in the source text

The third category “Conflictual or ancient documents” is the most skipped category in the gold sets, and it is the one that often produces the most production cases about a smooth, well-cited, but completely constructed answer to a document that no one has ever had access to in order to be archived. If your corpus accumulates old versions of things (many enterprise document stores do), your eval set needs to check for it, or your pipeline won't see it like your reviewers did.

Easy pass check before automation

Use the golden set in your pipe and read the answers near the ground truth before setting up any scoring tool. This step is often skipped because it feels too basic to be a real engineering exercise but it tells you what your dataset itself sounds like, and gives you a sense of what “wrong” looks like in your particular system before you trust the metric to do it for you.

results = []
for item in golden_set:
    answer = rag_pipeline.query(item["question"])
    results.append({
        "question": item["question"],
        "generated_answer": answer,
        "ground_truth": item["ground_truth"],
        "correct": None,  # fill in manually
    })

This catches embarrassing failures early like a broken data template, a finder that doesn't return anything, a model that completely ignores the context, etc. Automatically ignoring broken pipe points gives you an accurate number for something that wouldn't be useful.

Make goals automatically with RAGAS

Once the basics are in place, RAGAS gives you a quick, repeatable way to pipe in four dimensions without having to learn every answer by hand.

from ragas import evaluate
from ragas.metrics import (
    context_precision,
    context_recall,
    faithfulness,
    answer_relevancy,
)
from datasets import Dataset

eval_dataset = Dataset.from_list([
    {
        "question": item["question"],
        "answer": rag_pipeline.query(item["question"]),
        "contexts": rag_pipeline.retrieve(item["question"]),
        "ground_truth": item["ground_truth"],
    }
    for item in golden_set
])

result = evaluate(
    eval_dataset,
    metrics=[context_precision, context_recall, faithfulness, answer_relevancy],
)
print(result)

• Content accuracy – of the returned sections, how many were accompanied by a punitive sound
• Remembering the content – estimates that the retrieval has captured the information needed to produce the indicative response
• Honesty – the answer is based on what was returned, to catch the hallucination
• Answer compatibility – does the answer address the actual question, not an approximation

A typical run looks like this (numbers to describe each metric):

A high reliability score is read as “the answer is correct,” and that is a poor reading that allows for bad answers. Reliability only checks whether or not the answer is supported by the returned context, it doesn't mean that the context was the right thing to retrieve. An old or out-of-date document, faithfully summarized, produces an answer that is both fully justified and completely wrong at the same time. This is the roof of the RAGAS structure: it is very good at catching false ideas and very bad at catching the wrong source with confidence.

Add a custom LLM judge to what RAGAS can check

Most of the RAGAS metrics rely on LLM-based evaluations using pre-defined information that is not aware of your background. If your accuracy depends on something specific such as exact figures, recency of source, required disclaimer, tone, etc. An automatic notification will not catch you, because it was not asked to do so.

To solve this problem you can use LLM as a judge and pass a custom command to it telling it to judge the responses based on your specific needs.

import json
from google import genai
from google.genai.types import GenerateContentConfig

client = genai.Client(
    vertexai=True,
    project="YOUR_GCP_PROJECT_ID",
    location="us-central1",
)

JUDGE_PROMPT = """
Compare the generated answer to the ground truth. Score 1-5 on each dimension.

Question: {question}
Ground Truth: {ground_truth}
Generated Answer: {answer}
Source Document Date: {doc_date}

1. numeric_accuracy - are all numbers and facts correct, not just plausible?
2. recency_awareness - if the source is outdated, does the answer flag
   uncertainty instead of stating it as current fact?

Return JSON only:
{{
    "numeric_accuracy": int,
    "recency_awareness": int,
    "reasoning": str
}}
"""

def custom_judge(question, ground_truth, answer, doc_date, client):
    prompt = JUDGE_PROMPT.format(
        question=question,
        ground_truth=ground_truth,
        answer=answer,
        doc_date=doc_date,
    )

    response = client.models.generate_content(
        model="gemini-2.5-flash",
        contents=prompt,
        config=GenerateContentConfig(
            temperature=0,
            max_output_tokens=250,
            response_mime_type="application/json",
        ),
    )

    return json.loads(response.text)

Do not use this on a full dataset all the time because it is slower and more expensive per call than RAGAS. It has reached the stages where the RAGAS test is limited, which means that by doing so conflicting_docs again no_answer_expected. Driving it everywhere pays for the precision you don't need in questions RAGAS has already earned points for reliability.

Using a person in a loop

Both RAGAS and the custom judge are still LLMs who get another LLM result, and they will not always agree with the person. The reliable number to know here, which most teams never measure, is how often your judge agrees with someone. Actually, the LLM-judge-to-person agreement in the 80% low to medium range is normal, not a sign that something is broken, but the real ceiling is worth knowing rather than thinking.

def needs_human_review(ragas_score, judge_score, threshold=1.0):
    return abs(ragas_score - judge_score) > threshold

This keeps the human review line small and focused, only in cases where the two scoring methods disagree, not all answers. It is also occasionally appropriate for two people to find answers to the same parameters independently. If domain experts disagree about a third of the time on a particular question type, that's not a scoring pipeline problem the ground truth itself is unclear, and no amount of tools fix that. It usually means that the entries of the gold set must be rewritten, not the judge.

View the drift after deploying the system

A pipeline that only runs against a fixed gold set has a blind spot because the live corpus changes beneath it. New documents are being added, old ones are being archived, and real user questions are moving away from whatever was initially tested. None of this is reflected in the dataset that was created on the date you type.

import random
from datetime import datetime, timedelta

def sample_production_queries(logs, n=50, days=7):
    recent = [q for q in logs if q["timestamp"] > datetime.now() - timedelta(days=days)]
    return random.sample(recent, min(n, len(recent)))

Sampling a small piece of live traffic every week and using it in the context of RAGAS accuracy and reliability, both of these metrics work without ground truth feedback, gives you a second signal. A sudden crash without a corresponding code change usually means something changed in the corpus, not the pipeline, and it's a different failure mode than anything a compile-time check will ever catch.

Running it like a pipe

Two things make it a real pipe instead of a single exercise: cost-conscious planningand a CI gateway.

Running the full stack: RAGAS, a custom judge, and a human review of every single commit is expensive enough that many teams quietly stop doing it within a month. Combining it works best in practice:

• RAGAS in the full gold set: all pull requests that affect retrieval or information
• Custom judge in flagged categories only: each pull request, reached in a few examples
• Human review: weekly, on the disagree line only
• Production samples: weekly, on a rotating piece of live traffic

def check_regression(current_scores, baseline_scores, threshold=0.03):
    regressions = [
        (metric, baseline_scores[metric], score)
        for metric, score in current_scores.items()
        if baseline_scores[metric] - score > threshold
    ]
    if regressions:
        raise SystemExit(f"Blocking merge — regressions found: {regressions}")
    print("No regressions. Safe to merge.")

Wire this into CI to apply to every pull request that affects retrieval, aggregation, or information, and let it fail the build if any metric drops past your threshold. This is the most important part of the whole setup, and it's also the one that prevents accidents. For example, a big change that silently breaks the query section is caught here, before it becomes a support ticket three weeks later instead of a failed check today.

The conclusion

None of the above six steps are impressive on their own when used independently, but what makes the difference is using them together, on a schedule, as something the team trusts rather than something one person remembers once in a while. That's the purpose of this article, from starting the test as a one-time gut test to making it part of the RAG system infrastructure, sitting quietly in CI and holding the change that was about to be deployed.

If you're starting out with nothing, don't try to build all six steps at once. The golden data set with twenty good questions and the RAGAS score that you check manually is already ahead of most RAG systems produced today. More can be added as your system and patience for reading the answers by hand grows.