Agent RAG: Let Agent Search

the constructive application is the RAG application.

The recipe is simple: chunk, embed, extract, and turn.

It looks clean on paper. But when you use it in real situations, things get complicated very quickly: the same search finds the same words but not useful fragments, the relevant evidence is not visible in the returned context as it is at a very low level, or the important context may be separated between parameters.

In an inadequate context, the LLM has little room for recovery.

So, how can we do the recovery which repeats? What if the model could search, learn, decide if it has enough evidence, and then search again when needed? Maybe we don't even need vector embedding in the first place.

That's the basics agent's RAG.

In this post, we will build a small RAG workflow with the OpenAI Agents SDK. We will examine how the agent searches, learns, and iteratively supports its response.

Finally, we will step back and briefly discuss the considerations for building an effective RAG solution.

1. Example: Answering a Policy Query with Agent RAG

In our research, we will build a RAG policy agent on top of a company's policy document collection.

1.1 Examining the Text Collection

Here, I have created six custom company policy documents. They are all markup files. Each has a title, start date, brief summary, and policy text.

To be precise, those documents cover 6 areas of general company policy:

1. approval_matrix.mdwhich contains approval standards for general business travel decisions, effective July 1, 2025.
2. conference_guidelines.mdwhich contains rules for attending foreign events, which take effect on May 15, 2025.
3. faq.mdcontaining informal answers to common travel questions, effective September 1, 2025.
4. policy_updates_2026.mdcontaining updates to accommodation, conference travel, and the 2026 accreditation period, effective January 1, 2026.
5. remote_work_policy.mdcontaining remote work rules, effective February 1, 2026.
6. travel_policy.mdwhich contains general rules for booking air travel, accommodation, meals, and transportation, comes into effect on March 1, 2025.

We have made the point that the answer to a policy question may not reside in a single volume. This allows us to see the behavior of the agent that we want.

You can find the full implementation documentation and the RAG agent implementation guide here.

1.2 Defining the Agent

Next, we configure the agent. For that, we use the OpenAI Agents SDK.

At a high level, an agent is:

# pip install openai-agents
from agents import Agent

agent = Agent(
    name="Policy research assistant",
    instructions=INSTRUCTIONS,
    model="gpt-5.4",
    tools=[list_docs, search_docs, read_doc],
)

There are two parts we need to go through: the agent's instructions, and the tools it has access to.

First, education. This is where we define the desired search behavior:

# Note: This instruction is iterated with AI
INSTRUCTIONS = """
[Role]
You are a careful internal policy research assistant.

[Research behavior]
Answer employee policy questions using the document tools.
Find enough relevant evidence to support the answer.
Keep conclusions grounded in the policy documents.

[Expected output]
Give a direct answer first.
Then briefly explain the evidence.
Cite the document filenames used for each important claim.
""".strip()

In this case study, we require that the agent can only touch documents with the three tools described earlier:

The first is a tool that gives the agent a quick overview of what documents are available:

@function_tool
def list_docs() -> list[dict]:
    """List available policy documents without returning their body text."""
    return [
        {
            "doc_name": doc["doc_name"],
            "title": doc["title"],
            "effective": doc["effective"],
            "summary": doc["summary"],
        }
        for doc in docs.values()
    ]

The second tool is the keyword search tool. We keep it simple here: each document is divided into subsections, and each query is matched to those subsections by token overlap:

@function_tool
def search_docs(query: str) -> list[dict]:
    """Search policy documents and return the top three short snippets."""
    query_tokens = tokenize(query)
    scored = []

    for chunk in chunks:
        score = len(query_tokens & chunk["tokens"])
        if score:
            scored.append((score, chunk))

    scored.sort(key=lambda item: item[0], reverse=True)

    results = []
    for score, chunk in scored[:3]:
        snippet = chunk["text"].replace("n", " ")
        if len(snippet) > 420:
            snippet = snippet[:417].rstrip() + "..."
        results.append({
            "doc_name": chunk["doc_name"],
            "title": chunk["title"],
            "section": chunk["section"],
            "snippet": snippet,
            "score": round(score, 2),
        })

    return results

The last tool is the one that allows the agent to open a single document by filename:

@function_tool
def read_doc(doc_name: str) -> str:
    """Read one policy document by filename."""
    if doc_name not in docs:
        valid = ", ".join(sorted(docs))
        return f"Unknown document: {doc_name}. Valid documents: {valid}"

    return docs[doc_name]["text"]

That is the perfect RAG agent.

1.3 Using a Single Policy Query

We now test the agent with one concrete query:

We use an agent with:

from agents import Runner

result = await Runner.run(agent, PROMPT, max_turns=12)

The agent produced the correct answer: yes, an employee can book an official conference hotel if there is a valid business reason. Get that information conference_guidelines.md.

For the approval part, the agent first found that approval was required since the hotel was over the normal cap. It then provided us with the corresponding approval criteria. The agent used travel_policy.md, approval_matrix.mdagain policy_updates_2026.md support its response, which is exactly what we would expect.

The most interesting part is the trace, where we can learn how the agent thinks. We can show the trace as follows:

for item in result.new_items:
    print(type(item).__name__, item)

result.new_items contains intermediate tool calls and tool outputs produced by the agent. In my run, I see that the agent called first search_docs() with keywords like conference hotel, hotel cap, permit, and Berlin. Then, it called list_docs() reviewing available policy documents. After that, open the appropriate files with read_doc(). Only then did she reveal the final answer.

This is exactly the agent loop we wanted to see.

3. What to Decide Before Building an Agenttic RAG

The case study we just went through just scratches the surface. To create an effective RAG solution, based on my experience, I suggest you answer the following 5 questions:

Q1: How much freedom should an agent have?

One common option is exactly what we did in previous research: we exposed a few carefully selected tools, and the agent was allowed to use those tools to perform the investigation. This is specific about control, testing, and research.

But we can also give the agent wider access, such as the shell and the file system. In this way, the agent can directly use scripts to search and examine files, and perhaps perform some data processing to generate useful artifacts, on its own.

This pattern can be very powerful, but it also increases risk and makes behavior difficult to predict.

So in most RAG programs, I would start with selected tools first, then add shell/filesystem access when the complexity of the task justified it.

Q2: Should the agent search only raw text?

Many RAG projects may start with plain text such as PDFs, wiki pages, manuals, etc. That's right.

But in practice, we can simplify the retrieval by finding a knowledge layer over the raw materials.

Those found information artifacts can be document metadata, summaries, links to documents, or we can go further and use an appropriate information graph.

These acquired knowledge artifacts help the agent to navigate the corpus, while the raw documents remain the source of truth.

Q3: Do we still need embedding?

Agent RAG does not mean embedding is gone.

Vector embedding is still an effective way to find relevant text, and often performs better than a pure keyword search technique.

In the agent's RAG, what has changed is that retrieval becomes an “action” that the agent can take. Under this framework, an “action” can still be powered by an embedding-based, keyword-based, or hybrid retriever.

So embedding can still be useful. They are just one way to power the agent's search tool.

Q4: Should one agent handle everything?

A simple RAG setup for an agent is just one agent that does the searching, reading, and responding.

But as the work becomes more complex, you may want to split the work between multiple agents. Specifically, you may need to take ia multi-agent strategy.

You can divide the work by the role. For example, i Planner-retriever-writer classification, when the editor decides what evidence is needed, the follower collects it, and the writer produces a final answer using the collected evidence.

You can also divide source typewhere each agent is equipped with customized tools and focuses on a specific type of resource.

Just remember: A multi-agent setup adds communication complexity, and there's no guarantee that it will perform better than a single-agent setup. Empirical testing is very important.

Q5: Should we always use agent RAG?

Maybe not always.

Just because agency RAG is becoming a trendy topic doesn't mean you should always default to it.

Agentic RAG offers more flexibility, but that comes at a cost. Those costs are not limited to delays or token costs, but also less predictable agent behavior.

Start simple, and add agent loops if the query needs to be repeated.