Is the famous feature you can use in the apps such as ChatGPT and Google Gemini. Allows users to ask the question as usual, and the app spends a long time researching the question and coming in better response rather than normal llm answers.
You can also use this in your text collection. For example, suppose you have thousands of information information about the information of the company, you may want to create a deeper research program that takes user questions, scanning all available (internal), and comes with a good response based on that information.
Content
Why have you formed a deep research program?
The first question you might ask yourself is:
Why do I need a deep research program?
This is a good question, for there are other ways that apply to many situations:
- Feed all the information to the llm
- Rag
- Searching the keyword
If you can find these simple programs, you probably should always do that. The easiest moving method is simply feeding all the information in the llm. If your information contains less than 1 million tokens, this is definitely a good option.
In addition, if the traditional RAG is effective, or you can find the right details for keyword search, you should also choose those methods. However, sometimes, even if these solutions are solid enough to solve your problem. Maybe you need to analyze many sources, and the restoration of chunk from the same as you are not good enough. Or you can't use keyword searches because you are not familiar enough about dataset to know which keywords you can use. In that, you should consider using a deep research program.
How to build a deep research program
You can naturally use a deeper research program from suppliers such as Opelai, which provides a deep study of API. This can be another good way if you want to keep things simple. However, in this article, I will discuss many details that the deep research system is constructed, and why is it useful. Anthropic wrote a very good article about their Alent Multi's agent's research program (a deep research), which I recommend to understand more details on the topic.
Gathering and identifying details
The first step for any application for access to collect all your information in one place. Perhaps you are informed of applications such as:
- Google Drive
- Comprehension
- Sale
You whether to collect this information in one place (converting all in the PDFS, for example, and keep them in the same folder), or you can contact these apps, as ChatGPT has applied for its application.
After collecting the information, we now need to prove to be readily available. Two front indices should create:
- The search reference in the kitchen. Example BM25
- Vector The same Index: Chunk Your Text, Shumek, and keep it in vectordb like a pinecone
This makes information easily accessible from the tools I will describe in the following principle.
Tools
The agents will be using later tools to download the relevant information. You must make a series of activities that make it easy for the llM to download the relevant information. For example, if user questions reports reports, the llm may want to search keyword and analyze the restored documents. These tools can look like this:
@tool
def keyword_search(query: str) -> str:
"""
Search for keywords in the document.
"""
results = keyword_search(query)
# format responses to make it easy for the LLM to read
formatted_results = "n".join([f"{result['file_name']}: {result['content']}" for result in results])
return formatted_results
@tool
def vector_search(query: str) -> str:
"""
Embed the query and search for similar vectors in the document.
"""
vector = embed(query)
results = vector_search(vector)
# format responses to make it easy for the LLM to read
formatted_results = "n".join([f"{result['file_name']}: {result['content']}" for result in results])
return formatted_resultsYou can also allow for an agent access to other functions, such as:
- Search Internet
- File name is only search
And other relevant activities
To put everything together
The deep research program contains an orchestrator agent and many enemies. How often usually:
- The Orchestrator agent receives a user question and programs approaching
- Most suaagents are sent to download the relevant information and consumed summarized information from the orchestrator
- The Orchestrator decides that you have enough information to answer the user's question. If not, we return to the last piece of letters; If so, we can give the final point of characters
- Orchestrator puts all the information together and provides user for feedback
In addition, you may have a clearer question, if the user's question is not clear, or reduces the width of the user question. You have probably received this if you use any deep research program from the Frontier website, where a deep research program remains begins with the meaningful question.
Usually, the orchestrator is a large / better model, for example, for example, or GPT-5 with great efforts to think. Small fellegents, such as GPT-4.1 and Claude Sonnet.
The primary benefit of this method (above traditional RAG, especially) that allows the program to check and analyze many detail, lower the opportunity to lose lost information. The fact that you should check many documents usually make the system slow. Naturally, this is a trade between time and the quality of the answers.
Store
In this article, I discussed how I am building a deep research program. I first covered the motivation to build such a plan, and where you should focus on creating simple programs, such as the keyword searches. Continuous, I discussed the basis of what is the deepest examination of the user program, which is used for the answer, sends low agents to download the appropriate information, and respond to the user.
👉 I have found in the community:
🧑💻 Contact your
🔗 LickDin
🐦 x / Twitter
✍️ Medium
You can also read some of my topics:
