Photo by the Author
# Introduction
Retrieval-augmented generation (RAG) systems are a natural evolution of independent linguistic models (LLMs). RAG addresses several important limitations of classical LLMs, such as hallucinations of models or the lack of up-to-date information, necessary to generate fact-based answers to users' questions.
In a related thread, Understanding RAGwe have provided an overview of RAG systems, their characteristics, practical considerations, and challenges. We now compile part of those studies and combine them with the latest trends and techniques to define seven key steps that are considered essential to enable the development of RAG programs.
These seven steps relate to different sections or parts of the RAG area, as indicated by the number labels ([1] to [7]) in the diagram below, which shows the structure of a classic RAG:
7 Steps to Mastering RAG Programs (see labels numbered 1-7 and write below)
- Select and clean data sources
- Separation and division
- Embedding/vectorization
- Fill in the vector information
- Query vectorization
- Bring back related content
- Create a reasoned response
# 1. Selecting and Cleaning Data Sources
The principle “garbage in, garbage out” takes its highest importance at RAG. Its value is directly proportional to the relevance, quality, and purity of the source text data it receives. To ensure high-quality databases, identify high-value data silos and periodically audit your databases. Before ingesting raw data, perform an effective cleaning process through robust pipelines that employ key steps such as extracting personally identifiable information (PII), removing duplicates, and addressing other noise factors. This is a continuous engineering process that should be used every time new data is compiled.
You can read everything This article for an overview of data cleaning techniques.
# 2. Consolidation and Classification of Documents
Many instances of textual data or documents, such as literary novels or PhD theses, are too large to be embedded as a single data instance or unit. Chunking consists of dividing long texts into smaller parts that preserve semantic value and maintain context integrity. It requires a careful approach: not too many chunks (which includes possible loss of context), but not too few – big chunks affect the semantic search later!
There are various methods of grouping: from those based on alphabetic counting to those driven by logical parameters such as categories or categories. The LlamaIndex again LangChainwith associated Python libraries, can really help in this task by using more advanced partitioning methods.
Chunking can also take into account the overlap between parts of a document to preserve consistency in the retrieval process. For the sake of illustration, here's what the combination might look like in a small, toy-sized document:
Merging documents in overlapping RAG systems | Photo by the Author
In this installment in the RAG series, you can also learn the additional role of document aggregation processes in controlling the size of the RAG installation context.
# 3. Embedding and Documenting
Once the documents are cut into pieces, the next step before they are safely stored in the database is to translate them into “machine language”: numbers. This is usually done by converting each text into an embedded vector — a dense, high-dimensional numerical representation that captures the semantic properties of the text. In recent years, special LLMs have been created to perform this task: they are called embedding models and include well-known open source options such as Confused Face all-MiniLM-L6-v2.
Learn more about embedding and its advantages over traditional text representation methods This article.
# 4. Filling the Vector Database
Unlike traditional relational databases, Vector databases are designed to efficiently enable the search process by using high-dimensional (embedded) arrays representing text documents – a key component of RAG systems for retrieving documents relevant to a user's query. Both open-source vector stores love it FAISS or other freemium alternatives such as Pine they exist, and they can provide excellent solutions, thus bridging the gap between human-readable text and vector-like displays.
This code snippet is used to parse the text (see point 2 earlier) and fill the local, free vector data using LangChain and Chroma — assuming we have a long document to store in a file called knowledge_base.txt:
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma
# Load and chunk the data
docs = TextLoader("knowledge_base.txt").load()
chunks = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50).split_documents(docs)
# Create text embeddings using a free open-source model and store in ChromaDB
embedding_model = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
vector_db = Chroma.from_documents(documents=chunks, embedding=embedding_model, persist_directory="./db")
print(f"Successfully stored {len(chunks)} embedded chunks.")Learn more about vector databases here.
# 5. Vectorizing Questions
User commands expressed in natural language are not exactly the same as stored document vectors: they must also be interpreted, using the same embedding method or model (see step 3). In other words, a single query vector is created and compared with the vectors stored in the knowledge base to find, based on similarity metrics, the most relevant or similar documents.
Some advanced methods of query vectorization and optimization are described in it this part of Understanding RAG series.
# 6. Retrieving Related Context
Once your query is specified, the RAG system finder performs a similarity-based search to find the closest vectors (document fragments). Although standard top-k methods are often effective, advanced methods such as combined retrieval and reordering can be used to improve how the returned results are processed and integrated into the final, enriched LLM information.
Check it out this is a related article to find out more about these advanced methods. Accordingly, managing context windows it is another important process to use when LLM's ability to handle very large inputs is limited.
# 7. Generating Reasonable Responses
Finally, the LLM comes to the scene, takes the user's query raised by the returned context, and is instructed to answer the user's query using that context. In a well-designed RAG structure, following the previous six steps, this often leads to more accurate, defensible responses that may even include citations to our data used to build the knowledge base.
At this point, evaluating the quality of the response is important to gauge how the overall RAG system is behaving, and to signal where the model may need to be. fine tuning. Assessment frameworks because this end has been established.
# The conclusion
RAG programs or architectures have become an almost essential element of LLM-based applications, and commercial, large ones are rarely missed today. RAG makes LLM applications more reliable and informative, and helps these models generate evidence-based answers, sometimes based on confidential data from organizations.
This article summarizes seven key steps to getting the process right for building RAG systems. Once you have this basic knowledge and skills down, you'll be in a good position to develop advanced LLM applications that enable business-grade functionality, accuracy, and transparency – something that's not possible with popular online models.
Iván Palomares Carrascosa is a leader, author, speaker, and consultant in AI, machine learning, deep learning and LLMs. He trains and guides others in using AI in the real world.
