Agentic RAG in Practice: From Principles to Code, Building an Intelligent Retrieval System Step by Step

Traditional RAG systems often fall into the trap of "answering the wrong question"—the retrieved content looks relevant, but once injected into the model it proves useless. More tricky still, when faced with meta-questions like "what documents are in the knowledge base," traditional RAG simply gives up, because it can only mechanically execute the fixed pipeline of "question → retrieval → concatenation → generation." In a recent tutorial, a Bilibili creator pinpoints the core problem: the timing, content, and number of retrievals are all hard-coded, leaving the model with no decision-making power. Agentic RAG, by contrast, wraps retrieval capabilities as tools and hands them to the LLM, letting the model decide on its own when to search, what to search for, and how many times. Based on that tutorial, this article breaks down the core principles and complete code implementation of Agentic RAG, guiding you from scratch to build an intelligent retrieval system capable of self-correction and multi-hop queries.

The Three Fatal Flaws of Traditional RAG

Traditional RAG's pipeline architecture is simple and direct from an engineering standpoint, but the creator summarizes three unavoidable defects.

Retrieval timing is hard-coded. Whatever the user asks—even just a "hello"—the system mechanically runs a vector database search. The model cannot judge whether the current question needs external knowledge, leading to massive ineffective retrievals and wasted tokens.

Retrieval content is hard-coded. Most traditional RAG systems query directly with the user's original question, without any rewriting. A semantic gap exists between user phrasing and knowledge base documents, greatly reducing retrieval precision. More fatally, the model can only passively accept retrieval results; even if the content is irrelevant to the question, it must grit its teeth and generate an answer, with no self-correction capability.

Retrieval count is hard-coded. Each conversation round has only one retrieval opportunity, and after recalling K snippets, it ends hastily. For semantically complex multi-hop questions, the failure rate of a single recall is extremely high, and the model has no second chance to补救. The creator emphasizes in the video: "The way to fix these shortcomings is simple—let the model decide for itself when to ask, when to search, and how many times to search."

The Core Idea of Agentic RAG: Retrieval as a Tool

The essential transformation of Agentic RAG lies in elevating retrieval from a pipeline stage to a tool the model can invoke. In traditional RAG, the model is merely a "text processing worker" at the end of the pipeline; under the agentic architecture, the model becomes a "conductor," autonomously orchestrating tool calls through multi-step reasoning.

According to the creator's analysis, this shift brings three key capabilities. On-demand retrieval: the model answers simple questions directly and only triggers retrieval for complex ones. Self-correction: the model decides, based on the quality of retrieval results, whether to rewrite the query and try again. Multi-hop queries: complex questions are decomposed into multiple sub-queries, and the model repeatedly calls tools to gradually approach the answer.

A detail worth noting: traditional RAG only calls the LLM at the generation stage, whereas Agentic RAG introduces LLM decision-making at the judgment stage. The creator compares two implementation patterns in industry: Pattern 1 uses a separate model to route and decide whether retrieval is needed; Pattern 2 registers the knowledge base directly as a tool, and the model uses it autonomously through the tool-calling protocol. The latter supports more complex reasoning chains and is the approach adopted in this tutorial.

Of course, Agentic RAG is not without cost. Multiple rounds of model calls bring higher latency and token costs—the creator admits this is a "very real cost," so engineering practice must set an upper limit on rounds as a fallback. For simple scenarios like company employee handbook Q&A, traditional RAG's low latency and low cost are actually more suitable.

The tool calling (Tool Calling / Function Calling) mechanism is the technical foundation for understanding Agentic RAG. The LLM itself only outputs text and cannot directly execute retrieval. The tool calling mechanism allows the model to output structured call requests during generation (usually JSON, containing the tool name and parameters), which external programs execute on its behalf, then pass the results back to the model as new messages. The model continues reasoning based on this, forming a "think—call—observe" loop. This is also the engineering realization of the ReAct (Reasoning + Acting) paradigm. After instruction fine-tuning, the model can judge when to call which tool and how to fill in parameters—based precisely on tool function signatures and docstrings. This explains why the following sections emphasize that tool descriptions and return value formats must be standardized.

Multi-hop Query refers to questions whose answers cannot be obtained directly from a single text block and require chaining multiple evidence fragments to reason out. For example, "What is the relationship between LangChain's function calling and Agent modules?" requires first locating function calling content, then Agent content, and finally synthesizing. Traditional RAG recalls K snippets at once; if these snippets do not simultaneously cover the evidence for each hop, the model can only answer based on incomplete information. Agentic RAG turns each hop into an independent tool call, and the model judges each round what is still missing and what to search next based on information already obtained, thereby splitting a one-time broad recall into multiple precise recalls, significantly improving evidence completeness for complex questions.

Four Tools to Build Agentic RAG

The tutorial project equips the model with four tools, covering the complete retrieval chain from coarse filtering to deep reading.

Semantic retrieval tool is the toolized encapsulation of traditional RAG functionality; the model can call it repeatedly with different query terms, recalling different text blocks each time. Query directory tool lists all files and chunk structures in the knowledge base in paginated form, letting the model know "what's in the library"—a question traditional RAG can never answer. Get file info tool returns the file name and structure summary based on file ID, allowing the model to evaluate document value before deep reading, avoiding blind reads that waste tokens. Deep read original text tool directly reads specified snippets by file ID and chunk ID; when semantic retrieval results are incomplete, the model can use this to obtain surrounding context and fill in evidence.

In the demo, the creator shows a typical round of calls: the first round of semantic retrieval has low hit rate, the model reasons and judges evidence is insufficient, rewrites the query and initiates a second retrieval, and after the hit rate improves, the third round calls the deep read tool to complete details. The whole process is not a fixed script—the model flexibly combines tools according to needs.

Code Implementation: From Knowledge Base Construction to Agent Loop

The project consists of two core files: the knowledge base construction script and the Agentic RAG main program.

The knowledge base construction follows the offline chain of traditional RAG. The creator uses a text splitter to chunk documents, sets chunk size and overlap, then calls Alibaba Bailian's Text Embedding V3 model for vectorization, storing into the Chroma database. The tutorial prepares four documents about LangChain as sample data.

The upper part of the Agent main program defines the tool layer. Four Python functions are registered as tools via decorators; the function signatures and docstrings are the "tool manuals" for the model. The creator specifically reminds that tool return values must be unified as JSON strings to ensure structured readability. The lower part is the Agent loop: initialize the LLM, mount all tools via the bind tools method, set the system prompt and user prompt, then enter a loop with a maximum round limit. Each round calls the model; if the model returns a tool call request, the system executes it and returns the result; if the model directly gives an answer, the loop terminates.

In the demo, the creator inputs the question "How is the function calling feature implemented in LangChain?" The model calls the semantic retrieval tool in the first round and gives a complete answer in the second round, with rigorous source attribution. The creator mentions that in a previous run on the same question, the model took three rounds to complete—"This is exactly the flexibility of Agentic RAG: the number of rounds is not fixed, scaling on demand."

Vectorization and vector databases are the underlying support for the retrieval chain. The text splitter cuts long documents into chunks with overlap; each chunk is mapped by the Embedding model into a high-dimensional vector, and semantically similar texts are closer in vector space. Vector databases like Chroma store these vectors and build approximate nearest neighbor indexes; during query, the user question is also vectorized, then the K nearest chunks are recalled. Chunk size and overlap settings directly affect retrieval quality: too large a chunk dilutes semantics, too small loses context; overlap prevents key information from being cut off. The semantic retrieval tool in the article can recall different chunks each time by changing the query term precisely because different query vectors point to different neighboring regions in space.

Practical Advice and Boundaries

Agentic RAG is not a comprehensive replacement for traditional RAG. The creator's criteria are clear: use traditional RAG for simple Q&A and low-latency scenarios; only adopt Agentic RAG for knowledge-intensive, variable, and complex scenarios. In engineering implementation, be sure to set a loop round limit to prevent the model from falling into infinite loops. Tool design should follow the "single responsibility" principle—each tool solves one type of retrieval need, giving the model clear option boundaries during planning. Prompts must clearly constrain output format; the creator points out that less intelligent models may arbitrarily add or remove content, causing tool call failures, which requires extra hardening.

One-line Summary

Agentic RAG wraps retrieval as tools handed to the LLM for autonomous decision-making, cracking the three fatal flaws of traditional RAG.

Paragraph Summary

Based on a Bilibili tutorial, this article breaks down the principles and implementation of Agentic RAG. Traditional RAG hard-codes retrieval timing, content, and count, leaving the model with no decision-making power, resulting in ineffective retrieval, answering the wrong question, and inability to answer meta-questions. Agentic RAG wraps retrieval capabilities as model-invokable tools, introduces LLM decision-making at the judgment stage, and achieves on-demand retrieval, self-correction, and multi-hop queries. The tutorial project provides four tools—semantic retrieval, query directory, get file info, and deep read original text—covering the complete chain from coarse filtering to deep reading; in code, tools are registered via decorators, return values are unified as JSON, and a maximum round loop is set. The author also notes that multi-round calls bring latency and token costs, traditional RAG remains better for simple scenarios, and offers practical advice such as single responsibility for tools and prompt format constraints.