Bilibili Tutorial Deep Dive: Large Model Agents from Principles to Agent Tuning in Practice

Why Can't Large Model Applications Work Without Agents?

The tutorial doesn't start with technology directly. Instead, it opens with a set of typical scenarios: looking up the source of "不为无等你遮妖", checking the time and weather, querying today's sales figures, asking why there's traffic ahead while driving, and asking how old Andy Lau is. These questions seem everyday, yet they expose three hard flaws of native large models.

The first is the hallucination problem. Large models are essentially probabilistic generative models, and bias is unavoidable—it's very difficult to guarantee answer correctness in serious scenarios relying solely on the model itself. The second is that parameters cannot be updated in real time; once trained, the model is static and disconnected from the real world, so real-time information can only be queried through external tools. The third is that complex business scenarios require task decomposition, multi-step execution, and interaction—a simple question-and-answer mode is simply inadequate.

The tutorial shows a compelling comparison: a month ago, when asked "how old is Andy Lau", ERNIE Bot answered 61 based on a webpage about his 61st birthday celebration in 2022, which was clearly wrong. Now ERNIE Bot presents the intermediate process of querying his birthday and calculating the age difference based on the current time, giving the correct answer. This is precisely the typical value of an Agent—not only giving an answer but also showing the thinking steps and intermediate results, making the answer verifiable and improving reliability.

Agent Technical Framework: Evolution from Prompt to Planning

The tutorial cites a summary diagram from a foreign website that divides large model applications into three levels. The most native is the Prompt mode, where user instructions enter the large model black box and an answer is given directly—simple but limited in capability. The middle layer is RAG (Retrieval-Augmented Generation): for questions requiring real-time or accurate information, relevant context is first searched from index libraries, knowledge bases, or the entire web, and then handed to the large model for reference in answering, alleviating hallucinations.

At the Agent level, the structure is noticeably more complex: the toolset is richer, combining various tools according to business needs; it has short-term and long-term Memory; the Planning process includes deep thinking, self-critique, and chain-of-thought, decomposing problems into subtasks and executing them step by step, iterating repeatedly until completion. The tutorial specifically points out that Planning is a capability an Agent must possess, while the rich toolset and Memory on the left support implementation across a wide range of scenarios.

RAG (Retrieval-Augmented Generation)的核心思路是"先检索、后生成"：面对用户问题时，系统先从外部知识库、索引库或搜索引擎中检索出与问题相关的文档片段，再将这些片段作为上下文与原始问题一起送入大模型，由模型基于这些参考材料生成答案。这样做的好处是，模型不必把全部知识都"背"在参数里，实时信息、企业私有知识都可以通过外挂检索库获得，从而显著缓解幻觉。代价是系统复杂度上升，检索质量直接决定回答质量——检索不到或检索错误，模型就会"基于错误材料一本正经地胡说"。RAG通常被视为从纯Prompt走向Agent的中间形态。

Planning (规划) is the key capability that distinguishes an Agent from an ordinary question-answering system. When a task cannot be completed in one step, the Agent needs to decompose the goal into several subtasks, determine the execution order, call different tools to advance step by step, and after each step evaluate the result and decide whether to adjust the next step—this is exactly the role played by Chain-of-Thought and Self-Critique. Chain-of-Thought lets the model explicitly write out reasoning steps, turning "implicit computation" into a "visible process"; Self-Critique lets the model check intermediate results and backtrack to correct errors. The combination forms a closed loop of "decompose—execute—check—iterate", enabling the Agent to handle complex business tasks that are multi-step, dependent on external information, and error-prone, rather than just one-off text generation.

Why Is Agent Tuning Needed?

Since Agent workflows can be implemented with complex instructions written through Prompt Engineering, why do Agent Tuning at all? The tutorial gives three reasons.

First, a Prompt is essentially a "manual" written for the large model, and the premise is that the model is smart enough. Practice has shown that, apart from GPT-4 and above, other models—including 3.5—cannot well understand complex Prompts or complete complex tasks. Second, practical applications often require private deployment, the strongest large model APIs may not be available, and inference costs must be controlled—so you can only build your own model. Third, through specialized Agent Tuning, small-parameter models (such as 3B, 7B) can also acquire decent Agent capabilities, with lower inference costs and easier deployment.

The tutorial uses a vivid analogy: before graduation, college students receive general education and possess general knowledge reserves, but entering the workplace still requires pre-job training. Agent Tuning is, on top of the manual, intensive training with real cases.

Agent Tuning (智能体微调) refers to further training a general large model with specially constructed Agent task data, so that the model learns to work in an Agent way: understanding complex instructions, calling tools in the required format, following multi-step planning processes, and self-correcting when errors occur. Its difference from ordinary instruction fine-tuning is that the training data is not simple "question—answer" pairs, but complete task records including tool-call trajectories, multi-step reasoning processes, and both successes and failures. It works because small-parameter models are themselves "not smart enough" and can hardly follow complex processes stably relying on Prompts alone; through specialized training on large numbers of real cases, these behavioral patterns can be "internalized" into the model parameters, thereby obtaining usable Agent capabilities on small models like 3B and 7B while reducing inference costs and the barrier to private deployment.

Agent Tuning R&D Process and Cost Considerations

The typical R&D process is: first use a strong model like GPT-4 to get the entire Agent workflow working, because the Prompt is strongly tied to subsequent training data; then build training data on top of the working workflow, with both automation and manual correction; next, do Agent Tuning to train your own model; finally, replace and validate to close the loop.

But the tutorial explicitly warns that Agent Tuning is not necessary for every business scenario. If a strong model API can be used and the cost is acceptable, there's no need to do Tuning, because training itself has costs. The tutorial estimates using a 7B model as an example: 4.7 million tokens of training data, five training rounds, estimated at Alibaba Cloud A100 prices requiring four cards, with specific numbers for training duration and cost—and this is only the case of training once successfully; adjustments require additional costs.

"Getting the workflow working with a strong model, then training a small model" is an extension of the knowledge distillation idea in the Agent scenario. Since the format of training data, tool-calling methods, and planning steps are strongly bound to the earlier Prompt design, first running the entire Agent workflow with a GPT-4-level model is equivalent to first determining "what the standard answer looks like", and then batch-generating high-quality training data accordingly. If the order is reversed—training the model first and then changing the workflow—it often means all data is invalidated and retraining is required. This is also why the tutorial emphasizes that "Prompts and training data are strongly tied". It should be added that training cost is only part of the total cost; data construction (automated generation plus manual correction) is usually more time-consuming and labor-intensive; and after one training run, if business tools or workflows change, additional data and retraining are needed—this implicit cost should also be included in decision-making.

Consensus on Agent Implementation from Multiple Perspectives

Combining the Bilibili tutorial with analyses from several Agent technology channels on YouTube, a consensus is forming: an Agent does not replace the large model but fills its shortcomings in real-time responsiveness, reliability, and complex task execution. For enterprises, the key decision point is not "whether to use an Agent" but "whether to use Prompt, RAG, or Tuning". When resources are sufficient and the scenario is general, a strong model API plus Prompt is the shortcut; when private deployment, cost sensitivity, or small-model deployment is involved, Agent Tuning is worth the investment. The value of this tutorial lies precisely in explaining this decision path concretely enough.