RAG Interview Essential: A Complete Guide to the Retrieval-Augmented Generation Workflow

A large model's knowledge is "frozen" once training is complete, and it often knows nothing about private documents or the latest information. RAG (Retrieval-Augmented Generation) was created to solve exactly this problem, and it is also a high-frequency topic in interviews for large-model positions. This tutorial from Bilibili breaks down the complete RAG workflow in considerable detail, and this article, drawing on its explanation, lays out a clear main line of knowledge.

Why RAG Is Needed: Knowledge Freezing and the Dilemma of Fine-Tuning

Today's large models are basically trained models, and their knowledge cutoff stays at some point in the past. The model knows nothing about what happened after the training data. Although some models can obtain the latest news through web-connected tools, they are still helpless when it comes to internal documents and enterprise private data.

Some people ask: why not use fine-tuning to update knowledge? Theoretically it is possible, but the cost is high. Fine-tuning is costly and time-consuming, and more importantly, once knowledge is written into the model parameters, updating it requires retraining the whole thing—it is like having to go through university all over again just to remember one piece of news.

RAG takes a completely different path: instead of stuffing knowledge into model parameters, it retrieves from an external knowledge base in real time when the user asks a question, puts the relevant content it finds into the prompt, and hands it to the large model to read. The large model itself has strong reading ability; as long as the content appears in the context, it can answer based on that content.

The Complete RAG Workflow: Offline and Online Stages

A complete RAG system consists of three parts: building the knowledge base, using the knowledge base, and large-model generation. The tutorial summarizes it as an offline stage and an online stage, with a clear division of labor—offline is responsible for building the database, and online is responsible for retrieval and generation.

Offline Stage: Building the Knowledge Base

The goal is to prepare the knowledge base before the user asks a question. This includes four steps.

Document loading: Read raw data in various formats—PDF, Word, Markdown, web pages, or database records—into memory through document readers.

Document splitting: Split documents into small pieces called chunks. Why can't the whole document be stored? First, embedding models have an input length limit, so a whole document cannot fit; second, even if the length were unlimited, compressing a document into a single vector would "average out" a lot of detailed information, leading to inaccurate retrieval results.

Chunking strategy is a key interview topic. The simplest approach is fixed-character splitting, but it easily truncates semantics. If a chunk is too large, the information is too mixed, and retrieval easily recalls a bunch of irrelevant content; if a chunk is too small, the semantics are incomplete and the context is cut off. The tutorial gives a rule of thumb of 500 to 1000 tokens, with some overlap to avoid complete semantics being cut off in the middle.

Vectorization: This is the most core step in the offline process. The embedding model converts a piece of text into a high-dimensional numeric vector, such as a list of 1536-dimensional floating-point numbers. You can think of it as a "semantic coordinate system": semantically similar texts are closer together, and unrelated ones are farther apart. Retrieval usually computes cosine similarity—the smaller the angle, the more relevant.

Why can vectors represent semantic relevance? The key lies in training. The embedding model learns on massive text to identify which sentences express the same meaning, then uses contrastive learning to adjust weights, pulling similar ones closer and pushing unrelated ones farther away. Once this spatial structure is formed, similar content can be found by computation during querying. Note that vector matching compares whether the semantics are similar, not whether the keywords are the same.

Storage: Persist each vector together with its corresponding original text in a vector database, such as Milvus or Chroma.

Online Stage: Retrieval and Generation

The online stage provides real-time retrieval for user questions and has requirements for response speed. It includes four core steps.

User input rewriting: User questions are often colloquial and vague, and direct retrieval may not work well. In practice, the question is usually first rewritten into a form better suited for retrieval, so that both the vector database and the large model can better understand the user's intent.

Vector retrieval (coarse ranking): The user question is also vectorized, and similarity matching is performed in the vector database to find the top-k chunks with the closest distance. This step is very fast, and even a million-level vector database can usually return quickly. But speed comes at a cost—vector retrieval is essentially comparing the distance between two vectors and does not deeply understand the semantic relationship between the query and the document, so the recalled results inevitably include some content that looks close but is actually irrelevant. Therefore, this step is generally called "coarse ranking."

Re-ranking: To make up for the shortcomings of coarse ranking, the re-rank model concatenates the user question with each candidate chunk, deeply understands their relevance, reorders them, and filters out irrelevant content. In actual production environments, multi-route recall is often used—in addition to vector retrieval, keyword matching and other methods are also used, and during re-ranking the data recalled from all routes must be uniformly fine-ranked to ensure that the most relevant content is returned to the user.

Generation: Concatenate the user question and the re-ranked chunks into the prompt and hand it to the large model to generate the final answer. The prompt usually explicitly requires: answer only based on the provided materials, and say you don't know if the materials don't contain the answer. This can effectively suppress the large model's tendency toward "hallucination"—without restrictions, the model may fabricate when faced with questions it is not good at, with seemingly coherent logic that does not actually hold up.

The Core Value and Engineering Practice of RAG

The core value of RAG is reflected in two points.

Knowledge can be hot-updated: Just update documents in the knowledge base; there is no need to retrain the model, and the cost is low.

Answers have sources: Every answer can be traced back to which chunk it came from, which is more reliable than pure large-model generation. Even if the answer is wrong, it is possible to clearly determine which piece of knowledge caused the problem.

A RAG system is not something you set up once and forget. The tutorial emphasizes that in actual business, continuous evaluation and optimization are needed, focusing on metrics such as recall accuracy, recall rate, and faithfulness, and tools such as RAGAS can be used for evaluation and analysis. In addition, RAG has variants such as GraphRAG, which uses graph databases to maintain associations between knowledge, and the entire knowledge system is quite large.

For students preparing for interviews for large-model positions, RAG is a core foundational point for which a clear knowledge system must be established, and it is best to prepare complete RAG project experience. Understand the main line of offline database building and online retrieval-generation, then go deeper into details such as chunking strategy, multi-route recall, and re-ranking, and you will be able to speak with depth in interviews.

Background Supplement

Embedding Model is the executor of the vectorization step, usually a neural network trained with contrastive learning (such as a Transformer-based encoder). Its input is a piece of text, and its output is a dense vector of fixed dimension. Common embedding models include OpenAI's text-embedding-ada-002, BGE, M3E, and others. When choosing an embedding model, pay attention to its supported input length, vector dimension, and retrieval performance across different languages and domains. It is worth noting that the embedding model and the large language model are separate, and in a RAG system the two each perform their own duties: the embedding model is responsible for mapping text into semantic space, and the large language model is responsible for reading and generating answers.

Cosine similarity is the most commonly used distance measure in vector retrieval. It calculates the cosine of the angle between two vectors, with values ranging from -1 to 1; the larger the value, the closer the direction and the more similar the semantics. Besides cosine similarity, dot product or Euclidean distance is sometimes used, depending on how the embedding model was trained. Top-k refers to returning the k results with the highest similarity during retrieval. The value of k needs to be tuned according to the scenario: if k is too small, relevant documents may be missed; if k is too large, it increases the burden of subsequent re-ranking and may introduce noise. Vector databases (such as Milvus and Chroma) usually use approximate nearest neighbor (ANN) indexes (such as HNSW and IVF) internally to accelerate similarity search over large-scale vectors, sacrificing a small amount of accuracy in exchange for a large increase in query speed.

Re-ranker is usually a model based on a cross-encoder (Cross-Encoder), such as BGE-reranker. Unlike embedding models, which encode the query and document separately and then compute similarity, a cross-encoder concatenates the query and candidate document and inputs them together into the model, allowing the two to fully interact through the attention mechanism, thereby judging relevance more precisely. This method has higher accuracy but also higher computational cost, so it is only suitable for fine-ranking a small number of candidates (such as the top-50 returned by coarse ranking). Multi-route recall refers to using multiple retrieval strategies simultaneously (such as vector retrieval, BM25 keyword retrieval, knowledge graph retrieval, etc.) to recall candidate sets separately, then merging and deduplicating them before unified re-ranking, so as to compensate for the blind spots of a single retrieval method.