RAG Interview Essentials: Breaking Down the Full Offline Indexing and Online Retrieval Pipeline

Why Do Large Models Need RAG?

A large model's knowledge is "frozen" once training is complete. Ask any large model for its knowledge cutoff date and you'll always get some point in the past. The model knows nothing about events after its training data cutoff, and it certainly cannot directly answer questions about internal enterprise documents or private data.

Some might ask: why not update knowledge through fine-tuning? Theoretically feasible, but the cost is high—fine-tuning is expensive, time-consuming, and once knowledge is written into model parameters, updating it requires retraining. To use the original analogy: it's like being asked to memorize a single news item, but having to go through college all over again.

RAG takes a completely different path: instead of cramming knowledge into model parameters, it retrieves from an external knowledge base in real time when the user asks a question, puts the found content into the prompt, and lets the large model read it before answering. The large model itself has strong reading comprehension—as long as relevant content appears in the context, it can answer based on that content. This is the essence of RAG.

RAG stands for Retrieval-Augmented Generation. The core problem it solves is: once a large model's knowledge is fixed, it cannot answer questions about private data or the latest information. The approach is to first retrieve from an external knowledge base before generating an answer, then submit the retrieval results together with the user's question to the large model.

The Complete RAG Workflow: Offline and Online Phases

A complete RAG system has two parts: the offline phase builds the knowledge base, and the online phase uses it. Here is a step-by-step breakdown.

Offline Phase: Building the Knowledge Base

The goal of the offline phase is to prepare the knowledge base before users ask questions. It consists of four steps.

Step 1: Document loading. Read in raw data in various formats—PDF, Word, Markdown, web pages, database records are all fine—and load it into application or operating system memory through document readers.

Step 2: Document splitting. Split documents into small pieces called chunks. Why not store whole documents? Two reasons: embedding models have input length limits, so a whole document won't fit; and even if length weren't limited, compressing an entire document into a single vector would "average out" a lot of detail, making retrieval results inaccurate.

Chunking strategy is a common interview topic. The simplest approach is fixed-character splitting, but it easily truncates semantics. If chunks are too large, retrieval tends to recall a bunch of irrelevant information; if too small, semantics become incomplete and context gets cut off. Generally, 500 to 1000 is a reasonable chunk size, and some overlap should be applied to avoid cutting complete meaning in the middle. The original screenshot also mentions that we can "define" specific chunking parameters, but actual values should be determined based on the specific scenario.

Step 3: Vectorization. This is the most core step in the offline pipeline. An embedding model converts a piece of text into a high-dimensional numeric vector, such as a 1536-dimensional list of floats. You can think of it as a semantic coordinate system: semantically similar texts are closer together in this coordinate system, while unrelated ones are farther apart.

Why can vectors represent relevance? Because the embedding model has seen massive amounts of text during training and learned to recognize which sentences express the same meaning, continuously adjusting weights through contrastive learning to pull similar ones closer and push unrelated ones apart. Once this spatial structure is formed, similar content can be found by computing distances during queries. A common approach is cosine similarity, where smaller values indicate greater relevance.

Key insight: vector retrieval is not keyword matching—it compares semantic similarity. In actual production environments, single-vector retrieval is rarely used alone; typically it's multi-route recall—vector similarity retrieval plus keyword matching, followed by fusion ranking.

Step 4: Indexing. Persist each vector together with its corresponding original text into a vector database. Vector databases are specifically optimized for high-dimensional vector storage and similarity search. Common ones include Milvus and Chroma, which support quickly finding the most similar few records among tens of millions of vectors.

Online Phase: Using the Knowledge Base

The online phase provides real-time retrieval for user questions and has latency requirements. It consists of four core steps.

Step 1: User input and rewriting. User questions are often colloquial and vague, so direct retrieval may not work well. Usually the question must first be rewritten into a form more suitable for retrieval, so that both the vector database and the large model can better understand user intent.

Step 2: Vector retrieval (coarse ranking). Convert the user question into a vector as well, perform similarity matching in the vector database, and find the top-k chunks with the closest distances. This step is fast—even million-scale vector databases usually return quickly. But speed comes at a cost: vector retrieval essentially compares distances between two vectors without deeply understanding the semantic relationship between query and document, so recall results inevitably include content that looks close but is actually irrelevant. That's why this step is also called coarse ranking.

Step 3: Reranking (fine ranking). To compensate for the shortcomings of coarse ranking, a rerank model concatenates the user question with each candidate chunk, deeply understands their relevance, reorders them, and filters out irrelevant content. Actual systems typically use multi-route recall—besides vector retrieval, there may be keyword matching and other methods, with all results uniformly reranked to ensure the most relevant data is returned to the user. The original screenshot also emphasizes that "we use multi-route recall."

Step 4: Generation. Concatenate the user question and reranked chunks into a prompt and hand it to the large model to generate the final answer. The prompt usually explicitly requires: answer only based on the provided materials, and say you don't know if the materials don't cover it. This effectively suppresses model hallucination.

Why add this constraint? Without it, the large model may fabricate content when encountering questions it's not good at—output that is logically sound but factually wrong in details. When using large models, you must have the ability to judge correctness; blindly trusting output is irresponsible for the system.

RAG's Core Value and Interview Points

RAG's core value lies in two points. First, knowledge can be hot-updated at any time: just update documents in the knowledge base—no need to retrain the model, and costs are low. Second, answers have provenance: every response can be traced back to which chunk it came from, making it more reliable than pure large model generation; even if an answer is wrong, you can clearly identify which piece of knowledge caused the problem.

RAG is a core foundational topic in large model job interviews and requires a clear knowledge system. When writing project experience, it's best to demonstrate a RAG project. Beyond basic RAG, there's also RAG Graph—using graph databases to maintain relationships between knowledge. After the system is built, evaluation and optimization are needed, focusing on metrics like recall accuracy, recall rate, and faithfulness, which can be evaluated and analyzed with tools like RAGAS.

No system is done once built—in practice, RAG systems must be continuously adjusted and optimized so retrieval can faster and better meet user needs. Only by solidifying the foundations of both offline indexing and online retrieval can you clearly explain the entire design flow and operational steps in an interview.

Background Supplement

Embedding models and contrastive learning: Models that map text to dense vectors are typically based on Transformer encoders (such as BERT-like structures) and output a fixed-dimension vector. Their training objective often uses contrastive learning: constructing positive sample pairs (sentences with the same or similar meaning) and negative sample pairs (unrelated sentences), then using a loss function to pull positive sample vectors closer and push negative ones apart. Typical methods include Sentence-BERT, SimCSE, and retrieval-oriented dense retrieval models like DPR. When selecting models in practice, pay attention to the training corpus domain, maximum input length, and vector dimension—these directly determine chunking strategy and retrieval quality.

Cosine similarity: Cosine similarity computes the cosine of the angle between two vectors, given by the dot product divided by the product of their magnitudes, ranging from -1 to 1. Values closer to 1 indicate more consistent direction and greater semantic similarity; values closer to -1 indicate opposite directions. Since most embedding models output normalized vectors, cosine similarity is equivalent to the dot product in that case, making computation more efficient. The article's mention that "smaller values indicate greater relevance" usually refers to using cosine distance (1 minus cosine similarity) as the distance metric.

Division of labor between coarse and fine ranking: The coarse ranking (recall) stage faces massive candidate documents and must complete filtering at millisecond levels, so it can only use lightweight methods with low computational cost, such as vector distance or inverted indexes—better to recall more than to miss key content. The fine ranking (reranking) stage only processes dozens to hundreds of candidates, so it can use computationally expensive but highly accurate models like Cross-Encoders: concatenating the query with each candidate document and feeding them together into the model, allowing word-level deep interaction and outputting relevance scores. This "broad first, strict later" two-stage architecture is a general paradigm for information retrieval systems; the cost is higher reranking latency, so the number of candidates must be controlled.

Rerank models: Common reranking models are based on cross-encoder structures (such as BGE-Reranker, Cohere Rerank). Unlike dual-tower embedding models, they do not pre-compute document vectors but score "query-document" pairs in real time, thus capturing finer-grained semantic matching relationships. However, they cannot build indexes offline and are only suitable for ranking small candidate sets.