Building an AI Knowledge Base Testing System from Scratch: A Hands-On Introduction to Tencent's Open-Source Tool

Why the AI Era Needs "Evaluation" Rather Than "Testing"

As AI increasingly takes over testing work, a new question surfaces: who ensures the correctness of AI itself? Traditional software testing focuses on "right or wrong," but in the AI domain we deal with probabilistic outputs and semantic understanding, so we need a new paradigm called "evaluation." According to a Bilibili creator's video breakdown, AI evaluation falls into three main directions: the large model itself, knowledge bases, and agents. Among these, knowledge base evaluation offers the best cost-effectiveness—it can serve as a standalone product and also as a component of agents, analogous to interface testing in traditional testing.

Based on that video, this article walks you through building a local knowledge base using Tencent's open-source WeKnora (produced by the WeChat AI team), and helps you understand where testing and evaluation come into play.

Why Choose Tencent's Open-Source Knowledge Base Tool?

The tool demonstrated in the video comes from Tencent's WeChat team, with the official website domain being weixin.qq.com. It has three clear advantages:

• Big-company backing: Rich Chinese-language materials and relatively well-developed community support.
• Fully open source: Source code is available on GitHub, supporting in-house enterprise customization.
• Feature-rich: Beyond traditional knowledge Q&A, it supports task execution, MCP (Model Context Protocol), skills, clients, and more—it can even generate Wiki pages with knowledge links and be invoked as a component by AI tools like Claude Code.

Because a knowledge base can be deployed standalone and embedded into agents, evaluating it covers the most essential retrieval and generation pipeline.

MCP (Model Context Protocol) is an open protocol proposed by Anthropic, aimed at standardizing how AI models connect with external tools and data sources. Traditionally, each AI application needs to write separate integration code for each tool; MCP provides a unified interface specification so models can call external resources like file systems, databases, and APIs in a consistent way. In the knowledge base context, MCP lets a knowledge base be called directly by AI coding assistants like Claude Code as a standardized tool, without a custom adapter layer. This protocol lowers the complexity of agent integration and adds another evaluation dimension for knowledge bases—protocol compatibility and invocation stability.

Deployment Strategy: Production Environment vs. Learning Environment

The official recommendation is containerized deployment, but the video creator points out that the standard approach has high memory requirements (about 4-8GB), which is tight for an ordinary 16GB development machine also running the OS, IDE, and other software. Therefore, during the learning phase, the binary single-file approach is more recommended.

The official documentation lists three deployment forms: production environment (containerized/K8s), learning environment (binary), and desktop application (not yet released). The learning environment uses a file-based database and in-memory storage, requires no additional dependencies, and works right after extraction.

Quick Start Steps on Windows

1. Get the compiled Windows binary package (the video creator has already compiled one; you can request it from the teaching assistant).
2. After extraction, copy the sample environment file and rename it to .env.
3. The default configuration needs no changes (the listen address stays at localhost:8215); just run the startup script (e.g., start.ps1).
4. If startup fails, try deleting the data directory (it's empty on first run) and let the program generate the key automatically.

The video specifically warns: the data encryption key corresponds one-to-one with the data directory. If you change the .env file or clear the data, decryption may fail.

Registration, Configuration, and a First Look at the Interface

After accessing the local address for the first time, the system guides you through account registration. Even personal use requires an account, because data is associated through a "space-user" system. Once registered, you enter the admin interface, which is beautifully designed and lives up to big-company standards.

The video creator mentions a detail: the current binary version has no "log out" button, but you can switch accounts by clearing cookies via the browser's developer tools. Since the project is open source, this feature can be added later.

The configuration step is key to understanding how a knowledge base works. Next, it's important to be clear: the knowledge base here is not simple keyword search but a semantic system based on RAG (Retrieval-Augmented Generation). The video introduces the RAG concept at the end, which is exactly the core to focus on when evaluating a knowledge base—retrieval accuracy, generation faithfulness, and so on.

RAG (Retrieval-Augmented Generation) is a technical architecture that combines information retrieval with large language model generation. Its typical flow is: after a user asks a question, the system first retrieves document chunks semantically relevant to the question from the knowledge base, then inserts these chunks as context into the prompt and hands it to the large model to generate an answer. The core value of RAG is that it makes model answers "evidence-based," thereby reducing hallucinations. But this also introduces new risk points: the retrieval step may recall irrelevant content (low recall), and the generation step may deviate from the retrieved content (insufficient faithfulness). Therefore, evaluating a RAG system requires measuring retrieval quality and generation quality separately—this is exactly the key to knowledge base testing.

Evaluation Perspective: What Should Knowledge Base Testing Measure?

Although the video doesn't elaborate on specific evaluation metrics, building on its ideas, we can approach it from several dimensions:

• Retrieval quality: whether relevant document chunks can be recalled.
• Generation quality: whether answers are based on retrieved content, and whether there are hallucinations.
• Component integration: whether interface calls are stable when used as an agent component.

Since knowledge bases are often used as an agent's "memory," evaluating them is effectively safeguarding the reliability of the entire AI application. The video emphasizes that the learning environment is simple to deploy and ideal for repeated experimentation—exactly the ideal sandbox for building a testing system.

Hallucination refers to a large language model generating content that seems plausible but doesn't match facts or the given context. In RAG systems, hallucinations can be divided into two types: first, "retrieval-failure type," where the retrieved chunks themselves don't contain the answer but the model forcibly fabricates one; second, "faithfulness-deficit type," where the correct chunks are retrieved but the model distorts or ignores their content during generation. When evaluating generation quality, the "faithfulness" metric is commonly used to measure whether answers strictly adhere to retrieved content, and "answer relevance" measures whether answers stay on topic. Ways to reduce hallucinations include optimizing prompts, setting refusal thresholds, and introducing human or automated model scoring in evaluation.

Conclusion

The next stop for AI testing is evaluation, and knowledge base evaluation is the most cost-effective entry point. With Tencent's open-source WeKnora, you can quickly build a customizable knowledge base locally and use it as a foundation to explore testing methods for RAG systems. The binary package provided in the video lowers the barrier to entry, letting developers get started without high-end hardware. Going forward, you can further study its source code and customize evaluation scripts to form a complete AI testing loop.

One-line Summary

In the AI era we need to shift from testing to evaluation, and Tencent's open-source WeKnora is a cost-effective entry point for knowledge base evaluation.

Paragraph Summary

The article points out that as AI takes over testing work, ensuring the correctness of AI itself requires shifting from the traditional "testing" paradigm to "evaluation," because AI outputs are probabilistic and semantic. AI evaluation falls into three directions—large models, knowledge bases, and agents—among which knowledge base evaluation offers the best cost-effectiveness, as it can be deployed standalone or used as an agent component. Using Tencent WeChat team's open-source WeKnora as an example, the article introduces its advantages—big-company backing, fully open source, and rich features (MCP, skills, Wiki generation)—compares production versus learning environment deployment strategies, and recommends the binary single-file approach for developers with limited memory. It then walks through the Windows startup steps and registration/configuration flow, introduces the RAG concept, and finally offers evaluation entry points for knowledge base testing across three dimensions: retrieval quality, generation quality, and component integration.