In the Era of Agent Self-Evolution, How Can Test Development Prove AI Is Getting Better?

When an Agent starts modifying its own Prompt, Skill, and Memory, a neglected question surfaces: who proves it actually got better?

This isn't science fiction. According to a sharing from Bilibili's Hogwarts software testing and development, Agent self-evolution is becoming reality, and test development is facing an entirely new challenge—when an Agent starts modifying itself, how do we continuously verify it's improving rather than drifting further off course.

What Exactly Is the Agent Changing? A Three-Layer Evolution Model

The first step to understanding Agent self-evolution is distinguishing the layers of change. From bottom to top, it can be divided into three layers:

The bottom layer is the Answer layer, which just improves this particular answer—a single-output optimization. Above that is the Harness layer, which modifies Prompt, Skill, Memory, Tool, and Workflow—these are the Agent's "external gear" and "behavior strategy." The highest layer directly modifies model parameters, touching the model itself.

For most business Agents, the middle layer—Harness—is what deserves the most attention at this stage. The reason is straightforward: model parameter fine-tuning is costly and slow, while Harness-layer changes are more frequent, closer to the business, and more likely to introduce hidden risks. Change one word in a Prompt, swap a Skill implementation, store one extra record in Memory—any of these can trigger cascading changes in Agent behavior.

Test development needs to build sensitivity to Harness-layer changes: which changes are safe incremental optimizations, and which might break existing capabilities. This itself is a new testing boundary.

Harness (translatable as "execution framework" or "behavior shell") is the sum of all controllable configurations at Agent runtime excluding model weights, including system prompts, tool definitions and invocation specifications, memory storage and retrieval strategies, workflow orchestration logic, and various Skill implementations. It becomes the main battleground for self-evolution because modifying Harness only requires changing text or configuration without retraining the model, shortening iteration cycles from weeks to minutes. The cost is that such changes lack gradient signal constraints and easily break scenario B while fixing scenario A, making change impact assessment and regression coverage more critical than in model fine-tuning scenarios.

Looking Only at the Final Answer Is the Biggest Trap in Agent Evaluation

What does traditional large model evaluation look at? Whether the final output is correct. But Agent testing has an essential difference from ordinary large model evaluation: you cannot look only at the final answer.

The answer may be correct, but the Agent might have chosen the wrong Skill, taken the wrong path, or even skipped key steps. This time it got lucky, but next time with a different input it could collapse.

So now we need not only result evaluation but also Trace and execution path analysis. Trace records every decision and tool call the Agent makes, and it can expose process problems masked by the final answer. Test development needs to incorporate Trace analysis into the regular evaluation system: is the tool call order reasonable, were necessary validation steps skipped, did it take a long detour on the wrong branch.

Correct results are only the passing line; correct paths are the guarantee of stability.

Trace (execution trajectory) refers to the complete decision sequence an Agent produces during a task, typically including the reasoning content at each step, selected tools and their parameters, tool return results, state transitions, and intermediate artifacts. Unlike logs that only record inputs and outputs, Trace preserves the causal chain of "why it reached this step," which can be used to locate wrong branches, redundant calls, and missing validations. In engineering practice, Trace is often persisted as a structured event stream, supporting replay by task ID and step-by-step assertions—it is the foundational data source for building path-level regression test cases. Without Trace, teams can only do black-box comparison on final answers and struggle to distinguish lucky success from stable correctness.

The Evaluation Pyramid: What Rules Can Solve, Don't Hand Everything to the Model

Given the complex behavior of Agents, how should evaluation strategies be designed? The sharing proposes a clear evaluation pyramid approach.

If rules and automated assertions can solve it, don't hand everything to the LLM. Semantic understanding and open-ended tasks go to model judgment, and a small number of high-risk results at the top get human intervention.

The value of this layered logic lies in balancing cost and reliability. Rule assertions are fast, precise, and repeatable—suitable for covering highly deterministic checks; model judgment is flexible but has fluctuations and bias—suitable for semantic problems that can't be described by rules; humans guard the final high-risk gate.

The mistake test development easily makes is "wanting to evaluate everything with models," which turns evaluation itself into an unreliable variable. The pyramid structure reminds us: the lower you go, the more automated; the higher you go, the more it depends on judgment. Resources should be spent where they matter most.

Memory Is Not a Chat Log Warehouse

How should Agent memory be stored? A common misconception is stuffing all historical conversations into the memory store. But truly useful memory should be layered and distilled: from Episode to Facts to Skills.

In other words, distilling "what happened" into "what is known," and finally solidifying it into "how to act in the future." Episode is raw experience, Facts are distilled facts, Skills are reusable behavior patterns.

The significance of this layering for testing is that Memory modifications also need to be verified. When the Agent writes new content into Memory itself, testing needs to confirm that what's written is valuable facts rather than noise, and that the distilled Skill is a correct pattern rather than an accidental experience. Memory contamination is hidden and persistent—once false facts are solidified, subsequent decisions will continuously drift off course.

Memory contamination refers to the phenomenon where erroneous or outdated information is written into an Agent's long-term memory and repeatedly retrieved and amplified in subsequent tasks, ultimately solidifying into incorrect behavior patterns. Its difference from ordinary hallucination lies in persistence: a single hallucination can be eliminated by retrying, but once contamination enters Memory, it affects decisions across sessions and is difficult to trace to its source. Typical triggers include treating a user's slip of the tongue as fact, distilling a one-off anomalous experience into a general Skill, and lacking timeliness annotations causing old facts to override new ones. Defenses include fact verification before writing, attaching source and timestamp to memory entries, and regularly sampling and auditing high-impact memories and building rollback capabilities.

Fitting Agent Self-Evolution with CICD

Here comes the most critical question: after the Agent discovers a problem, can it just fix it itself and go live? The answer is clearly no.

The solution proposed in the sharing is: manage Agent self-evolution like code CICD. The complete process includes diagnosing problems, generating candidates, independent evaluation, regression testing, safety gating, canary release, and finally feeding online failure samples back.

This analogy is precise. Self-generated modifications by the Agent are equivalent to a code commit and must go through independent evaluation and regression testing before entering the release stage. Safety gating intercepts high-risk changes, canary release controls the blast radius, and failure sample feedback forms a closed loop of continuous improvement.

For test development, this means a shift in focus: from manually verifying single outputs to building automated, repeatable, gated self-evolution verification pipelines. The quality of evaluators themselves, the coverage of regression test cases, and the rigor of gating rules all become new core assets.

Canary release in Agent scenarios, beyond scaling by traffic ratio, also requires layered control by task type, user group, and risk level. Unlike ordinary software releases, Agent behavior changes often have no clear code diff to review, so the canary stage should focus on path-level metrics rather than just success rate—for example, whether the distribution of tool calls has shifted, whether average step count has abnormally increased, whether the trigger rate of a specific Skill has suddenly changed. Once metric drift is detected, it should be possible to quickly roll back to the previous version's Harness snapshot. This requires teams to version-manage Prompts, Memory, and tool configurations, giving every self-evolution change the engineering properties of being traceable, comparable, and revocable.

Autonomy Is Not Full Automation: Control Must Stay with Humans

One last point about boundaries: controlling Agent autonomy is not the same as full automation.

Stable scenarios can gradually delegate authority, but safety boundaries, critical Skill modifications, and severe regressions still need human control. This isn't conservatism—it's a clear-eyed awareness of risk. The stronger the Agent's self-evolution capability, the larger the blast radius when something goes wrong. Human oversight at critical nodes is the final safety valve.

So what truly deserves the attention of testing colleagues in Agent self-evolution is not whether AI will make itself smarter, but after the Agent starts modifying itself, how do we continuously prove it's truly getting better. This is the emerging new challenge for test development in the Agent era, and the new capability the evaluation system needs to add.