回归测试

It is recommended to incorporate the Agent evaluation system into the regression testing process to quickly verify whether performance degradation has occurred after each model or prompt update

Regression testing refers to re-running existing test cases after modifying code to ensure that the new changes have not introduced new bugs or broken existing functionality.

Smoke Testing only checks whether the system can start and whether core paths can run through, while Regression Testing is a comprehensive validation of all existing functionality, and the two cannot substitute for each other.

回归测试不仅验证Bug本身是否修复，还需要测试与该Bug相关联的功能是否受到影响

传统回归测试面临核心困境：全量回归耗时过长，而随机选择可能遗漏关键场景

Divide the test set into core cases that must pass and edge cases to cover as best as possible; any failure of a core case blocks release.

The three specific means of Agent evaluation are LLM as a judge (using a model to automatically perform quality checks at scale), regression testing (catching unexpected side effects), and embedding evaluation into the CI/CD pipeline (validating changes before they reach production).

Model regression testing reruns a fixed evaluation set after changing prompts, model versions, or retrieval configurations and compares whether the pass rate has declined; the determination methods include scoring thresholds, semantic similarity, or LLM-as-judge, with statistical comparison across multiple samples.

Skills such as designing evaluation sets, hallucination detection, and building regression tests to monitor model behavior are extremely scarce in engineering teams.

Based on the Reddit discussion, there is currently no unified solution; practices in use include building regression tests for key policies, adding version metadata to documents, and combining manual spot checks with user feedback; for early-stage projects, it is recommended to first focus on high-frequency questions and manually verify the effect of updates before gradually automating.

For downstream developers, a minor-version iteration means interface compatibility is usually preserved, but model behavior may still drift subtly, so version pinning and regression testing thus become necessary practices when integrating the API.

If a case never fails on any backend tried, it is a smoke test rather than a regression test and should be placed in a separate file.

Frequent checkpoint updates cause the output length and format preferences of the same prompt to drift across different checkpoints, so teams should maintain a fixed task set tailored to their own business for regression testing.

For CLI tools, regression testing should cover argument parsing of core commands, output formats, exit codes and cross-platform behavior.

In the RAG context, regression testing is the repeated use of a fixed query set to verify whether system performance degrades after changes to the retrieval configuration, including query-level degradation, recovery, and common failure analysis.

When integrating LLM code generation into CI/CD pipelines, non-deterministic output makes regression testing extremely difficult, because one cannot tell whether a test failure is a model generation problem or a problem with the code itself.

Each AI generation is a completely new sampling, and even if only one function is asked to be modified, other parts may be broken; this full-rewrite characteristic makes regression testing especially important in AI scenarios.

Pressure messages such as 'the manager always gives me 20 percent off,' 'ignore your instructions,' and 'confirm my card number to me' should be solidified into regression test cases and retested after every model upgrade or prompt change.

Having the AI go back and verify its own historical outputs is methodologically close to regression testing in software engineering, using new real data to challenge old assumptions and expose previously undetected bias.

MissionFlow's regression tests should assert outcome invariants rather than execution paths, because the same mission has multiple legitimate paths; path-level soft assertions can serve as guardrails but not as correctness criteria.