Self-Evolving Agent ExLabel: Improving SWE Dataset Quality Assessment

Introduction

Large language models (LLMs) have demonstrated strong capabilities on software engineering (SE) tasks, and high-quality datasets are the foundation for reliable training and evaluation. SWE-bench-style benchmarks (SWE datasets) are widely used to evaluate the ability of LLM-based coding agents to solve real-world software problems, where task clarity and test coverage directly determine whether an instance is solvable and verifiable. However, existing SWE datasets commonly suffer from quality issues, and manual annotation is costly, prompting researchers to explore LLM-based automatic annotation approaches.

SWE-bench and SWE datasets: SWE-bench is a benchmark for evaluating the ability of LLMs to solve real-world GitHub software problems. Each instance contains a code repository, an issue description (issue), and test cases used to verify the fix. The model needs to generate a patch; if the patch makes the originally failing tests pass without breaking existing tests, it is considered successfully resolved. SWE-bench Verified is a manually verified subset that removes instances with unclear issue descriptions or unreliable tests, making evaluation more trustworthy. Multi-SWE-bench extends this paradigm to multiple programming languages, covering Java, Go, Rust, etc., to test the model's cross-language generalization ability. "Task clarity" in the dataset refers to whether the issue description is sufficient for a developer to locate and fix the defect; "test coverage" refers to whether the tests truly verify the fix, avoiding misjudging model success due to missing or weak tests. These two quality attributes directly determine the validity of the benchmark.

Limitations of Existing Methods

Current LLM-based annotation methods usually rely on static, overly generic rubrics, making it difficult to handle diverse and subtle scenarios, which often leads to high false positive rates. In contrast, human annotators can make context-aware judgments based on accumulated experience. This gap has given rise to a new idea: enabling LLM agents to learn from experience and dynamically adjust evaluation criteria, just like humans.

False positive rate: In dataset quality assessment, a false positive means that an evaluation method incorrectly judges an actually unsolvable or unverifiable instance as "passed" or "high quality." For example, when test coverage is insufficient, a model may be misjudged as having solved the problem because it passes weak tests; when the task description is vague, a model may pass due to a lucky guess. An excessively high false positive rate overestimates model capability and contaminates training data. Static scoring rules struggle to capture differences in the same wording across different repository contexts, and thus easily produce false positives. Reducing the false positive rate is the core motivation for ExLabel to introduce an experience bank and achieve context-aware judgment.

ExLabel: A Self-Evolving, Experience-Guided Evaluation Assistant

Inspired by the human annotation process, researchers proposed ExLabel—an LLM-based self-evolving, experience-guided assistant specifically designed for evaluating task clarity and test coverage in SWE datasets.

At the core of ExLabel is a continuously evolving experience bank. This experience bank distills cross-instance knowledge into stable and adaptive judgment criteria, and reuses relevant experience for guidance when evaluating new instances. This mechanism enables ExLabel to break free from the constraints of static rules and achieve more accurate and flexible evaluation.

Experience bank: An experience bank is an external memory mechanism that structures historical judgment experience into reusable knowledge. In ExLabel, it does not simply store instances; instead, it distills cross-instance evaluation patterns into stable and adaptive judgment criteria, such as "what patterns of missing tests indicate insufficient coverage" or "what issue description wording causes ambiguity in solvability." When evaluating a new instance, the system retrieves and reuses relevant experience and dynamically adjusts the scoring basis rather than applying fixed rules. This mechanism enables the agent to continuously learn from past annotations, gradually approaching the process by which human annotators make context-aware judgments based on experience, while avoiding the high false positive problem of static rules in new scenarios.

Construction of the SWE-TQ Benchmark

To systematically evaluate the performance of ExLabel, the research team constructed SWE-TQ—a new benchmark covering eight programming languages. This benchmark was obtained by cleaning manual annotations in SWE-bench and Multi-SWE-bench, providing more reliable ground truth for evaluating task clarity and test coverage.

Experimental Results

Experiments on three backbone LLMs show that ExLabel achieves the best F1 scores under all settings. On the Python and multilingual subsets, the F1 for task clarity evaluation improved by 44.2% and 49.9% relative to the best baseline, respectively; the F1 for test coverage evaluation improved by 24.0% and 18.4%, respectively.

More importantly, downstream results validate the practical value of ExLabel: fine-tuning a coding model on SWE-rebench data filtered by ExLabel solved 66.2% of the problems on SWE-bench Verified, 14.5% higher than the model trained on data filtered by the best baseline.

F1 score: F1 is the harmonic mean of precision and recall, used to comprehensively measure the accuracy of binary classification evaluation. In task clarity and test coverage evaluation, precision represents the proportion of instances judged as "clear" or "sufficiently covered" that are truly qualified, and recall represents the proportion of all truly qualified instances that are correctly identified. The higher the F1, the better the method balances reducing false positives and false negatives. ExLabel achieves the best F1 on all three backbone LLMs, indicating that its judgment strategy has consistent advantages across different model backbones, while the relative improvement reflects the gain of the experience-guided mechanism over the static-rule baseline.

Analysis and Outlook

The success of ExLabel validates the potential of the "experience-guided + self-evolving" paradigm in data quality assessment. Compared with static rules, dynamically accumulating and reusing experience is closer to the decision-making process of human annotators and can effectively reduce the false positive rate. The construction of the SWE-TQ benchmark also provides a reusable evaluation tool for subsequent research. In the future, this self-evolving agent approach is expected to extend to more data annotation scenarios that require nuanced judgment.

Conclusion

ExLabel significantly improves the accuracy of task clarity and test coverage evaluation in SWE datasets through a self-evolving experience bank, and directly benefits downstream model training. This work highlights the key role of high-quality data filtering in the development of coding agents and provides a new direction for LLM-assisted data annotation.

One-line Summary

ExLabel uses a self-evolving experience bank to improve SWE dataset quality assessment and significantly enhance downstream coding model performance.

Paragraph Summary

ExLabel is a self-evolving, experience-guided LLM assistant inspired by the human annotation process, designed to evaluate task clarity and test coverage in SWE datasets. To address the high false positive rate of static scoring rules, ExLabel builds an experience bank that distills cross-instance knowledge into reusable judgment criteria and dynamically adjusts evaluation. For systematic evaluation, researchers constructed the SWE-TQ benchmark covering eight programming languages. Experiments show that ExLabel achieves the best F1 on three backbone LLMs, with relative improvements of 44.2% and 49.9% in task clarity F1 and 24.0% and 18.4% in test coverage F1. A coding model fine-tuned on data filtered by ExLabel solved 66.2% of problems on SWE-bench Verified, 14.5% higher than the best baseline.