Reference-Free Evaluation: Ensemble Learning Boosts Trustworthiness of LLM Code Summaries

Are LLM-generated code summaries actually reliable? This question becomes especially tricky when there is no human-written reference summary. A paper from arXiv proposes a reference-free confidence estimation method based on ensemble thinking, specifically targeting the COBOL code summarization scenario, attempting to approximate "truth" through "consensus" among models.

Core Idea: Let Multiple Models "Vote" on Correctness

The paper's core intuition is not complicated: if summaries generated by multiple independent models are semantically consistent, then the summary is more likely to be correct. This assumption—that "inter-model consistency reflects truthfulness"—provides an actionable path for quality assessment in reference-free scenarios.

Unlike traditional methods that rely on string matching or embedding similarity, this research introduces LLM-as-a-judge for pairwise comparison, arguing that LLM judges can provide higher-fidelity semantic-level judgments than traditional metrics. This means the evaluation dimension upgrades from "literal similarity" to "semantic equivalence," better aligning with the actual quality requirements of code summaries.

LLM-as-a-judge is a method that uses large language models themselves as evaluators, typically through carefully designed prompts that ask the model to perform pairwise comparisons or scoring on two or more candidate outputs. Unlike traditional metrics based on n-gram overlap such as BLEU and ROUGE, LLM judges can capture higher-level attributes such as semantic equivalence and logical coherence, making them particularly suitable for tasks like code summarization that require understanding program intent. The typical workflow is: place the two summaries to be compared along with the original code into a prompt, ask the model to determine which summary is better or whether they are semantically equivalent, and output structured results (e.g., "A is better than B," "equivalent," "contradictory"). The advantage of this approach is that it requires no human-annotated reference summaries and offers finer judgment granularity; the risk is that the judge may inherit biases from training data or exhibit decreased judgment stability in unfamiliar domains (such as COBOL).

Three-Step Methodology: From Generation to Aggregation

The method consists of three interconnected stages, forming a complete evaluation pipeline.

1. Controlled Summary Generation

Summaries are generated from multiple diverse models rather than relying on a single model's output. This diversity is the foundation for subsequent "consensus" judgments—if all summaries come from the same model, consistency loses its reference value.

2. LLM Judge Pairwise Comparison

An LLM evaluator is responsible for comparing summaries pairwise and outputting structured judgments. Unlike traditional scalar scoring, pairwise comparison can capture finer-grained semantic differences, and structured outputs facilitate subsequent aggregation.

3. Calibrated Aggregation Rules

This is the final stage of the method: combining supporting evidence while penalizing only "confirmed contradictions." This design is quite thoughtful—not all disagreements are equivalent; only cross-validated contradictions should reduce confidence, avoiding misjudgments caused by noise.

Aggregation of pairwise comparisons is a key technical component of this method. When N models generate summaries, the LLM judge performs pairwise comparisons of summary pairs, producing a pairwise judgment matrix. The aggregation rules need to transform these local comparisons into a global confidence score for each summary. Common aggregation approaches include: graph-theoretic methods (e.g., treating "equivalence" as edges and finding the maximum consistent subset), voting-based methods (counting the number of "support votes" each summary receives), or probabilistic model-based methods (treating pairwise judgments as noisy observations to infer the underlying true quality ranking). The paper emphasizes "penalizing only confirmed contradictions," meaning aggregation is not simple vote counting but involves cross-validating contradictory judgments—confidence is reduced only when multiple independent sources point to the same contradiction, thereby suppressing misjudgments caused by individual judge noise. This design draws on robust aggregation ideas from ensemble learning.

Why This Is Especially Important in the COBOL Context

The paper's choice of COBOL as the application scenario is not coincidental. COBOL code exists in large quantities in legacy systems, but its syntax and business logic are relatively unfamiliar to modern LLMs, making the reliability of generated summaries all the more in need of verification. At the same time, the COBOL domain lacks large-scale human-annotated reference summaries, giving reference-free evaluation methods practical value.

The method's applicability may not be limited to COBOL. Any code summarization task lacking human reference summaries can draw on the approach of "diverse generation + LLM judging + calibrated aggregation" to build automated confidence estimation mechanisms.

COBOL (Common Business-Oriented Language) is a business-oriented programming language created in 1959 that still supports a large number of critical legacy systems worldwide in finance, insurance, government, and other sectors. It is estimated that hundreds of billions of lines of COBOL code are still running globally, yet developers who know the language are increasingly scarce. COBOL's syntax style differs significantly from modern languages: it uses fixed-format column layout, makes heavy use of natural-language-style verbs (such as MOVE, PERFORM, COMPUTE), and business logic is often implicit in verbose procedural code. Modern LLM training corpora are dominated by mainstream languages like Python and Java, with relatively sparse coverage of COBOL, making generated summaries more prone to semantic deviations. Meanwhile, COBOL legacy systems generally lack documentation and human reference summaries, making reference-free evaluation an urgent necessity. This background explains why the paper chose COBOL as the validation scenario—it is both a weak spot for LLMs and the domain where reference-free evaluation needs are most pressing.

Trade-offs and Limitations Behind the Method

It is worth noting that this method depends on the judgment quality of the LLM judge. If the judge itself has biases, the credibility of the aggregation results will also be compromised. The paper mitigates this issue through "penalizing only confirmed contradictions," but the judge's capability ceiling remains an implicit constraint of the method.

Additionally, the computational overhead from multi-model generation and pairwise comparison is also a cost factor to consider in practical deployment.

For researchers and engineers focused on evaluating LLM code understanding capabilities, this work provides a reusable methodological framework: replacing human references with ensemble consensus, and replacing literal matching with semantic judgment.

One-line Summary

The paper proposes an ensemble consensus method that uses multi-model generation and LLM judging to achieve reference-free confidence estimation of COBOL code summaries.

Paragraph Summary

This paper addresses the difficulty of evaluating LLM-generated code summaries when human references are unavailable, proposing a reference-free confidence estimation method based on ensemble thinking, with COBOL code summarization as the application scenario. The core assumption is: if summaries generated by multiple independent models are semantically consistent, they are more likely to be correct. The method consists of three stages: controlled summary generation from diverse models; pairwise comparison using LLM-as-a-judge to obtain higher-fidelity semantic judgments than traditional string or embedding similarity; and finally, calibrated aggregation rules that combine supporting evidence while penalizing only cross-validated contradictions, avoiding confidence damage from noisy disagreements. The paper notes that this method has practical value in COBOL legacy systems lacking human reference summaries and can be transferred to other code summarization tasks, but it depends on judge quality and incurs significant computational overhead.