LLMs Challenge Fine-Tuned Models: New Findings from KG-to-Text Benchmarks

Knowledge Graph-to-Text: LLMs vs. Fine-Tuned Models Head to Head

The knowledge graph-to-text (KG-to-text) task requires models to convert a set of knowledge graph triples into fluent natural language. For a long time, fine-tuned encoder-decoder models were the standard approach for this task, but as large language models (LLMs) rapidly improve, the gap between the two is being redefined. A recent empirical study systematically compared the performance of recent LLMs and fine-tuned models on standard benchmarks, revealing how conclusions differ depending on the evaluation dimension.

Research Methodology: A Dual-Track Evaluation Framework

The study adopted two parallel evaluation frameworks. The first is traditional reference-based metrics, which compare model outputs against human-written reference texts. The second is graph-grounded reference-free LLM-based evaluation, validated through human annotation. This dual-track design is itself a reflection on existing evaluation methods—a single metric may not fully capture generation quality.

The research team conducted a head-to-head comparison of recent LLMs and fine-tuned encoder-decoder models on the same benchmarks, covering multiple standard datasets. The evaluation focused not only on text fluency but also on factual consistency between generated content and the input graph.

Reference-based metrics (such as BLEU, ROUGE, and METEOR) score by computing n-gram overlap or semantic similarity between generated text and human reference texts. These metrics are computationally efficient and highly reproducible, but have inherent limitations: they reward surface-level imitation of reference wording while failing to determine whether the generated content is faithful to the input graph. For the same set of triples, different annotators may write reference texts with vastly different wording, and a model output that is factually completely correct may still be penalized for different lexical choices.

Graph-grounded reference-free LLM evaluation bypasses reference texts and directly asks an LLM to judge factual consistency between generated text and the input graph—for example, whether triples are omitted or whether information not present in the graph is introduced (contradiction). This type of evaluation is closer to the core goal of KG-to-text—faithfully conveying graph content—but its reliability depends on the evaluator LLM's capabilities and prompt design. The study therefore introduced human annotation for validation to calibrate the credibility of LLM evaluation.

Key Findings: Gap Narrowed but Evaluation Dimensions Diverge

On reference-based metrics, recent LLMs have substantially narrowed the gap with fine-tuned models, and in some settings have even caught up. This result is not surprising—LLMs' advances in language fluency and surface-form matching are evident.

More noteworthy, however, is the reversal on graph-grounded metrics: LLMs outperform fine-tuned models in reducing contradictions and triple omissions. In other words, while fine-tuned models are better at "imitating" the wording of reference texts, they are actually more prone to errors when it comes to faithfulness to the input graph.

The study also found limited correlation between LLM-based evaluation scores and reference-based metrics. This suggests that the two approaches capture different aspects of generation quality: the former focuses more on semantic consistency and factual accuracy, while the latter leans toward lexical and syntactic similarity. For tasks like KG-to-text that emphasize faithfulness, relying solely on reference metrics may mask model deficiencies at the factual level.

Triple omission refers to a triple in the input graph that is completely unmentioned in the generated text—an information-missing error. Contradiction refers to generated text that expresses content conflicting with the input graph—for example, reversing relation direction or misstating entity attributes—an information-distortion error. In KG-to-text tasks, these two error types directly undermine the factual reliability of generated text, far more seriously than lexical mismatches. Because reference-based metrics mainly measure surface similarity, they are insensitive to penalizing omissions and contradictions—a model can easily produce sentences highly similar to reference text while omitting key triples. This is precisely why the study introduced graph-grounded evaluation: only by anchoring directly to the input graph can these two types of factual defects be systematically detected.

Graph Scale Effects: A Hidden Concern for Fine-Tuned Models

A particularly critical finding is the difference in error rate growth as graph size increases. As input graphs grow larger (more triples), contradictions and omission errors in fine-tuned models grow faster, while errors in prompt-based LLMs grow relatively gently.

This points to an essential difference between the two modeling paradigms: fine-tuned models perform excellently within the training distribution but have limited generalization ability when facing more complex, longer inputs; LLMs, with stronger context understanding and instruction-following capabilities, maintain better robustness under scaled inputs. For knowledge graphs that often contain dozens of triples in practical applications, this difference may directly affect deployment choices.

Implications for KG-to-Text Practice

This study conveys several signals worth attention from practitioners. Evaluation systems need upgrading: relying solely on reference metrics like BLEU and ROUGE may underestimate LLMs' true capabilities and also overlook fine-tuned models' factual deficiencies. Graph-grounded evaluation should become a standard component of KG-to-text tasks.

Model selection requires scenario-based trade-offs: if the application scenario has strict reference template requirements for text style, fine-tuned models still have advantages; but if greater emphasis is placed on factual faithfulness and robustness to large-scale graphs, prompt-based LLMs are the safer choice. One detail: LLMs' advantages are built on good prompt design, and practical deployment still requires prompt optimization for specific domains.

The study also leaves an open question: how to feed graph-grounded evaluation signals back into model training so that fine-tuned models can improve factual consistency while maintaining their advantage on reference metrics—this may be a direction worth exploring in the next phase.