Can Lightweight LLMs Handle Intent-to-SLO Translation? Five-Dimensional Evaluation Reveals the Answer

Intent-Based Networking (IBN) automatically generates network configurations from natural language intents, and its first step—translating intents into executable SLOs—is increasingly relying on large language models. However, the cost and energy consumption of large models have prompted researchers to explore the potential of lightweight open-source models. A recent paper conducted a five-dimensional evaluation of 10 LLMs with fewer than 10B parameters from 5 families, covering translation performance, intent validation, hallucination, energy impact, and robustness to diverse phrasings.

Evaluation Framework and Experimental Setup

The research team tested lightweight models on 716 placement intents using four prompt settings (including zero-shot and few-shot reasoning). Evaluation dimensions included: strict F1 score, invalid intent rejection rate, hallucination rate, GPU energy consumption, and paraphrase robustness. All models had fewer than 10B parameters, aiming to verify whether they can replace large models in resource-constrained environments.

Strict F1 Score: F1 score is the harmonic mean of precision and recall, commonly used to measure overall performance in classification or sequence labeling tasks. In the intent-to-SLO translation task, if the model's output SLO is treated as a predicted "entity" or "segment," strict F1 requires the prediction to exactly match the ground truth in both boundary and content to be counted as correct; any partial match or format deviation is considered an error. Therefore, a strict F1 score of 0.943 means that the model-generated SLOs fully match the human-annotated expected results in the vast majority of cases, demonstrating high-quality structured output.

Few-shot Reasoning: Few-shot reasoning refers to providing a small number of input-output examples in the prompt to guide the LLM to complete a new task following the example pattern, without updating model parameters. Compared to zero-shot reasoning (which only provides a task description without examples), few-shot reasoning can significantly improve the model's understanding of output format and task constraints, reducing hallucination rates. In this study, the few-shot setting helped lightweight models follow translation rules more accurately by showing several intent-to-SLO conversion examples.

Translation Performance and Intent Validation

The best configuration achieved a strict F1 score of 0.943 on the intent translation task, indicating that lightweight models can translate natural language intents into SLOs with high accuracy. In terms of invalid intent rejection, the best model correctly rejected 191 out of 195 invalid intents, demonstrating validation capability approaching practical levels.

Impact of Hallucination and Prompting Strategies

Few-shot reasoning significantly reduced hallucination rates: the strongest model produced unsupported QoS requirements in about 1% of responses, while performance under zero-shot settings was noticeably worse. This shows that prompt engineering is crucial for the reliability of lightweight models, and proper example guidance can greatly improve output accuracy.

Hallucination: In the context of LLMs, hallucination refers to the model generating plausible but unfounded content that is inconsistent with the input or facts. In the intent-to-SLO translation task, hallucination specifically manifests as the model outputting QoS (Quality of Service) requirements not mentioned in the original intent, such as fabricating bandwidth, latency, or reliability metrics. Such hallucinations cause the generated SLOs to deviate from the user's true intent, leading to incorrect network configurations. Therefore, the study uses "unsupported QoS requirements" as a quantitative indicator of hallucination and finds that few-shot reasoning can reduce the hallucination rate to about 1%.

Energy Analysis and Cost-Effectiveness

Without using reasoning, translating one intent consumes 0.044-0.148 Wh of GPU energy. For weaker models, the additional energy cost of adding reasoning steps yields performance improvements, making it cost-effective. This finding provides a basis for deploying lightweight models on edge devices or in resource-constrained environments.

Paraphrase Robustness Challenge

Real-world paraphrasing reduces the rejection rate of invalid intents: in 15 out of 16 configurations, the best model showed a decrease in rejection rate. This indicates that the robustness of intent validation remains a key open challenge for reliable IBN, as models are prone to misjudgment when faced with diverse phrasings.

Conclusion and Outlook

Lightweight LLMs show considerable potential in intent-to-SLO translation, especially with few-shot reasoning assistance, achieving high-precision translation and low hallucination rates. However, insufficient paraphrase robustness suggests the need for further research on intent validation mechanisms. This study provides empirical evidence for deploying lightweight models in cost-sensitive scenarios and has open-sourced code and data to facilitate future work.

One-line Summary

Lightweight LLMs can translate intents to SLOs with high accuracy; few-shot reduces hallucination, but paraphrase robustness remains unsolved.

Paragraph Summary

This paper evaluates 10 lightweight LLMs with fewer than 10B parameters on the intent-to-SLO translation task in Intent-Based Networking (IBN). The study uses four prompt settings on 716 placement intents and examines five dimensions: translation performance, intent validation, hallucination, energy consumption, and paraphrase robustness. The best configuration achieves a strict F1 score of 0.943 and correctly rejects 191 out of 195 invalid intents; few-shot reasoning reduces the hallucination rate to about 1%. In terms of energy, translating one intent consumes 0.044-0.148 Wh of GPU energy, and adding reasoning steps is cost-effective for weaker models. However, real-world paraphrasing reduces the invalid intent rejection rate, with decreases in 15 out of 16 configurations, indicating that the robustness of intent validation remains an open challenge. The study provides empirical evidence for deploying lightweight models in cost-sensitive scenarios.