Pterodactyl: A New Ultra-Low-Cost Approach to Machine Translation Quality Evaluation

In the field of machine translation quality evaluation, accuracy and cost are often hard to reconcile. The Pterodactyl system submitted by 1020 Group to the WMT26 shared task attempts to break this deadlock. The system is designed specifically for Subtask 2 (span-level quality score prediction) and Subtask 3 (error-free span detection), emphasizing high customizability and ultra-low cost, with particular optimization for binary classification scenarios in translation filtering and data cleaning.

Core Idea: Extreme Tuning of LLM-as-a-judge

Pterodactyl does not train a new neural network. Instead, it adopts the "LLM-as-a-judge" paradigm, building an evaluator through prompt engineering and model selection. The research team uses ROC analysis to guide prompt design and model selection, enabling the system to achieve competitive evaluation performance while maintaining extremely low inference costs. This approach transforms the evaluation task into a classification problem under prompting, avoiding reliance on large-scale annotated data and expensive model fine-tuning.

According to the paper, Pterodactyl's standout feature is its high customizability—users can adjust prompt templates, model parameters, and decision thresholds according to specific scenarios, thereby adapting to different translation quality standards and data distributions.

LLM-as-a-judge refers to not training a dedicated evaluation model, but instead directly calling a general-purpose large language model and using prompts to have it make judgments or assign scores to inputs (such as translation quality). Its typical workflow is: design evaluation criteria and output format → construct prompt templates → have the model output structured results → calibrate thresholds with a small amount of annotated data if necessary. Compared with fine-tuning a dedicated model, this paradigm requires no large-scale annotated data or training compute, and deployment costs mainly come from inference calls; however, the trade-off is sensitivity to prompt wording, model version, and context order, which may cause position bias or score drift. Therefore, statistical methods such as ROC analysis are often needed to select prompts and models.

ROC analysis (receiver operating characteristic curve analysis) was originally used to evaluate the discriminative ability of binary classifiers: plotting the true positive rate against the false positive rate at different thresholds, where a larger area under the curve (AUC) indicates a stronger ability to distinguish positive from negative samples. In systems like Pterodactyl, ROC analysis can be used to compare the discriminative power of different prompt templates and candidate models on the judgment of "whether a translation contains errors," and to help select decision thresholds that balance recall and precision. Compared with looking only at a single accuracy metric, ROC can reveal system performance under different misjudgment costs, which is especially important for translation filtering scenarios—the costs of missing low-quality translations and falsely rejecting acceptable translations are often asymmetric.

Variant Design for Subtask 3

For Subtask 3 (detecting error-free spans), the team developed multiple Pterodactyl variants. These variants are targeted adaptations on top of the base system to better handle the binary classification judgment of "whether an error exists." Although the paper does not disclose the specific technical details of the variants, it clearly states that these variants were adjusted specifically for Subtask 3, demonstrating the flexibility of the system architecture.

The practical value of this design approach lies in the fact that translation companies and data annotation teams often need to quickly filter out low-quality translations. The ultra-low-cost solution provided by Pterodactyl makes large-scale automated filtering possible without investing substantial computing resources.

The WMT automatic quality evaluation shared task typically divides subtasks by evaluation granularity: sentence-level or document-level quality scoring, span-level quality score prediction, and error-free span detection. Span-level evaluation requires the system to locate erroneous spans in the translation and provide local quality judgments, rather than scoring only the whole sentence; error-free span detection focuses more on binary classification—determining whether a given span is completely correct, often used to select high-quality spans from parallel corpora that can be directly used for training. Such binary classification tasks have high precision requirements because misjudgments directly contaminate subsequent data cleaning or model training pipelines, which also explains why Pterodactyl designed separate variants for Subtask 3.

An Evaluation Philosophy Prioritizing Efficiency

Pterodactyl's core competitiveness lies in efficiency. In practicality-oriented evaluations like WMT, low-cost, easy-to-deploy systems are often more valuable in real-world deployment than solutions that pursue extreme scores at the cost of enormous resource consumption. 1020 Group's choice reflects a pragmatic engineering orientation: using mature LLM capabilities combined with refined prompt engineering to find a balance between cost and effectiveness.

For teams that need to continuously process massive amounts of translation data, this "good enough and cheap" evaluator may be more attractive than high-scoring models in the lab. Pterodactyl's exploration also provides a reusable technical path for automated translation evaluation—driving prompt optimization through ROC analysis, allowing general-purpose large models to deliver performance close to dedicated systems on specific evaluation tasks.

Limitations and Outlook

The current information mainly comes from the system description, lacking detailed experimental data and comparative results. Pterodactyl's specific performance in the WMT26 shared task, its gap compared with other participating systems, and its stability across different language pairs remain to be disclosed in the full paper. In addition, how to mitigate the prompt sensitivity, position bias, and other issues inherent in the LLM-as-a-judge paradigm under ultra-low-cost constraints is also a direction worth following up on.

One-line Summary

Pterodactyl combines LLM-as-a-judge with ROC tuning to achieve span-level quality evaluation at ultra-low cost.

Paragraph Summary

1020 Group submitted the Pterodactyl system to the WMT26 automatic quality evaluation shared task, targeting Subtask 2 (span-level quality score prediction) and Subtask 3 (error-free span detection). The system does not train a new model; instead, it adopts the LLM-as-a-judge paradigm, building an evaluator through prompt engineering and model selection, and uses ROC analysis to guide prompt design and model selection, achieving competitive performance at extremely low inference cost. Its core advantages are high customizability and ultra-low cost, aimed at binary classification scenarios in translation filtering and data cleaning, and the team also developed multiple adapted variants for Subtask 3. The article also notes that specific experimental data and cross-lingual stability await disclosure in the full paper, and issues such as prompt sensitivity and position bias also warrant follow-up attention.