Surrogate Quality Score: A New Metric for Model Selection that Balances Accuracy and Complexity

In an era where engineering decisions increasingly rely on high-fidelity machine learning models, how to systematically select a model has become an underestimated yet critical problem. The traditional approach tends to focus solely on prediction accuracy, but a paper from arXiv proposes a different idea: the Surrogate Quality Score (SQS), a composite performance metric that incorporates both prediction accuracy and model complexity.

Why a Single Accuracy Metric Is Not Enough

The paper opens by pointing out a real-world contradiction: while high prediction accuracy at untrained points is certainly a necessary trait of a final model, model complexity, predictability, interpretability, scalability, and physics-awareness may be equally important—and in some application scenarios, even more critical.

This means that if a model achieves only a slight gain in accuracy while paying an order-of-magnitude higher complexity cost, it may not be the better choice from an engineering decision-making perspective. Model selection is inherently a multi-attribute trade-off problem, yet existing practice lacks a quantitative tool that can compare accuracy and complexity within the same framework. SQS is proposed precisely to fill this gap.

Surrogate models are a common tool in engineering optimization and uncertainty quantification: when a single run of high-fidelity simulation (such as finite element analysis or CFD) is too expensive, researchers train a computationally cheap approximation model on a small number of simulation samples to replace the true simulation for optimization iterations, sensitivity analysis, or Monte Carlo sampling. Common surrogate models include Gaussian processes (Kriging), polynomial response surfaces, radial basis functions, and various neural networks. The quality of a surrogate model directly determines the reliability of subsequent engineering decisions, and that quality is twofold: it must approximate the true response at unsampled points (accuracy) while remaining lightweight, stable, and interpretable (complexity and robustness). These are precisely the two aspects that SQS attempts to characterize simultaneously.

The Core Construction of SQS: Weighted Product and Adjustable Trade-off

The formulation of SQS is not complicated; its core idea is to normalize prediction accuracy and model complexity separately, then integrate the two into a unified score via a weighted product to reflect the overall quality of the model.

One detail worth noting: the metric design introduces a user-specified parameter to explicitly control the trade-off between accuracy and complexity. In other words, SQS is not a static score with fixed weights, but allows users to adjust the balance according to specific application scenarios and preferences—whether to prioritize accuracy or to favor a lightweight model. The paper emphasizes that this design incorporates human judgment and preferences into the model selection process, making the score both systematic and flexible.

Choosing a weighted product rather than a weighted sum is intentional in the metric construction. The product form means that poor performance in either dimension will significantly lower the total score, thereby preventing extreme choices—such as "extremely high accuracy but uncontrolled complexity" or "extremely simple model but concerning accuracy"—from being masked by high scores.

Normalization is a key prerequisite for constructing composite metrics. Accuracy metrics (such as R², RMSE, MAE) and complexity metrics (such as parameter count, training time, inference latency) have completely different dimensions and vastly different numerical ranges; direct multiplication or addition would let the larger-magnitude side dominate the result. Common approaches include min-max normalization (mapping to the [0,1] interval), z-score standardization, or rank normalization relative to the set of candidate models. The choice of normalization method affects the absolute value of SQS but typically does not change the relative ranking among candidate models—which is one reason the paper positions it as a comparison framework rather than an absolute quality standard.

From Heuristic to Operational: The Positioning and Boundaries of SQS

The paper candidly describes SQS as a heuristic integration method rather than a theoretically optimal metric derived from first principles. The authors thoroughly discuss and justify the construction of SQS and demonstrate its practical utility through an example.

This positioning is actually quite pragmatic. Model selection itself involves a great deal of subjective judgment, and attempting to solve it once and for all with a theoretically complete formula is unrealistic. The value of SQS lies in providing a systematic and objective comparison framework: different models can be scored under the same set of normalization standards, and trade-off preferences are explicitly expressed through parameters, reducing the room for "gut-feeling" decisions.

Heuristic methods refer to approaches constructed based on empirical rules and reasonable assumptions rather than rigorously derived from an axiomatic system. In the fields of model selection and multi-criteria decision-making, heuristic metrics are very common—for example, AIC and BIC balance goodness of fit against parameter count with penalty terms, while TOPSIS and the weighted sum method are used for multi-attribute ranking. The legitimacy of such methods does not come from mathematical proofs of optimality, but from whether they satisfy certain reasonable properties (monotonicity, consistency, interpretability) and whether they produce intuitive rankings in practice. By explicitly positioning SQS as heuristic, the paper is in effect managing reader expectations reasonably: it provides a reproducible comparison tool, not a universal optimal criterion.

Implications for Engineering Practice

For teams engaged in engineering modeling and surrogate model development, the significance of SQS lies in quantifying the trade-off relationships that have long existed in intuition. When the number of candidate models increases and accuracy gaps narrow, hidden costs such as complexity and interpretability often become decisive factors. SQS provides a reproducible scoring method that shifts model selection from "who has the highest accuracy" to "whose overall quality best matches the requirements."

The paper also hints at the metric's room for expansion: the current version focuses on the two dimensions of accuracy and complexity, while attributes such as interpretability and physics-awareness are also selection criteria acknowledged by the authors. Whether these dimensions can be incorporated into a unified scoring framework in the future is worth continued attention.

Conclusion

SQS is not intended to replace accuracy metrics, but to remind us that model selection is a multi-dimensional decision problem. Placing accuracy and complexity on the same table through an adjustable-weight composite score is the most direct contribution of this metric. For engineering teams that need to choose among multiple high-fidelity models, it offers a quantitative tool worth trying.

One-line Summary

The paper proposes the SQS composite metric, which uses a weighted product to unify accuracy and complexity into a single score to aid model selection.

Paragraph Summary

The article introduces a Surrogate Quality Score (SQS) proposed in an arXiv paper—a composite performance metric for surrogate model selection. The article points out that selecting models solely based on prediction accuracy has clear shortcomings in engineering practice: model complexity, interpretability, scalability, and physics-awareness are equally important, and a model that leads only slightly in accuracy while paying an order-of-magnitude complexity cost may not be better. SQS normalizes prediction accuracy and model complexity separately, then integrates them into a unified score via a weighted product, and introduces a user-specified parameter to explicitly adjust the trade-off between the two. The product form ensures that poor performance in either dimension significantly lowers the total score. The paper acknowledges that SQS is a heuristic method rather than a theoretically optimal metric; its value lies in providing a systematic, objective, and reproducible comparison framework, and it leaves room for incorporating more dimensions such as interpretability in the future.