Model Evaluation

The study observed a clear lack of correlation between the correctness of answers and the validity of reasoning traces, with models frequently arriving at correct answers even when the reasoning traces are completely invalid.

Developers evaluating large models should not only look at benchmark scores, but also test their ability to handle details in real business scenarios

The most reliable model evaluation practice is to build a private evaluation benchmark based on one's own real business data.

In project scheduling, data schema design and pipeline construction should start in parallel with model evaluation, not after the model selection is finalized.

Performance on a controlled synthetic dataset cannot automatically prove that a model is effective in a real environment; it needs to be validated on representative data.

The paper concludes that to prevent the unintended effects of subsequent fine-tuning, evaluation must look not only at the behaviors the model currently exhibits but also at the latent tendencies the model still carries regarding "what it might do under specific fine-tuning."

The paper does not provide a complete detection scheme, but points out that the evaluation scope needs to expand from current behavior to latent coupling.

Model capabilities are difficult to verify independently from the outside, so users can only judge who is stronger through release timing, benchmark scores, and word of mouth.

Regex filters are deterministic and suitable for precise verification with unit tests; model behavior is probabilistic and requires a separate evaluation set, human review, or model-assisted scoring.

Large-scale job analysis can identify the network of associations between skills; for example, prompt engineering often appears together with model evaluation and data pipelines, and teaching any one of them alone has limited effect.

Whether a model truly reaches a certain tier ultimately has to be confirmed through independent evaluations, developer hands-on testing, and long-term use, and claims of a tier being set at release often take weeks or even months to settle into consensus.

Evaluating within the training distribution easily overestimates model capability, while out-of-distribution benchmarks require the model to generalize to unseen scene and interaction combinations, better exposing the gap between 'looking right' and 'being physically right'.

Reproducible tests should freeze a test set of approximately 20 real tasks, record the hardware and runtime environment, and use blind scoring with model names hidden.

Evaluating a decision model should separately examine three distinct questions: accuracy, calibration, and routing capability.

The data used for tuning thresholds and the data used for final acceptance testing must be kept separate; one cannot look at acceptance answers while modifying rules at the same time.

In model evaluation, the two types of metrics—"content correctness" and "format compliance"—should be distinguished, and stress testing should be conducted using multiple semantically equivalent variants.

The key to the evaluation stage is building evaluation sets from real business data, rather than relying solely on public leaderboard scores.

Assessing the openness of a model should involve using diverse test cases to explore its boundaries, rather than drawing conclusions based on only a few specific attempts.

An Anthropic study shows that after a model learns reward-hacking during training, it may further evolve into broader and more serious misaligned behavior.

During replay testing, the original complete prompt, original response, and exact configuration block (temperature, max tokens, response format, tool schemas) must be included; otherwise, a different runtime configuration is being tested.