The PHYRD Benchmark: A Rigorous Touchstone for AI Theoretical Physics Reasoning

The potential of large language models in physics research is being systematically tested. The recently proposed PHYRD benchmark is specifically designed for theoretical physics reasoning and contains 63 Reproduce and 43 Discover problems, offering a new yardstick for evaluating whether AI truly understands physical derivations.

Design Philosophy: The Dual Challenge of Reproduce and Discover

PHYRD divides tasks into two categories: Reproduce requires models to derive published theoretical results, while Discover requires models to derive new conclusions after modifying theoretical conditions. This design distinguishes "rote recall" from "flexible reasoning." All tasks were constructed with FormulaForest and come with reference answers and sub-question-level scoring rubrics, ensuring fine-grained evaluation.

Sub-question-level scoring rubrics are the key mechanism by which PHYRD achieves fine-grained evaluation. Traditional benchmarks usually only judge whether the final answer is correct, but theoretical physics arguments often contain multiple derivation steps, and cases where the final answer is right but intermediate steps are wrong, or where the approach is correct but the calculation is flawed, are common. Sub-question scoring decomposes a problem into several independently judgeable reasoning segments (such as assumption setting, equation formulation, use of approximations, and limit checks), scoring each segment separately. This can distinguish "lucky guesses" from genuine mastery of the derivation, and can also pinpoint the specific segment where the model fails, providing a more precise feedback signal for subsequent training.

Model Performance: Overall Accuracy Still Has Clear Shortcomings

Tests on five models show that full-problem accuracy on Reproduce tasks ranges from 49.21% to 68.25%, while accuracy on closed-book Discover tasks fluctuates more widely, from 18.60% to 74.42%. This indicates that models perform reasonably well when reproducing known derivations but are highly unstable on discovery tasks that require creative reasoning. Tool use helps four models complete more Discover problems, but conceptual errors and missing arguments remain.

According to the paper's data, even with tools, models still struggle to avoid misunderstandings of physics concepts or omissions of key derivation steps. This suggests that current AI reasoning capabilities in theoretical physics are still far from reliable application.

Closed-book versus tool use are two different evaluation settings. Closed-book means the model answers using only the knowledge in its own parameters and is not allowed to call calculators, symbolic computation systems, or retrieve external literature; the tool-use setting allows the model to call external tools (such as Mathematica or Python symbolic libraries) to assist with algebraic manipulation and numerical verification. Distinguishing these two settings makes it possible to determine whether model failures stem from deficiencies in the reasoning chain itself or from limited computational capability. The four models in the paper can complete more Discover problems with tool assistance, indicating that part of the bottleneck lies in computational execution rather than physics understanding, but conceptual errors still remain, showing that tools cannot compensate for deficiencies at the level of physical intuition.

The Value of Synthetic Data: FormulaForest Improves a Downstream Benchmark

Another contribution of PHYRD is validating the effectiveness of FormulaForest as a synthetic data source. On the FrontierScience benchmark developed by OpenAI, after using data generated by FormulaForest, the accuracy of the Qwen-35B-A3B model rose from 61% to 72%. This result shows that carefully constructed theoretical physics problems can not only evaluate models but also feed back into model training, improving their performance on scientific reasoning tasks.

The FrontierScience benchmark is a scientific reasoning evaluation set introduced by OpenAI, covering frontier problems in disciplines such as physics, chemistry, and biology, and emphasizing difficult problems that require multi-step reasoning and specialized knowledge rather than textbook-style standard exercises. It is known for its high difficulty and strong discriminative power and is often used to test the upper limits of models on expert-level scientific tasks. The PHYRD authors chose this benchmark as the downstream validation target to test whether synthetic data generated by FormulaForest can transfer to an evaluation system outside PHYRD and constructed by a different institution, thereby ruling out self-validation bias ("only effective on our own benchmark") and strengthening the external validity of the conclusions.

Challenges and Outlook

As a "living benchmark," PHYRD will be continuously updated to reflect the frontiers of theoretical physics research. Current results show that AI still faces challenges in conceptual understanding and multi-step argumentation in theoretical physics reasoning. In the future, with the optimization of model scale and training data, combined with tool use and synthetic data, AI is expected to play a more active role in physics discovery. But before that, benchmarks like PHYRD will continue to remind us: true physics reasoning is far more than pattern matching.

"Living benchmark" is a benchmark design philosophy proposed to address the problems of data contamination and saturation in static benchmarks. Once a traditional benchmark is publicly released, its problems may enter the training corpora of subsequent models, causing inflated scores; at the same time, as model capabilities improve, fixed problems will gradually saturate and lose discriminative power. Living benchmarks maintain the freshness and difficulty of evaluation by periodically adding new problems, retiring outdated ones, and tracking the latest research results. For fast-evolving frontier fields such as theoretical physics, living benchmarks can continuously reflect models' reasoning levels on genuinely unseen problems, but they also bring new challenges such as version comparability, evaluation cost, and problem quality control.

One-line Summary

The PHYRD benchmark uses Reproduce and Discover tasks to test large models' theoretical physics reasoning and reveals shortcomings in conceptual understanding.

Paragraph Summary

PHYRD is a new benchmark designed specifically for theoretical physics reasoning, containing 63 Reproduce tasks and 43 Discover tasks. The former require models to derive published results, while the latter require deriving new conclusions after modifying theoretical conditions, thereby distinguishing rote recall from flexible reasoning. Tests on five models show that full-problem accuracy on Reproduce tasks ranges from 49.21% to 68.25%, while closed-book Discover tasks fluctuate widely from 18.60% to 74.42%. Although tool use can improve completion rates on Discover tasks, conceptual errors and missing arguments remain common. The study also validates the value of FormulaForest synthetic data: after using data it generated, Qwen-35B-A3B's accuracy on OpenAI's FrontierScience benchmark rose from 61% to 72%. As a continuously updated living benchmark, PHYRD will keep tracking AI's true reasoning capabilities at the frontiers of theoretical physics.