FrontierPhysics: A New Benchmark for End-to-End Evaluation of Physics Research Agents

AI agents are playing an increasingly important role in physics research, but existing benchmarks often only evaluate fixed problems, isolated subtasks, or paper reproduction with given solutions, and rarely examine research quality and correctness at the same time. To address this, researchers have introduced FrontierPhysics—an end-to-end benchmark built by 42 physicists based on their own research, comprising 56 tasks spanning nine domains across theoretical, computational, and experimental physics.

Limitations of Existing Physics Benchmarks

Most current AI evaluations in physics focus on problem-solving ability: given a physics problem, the model is asked to output an answer. Such tests can measure knowledge reserves and computational skills, but they cannot reflect the complexity of real research workflows. Real research work goes far beyond problem-solving—it requires researchers to search the literature, identify gaps, design feasible approaches, execute experiments or simulations, analyze data, and ultimately produce a publishable manuscript. FrontierPhysics starts precisely from this gap, shifting the evaluation target from "answerer" to "researcher."

Deterministic verifiers refer to check programs that make automatic, reproducible judgments on task outputs—for example, checking whether numerical results fall within preset tolerances, whether code runs successfully and produces data files matching expectations, or whether cited references actually exist. They complement scoring methods that rely on subjective human expert judgment: verifiers guarantee objective correctness, while expert rubrics assess whether the research design is reasonable, the analysis is in-depth, and the manuscript reaches a publishable standard. In an end-to-end benchmark like FrontierPhysics, a single verifier can hardly cover every part of an open research workflow, so the two must be combined to prevent models from passing evaluations by "guessing answers" or piling on rhetoric.

Design and Composition of FrontierPhysics

All 56 tasks in this benchmark were designed by 42 physicists based on their own ongoing research, rather than drawn from textbooks or question banks. The tasks span nine domains, covering theoretical, computational, and experimental physics. Each task requires the agent to complete the full research loop: reviewing the literature, formulating a research plan, conducting research and experiments, and generating structured outputs and a scientific manuscript. Evaluation combines deterministic verifiers with expert-written rubrics, checking both the correctness of results and whether the research reaches a publishable standard.

Performance and Shortcomings of Frontier Models

Researchers tested 22 models in 26 configurations and found that no configuration passed more than 46 of its 168 runs. This result intuitively demonstrates that end-to-end physics research remains a serious challenge for current AI. Manual review of agent trajectories further revealed the root causes of failure: the main weaknesses of frontier agents lie in insufficiently rigorous reasoning and a lack of expert intuition, rather than insufficient physics knowledge or execution errors. In other words, models "know" physics but are not very good at "doing" physics research—they struggle to make judgments in ambiguous situations, become alert to data anomalies, and make tasteful trade-offs in research directions the way human experts do.

Agent trajectories refer to the complete sequence of actions an agent produces while completing a task, including every tool call, intermediate reasoning step, file read/write, code execution, and result observation. Reviewing trajectories rather than only final answers makes it possible to locate where failures occur—whether literature retrieval missed key work, the research plan contains logical leaps, or anomalies were ignored during data analysis. The manual review of trajectories in FrontierPhysics is based precisely on this: the model's shortcomings are not in physics knowledge itself, but in reasoning rigor and expert intuition, and such problems often manifest in intermediate steps yet are not reflected in whether the final output is correct.

Implications for AI Research Agents

The value of FrontierPhysics lies not only in providing a leaderboard, but also in pointing out directions for improvement. Knowledge injection and tool use are certainly important, but the core of research capability may lie in rigorous chains of reasoning and the cultivation of domain intuition. For teams building research agents, this means emphasizing the model's ability to self-verify within open research workflows and to form reasonable scientific judgments from limited information. The benchmark also provides a reusable evaluation framework for subsequent research, driving AI's evolution from a "problem-solving tool" to a "research partner."

Expert intuition in the research context refers to experts' ability to make reasonable judgments when information is incomplete and results are ambiguous—for example, sensing that a data distribution "looks off," judging that one research direction is worth pursuing while another should be abandoned early, or choosing among multiple feasible approaches based on taste and experience. It is difficult to acquire through standard question-bank training and hard to measure with automated metrics. FrontierPhysics lists the lack of expert intuition as a primary cause of failure, suggesting that the evaluation and training of research agents need to introduce open-ended situations without standard answers, allowing models to practice self-verification and scientific judgment under uncertainty, rather than merely improving knowledge coverage and tool-call success rates.

One-line Summary

42 physicists built the end-to-end benchmark FrontierPhysics, revealing that AI still has obvious shortcomings in doing physics research.

Paragraph Summary

FrontierPhysics is an end-to-end physics research benchmark built by 42 physicists based on their own research, comprising 56 tasks spanning nine domains of theoretical, computational, and experimental physics. Unlike existing benchmarks that only test problem-solving or reproduction, it requires agents to complete the full research loop from literature review and research planning to experiment execution and a scientific manuscript, and uses deterministic verifiers plus expert rubrics to check both correctness and publishability. Testing 22 models in 26 configurations, no configuration passed more than 46 of 168 runs. Trajectory review shows that the main weaknesses of frontier agents are insufficient reasoning rigor and a lack of expert intuition, rather than insufficient physics knowledge, which points the way forward for improving research agents.