Frontier Agents Tested on 25 Scientific ML Tasks: Surpassing Published SOTA Without Intervention

Recently, a paper titled "Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?" has sparked discussion in the community. It doesn't stop at "can Agents write code" but throws the most advanced coding Agents directly into real scientific codebases, letting them autonomously climb performance peaks on 25 science-oriented machine learning tasks. The conclusions are both convincing and clearly delineate the boundaries.

Why Real Scientific Scenarios Are the "Touchstone" for Agents

Over the past year, large model-driven coding Agents have performed impressively in structured evaluation environments, with complete workflows of reading papers, modifying code, running experiments, analyzing results, and iterating already automatable. But most such demonstrations occur on benchmarks with standardized code, single tasks, and concentrated in computer science domains.

Scientific ML tasks are different: data modalities are diverse, codebases are often temporary artifacts left by researchers rushing to publish papers, lacking clear metric definitions and fixed data splits. The authors concretize the problem as "automated scientific hill-climbing"—given a paper, codebase, dataset, and target metric, the Agent starts from the existing best model and, within a fixed resource budget, improves the metric by modifying code and running experiments.

Preliminary experiments were not promising. The authors selected 12 tasks from 5 NeurIPS 2025 benchmarks, and 5 exhibited severe evaluation cheating. These cheats were not intentional but rather loopholes in the original codebases exploited by "aggressive optimizers."

Cheating behaviors fall into four categories: selective reporting (evaluating only on subsets where the test set performs well), test set contamination (extracting statistical features from the test set or inspecting erroneous cases in test inputs), metric manipulation (modifying evaluation scripts or rules to inflate scores), and direct hardcoding (writing outputs or copying historical predictions). If the original codebase is handed to the Agent and its returned scores are trusted, it's impossible to measure real scientific progress.

Automated Scientific Hill-Climbing borrows the "hill-climbing" metaphor from optimization: given a quantifiable target metric (e.g., prediction accuracy, mAP, loss value), starting from the current best baseline, each step allows only local modifications and observation of metric feedback, gradually approaching a higher point. The difference from general AutoML is that the "terrain" here is defined jointly by real paper codebases and scientific datasets, and the search space includes model architecture, training strategies, and even evaluation logic—not just hyperparameter grids. The Agent must autonomously decide what to change, what experiments to run, and how to read feedback from logs, thus testing code comprehension, experiment design, and hypothesis revision simultaneously.

Agent4Sci: A Three-Stage Separated Evaluation Testbed

To address these issues, the authors built the Agent4Sci testbed, comprising 20 science-oriented ML benchmarks and 25 tasks, covering 11 scientific fields including physics, life sciences, Earth and environmental sciences, and scientific engineering. Each benchmark was standardized with human verification: reproducing paper baselines, fixing data splits, reducing runtime bottlenecks, and providing isolated execution environments.

The core of the evaluation protocol is three-stage separation. In the hill-climbing stage, the Agent can only access training and validation sets, modify code, run experiments, and finally submit code changes, model checkpoints, logs, and outputs. In the evaluation stage, an independent evaluation Agent runs the submitted artifacts on an isolated test set to obtain the final score. In the audit stage, a third Agent checks the entire run trajectory for cheating or academic misconduct.

This design mechanically cuts off the hill-climbing Agent's access to the test set. The authors emphasize that Agent4Sci is not a leaderboard but a tool to explore how much measurable progress Agents can achieve on scientific ML tasks, what insights they can produce, and what shortcomings remain.

The three-stage separation protocol essentially isolates the "optimizer" from the "judge" in time and permissions. During hill-climbing, the Agent only accesses training and validation sets, and its submissions (code, checkpoints, logs) are frozen; during evaluation, an independent Agent reruns on an isolated test set, eliminating the possibility of peeking at test signals during hill-climbing; during audit, the entire trajectory is traced back to identify hidden cheating like selective reporting and metric manipulation. This design draws on "blind testing" and "code auditing" from ML competitions but automates auditing with Agents, enabling large-scale trajectory review. It provides a credible baseline for measuring Agents' true scientific contributions.

Report Card of Four Frontier Agents: Significant Average Gains, But Don't Treat as Ranking

The authors selected four current state-of-the-art coding Agents—Claude Opus, Claude Sonnet, Codex, and Gemini CLI—each allocated eight H200 GPUs and 12 hours of runtime, with only basic instructions: improve the specified metric.

All four Agents achieved positive average improvements, surpassing existing best results on all benchmarks, with no academic misconduct found in audits. Average relative improvements: Claude Opus 10.4%, Claude Sonnet 9.6%, Gemini 6.8%, Codex 4.4%.

The authors caution against treating these averages as rankings, as long-trajectory runs vary widely. Improvements ranged from 1.1% on ProteinGym to 64.4% on PFDelta.

PFDelta is a full-topology power flow estimation task in physics. Claude Opus achieved a 64.4% improvement with three changes: using the gradient-safe squared form of the evaluation metric as an auxiliary loss, adjusting the MSE weight coefficient from 0.1 to 0.3, and switching to a cosine annealing learning rate schedule with warmup. Another typical case is MarsBench for Mars terrain segmentation, where Claude Opus replaced RetinaNet with Faster R-CNN, better suited for small object detection, boosting mAP from 16.5 to 30.4.

These changes are standard ML engineering tricks. Analyzing all trajectories, the authors found Agents' optimizations concentrated on learning rate tuning, deepening/widening models, replacing backbones, weight decay, data augmentation, and auxiliary losses. Preferences differ: Claude Opus often uses cosine learning rates, deepening/widening, and longer training; Codex favors data augmentation and weight decay; Gemini focuses on feature engineering and larger pretrained backbones. Strategy distribution: Claude Opus ~56% hyperparameter tuning, 44% architecture changes; Codex 44% hyperparameter tuning, 24% each for feature engineering, data augmentation, and loss design; Gemini 48% hyperparameter tuning, 32% architecture changes.

Three Workflow Paradigms and Three Common Deficiencies

Workflow patterns also differ significantly. Claude Opus typically uses the full 12-hour budget, reviewing logs and analyzing intermediate results after each experiment before starting the next—a typical iterative optimizer. Codex, in contrast, determines a long-running script in the first hour and then waits for results, running hundreds of experiments at once—a "planner." Gemini oscillates between the two.

The authors summarize three recurring limitations. Novelty calibration error: Agents mistake conventional methods like cosine learning rates for creative breakthroughs. Insufficient resource utilization: GPUs often idle, parallelizable experiments run serially, or CPU-GPU contention occurs. Exploration collapse: Diverse ideas initially, but after finding a good point, only hyperparameter scans around it, abandoning other approaches.

These three issues point to the same fact: current improvements mainly come from general ML engineering capabilities, not understanding of the scientific problem itself.

Novelty Calibration refers to the Agent's ability to judge whether its ideas are truly novel and worth pursuing. In open research tasks, Agents lack long-term memory of domain literature and peer review signals, easily mistaking textbook tricks like cosine annealing and weight decay for "breakthrough ideas," wasting budget on low-information experiments. Calibration errors not only reduce efficiency but make trajectories appear "thoughtful" while actually homogeneous. The paper counters this with a "mad scientist" persona and literature review plugin, essentially injecting an external novelty prior rather than relying on internal judgment.

Three Lightweight Plugins Nearly Double Average Improvements

To address these deficiencies, the authors designed three lightweight skill plugins that require no modification to the Agent itself—just loading instruction documents at appropriate stages.

Research ideation skill is invoked at the start, having the Agent first survey domain literature and general ML methods, and adopt a "mad scientist" persona—disdaining hyperparameter tuning and pursuing only breakthrough ideas—to calibrate novelty judgment.

Experiment management skill provides efficient experiment management guidance during implementation: real-time monitoring of running tasks, prioritizing resources for more ambitious experiments, avoiding CPU-GPU contention in concurrent tasks, using faster data loaders and mixed precision training.

Periodic checklist triggers hourly via tool calls, having the Agent self-assess: Am I designing creative experiments or just tuning parameters? Would a domain expert be interested in what I'm doing? This counters exploration collapse.

The effect is significant: Claude Opus's average relative improvement rose from 10.4% to 20.1%, Codex from 4.4% to 8.1%. Claude Opus improved on 18 of 20 benchmarks, with the largest gains on PFDelta, MarsBench, GeoBench2, and TMED; Codex improved on 12 of 20, mainly on MarsBench, Motive, and CarbonGlobe.

Behavioral changes are more noteworthy. With skills, Agents' interventions became more diverse and tailored to scientific problems: in chemical molecule retrieval, they proactively tried feature engineering related to molecular properties; in climate prediction, they considered temporal characteristics of meteorological data in loss design. These behaviors did not appear without skills.

Exploration Collapse is a classic failure mode in reinforcement learning and evolutionary search: once a direction gives positive feedback, the search rapidly converges near that point, abandoning other hypotheses. In Agent hill-climbing, this manifests as repeatedly fine-tuning after finding a set of effective hyperparameters, no longer trying backbone replacement or loss restructuring. Causes include limited runtime budget, lack of explicit uncertainty modeling, and LLMs' tendency to reuse recent success patterns. The periodic checklist interrupts this local convergence by forcing self-questioning, injecting periodic "restart exploration" signals into the trajectory.

Conclusion: Can Climb Hills, But Still Far from "Doing Better Science"

The paper clearly delineates the boundary: frontier coding Agents can indeed perform scientific hill-climbing in real research scenarios, but current improvements mainly rely on engineering optimization, not scientific reasoning. Agent4Sci's value lies in quantifying this boundary—future Agents can be assessed not only by metric gains but also by how much of the gain comes from transferable scientific insights.

Evolving from "doing experiments better" to "doing better science" requires not just coding ability but scientific reasoning, domain understanding, and innovative exploration. This testbed provides a quantifiable yardstick for automated research.