Baidu ERNIE Evolve Automated Research System Beats NanoChat and NanoGPT Records

AI doing research autonomously is moving from proof of concept to quantifiable engineering practice. Baidu's ERNIE Evolve is an automated research system whose workflow can be summarized as: rewrite code, run experiments, and keep only changes that are verified to be effective. According to MarkTechPost, the system set new best records on two public benchmarks—NanoChat and NanoGPT Speedrun—and accelerated inference for ERNIE-5.5-Flash.

What ERNIE Evolve Actually Does

ERNIE Evolve is positioned not as an "AI assistant" but as a closed-loop automated research pipeline. It edits code itself, launches experiments itself, and then filters the results—only modifications that are experimentally verified to bring gains are kept, while the rest are discarded.

This "keep only verified gains" mechanism is the most critical design of the entire system. Automatically generating code is nothing new; the hard part is determining whether a change actually makes the system better. ERNIE Evolve delegates that judgment to experimental data, forming a self-iterating feedback loop rather than one-off code completion.

In terms of capability boundaries, it covers engineering research tasks with clear evaluation metrics, such as model training and inference optimization—tasks that share a common trait: experiments can be run automatically, and scores can be obtained automatically.

Performance on NanoChat and NanoGPT Speedrun

The report mentions that ERNIE Evolve surpassed existing best records on two public benchmarks: NanoChat and NanoGPT Speedrun.

The significance of these two benchmarks in the community is worth explaining. Benchmarks like NanoGPT Speedrun focus on compressing the training pipeline to be as fast as possible under fixed conditions—it tests not model scale but optimization capability at the training-script level, including kernels, parallelism strategies, hyperparameters, and data pipeline adjustments. Such tasks have single, well-defined evaluation metrics, making them ideal for automated systems to repeatedly trial and error.

NanoChat is likewise an automatically evaluable dialogue/training benchmark. ERNIE Evolve's ability to lead on these two public leaderboards shows that its "edit code—experimentally verify—keep gains" loop has usable output capability in real engineering tasks, not just in running through a pipeline in a synthetic environment.

NanoGPT Speedrun is a class of community challenges aimed at "training GPT to a target loss as fast as possible," with representative projects such as modded-nanogpt. It fixes model scale, dataset, and evaluation method; participants can only compress training time by improving the training script itself. Common optimization techniques include using more efficient attention kernels (such as FlashAttention), adjusting optimizers and learning rate schedules, improving data loading and parallelism strategies, and adopting more aggressive hyperparameter combinations. Because each submission automatically yields clear training time and loss metrics, such benchmarks are naturally suited as a testing ground for automated trial-and-error systems.

NanoChat, on the other hand, is a lightweight dialogue model training and evaluation benchmark that focuses on completing the full pipeline from data to dialogue capability under a small compute budget, with evaluation dimensions typically including dialogue quality and training efficiency. What it shares with NanoGPT Speedrun is: clear task boundaries, automatically computable metrics, and reproducible experiments—making it suitable for measuring the output capability of automated research systems in real training engineering, rather than merely verifying whether a pipeline runs.

Accelerating Its Own Model: ERNIE-5.5-Flash Inference Optimization

More noteworthy than topping leaderboards is the second result: ERNIE Evolve accelerated inference for ERNIE-5.5-Flash.

This is an internal deployment case. If an automated research system can only produce results on public benchmarks, its value is limited; being able to deliver actual inference performance improvements for its own production model means the system has been integrated into real engineering pipelines. Inference acceleration typically involves a large number of fine-grained changes such as operator fusion, scheduling strategies, memory management, and batching configurations—exactly the type of work suited for automated systems to experiment on at high frequency and verify item by item.

However, the report does not disclose the magnitude of the acceleration, which specific components were optimized, or whether these changes have been deployed to production services. The absence of this information makes it difficult for outsiders to assess the scale of its actual gains.

Large model inference acceleration is a highly engineered optimization effort. Common approaches include: operator fusion (merging multiple small operators to reduce kernel launch and memory access overhead), KV Cache management and paged attention, quantization (low-precision inference such as INT8/FP8), continuous batching and scheduling strategy adjustments, and kernel rewriting for specific hardware. These changes often yield only a few percent of gain at each point and interact with one another, requiring extensive ablation experiments to confirm whether they are truly effective—exactly the typical scenario where the automated "edit—experiment—keep gains" loop can play a role. But precisely because per-point gains are small and experimental noise is large, if the evaluation protocol is not rigorous enough, it is easy to mistake fluctuations for improvements.

Value and Boundaries of Automated Research Systems

The imaginative space for such systems lies in lowering the marginal cost of research iteration. A manual ablation experiment—from modifying code to running it to analyzing results—takes hours or even days per cycle; an automated system can try more directions in parallel and eliminate ineffective ones promptly. For problems with clear objective functions, such as inference optimization and training acceleration, the gains are most direct.

The boundaries are equally clear. First, it relies on automatically evaluable metrics—open-ended research questions and creative work requiring human judgment currently cannot fit into this loop. Second, "keep only verified gains" means it is sensitive to noise; when experimental design is not rigorous, random fluctuations can easily be mistaken for real improvements. Third, leading on public benchmarks does not equal leading in general research capability—benchmarks themselves can be targeted for optimization.

Automatically evaluable metrics are the prerequisite for such systems to operate. "Automatically evaluable" means the task has a score that can be computed without human intervention, such as training loss, throughput, latency, or benchmark accuracy. The opposite is open-ended research questions—for example, proposing motivations for new architectures, designing new evaluation dimensions, or judging whether an idea is worth pursuing—which depend on researchers' judgment and domain intuition and are difficult to compress into a scalar reward. Moreover, a single scalar reward also brings the risk of "reward hacking": the system may find solutions that score higher under the evaluation protocol but do not represent real capability improvements, which is especially a concern on public benchmarks.

Summary

ERNIE Evolve demonstrates a pragmatic path: rather than pursuing the narrative of an "AI scientist," it converges automated research onto engineering tasks with clear evaluation signals, using experimental verification as the gatekeeper. The records on NanoChat and NanoGPT Speedrun, along with the inference acceleration of ERNIE-5.5-Flash, are two types of evidence this path has produced so far—one proving capability externally, the other proving usability internally. What is truly worth watching next is whether this system can continuously produce results, and whether its optimized outcomes can stably transfer to larger-scale production models.