Quant-Arena: An LLM Quantitative Trading Research Benchmark — Nine Frontier Models Still Can't Beat the Equal-Weight Market

Can LLMs Complete the Closed Loop of Quantitative Research?

Quantitative research isn't a contest of writing formulas — it's a complete iterative closed loop: researchers propose market hypotheses, implement them as executable signals, backtest on historical data, and then decide which strategies are worth keeping. Most existing benchmarks only examine one link in this chain — whether a formula is correct, whether code runs, or whether a single trade decision is reasonable. But a more critical question remains unresolved: can large language models independently complete this entire loop and produce strategies that remain effective beyond the research period?

Quant-Arena is designed to answer exactly this question. According to its paper, it is an end-to-end quantitative research benchmark built on real A-share data, accompanied by a public leaderboard. Its evaluation logic isn't complicated, but it's sufficiently brutal: models must construct features themselves, write stock-scoring code, receive feedback only from 2022–2023, then freeze a strategy and face final evaluation on the 2024 "sealed year."

Evaluation Design: Limited Feedback, Sealed Assessment

The core constraint of Quant-Arena lies in information isolation. During the development phase, models can only see data feedback from 2022–2023; once a strategy is frozen, it is evaluated on completely unseen 2024 data. This design directly targets the most common failure mode in quantitative research — overfitting. Strategies that perform excellently during the research period often rapidly fail out-of-sample, and the sealed-year mechanism turns this into a hard test.

The evaluator itself also mirrors real trading. According to the paper, the system employs a cash account model and imposes trading constraints and costs, including market impact estimated from a simulated order book. This means strategies cannot win through idealized execution assumptions — slippage and capacity constraints are both counted. Scoring is dominated by realized excess returns, supplemented by signal quality and the diversity of features used by the strategy as secondary dimensions.

Out-of-Sample Testing and Overfitting

In quantitative research, overfitting refers to a strategy excessively fitting the noise in historical data rather than genuine market patterns. A common detection method is to split data into in-sample (for development) and out-of-sample (for validation) portions. Quant-Arena's "sealed year" design is a strict version of out-of-sample testing: models can only access 2022–2023 data during development, and the final strategy is evaluated once on 2024 data with no ability to adjust based on 2024 results. This design simulates the real scenario a strategy faces an unknown future after going live, and can effectively expose spurious signals obtained merely through data snooping.

Market Impact and Order Book Simulation

Market impact refers to the effect trading behavior itself has on prices: large buy orders push prices up, large sell orders push prices down. Ignoring market impact in backtesting leads to overestimated strategy capacity and live performance far worse than backtests. Quant-Arena estimates impact costs by simulating an order book: the order book records buy and sell order quantities at different price levels, and the simulator estimates execution price slippage based on the ratio of the strategy's order size to the depth of resting orders. This brings evaluation closer to real trading environments, particularly constraining high-turnover or large-capital strategies.

Nine Frontier Models, None Beat the Equal-Weight Market

The paper evaluated nine frontier models, with GPT-6 Astra scoring highest. But the conclusion is not encouraging: no model beat the equal-weight market benchmark in the 2024 sealed test. This result deserves closer examination. The equal-weight market is an extremely simple benchmark that requires almost no research capability, yet nine frontier models — given complete development feedback — still failed to surpass it.

This doesn't mean the models have no research capability at all. The scoring includes signal quality and feature diversity, indicating that some models did construct structurally reasonable signals — they just didn't translate into realizable excess returns out-of-sample. In other words, the models may have learned the form of "looking like quantitative research" without grasping the market insights that can truly survive across cycles.

Equal-Weight Market Benchmark

The equal-weight market benchmark refers to a strategy that allocates capital equally across all constituent stocks in an index and rebalances periodically. It requires no stock-selection ability and only captures the market's average return (beta), so it is often regarded as the floor of "passive investing." In quantitative research, the equal-weight benchmark is used to distinguish a strategy's excess return (alpha) from overall market movements. If an active strategy cannot beat the equal-weight benchmark after deducting trading costs, its stock-selection signals are not creating additional value. Quant-Arena uses the equal-weight market as a comparison to directly test whether models possess research capability beyond naive allocation.

Why This Benchmark Matters

Quantitative trading has long been a touchstone for evaluating LLM capabilities. Financial data is highly noisy, has a low signal-to-noise ratio, and is strongly non-stationary — any overfitting will be ruthlessly amplified out-of-sample. Quant-Arena shifts the evaluation focus from "is the code written correctly" to "is the strategy usable," which is closer to the value judgment of actual research work.

At the same time, it adopts a public leaderboard and open submission format, allowing any model or agent harness to participate. This openness helps form a comparable community baseline and prevents everyone from separately claiming their models possess quantitative capabilities. For a field easily surrounded by marketing rhetoric, a public benchmark with a sealed test set is especially important.

Limitations and Open Questions

Currently Quant-Arena focuses on the A-share market, and whether its conclusions transfer to other markets still needs verification. A single sealed year (2024), while able to test overfitting, also carries the contingency of a single year's market environment — strategies of different styles may perform very differently depending on market conditions. Additionally, scoring is dominated by excess returns, which may lead participants to over-pursue returns while ignoring risk-adjusted performance, although signal quality and feature diversity provide some balance.

The most core question remains open: what capabilities do models need to truly complete the loop from hypothesis to deployable strategy? Quant-Arena provides a clear negative starting point — at least for now, frontier models haven't achieved it.

Risk-Adjusted Returns

Excess return is the difference between strategy return and benchmark return, but high returns may come with high risk. Risk-adjusted return measures the return per unit of risk through metrics such as volatility, maximum drawdown, or the Sharpe ratio. For example, Sharpe ratio = (strategy return - risk-free rate) / strategy return volatility. Looking only at excess returns may incentivize models to take excessive risk, whereas risk-adjusted returns can evaluate strategy quality more comprehensively. Quant-Arena currently is dominated by excess returns; introducing risk-adjusted metrics in the future would bring it closer to actual fund management needs.