When News Lags the Market: Should LLM Trading Agents Track or Follow?

Introduction: News and Prices—Which Comes First?

In the world of quantitative trading, the relationship between news and prices is far more subtle than one might imagine. Most online news about public companies is often published after stock price movements have already occurred, and the content reported is precisely the price change that has already happened. This means that an LLM agent reading the news can perfectly align with the market simply by "following" it, without needing to genuinely "understand" the news event itself.

This is exactly the core question that the paper Track or Follow? A Jev-Compatible Multi-Agent Trading System When Web News Lags the Market aims to answer: Is the LLM's judgment of news tracking (track) the market's reaction to events, or is it following (follow) price movements that have already been reported?

An Isolated Multi-Agent Trading System

To rigorously distinguish between "tracking" and "following," the researchers built a multi-agent trading system. Each agent in the system can only access a point-in-time data store and answers typed questions through Jev—a hosted typed decision model interface. The code sets each position within the proof boundary of the text's impact range on positions, ensuring that the effect of textual judgments is strictly constrained.

The key to this design is information isolation: agents cannot access future information, nor can they bypass the interface to directly manipulate positions. The researchers also employed a hash-committed testing approach to prevent post-hoc tampering with judgment records, ensuring the reproducibility and credibility of the experiments.

Point-in-time data store is a critical infrastructure used in quantitative backtesting to avoid look-ahead bias. It only exposes information that was genuinely available at a given historical moment, strictly shielding data that was published or revised only after that moment. In news and trading scenarios, this means that agents can only see news, prices, and announcements that were publicly available at the time of decision-making, and cannot "peek" at subsequent corrections or supplementary disclosures. Point-in-time data is a prerequisite for distinguishing "tracking" from "following," because if future information leakage were allowed, agents might inadvertently exploit price movements that were only confirmed after the fact, distorting the evaluation results.

Hash commitment is a cryptographic commitment mechanism: experimenters compute and publicly disclose the hash value of judgment records or datasets before the experiment begins, and afterward can only reveal the original content without being able to tamper with it, otherwise the hash won't match. In the tests in this paper, hash commitment is used to lock down the agents' decision records and test data, preventing researchers from modifying judgments or data after seeing the results to manufacture statistical significance. It enhances the reproducibility and credibility of experiments and is an important engineering technique for auditing and adversarial testing of multi-agent trading systems.

Core Finding: Judgments Can Indeed Track Reactions

The research team built a Jev-compatible reader based on fixed LLM option probabilities and conducted hash-committed tests. The results showed a statistical association between item judgments and abnormal returns in announcement windows (weekly Spearman correlation coefficient 0.119), and this association still holds after the LLM's own publication. Event type judgments were also consistent with the issuer's 8-K filing types (AUC 0.833).

More specifically, Jev's judgments of news that appeared only after its own publication could also track market reactions (descriptive results). This indicates that the system does not simply rely on historical price memory to "cheat," but possesses a certain degree of semantic understanding capability.

In adversarial attack tests, decision cards successfully traced 88% of attacks back to the implanted news items, demonstrating strong interpretability and traceability of the system.

Spearman correlation coefficient is a non-parametric rank correlation measure used to assess the strength of monotonic relationships between two variables, ranging from -1 to 1, and does not require data to follow a normal distribution. In financial text research, it is often used to test whether LLM judgment rankings are consistent with abnormal return rankings: 0.119 is a weak positive correlation, but if statistically significant with sufficient sample size, it still indicates that textual judgments contain some ranking information. AUC (Area Under the Curve) measures the discriminative ability of binary classification models, where 0.5 is equivalent to random guessing and 1.0 is perfect discrimination; AUC 0.833 indicates that event type judgments are highly consistent with 8-K filing types, demonstrating good classification performance.

Where Does the Association Come From? Exploratory Analysis Reveals the Truth

Although confirmatory tests showed statistical significance, exploratory analysis revealed the true source of the association and exposed the problems:

• 72% of earnings news appears after the issuer's filing: That is, most earnings-related news is itself "old news"—it reports events that have already occurred and been absorbed by the market.
• Items written by contributors often directly report price changes: The news content itself contains price information, so when the LLM makes judgments, it is actually reading prices rather than events.
• From the first trade after publication, the association drops to 0.023: Once the market begins trading, the information value in the news rapidly decays, indicating that the association mainly comes from pre-publication price momentum.

These findings point to a conclusion: the LLM's "tracking" ability is largely the extraction of price information already contained in the news, rather than independent judgment of event fundamentals.

Abnormal return refers to the excess return of an individual stock after removing the portion explained by the overall market or risk factors, typically estimated using market models or Fama-French factor models. Announcement window abnormal return is the cumulative abnormal return within the event window, used to measure the market's reaction to specific information. This paper uses it as a proxy variable for "market reaction" to test whether LLM judgments are related to the true impact of events. If the news itself merely reports price changes that have already occurred, then the association between abnormal returns and LLM judgments may come from price momentum rather than semantic understanding—this is exactly what exploratory analysis aims to disentangle.

Methodological Insight: Evaluate from Publication Time, Not Event Time

On weekly time windows, confirmatory tests did not detect ranking information beyond pure code features. This means that after controlling for price features, the LLM's textual judgments did not provide additional predictive power.

The researchers therefore propose an important recommendation: Market scoring tests for web text readers should use the news publication time as the starting point for evaluation criteria, not just the event occurrence time. This methodological correction can more realistically reflect the value of text readers in real trading scenarios and avoid overestimating their event understanding capabilities.

Implications for AI Trading Systems

This study sounds a warning for LLM financial applications. When news lags the market, seemingly intelligent textual judgments may simply be carriers of price information. When building trading systems, it is necessary to rigorously distinguish between "reading events" and "reading prices," and to verify agents' true capabilities through information isolation, point-in-time data constraints, and adversarial testing. Typed decision interfaces like Jev provide position constraints and traceability, offering an engineering foundation for the safety of multi-agent systems.

For researchers and practitioners, when evaluating LLM financial text understanding capabilities, it is essential to consider the time gap between news publication and event occurrence; otherwise, it is easy to mistake "following" for "tracking."

One-line Summary

Research using an isolated multi-agent system reveals that LLM news judgments are mostly following already-reported prices rather than genuinely tracking events.

Paragraph Summary

The paper Track or Follow? builds an information-isolated multi-agent trading system that, through the Jev typed decision interface and point-in-time data store, rigorously distinguishes whether LLM judgments of news are "tracking" the market's reaction to events or "following" already-reported price movements. Confirmatory tests found a weak correlation between item judgments and announcement window abnormal returns (weekly Spearman 0.119), consistency between event types and 8-K filings (AUC 0.833), and 88% traceability in adversarial attacks. However, exploratory analysis revealed that 72% of earnings news is published after the issuer's filing, contributors often directly report price changes, the association drops to 0.023 from the first trade after publication, and after controlling for price features, there is no additional ranking information in weekly windows. The study recommends that market scoring tests should start from the news publication time rather than the event occurrence time.