30-Day Test: How Robust Are Chatbot Policy Judgments?

Chatbots are increasingly used to advise on public affairs, but their recommendations are only truly useful if their judgments are robust—that is, stable and reproducible. A recent 30-day empirical study systematically evaluated the stability of five mainstream chatbots' ratings of U.S. presidential actions and hypothetical reforms through ordinary consumer chat interfaces, revealing the reliability boundaries and potential risks of model judgments.

Experimental Design: Policy Ratings in Everyday Conversations

The research team simulated ordinary user scenarios, asking five chatbot configurations twice daily for 30 days: Mistral, Gemini Flash, Claude Sonnet, and two Claude Opus settings with different effort levels. The questions asked models to rate 40 actions and 15 hypothetical reforms across the past four U.S. presidential administrations, providing scores of long-term impact on the U.S., confidence levels, and brief commentary. The experiment ultimately collected 16,500 judgment data points, which were publicly released.

The key to this design is the "consumer interface"—no API calls, no system prompt modifications, fully simulating how real users interact with chatbots. As a result, the findings more closely reflect the quality of advice ordinary people actually receive, rather than performance under idealized laboratory conditions.

Robustness Performance: Highly Consistent, but with About 4% Divergence

Overall, the chatbots' judgments demonstrated impressive robustness. All configurations were completely consistent in the sign (positive or negative) and ranking of average government ratings. This means that regardless of which model was used, there was no disagreement on the relative ordering of administrations' overall performance.

However, divergence did exist. The study found that cases where two different configurations gave opposite sign judgments on the same action on the same day accounted for about 4%. In other words, if you and a friend separately ask different chatbots about the same policy question, roughly 1 in 25 times you will get conclusions pointing in opposite directions. This low-probability but non-negligible inconsistency remains a signal worth heeding for users who rely on a single model's advice.

Robustness in the machine learning context refers to the degree to which a model's output remains stable and reproducible when inputs undergo minor perturbations, environmental changes, or repeated calls. It differs from accuracy: a model may predict very accurately, but if each answer differs slightly, its robustness is low. For chatbots, robustness involves two aspects—the self-consistency of the same model on the same question (whether multiple calls yield the same judgment), and consistency across different models (whether different models reach the same conclusion). This article uses "sign agreement rate" and "ranking consistency" as robustness measures, which are evaluations at the external behavioral level rather than internal parameter stability. Understanding this helps distinguish two different dimensions: "correct judgment" and "stable judgment." High robustness does not guarantee correct judgment, but low robustness means advice cannot be relied upon.

Temporal Dimension: Greater Cross-Day Variation, Traces of Service Changes

The study further compared same-day and cross-day judgment differences. Results showed that answers obtained on different dates differed more than those within the same day, but differences grew slowly over time unless the service itself changed.

The research team identified three server-side changes that left almost no trace on the user interface yet significantly altered model judgments. This finding is particularly important: users often cannot perceive updates behind the model, but judgments may have already shifted. For ordinary users who treat chatbots as policy reference tools, this "invisible drift" means that advice trustworthy yesterday may not be equally reliable today.

A server-side update refers to a model provider updating model weights, inference configurations, safety policies, or system prompts in the background, while the user interface and API version numbers may remain completely unchanged. Common triggers include continuous fine-tuning, alignment strategy adjustments, load balancing switches, or A/B testing. Because consumer interfaces typically do not expose model version information, users cannot perceive which specific version they are interacting with. This opacity leads to "invisible drift": the same entry point and the same question format may yield systematically shifted judgments due to backend updates. In public affairs consulting scenarios, this drift is especially dangerous because users often continue using the same product based on historical trust without realizing that the basis for advice has changed. Establishing model change logs or version notifications is a common engineering practice to mitigate this problem.

Commentary Quality: Specific and Consistent Across Runs

Beyond ratings, the study also analyzed the brief commentaries generated by models as a retrieval task. Results showed that commentary content was specific to each individual item and remained consistent across multiple runs and different chatbots. Vocabulary similarity and embedding similarity analyses yielded concordant results, indicating that models are stable not only in scoring but also in the reasoning they provide.

This finding has positive implications for practical applications: when users ask a chatbot to explain its evaluation of a policy, the reasons obtained are not randomly generated but closely related to the specific policy content and reproducible. This enhances the interpretability and credibility of model judgments.

Embedding similarity refers to the method of mapping text into high-dimensional vectors and comparing semantic proximity through measures such as cosine similarity. Unlike vocabulary similarity (e.g., Jaccard coefficient, edit distance), which focuses only on literal overlap, embedding similarity can capture cases where "the wording is different but the meaning is the same." This article uses both vocabulary and embedding metrics to analyze commentary consistency; if the two conclusions align, it indicates that the model is stable not only in word choice but also in semantic content, rather than creating an illusion of consistency through templated phrasing alone. This dual-metric cross-validation is relatively rigorous in text reproducibility research, because a single vocabulary metric may overestimate consistency (the model repeatedly uses the same phrases but with empty semantics), while a single embedding metric may be affected by vector space bias. Mutual corroboration of the two enables more reliable judgment of whether commentary truly addresses specific policy content.

Implications for Users and Developers

This study sends mixed signals. On one hand, mainstream chatbots overall demonstrate high robustness in policy judgments, and cross-model, cross-time ranking consistency is commendable. On the other hand, the approximately 4% opposite judgments, the invisible shifts caused by server-side changes, and the accumulation of cross-day differences all remind users not to treat a single model's single response as absolute authority.

For developers, the lack of transparency around service changes revealed by the study deserves attention. When model updates cause judgment shifts, if the interface provides no notification, users may unknowingly receive advice different from before. Establishing change logs or version notification mechanisms may be a necessary step to enhance user trust.

The 16,500 judgment dataset released by the study also provides a valuable benchmark for future research, helping to continuously monitor the evolution of chatbot stability in public affairs judgments.

One-line Summary

30-day test of five chatbots' policy ratings: overall high robustness, but about 4% opposite judgments, and invisible server-side changes cause drift.

Paragraph Summary

A 30-day empirical study systematically evaluated the rating stability of Mistral, Gemini Flash, Claude Sonnet, and two Claude Opus configurations on U.S. presidential actions and hypothetical reforms through ordinary consumer chat interfaces, collecting 16,500 judgment data points. Results showed that all configurations were completely consistent in the sign and ranking of average government ratings, indicating high overall robustness; however, cases where different configurations gave opposite sign judgments on the same action on the same day accounted for about 4%. Cross-day differences exceeded same-day differences, and the study identified three server-side changes that left no trace on the interface yet significantly altered model judgments. Model-generated commentaries were specific to individual items and consistent across runs and models, with vocabulary and embedding similarity analyses corroborating each other. The study publicly released the dataset, providing a benchmark for future robustness monitoring.