Can LLMs Autonomously Discover Hidden Physical Laws? A New Study Weighs In

Scientific discovery often begins with an unexpected observation that prompts researchers to ask new questions. So, if the physical laws governing the world were quietly altered, could large language models (LLMs) notice anomalies like human scientists would (for example, heavier objects falling more slowly) and uncover the underlying laws through autonomous experimentation? A new study offers a preliminary, exploratory answer to this question.

An "Alternate World" Testbed With No Research Task

To evaluate LLMs' autonomous discovery capabilities, researchers built an automated framework called LongChang. The framework generates a series of simulated worlds in which physical laws have been modified—one or more laws differ from everyday physics. The LLMs under evaluation play the role of "explorers": they can manipulate objects, build tools, and independently decide what to investigate, all without being given a research task or any hints about these laws.

To ensure the exploration process is closed-loop and objective, LongChang also deploys independent LLM agents responsible for generating and reviewing scenarios, auditing tool designs, and evaluating reports for correctness and experimental support based on the altered physical laws and exploration records. These exploration records further guide scenario evolution, allowing the entire exploration loop to run continuously without human intervention.

Multi-Agent System refers to a computational system composed of multiple agents with some degree of autonomy, each possessing independent perception, decision-making, and action capabilities, and completing tasks that a single agent cannot handle through communication and collaboration. In the LLM context, a multi-agent system typically means having multiple LLM instances each play different roles (such as generator, reviewer, and evaluator), achieving division of labor and checks and balances through structured interaction. The core advantage of this architecture is twofold: on one hand, role separation can reduce the subjective bias of a single model, making evaluation closer to the objectivity of "peer review"; on the other hand, each agent can process different subtasks in parallel, improving the efficiency and scalability of the overall process. LongChang leverages this idea by splitting scenario generation, tool review, and report evaluation among different agents, thereby building an automated evaluation loop that requires no human intervention.

Discovery Rate of Only 16.7%: Most Models Came Up Empty

The research team tested 12 LLMs. The results were quite stark: only GPT-6-Astra achieved a discovery rate of 16.7%, meaning it successfully discovered the modified physical laws; the other 11 models all had a discovery rate of 0%. This means that under the free exploration setting, the vast majority of LLMs failed to proactively identify anomalous phenomena and further derive the hidden physical laws.

This data reveals the enormous gap in current LLMs' open-ended scientific discovery capabilities. Even if models possess certain reasoning and planning abilities, they still struggle to continuously track anomalous clues and design effective experiments to test hypotheses without explicit goal guidance.

Clues in the Exploration Records: Anomalies Ignored, Laws Only Partially Identified

Beyond the discovery rate, researchers also analyzed the exploration records to understand why models failed. The records show that models frequently encountered uninvestigated anomalies—they observed strange results but did not follow up on them. Moreover, even when models did take action, they could only partially identify the altered physical laws and were unable to fully derive the underlying regularities.

These findings indicate that a core bottleneck for LLMs in autonomous scientific discovery lies in "problem generation": they may be able to execute predefined experimental tasks, but struggle to proactively propose research questions worth investigating from passive observations and to integrate scattered clues into complete law descriptions.

Hypothesis Generation is a key step in the scientific discovery process, referring to researchers proposing testable explanatory propositions based on observed phenomena. In open-ended scientific discovery, hypothesis generation is closely tied to "problem generation": scientists first need to identify anomalies worth investigating from a multitude of observations, then translate them into explicit hypotheses, and then design experiments to verify them. For human scientists, this process often relies on intuition, domain knowledge, and analogical reasoning; for LLMs, hypothesis generation requires the model not only to understand current observations but also to judge their deviation from expected laws and to proactively construct falsifiable candidate explanations. The article points out that most models "did not follow up" when encountering anomalies, which is essentially a missing hypothesis generation step—the model lacks the intrinsic drive and reasoning mechanism to convert anomalous signals into research questions.

The Significance of the Automated Evaluation Framework and Multi-Agent Collaboration

The design of the LongChang framework itself is also noteworthy. It achieves a fully automated evaluation loop through multiple LLM agents with clearly defined roles (scenario generation and review, tool design review, and report evaluation), avoiding the bottlenecks caused by manual annotation and intervention. This architecture provides a scalable testbed for future research on LLMs' scientific reasoning capabilities and also demonstrates the potential of multi-agent collaboration in complex evaluation tasks.

At the same time, the framework emphasizes that reports must be supported by experimental evidence, which to some extent simulates the peer review process in the scientific community, making the evaluation more rigorous and closer to real research scenarios.

Peer Review is a core mechanism in the scientific community for ensuring research quality, referring to the critical examination of a research paper's methods, evidence, and conclusions by independent experts in the same field. Its key features include: reviewers being independent of authors, evaluation based on reproducible evidence, and conclusions withstanding scrutiny. LongChang requires exploration reports to be supported by experimental records and reviewed by independent LLM agents based on the altered physical laws, which structurally simulates the "evidence support" and "independent review" principles of peer review. The significance of this design is that it extends the evaluation criteria from "whether the answer is correct" to "whether the conclusions have sufficient experimental basis," thereby getting closer to the requirements for scientific rigor in real research scenarios and making automated evaluation results more credible.

Outlook: How Far Are LLMs From Becoming Autonomous Scientists?

The message conveyed by this study is twofold. On one hand, LLMs' ability to autonomously discover physical laws through free exploration remains very limited, with notable shortcomings especially in proactive anomaly detection and hypothesis generation. On the other hand, frameworks like LongChang provide tools for systematically measuring and tracking this capability, helping to drive models toward more autonomous scientific discovery.

Future improvement directions may include: enhancing models' sensitivity to unexpected results, encouraging more proactive experimental design, and improving reasoning abilities to induce complete laws from partial observations. For AI researchers, this study also suggests that merely scaling up models or improving benchmark scores does not automatically translate into open-ended scientific discovery capabilities.

One-line Summary

LongChang framework tests 12 LLMs' autonomous discovery in worlds with altered physical laws; only GPT-6-Astra achieved a 16.7% discovery rate.

Paragraph Summary

The study built the LongChang automated framework, which generates simulated worlds with modified physical laws and lets LLMs freely explore and attempt to discover hidden laws without task hints. After testing 12 models, only GPT-6-Astra achieved a 16.7% discovery rate, with all others at 0%. Exploration records show that models often ignore anomalies and can only partially identify altered laws, with the core bottleneck being proactive problem generation and hypothesis integration. The framework uses multi-agent division of labor to achieve a fully automated closed-loop evaluation that simulates peer review, providing a scalable platform for measuring LLMs' scientific discovery capabilities.