LongChang
An automated multi-agent framework for evaluating the autonomous scientific discovery capability of LLMs. It generates simulated worlds in which physical laws are modified and deploys multiple LLM agents to perform scenario generation, tool review and report evaluation, enabling closed-loop evaluation without human intervention.
Timeline (last 90 days)
The researchers built an automated framework called LongChang to generate simulated worlds with modified physical laws in order to evaluate LLMs' autonomous discovery capabilities.
The LongChang framework deploys multiple independent LLM agents, respectively responsible for generating and reviewing scenarios, auditing tool designs, and evaluating report correctness and experimental support.
The research team tested 12 LLMs, and only GPT-6-Astra achieved a discovery rate of 16.7%, while the remaining 11 models all had a discovery rate of 0%.
LongChang implements a fully automated evaluation loop through multiple LLM agents with clearly divided roles, avoiding the bottleneck caused by human annotation and intervention.
LongChang requires that exploration reports be supported by experimental records and reviewed by an independent LLM agent against the altered physical laws, structurally simulating the "evidence support" and "independent review" principles of peer review.
All Facts (5)
The researchers built an automated framework called LongChang to generate simulated worlds with modified physical laws in order to evaluate LLMs' autonomous discovery capabilities.
50%UnverifiedThe LongChang framework deploys multiple independent LLM agents, respectively responsible for generating and reviewing scenarios, auditing tool designs, and evaluating report correctness and experimental support.
50%UnverifiedThe research team tested 12 LLMs, and only GPT-6-Astra achieved a discovery rate of 16.7%, while the remaining 11 models all had a discovery rate of 0%.
50%UnverifiedLongChang implements a fully automated evaluation loop through multiple LLM agents with clearly divided roles, avoiding the bottleneck caused by human annotation and intervention.
50%UnverifiedLongChang requires that exploration reports be supported by experimental records and reviewed by an independent LLM agent against the altered physical laws, structurally simulating the "evidence support" and "independent review" principles of peer review.
50%