OptionWorld
A benchmark for diagnosing the failure modes of LLMs as long-term personal assistants. By simulating everyday tasks spanning multiple days, it traces the full chain of information acquisition, judgment, action and outcome, precisely locating the stage at which a model fails to maintain task state.
Timeline (last 90 days)
The paper "OptionWorld: A Benchmark for Diagnosing Failures of LLMs as Long-Term Personal Assistants" proposes a diagnostic framework for evaluating the ability of LLMs to serve as long-term personal assistants.
OptionWorld simulates everyday life tasks and traces the full chain of information acquisition, judgment, action, and outcome to precisely locate where the model fails.
In 35.5% of failure cases, the key facts needed to complete the task had already appeared in the model's last input before its cutoff time.
Experiments supplementing missing facts show that directly providing the key facts the model lacked can restore success on some tasks, indicating that information gaps are a direct cause of some failures.
Action-reminder experiments show that sending the model an action reminder can restart work, but subsequent steps may still miss the deadline, so a reminder alone is insufficient to solve the time management problem.
Experiments correcting erroneous constraints show that when the model receives an erroneous note prohibiting contact, correcting that note can restore the permitted inquiry behavior, confirming that the model blindly accepts externally assumed constraints.
All Facts (6)
The paper "OptionWorld: A Benchmark for Diagnosing Failures of LLMs as Long-Term Personal Assistants" proposes a diagnostic framework for evaluating the ability of LLMs to serve as long-term personal assistants.
50%UnverifiedOptionWorld simulates everyday life tasks and traces the full chain of information acquisition, judgment, action, and outcome to precisely locate where the model fails.
50%UnverifiedIn 35.5% of failure cases, the key facts needed to complete the task had already appeared in the model's last input before its cutoff time.
50%UnverifiedExperiments supplementing missing facts show that directly providing the key facts the model lacked can restore success on some tasks, indicating that information gaps are a direct cause of some failures.
50%UnverifiedAction-reminder experiments show that sending the model an action reminder can restart work, but subsequent steps may still miss the deadline, so a reminder alone is insufficient to solve the time management problem.
50%UnverifiedExperiments correcting erroneous constraints show that when the model receives an erroneous note prohibiting contact, correcting that note can restore the permitted inquiry behavior, confirming that the model blindly accepts externally assumed constraints.
50%