QERA: Ask First, Answer Later—Solving the Implicit Context Reasoning Challenge

When AI assistants need to reason over a user's personal digital traces—group chat logs, community posts, activity logs, revision histories—their performance is often disappointing. On CL-bench Life, a benchmark for evaluating real-life in-context learning, frontier large models solve fewer than one-fifth of the tasks. What's the problem? A new paper offers a key insight: a user's request is not the same as a complete requirements specification.

The Overlooked Implicit Requirements: Rules Beyond the Request

The researchers found that in the CL-bench Life benchmark, over 80% of grading criteria have less than 60% content-word overlap with the final request. This means a large amount of critical information is buried in the conversation history rather than in the current request. For example, a user may have mentioned a set of rules two turns earlier, or may expect the model to find all records matching their criteria—none of which is explicitly stated in the final request.

The data confirms this: when large models answer directly, their pass rate on implicit grading criteria is only 67%, compared with 81% on explicit criteria. This gap of nearly 14 percentage points reveals a systematic blind spot in current in-context learning—models focus too much on the immediacy of "question answering" and neglect the completeness of "evidence."

In-Context Learning refers to a learning paradigm in which the model does not update its parameters and relies solely on the examples or information provided in the prompt to complete a task. It performs well with short prompts and single-turn Q&A, but faces challenges in real-world scenarios involving long conversations and multi-turn accumulation: relevant information is scattered across historical turns, is semantically distant from the current request, and the model tends to answer based only on the last turn's request, thereby missing preconditions that appeared earlier. CL-bench Life is precisely designed to characterize such "long-span, multi-source, incomplete-requirement" real-life scenarios, so the problems it exposes are representative rather than accidental failures of a particular model.

The QERA Architecture: Asking Before Answering

To address this pain point, the researchers propose QERA (Question-driven Evidence Reasoning Architecture). Its core idea is straightforward yet effective: before answering, ask questions of the context.

The QERA workflow consists of several key steps:

Requirement checklisting: It transforms the user request into a requirements checklist and typed information needs, and applies code-based filtering to ensure these requirements remain faithful to the user's original wording—avoiding the model hallucinating or drifting from the user's intent.

Full-conversation retrieval and self-checking: QERA searches for answers across the entire conversation, then re-asks the same questions with code-verified citations. This "ask–verify–ask again" loop ensures the accuracy and relevance of evidence.

Guarded local repair: After forming a preliminary answer, QERA applies a guarded local repair mechanism to refine the response in a targeted way.

The entire architecture requires no training and no predefined grading criteria, averaging only about four LLM calls per task.

Inference-time baselines refer to methods that do not train the model but improve performance at inference time through prompting strategies or workflow orchestration, such as chain-of-thought, self-consistency sampling, self-reflection, and multi-turn self-questioning. PSCI and Ctx2Skill, which are compared with QERA, fall into this category. Their common limitation is that the optimization focus is usually "how to better answer the current question" rather than "whether the current question covers all requirements." QERA's differentiator is that it moves requirement identification and evidence retrieval into separate stages up front, rather than remedying things after generating an answer—which also explains why the gains in the ablation study concentrate before answer generation.

Experimental Results: Consistently Outperforming Baselines

On all 405 tasks, the researchers tested QERA with four frontier large models. The results show that QERA achieves the highest solve rate on every model—21.0% on average, compared with 18.2% for the strongest of the three inference-time baselines and 14.6% for direct answering. Aggregated across models, QERA significantly leads all baselines.

Efficiency is also notable: QERA uses fewer tokens than PSCI and only 4% to 5% of Ctx2Skill's token count. For real-world deployment, this means lower cost and faster response.

Ablation experiments further reveal where the performance gains come from. Analysis on GPT-6-Luna shows that the gains are established before answer generation: the fixed answering protocol contributes 3.8 grading points, while request-specific questions contribute 1.6 grading points. More importantly, these gains are concentrated on implicit grading criteria, while post-hoc checking mainly refines numerical requirements.

Ablation study involves removing or replacing one component of a system at a time and observing the performance change to determine each part's actual contribution. The key here is distinguishing two types of contribution: the "fixed answering protocol" and the "request-specific generated questions." The former is a general structural constraint on answering that applies to all tasks; the latter is dynamically generated based on the specific request and is more targeted. The results show the former contributes more (3.8 grading points), indicating that in long-context tasks, a stable answering and verification process alone can significantly reduce omissions, while the latter mainly fills in request-specific details. The engineering takeaway is that even without implementing full QERA, fixing a "list requirements first, then gather evidence, then answer" protocol for long-context Q&A can yield substantial gains.

Implications: From "Ask-and-Answer" to "Ask First, Answer Later"

The value of QERA is not just about topping the leaderboard. It points to a widely overlooked design principle: in complex context scenarios, understanding the question is more important than answering it. Current AI assistants are often optimized for "fast response," but when faced with users' long-accumulated fragmented information, this mode easily misses key preconditions.

By making requirements explicit and moving evidence retrieval earlier, QERA achieves more reliable reasoning at the cost of very few additional calls. For building personal knowledge assistants, customer service bots, or any system that needs to handle long conversation histories, this approach offers a reusable engineering paradigm.

Of course, the absolute solve rate of 21.0% also reminds us that real-life in-context learning remains a hard problem. But QERA at least proves one thing: teaching models to "ask before answering" is an effective path to higher accuracy.

One-line Summary

QERA has models question the context before answering, closing the implicit-requirement blind spot.

Paragraph Summary

The article centers on a problem revealed by the CL-bench Life benchmark: frontier large models solve fewer than 20% of real-life in-context learning tasks, because the user's final request is not the same as a complete requirements specification. Over 80% of grading criteria have less than 60% content-word overlap with the request, and models' pass rate on implicit criteria is only 67%, clearly below the 81% on explicit criteria. To address this, the researchers propose the QERA architecture, which transforms the request into a requirements checklist and typed information needs, applies code-based filtering to ensure faithfulness, retrieves evidence from the full conversation and repeatedly asks questions with verified citations, and refines the answer through guarded local repair. The method requires no training or predefined grading criteria, averaging about four LLM calls per task. On 405 tasks, QERA achieves an average solve rate of 21.0%, surpassing the strongest inference-time baseline's 18.2% and direct answering's 14.6%, while consuming far fewer tokens than compared methods.