TWIST
A benchmark suite for evaluating the intervention quality of conversational memory systems. Through ingestion, recall, and review interfaces, it examines a memory system's active intervention capabilities across four dimensions: belief-conflict detection, draft review, belief-update tracking, and sensitive-information governance.
Timeline (last 90 days)
TWIST is a benchmark suite for evaluating the "intervention quality" of dialogue memory systems, measuring a system's ability to respond correctly at inflection points where user beliefs change.
TWIST examines system behavior through the memory system's own ingest/recall/vet interface and designs four evaluation tracks: unprompted contradiction detection, draft vetting, responding based on current beliefs, and sensitive information recall governance.
TWIST extends the LoCoMo (Long Conversation Memory) corpus and evaluation framework.
TWIST pairs every detection/interception metric with a matching "do not over-detect" control group, and by introducing surface-matched hard negatives that are highly similar on the surface, it prevents systems from gaming the score by flagging everything as suspicious.
TWIST employs an independent, ground-truth-blind dual-annotation plus arbitration process, together with judge decoy calibration and separability audits, to verify the reliability of the benchmark itself.
On the manually validated Track B v1.0 gold set (161 entries), the post-arbitration Cohen's kappa value reached 0.85.
On Track B, the flat RAG baseline can detect 76%–97% of true contradictions, but produces false positives on 16%–43% of surface-similar safe drafts (the specific ratio depends on the backend).
One deployed system optimized for coherence achieves a specificity of 0.98–1.00 but captures only 42% of true contradictions.
No tested configuration is able to simultaneously achieve high recall, high hard-negative specificity, and high attribution accuracy.
The paper constructs a 13-configuration baseline ladder to decompose the root causes of the problem.
4 more timeline events
All Facts (14)
TWIST is a benchmark suite for evaluating the "intervention quality" of dialogue memory systems, measuring a system's ability to respond correctly at inflection points where user beliefs change.
50%UnverifiedTWIST examines system behavior through the memory system's own ingest/recall/vet interface and designs four evaluation tracks: unprompted contradiction detection, draft vetting, responding based on current beliefs, and sensitive information recall governance.
50%UnverifiedTWIST extends the LoCoMo (Long Conversation Memory) corpus and evaluation framework.
50%UnverifiedTWIST pairs every detection/interception metric with a matching "do not over-detect" control group, and by introducing surface-matched hard negatives that are highly similar on the surface, it prevents systems from gaming the score by flagging everything as suspicious.
50%UnverifiedTWIST employs an independent, ground-truth-blind dual-annotation plus arbitration process, together with judge decoy calibration and separability audits, to verify the reliability of the benchmark itself.
50%UnverifiedOn the manually validated Track B v1.0 gold set (161 entries), the post-arbitration Cohen's kappa value reached 0.85.
50%UnverifiedOn Track B, the flat RAG baseline can detect 76%–97% of true contradictions, but produces false positives on 16%–43% of surface-similar safe drafts (the specific ratio depends on the backend).
50%UnverifiedOne deployed system optimized for coherence achieves a specificity of 0.98–1.00 but captures only 42% of true contradictions.
50%UnverifiedNo tested configuration is able to simultaneously achieve high recall, high hard-negative specificity, and high attribution accuracy.
50%UnverifiedThe paper constructs a 13-configuration baseline ladder to decompose the root causes of the problem.
50%UnverifiedEach gold-standard contradiction is detectable on the basis of its evidence alone (recall 1.000), indicating no information gap at the data level and pointing to the retrieval stage rather than reasoning ability as the source of the problem.
50%UnverifiedCalibrated models almost resolve the contradiction detection track when given access to the complete conversation history, consistent with the finding that there is a significant gap in retrieval coverage.
50%UnverifiedTWIST draws an intervention profile (TWIST profile) for memory systems, measuring whether the system knows when to intervene and when to hold back.
50%UnverifiedTWIST was released as an arXiv preprint, and its conclusions are pending peer review and further validation.
50%