KEEP-Bench
A diagnostic benchmark for directly evaluating the faithfulness of context compression; by requiring cases in which specific evidence must be retained for a correct answer, it decouples evidence retention from question-answering accuracy.
Timeline (last 90 days)
KEEP-Bench is a diagnostic benchmark that directly evaluates the faithfulness of context compression.
KEEP-Bench contains 4,102 cases covering three challenging scenarios: retaining evidence to support plausible referential resolution, explicit qualifiers, and minority viewpoints.
In the experiments, the input averages only 3.1K tokens, even though the context window exceeds 100K tokens.
On KEEP-Bench, the best open-source model (31B) retained all required evidence in only 66.8% of cases.
On KEEP-Bench, the best closed-source model retained all required evidence in only 61.6% of cases.
Even with customized prompts, the strongest models' evidence retention on KEEP-Bench remains unreliable.
Performance varies significantly across evidence structures, meaning conclusions based on a limited set of structures may be biased.
All Facts (7)
KEEP-Bench is a diagnostic benchmark that directly evaluates the faithfulness of context compression.
50%UnverifiedKEEP-Bench contains 4,102 cases covering three challenging scenarios: retaining evidence to support plausible referential resolution, explicit qualifiers, and minority viewpoints.
50%UnverifiedIn the experiments, the input averages only 3.1K tokens, even though the context window exceeds 100K tokens.
50%UnverifiedOn KEEP-Bench, the best open-source model (31B) retained all required evidence in only 66.8% of cases.
50%UnverifiedOn KEEP-Bench, the best closed-source model retained all required evidence in only 61.6% of cases.
50%UnverifiedEven with customized prompts, the strongest models' evidence retention on KEEP-Bench remains unreliable.
50%UnverifiedPerformance varies significantly across evidence structures, meaning conclusions based on a limited set of structures may be biased.
50%