KEEP-Bench: A New Benchmark for Evidence Preservation in Context Compression

Context compression techniques aim to improve the efficiency of long-context processing, but the information bottleneck may introduce faithfulness issues—the compressed text may answer some questions yet lose critical evidence. Existing evaluations rely on question-answering accuracy; however, even when compression is unfaithful, readers may still answer correctly, thereby underestimating information loss. To address this, researchers introduce KEEP-Bench, a diagnostic benchmark that directly evaluates compression faithfulness.

Why Do We Need Dedicated Evaluation of Evidence Preservation?

Traditional QA metrics have blind spots: models may answer questions correctly based on erroneous cues in the compressed content or external knowledge, rather than genuinely preserving the original evidence. KEEP-Bench uses 4,102 cases covering three challenging scenarios: preserving evidence to support plausible reference resolution, explicit qualifiers, and minority viewpoints. These scenarios require the compressed text to contain specific evidence in order to answer correctly, thereby exposing unfaithful compression.

Information bottleneck and faithfulness: Context compression typically conveys information from the original long text through a capacity-limited intermediate representation (e.g., a fixed-length summary or latent vector), a process known as the information bottleneck. When the bottleneck is too narrow, the compressor is forced to discard some details, potentially prioritizing surface fluency over critical evidence. Faithfulness requires that the compressed text support reasoning based on the original text, not merely generate plausible answers. KEEP-Bench is designed specifically to directly detect whether the bottleneck severs the chain of evidence necessary for answering.

Decoupling evidence from QA accuracy: In traditional evaluations, models can rely on parametric knowledge or semantic cues in the compressed text to guess the correct answer, even if critical evidence has been removed. This phenomenon is called "shortcut learning." KEEP-Bench decouples evidence preservation from answer correctness by constructing cases that must cite specific evidence to be answered correctly, thereby exposing unfaithful compressions that "answer correctly but do not preserve evidence."

Model Performance Revealed by Experiments

Even with custom prompts, the strongest models remain unreliable. Despite context windows exceeding 100K tokens and inputs averaging only 3.1K tokens, the best open-source model (31B) preserved all required evidence in only 66.8% of cases, and the best closed-source model in only 61.6%. Performance varies significantly across different evidence structures, meaning conclusions based on a limited set of structures may be biased.

Evidence Preservation Return (EPR) Metric

Researchers introduce the EPR metric, which relates evidence recall to inference FLOPs. They find that EPR peaks at medium scale (4B–8B), but recall remains low at that point. Increasing the output budget mainly benefits already-strong models and widens capability gaps, while stronger evidence preservation comes with steep computational costs.

Inference FLOPs and computational efficiency: EPR relates evidence recall to the number of floating-point operations (FLOPs) consumed during inference, measuring the level of evidence preservation achieved per unit of computational cost. FLOPs is a common metric for measuring model inference computation, related to the number of output tokens, model parameter count, and attention computation complexity. The peak of EPR occurs at the 4B–8B scale, meaning that within this parameter range, models achieve the highest computational efficiency in evidence preservation, but absolute recall remains unsatisfactory. This suggests that simply increasing the output budget or model scale is not an efficient path to improving faithfulness.

Implications for Future Research

These findings emphasize that future work should explore compressors with strong information preservation capabilities under small computational budgets, rather than relying on large output budgets or model scaling. KEEP-Bench provides a key tool for diagnosing compression faithfulness, driving more reliable long-context processing.

One-line Summary

KEEP-Bench directly evaluates the evidence-preservation faithfulness of context compression, revealing that even the strongest models preserve evidence only about 60% of the time.

Paragraph Summary

KEEP-Bench is a new benchmark for diagnosing the faithfulness of context compression. It contains 4,102 cases covering three scenarios that require evidence preservation: plausible reference resolution, explicit qualifiers, and minority viewpoints. Experiments show that even with context windows exceeding 100K tokens and inputs of only about 3.1K tokens, the best open-source model (31B) preserves all required evidence in only 66.8% of cases, and the best closed-source model in only 61.6%. The researchers propose the Evidence Preservation Return (EPR) metric, finding that it peaks at the 4B–8B scale but recall remains low; increasing the output budget mainly benefits strong models and widens gaps. The conclusion calls for research on compressors with strong information preservation under small computational budgets.