RULER
A long-context evaluation suite introduced by NVIDIA. It contains a variety of synthetic tasks, including retrieval, multi-hop tracing and aggregation, and supports custom context lengths such as 4K to 128K. It is widely used to measure how effectively a model uses information across long texts.
Timeline (last 90 days)
On RULER needle-in-a-haystack and long-context question answering tasks, MERLIN improves over the baseline by 8.3% on Gated DeltaNet-1.3B.
RULER is a long-context evaluation benchmark proposed by NVIDIA and other institutions in 2024, comprising four major categories of tasks: retrieval, multi-hop tracing, aggregation, and question answering.
On the 10M-length single-needle retrieval task of the RULER benchmark, Fern 0.1 achieved a score of 100%.
On six RULER needle-in-a-haystack tasks at a one-million-token setting, the paper reports that GDN's average accuracy improved from 1.7% to 70.3%.
Latent Compaction achieves 96.7% accuracy on the RULER benchmark at a 32x compression rate, while KVZip+ achieves only 40.0%.
The RULER benchmark was proposed by NVIDIA and includes various synthetic tasks such as retrieval, multi-hop tracing, and aggregation, supporting context lengths from 4K to 128K.
Latent Compaction demonstrates advantages on benchmarks such as RULER, LongBench, and Phonebook Lookup.
All of SubQ Snap's performance data comes from Subquadratic itself and has not been independently tested by an external lab, and its long-context tests are its own modified versions of RULER and LCR.
Benchmark tests commonly used to evaluate long-memory systems include RULER, NIAH (Needle-in-a-Haystack), and LongBench.
RULER is a benchmark designed specifically to evaluate long-context model capabilities, covering tasks such as single-hop and multi-hop retrieval, question answering, summarization, and verbatim copying, typically evaluated at context lengths of 16K to 128K.
5 more timeline events
All Facts (15)
On RULER needle-in-a-haystack and long-context question answering tasks, MERLIN improves over the baseline by 8.3% on Gated DeltaNet-1.3B.
50%UnverifiedRULER is a long-context evaluation benchmark proposed by NVIDIA and other institutions in 2024, comprising four major categories of tasks: retrieval, multi-hop tracing, aggregation, and question answering.
50%UnverifiedOn the 10M-length single-needle retrieval task of the RULER benchmark, Fern 0.1 achieved a score of 100%.
50%UnverifiedOn six RULER needle-in-a-haystack tasks at a one-million-token setting, the paper reports that GDN's average accuracy improved from 1.7% to 70.3%.
50%UnverifiedLatent Compaction achieves 96.7% accuracy on the RULER benchmark at a 32x compression rate, while KVZip+ achieves only 40.0%.
50%UnverifiedThe RULER benchmark was proposed by NVIDIA and includes various synthetic tasks such as retrieval, multi-hop tracing, and aggregation, supporting context lengths from 4K to 128K.
50%UnverifiedLatent Compaction demonstrates advantages on benchmarks such as RULER, LongBench, and Phonebook Lookup.
50%UnverifiedAll of SubQ Snap's performance data comes from Subquadratic itself and has not been independently tested by an external lab, and its long-context tests are its own modified versions of RULER and LCR.
50%UnverifiedBenchmark tests commonly used to evaluate long-memory systems include RULER, NIAH (Needle-in-a-Haystack), and LongBench.
50%UnverifiedRULER is a benchmark designed specifically to evaluate long-context model capabilities, covering tasks such as single-hop and multi-hop retrieval, question answering, summarization, and verbatim copying, typically evaluated at context lengths of 16K to 128K.
50%UnverifiedAggregate metrics (such as multi-task average scores) have an inherent ability to dilute failures; when simpler tasks constitute the majority, the average score can mask significant degradation on a small number of high-difficulty tasks.
50%UnverifiedRULER is a long-context evaluation benchmark proposed by NVIDIA's research team in 2024, comprising 13 synthetic tasks.
50%UnverifiedAmong RULER's 13 tasks, 6 NIAH tasks exhibit cooperative characteristics, 2 QA tasks use datasets from years ago, and the VT task contains a large amount of irrelevant context.
50%UnverifiedPapers can conceal performance degradation on difficult subtasks such as NIAH-MK3 by reporting only aggregated average scores.
50%UnverifiedRULER is a long-context evaluation benchmark proposed by the NVIDIA research team in 2024, containing 13 synthetic tasks divided into four categories: NIAH, Variable Tracking, Common/Frequent Words Extraction, and question answering based on SQuAD/HotpotQA.
50%