FRAMES
A multi-hop question-answering benchmark dataset released by Google DeepMind in 2024, comprising 824 multi-hop reasoning questions. It is designed to evaluate RAG systems on tasks requiring integrated reasoning across multiple documents, and is regarded as a rigorous testbed for measuring the practical upper bound of RAG system performance.
Timeline (last 90 days)
In the Google FRAMES benchmark, a simple agent loop (capable of reading retrieved results and searching again) achieved an accuracy of 92.7%.
In the same test, the best carefully tuned traditional RAG pipeline reached only 78.9% accuracy.
This test covered all 824 multi-hop questions in FRAMES and built 18 pipeline variants to break down the contribution of each component.
FRAMES (Factuality, Retrieval, And Multi-hop Evaluation of Systems) is a multi-hop question-answering benchmark dataset released by Google DeepMind in 2024.
The conclusions are based on the single dataset FRAMES and focus on multi-hop question answering; whether they can be generalized to other task types still requires further validation.
All Facts (5)
In the Google FRAMES benchmark, a simple agent loop (capable of reading retrieved results and searching again) achieved an accuracy of 92.7%.
50%UnverifiedIn the same test, the best carefully tuned traditional RAG pipeline reached only 78.9% accuracy.
50%UnverifiedThis test covered all 824 multi-hop questions in FRAMES and built 18 pipeline variants to break down the contribution of each component.
50%UnverifiedFRAMES (Factuality, Retrieval, And Multi-hop Evaluation of Systems) is a multi-hop question-answering benchmark dataset released by Google DeepMind in 2024.
50%UnverifiedThe conclusions are based on the single dataset FRAMES and focus on multi-hop question answering; whether they can be generalized to other task types still requires further validation.
50%