HotpotQA
A multi-hop question answering benchmark dataset that requires models to answer complex questions through multi-step reasoning and information retrieval. It is commonly used to evaluate multi-step reasoning and retrieval capabilities of language models.
Core Facts
ReAct significantly outperforms methods that use CoT or action alone on complex reasoning benchmarks such as HotpotQA and FEVER.
Through experiments on multi-hop reasoning benchmarks such as HotpotQA and FEVER, the ReAct paper demonstrated that compared to pure Chain-of-Thought reasoning, ReAct improved accuracy by approximately 10%–30% on tasks requiring external information retrieval.
Timeline (last 90 days)
The example teaches Qwen2.5-3B-Instruct to retrieve from a BM25 corpus while answering HotpotQA questions.
SAG2 uses no embeddings, adopting SQLite plus BM25 together with a lightweight "System One" model to make contextual evidence decisions at query time; indexing about 11,600 passages takes only 2.36 seconds with zero model calls, matching the recall level of dense retrieval on MuSiQue, 2Wiki, and HotpotQA.
Multi-hop QA is a class of questions that require reasoning across multiple independent evidence fragments to answer, with mainstream benchmarks including HotpotQA, MuSiQue, and 2WikiMultiHopQA.
The experiment was conducted on 400 samples drawn from three multi-hop QA datasets: MuSiQue, HotpotQA, and 2WikiMultiHopQA.
The improvements of RLDS on HotpotQA and DeepResearch fall within the noise range, consistent with the diagnostic prediction that these two tasks have limited recoverable information.
The authors evaluated RLDS on four agent benchmarks: FrozenLake, HotpotQA, ScienceWorld, and DeepResearch.
The HotpotQA dataset released by Yang Zhilin et al. in 2018 was the first to systematically define the multi-hop reasoning challenge, with each question requiring the extraction and combination of information from at least two Wikipedia articles.
HotpotQA is a multi-hop question-answering dataset released by Stanford in 2018, in which questions require integrating information from multiple Wikipedia documents to answer.
ReAct outperforms chain-of-thought on the fact-checking task FEVER, but does not always win on HotpotQA.
All Facts (12)
ReAct significantly outperforms methods that use CoT or action alone on complex reasoning benchmarks such as HotpotQA and FEVER.
80%VerifiedThrough experiments on multi-hop reasoning benchmarks such as HotpotQA and FEVER, the ReAct paper demonstrated that compared to pure Chain-of-Thought reasoning, ReAct improved accuracy by approximately 10%–30% on tasks requiring external information retrieval.
75%UnverifiedReAct论文在HotpotQA和FEVER等基准测试上证明ReAct模式相比纯推理或纯行动方法都有显著提升
85%UnverifiedThe example teaches Qwen2.5-3B-Instruct to retrieve from a BM25 corpus while answering HotpotQA questions.
50%UnverifiedSAG2 uses no embeddings, adopting SQLite plus BM25 together with a lightweight "System One" model to make contextual evidence decisions at query time; indexing about 11,600 passages takes only 2.36 seconds with zero model calls, matching the recall level of dense retrieval on MuSiQue, 2Wiki, and HotpotQA.
50%UnverifiedMulti-hop QA is a class of questions that require reasoning across multiple independent evidence fragments to answer, with mainstream benchmarks including HotpotQA, MuSiQue, and 2WikiMultiHopQA.
50%UnverifiedThe experiment was conducted on 400 samples drawn from three multi-hop QA datasets: MuSiQue, HotpotQA, and 2WikiMultiHopQA.
50%UnverifiedThe authors evaluated RLDS on four agent benchmarks: FrozenLake, HotpotQA, ScienceWorld, and DeepResearch.
50%UnverifiedThe improvements of RLDS on HotpotQA and DeepResearch fall within the noise range, consistent with the diagnostic prediction that these two tasks have limited recoverable information.
50%UnverifiedThe HotpotQA dataset released by Yang Zhilin et al. in 2018 was the first to systematically define the multi-hop reasoning challenge, with each question requiring the extraction and combination of information from at least two Wikipedia articles.
50%UnverifiedHotpotQA is a multi-hop question-answering dataset released by Stanford in 2018, in which questions require integrating information from multiple Wikipedia documents to answer.
50%UnverifiedReAct outperforms chain-of-thought on the fact-checking task FEVER, but does not always win on HotpotQA.
50%