Spider
Spider is a large-scale cross-domain benchmark dataset for Text-to-SQL tasks, designed to evaluate the ability of natural language processing models to convert natural language questions into structured SQL queries. The dataset covers databases from multiple domains, includes complex multi-table join scenarios, and requires cross-domain generalization, serving as a widely referenced standard for measuring Text-to-SQL system performance in both academia and industry.
Core Facts
Timeline (last 90 days)
Beaver is a benchmark dataset for enterprise-level Text-to-SQL, collected from real enterprise databases, with incomplete schema documentation and a large semantic gap between table names and business terms, making it closer to production environments than academic benchmarks such as Spider.
Spider is a cross-database Text-to-SQL benchmark whose outputs are highly structured with high template repetition, so n-gram hit rates are naturally favorable.
On the Spider task, NgramRL's measured rollout throughput improves 1.38 times over the synchronous version, approaching the theoretical CPU-GPU overlap limit of 1.40 times.
On the Spider structured generation task, with a candidate budget of 16, the prefetcher covered 96.6% of Qwen3-8B's verification results.
Spider is the most widely used cross-domain benchmark in the Text-to-SQL field, containing about 10,000 natural language questions with corresponding SQL queries across multiple database schemas; the original Spider does not involve access control.
On the Spider dataset, schemagate tied with the same model (Claude Sonnet 4.5) at 93.0; the paper's best result on Spider came from DeepSeek V3.2-Reasoning at 95.9.
On both datasets, the number of queries in which schemagate actually read a forbidden column was 0.
On Spider, schemagate and the paper's best configuration both achieved an AC-F1 of 93.0, but schemagate's wrong-answer rate was 5.1%, slightly higher than the paper's 4.6%.
On both the BIRD and Spider datasets, no SQL reading forbidden data appeared in schemagate.
For the schemagate benchmark's paired McNemar test, the p-value is about 1e-49 on Spider and about 1e-36 on BIRD.
4 more timeline events
All Facts (20)
The Spider benchmark was released by Yale University in 2018 and contains more than 10,000 natural language questions and corresponding SQL queries, covering 200 databases.
85%UnverifiedOn the Spider benchmark, the accuracy of state-of-the-art models rose from under 20% in 2018 to over 80%
85%UnverifiedIRNet和RATSQL等基于Seq2Seq和Transformer的模型在Spider基准数据集上取得了Text-to-SQL的突破
85%UnverifiedGPT-4等大模型在Spider等基准测试上的零样本或少样本SQL生成准确率已超过80%
75%Unverified在Spider基准上，早期单模型方案准确率约为70%，GPT-4等大模型将其提升到80%以上，多智能体协作策略的系统已能达到85%甚至更高的执行准确率
75%UnverifiedMainstream Text-to-SQL solutions rely on models fine-tuned on standard benchmark datasets such as WikiSQL and Spider.
70%UnverifiedBeaver is a benchmark dataset for enterprise-level Text-to-SQL, collected from real enterprise databases, with incomplete schema documentation and a large semantic gap between table names and business terms, making it closer to production environments than academic benchmarks such as Spider.
50%UnverifiedSpider is a cross-database Text-to-SQL benchmark whose outputs are highly structured with high template repetition, so n-gram hit rates are naturally favorable.
50%UnverifiedOn the Spider structured generation task, with a candidate budget of 16, the prefetcher covered 96.6% of Qwen3-8B's verification results.
50%UnverifiedOn the Spider task, NgramRL's measured rollout throughput improves 1.38 times over the synchronous version, approaching the theoretical CPU-GPU overlap limit of 1.40 times.
50%UnverifiedSpider is the most widely used cross-domain benchmark in the Text-to-SQL field, containing about 10,000 natural language questions with corresponding SQL queries across multiple database schemas; the original Spider does not involve access control.
50%UnverifiedOn the Spider dataset, schemagate tied with the same model (Claude Sonnet 4.5) at 93.0; the paper's best result on Spider came from DeepSeek V3.2-Reasoning at 95.9.
50%UnverifiedOn both datasets, the number of queries in which schemagate actually read a forbidden column was 0.
50%UnverifiedOn Spider, schemagate and the paper's best configuration both achieved an AC-F1 of 93.0, but schemagate's wrong-answer rate was 5.1%, slightly higher than the paper's 4.6%.
50%UnverifiedOn both the BIRD and Spider datasets, no SQL reading forbidden data appeared in schemagate.
50%UnverifiedIn schemagate's benchmark, on the Spider dataset's 1034 questions the pure mathematical ranking accuracy is 80.3%, and 96.5% after re-ranking; on the BIRD dataset's 1534 questions the pure mathematical ranking is 88.6%, and 96.7% after re-ranking.
50%Unverifiedschemagate's benchmark conditions: all 876 tables of Spider dev and BIRD dev are placed in the same pool, the model does not know which database a question belongs to, and a table must all appear in the set of tables used by the correct SQL to count as a top-5 hit.
50%UnverifiedFor the schemagate benchmark's paired McNemar test, the p-value is about 1e-49 on Spider and about 1e-36 on BIRD.
50%UnverifiedThe commonly used benchmarks for measuring NL2SQL system accuracy are the Spider and BIRD datasets, where Spider tests cross-domain generalization ability and BIRD more closely reflects the complexity of real enterprise databases.
50%UnverifiedSpider爬虫工具用Rust原生编写，处理速度比同类产品快7倍
50%