BIRD

On BIRD's seven-table database, SchemagateRetriever reduced prompt size by only 8.8%.

Both 92.9% and 8.8% are medians from the author's own testing, and were obtained only on specific schemas in Spider 2.0 and BIRD; no third-party independent reproduction has yet been seen.

On the same pipeline and the same 135 Spider 2.0-lite SQLite questions, Sonnet got 71.9% while Opus got only 67.4%, but Opus performed better on BIRD.

On all 1534 questions of the BIRD dev set, Claude Opus 5.5 achieved 77.1% execution accuracy.

BIRD is one of the most influential public benchmarks in the Text-to-SQL field, its dev set contains 1534 natural-language questions across multiple databases, and evaluation is typically based on execution accuracy.

UniQL is a new benchmark that translated 1,534 questions from the BIRD dataset into 16 SQL dialects.

The schemagate authors note its limitations: only one model was tested, and the prompt wording was chosen on 100 BIRD questions; removing these two factors leaves the score essentially unchanged (90.08).

The paper tested the best pure schema-filtering configuration on the BIRD dataset, and still about 27% of questions that should have been refused were answered by the model.

By adding to the prompt 'you are seeing part of this database; if you need more information, refuse to answer', schemagate reduced incorrect answers from 16% to 5% in a 100-question pilot.

On the BIRD dataset, schemagate achieved an AC-F1 of 90.1, while the best of the 13 models in the paper scored 88.1 (that model could see the full schema and policies).

On both datasets, the number of queries in which schemagate actually read a forbidden column was 0.

After removing the scope check, schemagate's wrong-answer rate on BIRD rose from 5.4% to 11.3%.

On the BIRD dataset, the paper's best "hidden schema" configuration still answered 27% of questions that should have been refused.

Running five times with Claude Sonnet 4.5, schemagate achieved an AC-F1 of 90.1 on BIRD, versus 88.1 for the paper's best configuration.

On BIRD, the proportion of questions schemagate should have refused but answered was 5.4%, lower than the 7.4% of the paper's best configuration.

On both the BIRD and Spider datasets, no SQL reading forbidden data appeared in schemagate.

Providing the model with BIRD's column descriptions actually made schemagate's results worse.

The wording of the prompt was chosen on 100 BIRD questions that are part of the test set, so there is tuning contamination.

The BIRD database version is the 2024 edition, while the paper used the 2025 edition.

For the schemagate benchmark's paired McNemar test, the p-value is about 1e-49 on Spider and about 1e-36 on BIRD.