schemagate
A table-filtering tool for text-to-SQL scenarios. It first performs coarse filtering of candidate tables using mathematical methods such as BM25, embeddings and table-name matching, then fuses and ranks results via RRF, and can optionally call an LLM to re-rank the candidate tables.
Timeline (last 90 days)
On the 205 BigQuery questions of Spider 2.0-lite, schemagate's accuracy was only 45.9%; the Snowflake portion has not yet been run.
schemagate is open-sourced under the Apache-2.0 license and can be installed via pip install schemagate.
The SQL scope check once misread the CTE name periods in a WITH RECURSIVE clause as a table the caller had no permission to access, causing recursive queries from all users to be wrongly rejected; the issue has been fixed and tests added.
In a database containing 366 ga_sessions_YYYYMMDD date-sharded tables, providing the tables one by one to the model filled all 12 slots with random dates and the model refused to answer; after collapsing each table family into a wildcard table (ga_sessions_* with _TABLE_SUFFIX filtering), most refusals disappeared.
The combination of multi-candidate queries plus majority voting plus a judge model appeared to yield an improvement of about +3 points on half the questions, but on the held-out half the gain was exactly 0, while the cost was 5.5 times that of a single call.
The author states that the best-performing public Agent he knows of on the Spider 2.0-lite SQLite subset reports 57.8% Pass@1, and that it uses a different model.
After injecting two Oracle-specific rules into the prompt, first-attempt failed queries dropped from 694 to 85.
The 64.5% first-attempt accuracy is still far from production-ready, and the error-repair loop remains indispensable in actual deployment.
The developer proactively reported the repaired 67.6% separately from the first-attempt 64.5% and submitted them to the UniQL leaderboard (issue #2).
Using the UniQL evaluator on Oracle Free 23ai, the schemagate + Gemini 2.5 Flash combination reached 69.9% accuracy after adding an error-repair loop, but this score is mainly attributable to the repair loop rather than the model's own capability.
40 more timeline events
All Facts (20)
schemagate is open-sourced under the Apache-2.0 license and can be installed via pip install schemagate.
50%UnverifiedThe combination of multi-candidate queries plus majority voting plus a judge model appeared to yield an improvement of about +3 points on half the questions, but on the held-out half the gain was exactly 0, while the cost was 5.5 times that of a single call.
50%UnverifiedOn the 205 BigQuery questions of Spider 2.0-lite, schemagate's accuracy was only 45.9%; the Snowflake portion has not yet been run.
50%UnverifiedIn a database containing 366 ga_sessions_YYYYMMDD date-sharded tables, providing the tables one by one to the model filled all 12 slots with random dates and the model refused to answer; after collapsing each table family into a wildcard table (ga_sessions_* with _TABLE_SUFFIX filtering), most refusals disappeared.
50%UnverifiedThe author states that the best-performing public Agent he knows of on the Spider 2.0-lite SQLite subset reports 57.8% Pass@1, and that it uses a different model.
50%UnverifiedA developer using the lightweight model Gemini 2.5 Flash together with the schemagate tool raised first-attempt accuracy on the UniQL Oracle benchmark from 35.0% to 64.5%.
50%UnverifiedThe 64.5% first-attempt accuracy is still far from production-ready, and the error-repair loop remains indispensable in actual deployment.
50%UnverifiedAfter injecting two Oracle-specific rules into the prompt, first-attempt failed queries dropped from 694 to 85.
50%Unverifiedschemagate's solution is: when the target database is Oracle, explicitly tell the model two rules in the prompt (no AS before table aliases, quote reserved words such as DATE/NUMBER when used as column names), while leaving prompts for other databases unchanged.
50%UnverifiedUsing the UniQL evaluator on Oracle Free 23ai, the schemagate + Gemini 2.5 Flash combination reached 69.9% accuracy after adding an error-repair loop, but this score is mainly attributable to the repair loop rather than the model's own capability.
50%UnverifiedAfter the bug fix, schemagate ran all 19,624 records of Spider-ACL using Claude Sonnet 5.5, reaching a rejection F1 of 0.984.
50%Unverifiedschemagate is a middle layer between the LLM and the database that only shows the model the tables and columns the user is authorized to read, and checks whether a query touches anything else before any generated SQL is executed.
50%UnverifiedThe systems in the paper used 2024-25 models, whereas this test used a 2026 model (Claude Sonnet 5.5), so part of the performance gap comes from capability improvements in the model itself.
50%UnverifiedIn the full Spider-ACL test, schemagate answered 1.8% of the forbidden questions, but the records of unauthorized data reads were 0/8,128.
50%UnverifiedThe lesson the author summarizes is: self-built test sets easily fall into the blind spot of "you test what you thought of," and the greatest value of an external benchmark is attacking the system with boundary conditions you never imagined.
50%UnverifiedOn the 600 records never touched during the pilot, schemagate's rejection F1 was 0.986.
50%Unverifiedschemagate performs a policy check on each generated query before execution, and removing this step doubles the violation rate.
50%UnverifiedThe schemagate authors note its limitations: only one model was tested, and the prompt wording was chosen on 100 BIRD questions; removing these two factors leaves the score essentially unchanged (90.08).
50%UnverifiedOn the Spider dataset, schemagate tied with the same model (Claude Sonnet 4.5) at 93.0; the paper's best result on Spider came from DeepSeek V3.2-Reasoning at 95.9.
50%UnverifiedOn the BIRD dataset, schemagate achieved an AC-F1 of 90.1, while the best of the 13 models in the paper scored 88.1 (that model could see the full schema and policies).
50%