Text-to-SQLExecution Accuracy, Not String Match
Natural-language questions to SQL over a real SQLite database, scored by execution accuracy across four prompting strategies, with the failure modes tagged so each strategy is credited for what it actually fixes.
Headline results
- Execution accuracy
- 83.3% → 100% few-shot
- Self-correction
- No-op on the real model
- Exact match
- 25%, under-counts by 58 points
- Scoring
- Run both queries, compare rows
System architecture
Problem
Text-to-SQL benchmarks routinely grade by string match against a gold query, and that metric is broken in both directions. A correct query written with a different alias, column order, or whitespace mismatches the gold yet returns identical rows, while a syntactically valid query can silently return the wrong rows when a join condition goes missing. Only running both queries and comparing result sets tells you whether the SQL is actually right.
Approach
Every prediction executes against a seeded in-memory SQLite e-commerce database (customers, products, orders) and its rows are compared to the gold query's rows as an order-insensitive multiset, order-sensitive only when the gold query has an ORDER BY. Four prompting strategies run over the same 12 questions: zero-shot builds the prompt from the schema, few-shot prepends worked examples, self-correction re-prompts with the SQLite error message on an execution failure, and the fourth combines both. Questions are tagged by the failure mode they trigger, semantic (valid SQL, wrong rows) or syntax (SQL that throws), so the benchmark reports accuracy within each subset rather than one blended number. The real path runs Claude Haiku 4.5 on AWS Bedrock.
Impact
On the real model self-correction was a no-op. Claude Haiku made essentially no syntax errors, syntax-subset accuracy sat at 1.00 even zero-shot, so self-correction scored 83.3%, identical to zero-shot. The residual failures were all semantic, the semantic subset sat at 0.75, and few-shot examples alone repaired those to 1.00 and lifted the whole set to 100%. String match under-counts hard: zero-shot scored 83.3% by execution but 25.0% by exact match, a 58-point gap. The calibrated mock assumed both fixes were needed and the live run overturned it.
Decisions & tradeoffs
Score by executing both queries
Exact match is still computed, but only to contrast. Running the prediction and the gold against the same database and comparing rows is the only check that catches a valid query returning wrong rows while crediting a correct query that happens to be worded differently.
Tag the failure mode per question
An aggregate accuracy cannot say why a strategy helped. Splitting questions into semantic traps and syntax traps is what exposed that the real model's syntax subset was already at 1.00, which is the whole reason self-correction had nothing to catch.
Keep the mock as a contrasting fixture
The offline mock has designed orthogonal failure modes and reproduces a different table, zero_shot 41.7% rising to 100.0% only with both fixes. Keeping it next to the real numbers makes the gap between the assumed story and the measured one explicit, and lets CI reproduce every mock number with no key.
Build spec
- Questions
- 12 over seeded SQLite
- Best strategy
- few_shot, 100.0% exec
- Zero-shot
- 83.3% exec / 25.0% exact
- Model
- Claude Haiku 4.5 on Bedrock
- Stack
- SQLite, Bedrock, pytest
System notes
- Few-shot alone hit 100% exec accuracy, self-correction was a no-op at 83.3%, tied with zero-shot
- String match under-counts by 58 points: 83.3% exec vs 25.0% exact on zero-shot
- Result sets compared as an order-insensitive multiset, order-sensitive only under ORDER BY
- Questions tagged semantic vs syntax so subset accuracy shows which strategy fixes which mode
What this does not show
- The real model makes almost no syntax errors, so the self-correction stage that helps the mock does nothing here. The mock is kept as a contrasting fixture.
Stack
Python · SQLite · Claude · AWS Bedrock · Evaluation · pytest