July 22, 2026
SQL and the City: Data Drama
Any text-to-SQL benchmark should address difficulties of real-world data stores
AI can ace test questions, but commenters say real company data will chew it up
TLDR: The article says AI database tests are too neat and don’t reflect the confusing, messy systems real companies actually use. Commenters mostly agreed, with some saying the tools can work, but many warning that bad AI answers could get repeated as truth in meetings and that flashy benchmarks prove very little.
The article’s big claim is a classic "looks great in the demo, melts down in real life" moment. Researchers argue that today’s text-to-database tests make artificial intelligence look smarter than it really is, because the practice data is too clean, too public, and way too tidy compared with the chaotic mess inside real companies. In the real world, databases are full of confusing labels, years of patchwork changes, and weird local language that only insiders understand. In other words: the benchmark says genius, the office basement full of mystery spreadsheets says good luck, buddy.
And the comments? Absolute reality-check mode. One person chimed in with a startup success story, saying a friend built a tool that lets non-technical people query databases in plain English and that it seems “fairly robust,” which gave the thread a little maybe the dream is alive after all energy. But the louder camp was waving red flags. A commenter working with a 20-year-old database with a rotting schema basically confirmed the article’s nightmare scenario and said the only way forward was to rebuild the whole thing before layering AI on top. Another brought the most terrifying office drama angle: if a non-expert asks an AI for numbers, gets the wrong answer, and then shares it in meetings as truth, you’ve got spreadsheet fan fiction becoming corporate fact.
The spiciest mini-fight was over the article’s line that an AI can only find data it has seen before. One commenter flatly asked, what does that even mean? Another dunked on benchmarks entirely, saying they’re all gameable unless the test is secret. The mood was clear: people love the idea, but the comment section is deeply suspicious of the hype machine.
Key Points
- •The article says enterprise data warehouses are heavily used for ad hoc queries and have historically been accessed through BI tools such as Business Objects, Power BI, and Tableau.
- •It argues that current text-to-SQL benchmark scores on Spider 1.0 and Bird-SQL do not adequately represent real enterprise conditions.
- •The article identifies three major realism gaps in benchmarks besides query complexity: public benchmark data, schema rot in enterprise warehouses, and idiosyncratic organization-specific data.
- •It states that real warehouse queries are often more complex than benchmark tasks and frequently involve two or three joins.
- •The authors report that they have spent two years testing LLM-based text-to-SQL on real workloads, starting with MIT's Oracle warehouse containing more than 1,400 tables.