July 8, 2026
Code Wars: Cheap Thrills Edition
Benchmarking coding agents on Databricks' multi-million line codebase
Databricks’ AI code showdown sparked cheers, side-eyes, and one big “wait, how huge is that code?”
TLDR: Databricks says the best artificial intelligence coding tools are now a mix of pricey giants and cheaper open options, with GLM 5.2 emerging as a standout. Commenters loved the upset but argued hardest over hidden costs, simple tools beating fancy setups, and just how absurdly huge Databricks’ codebase really is.
Databricks dropped a big internal bake-off of artificial intelligence coding helpers, testing them on real jobs inside its multi-million-line software empire, and the crowd immediately turned the results into a full-on comment-section spectator sport. The company’s big takeaway was simple enough for non-experts: no single tool rules everything. Some expensive systems were great, some cheaper ones punched above their weight, and one open model called GLM 5.2 got people especially worked up because it apparently hung with the big names while costing less per task. That sparked instant applause from the open-source fans, with one commenter basically summing up the mood as: top-tier tools are no longer just for the giant closed companies.
But the real drama? The community zeroed in on the messy details Databricks exposed: sticker price is a trap. Several readers loved the point that a cheaper price per chunk of text doesn’t always mean a cheaper finished job, because some models wander around, burn time, and rack up a bigger bill anyway. One commenter said they’d seen the same thing firsthand: the “cheap” option can end up taking way more tries to solve the same problem. Others immediately demanded a sequel, especially around whether wordier, stricter programming languages make these tools cheaper or pricier to use.
And of course, there was some classic internet side-eye. One skeptical commenter stared at the phrase “multi-million line codebase” and basically asked, excuse me, why is the wrapper bigger than the product? Meanwhile another chimed in with a very online plot twist: a simple tool called Pi is spreading fast inside companies because it’s easy to tweak, and apparently gets better as more people feed it context. Translation: the benchmark was interesting, but the comments turned it into a soap opera about cost, bloat, and whether the underdogs are coming for the crown.
Key Points
- •Databricks built an internal benchmark to evaluate coding agents on real engineering tasks from its multi-million line codebase across languages such as Python, Go, TypeScript, and Scala.
- •The company concluded that the coding-task Pareto frontier includes models from OpenAI, Anthropic, and open-source systems rather than a single provider.
- •Databricks reported that open models, particularly GLM 5.2, can handle high-difficulty coding tasks and that GLM 5.2 reached the top capability tier.
- •The benchmark results clustered models and harnesses into three capability tiers, leading Databricks to shift more everyday work toward Haiku- and GPT 5.4 Mini-class models.
- •Databricks found that token price is a poor proxy for total task cost and that harness choice, including simple options like Pi, can significantly change both quality and cost.