Databricks Benchmarks Coding Agents: Open Models Emerge as Cost-Effective Alternatives
Key Takeaways
- ▸Open-source models like GLM 5.2 now compete with premium models on coding tasks at significantly lower cost, with GLM achieving comparable quality to Opus 4.8 while saving 34% per task
- ▸Model token price is a poor predictor of actual task costs due to variance in reasoning efficiency and problem-solving approach across models
- ▸Three distinct capability tiers emerged, with routine engineering tasks best served by smaller, cheaper models (Haiku, GPT 5.4 Mini) rather than top-tier models
Summary
Databricks has published the results of an internal benchmark evaluating coding agents and AI models on real-world engineering tasks performed on their multi-million line codebase. The study tested models from OpenAI, Anthropic, and open-source providers across multiple programming languages including Python, Go, TypeScript, and Scala. The research revealed three distinct capability tiers, with the Pareto frontier for coding tasks including models from all three categories.
A key finding is that open-source models, particularly GLM 5.2, have reached competitive performance levels with premium proprietary models. GLM 5.2 achieved quality scores statistically tied with Anthropic's Opus 4.8 while costing 34% less per task ($1.28 vs. $1.94). The benchmark also uncovered that token price is a poor indicator of actual task costs, with larger models often proving more token-efficient than smaller alternatives.
Databricks' analysis showed that models cluster into rough capability tiers, with the highest-performing models excelling across all task types but at premium costs. Medium and lower-tier models like Anthropic's Haiku and OpenAI's GPT 5.4 Mini perform highly effectively on common operational tasks and are significantly cheaper, leading the company to shift more routine engineering work to these models. Notably, the choice of harness (the execution environment) was found to impact both cost and quality nearly as much as model selection itself.
- The execution harness impacts performance and cost as much as the model choice itself, with simpler harnesses like Pi often performing best on real workloads
- A mix of tools from OpenAI, Anthropic, and open-source providers is needed to achieve frontier performance on coding tasks
Editorial Opinion
Databricks' benchmark is a timely reality check for organizations overspending on premium AI models for routine coding tasks. The finding that open-source models like GLM 5.2 can match proprietary alternatives at a third of the cost suggests the coding AI market is approaching a commoditization inflection point. Most significantly, the data validates what many practitioners suspected: bigger and more expensive doesn't always mean better for production workloads. Organizations should now feel confident building polyglot AI stacks that reserve premium models for genuinely hard problems.
