Supabase Open Sources Evals, Benchmark Framework for Testing AI Agents
Key Takeaways
- ▸Supabase open sourced its evals framework, a benchmark tool for measuring AI agent performance on real Supabase development tasks
- ▸The framework tests multiple agents including Claude Code and Codex using public benchmarks and daily regression monitoring against real Supabase environments
- ▸Early results show AI agents perform well on Supabase tasks even without specialized skills, indicating strong foundational capabilities
Summary
Supabase has open sourced its evals framework, a comprehensive benchmarking and evaluation tool designed to measure how effectively AI agents complete real-world Supabase tasks. The framework tests popular coding agents—including Claude Code, OpenAI's Codex, and others—against authentic scenarios such as schema building, debugging Edge Functions, and fixing row-level security (RLS) policies. As AI agents increasingly become the primary way developers build Supabase projects, Supabase created this tool to empirically measure agent performance rather than relying on assumptions.
The evals framework operates in two modes: a public benchmark suite demonstrating breadth across diverse real-world scenarios, and an internal regression suite monitored daily to detect performance regressions and validate new features. All tests run against actual Supabase environments—both hosted and local—by spinning up a hosted-like stack and local CLI projects in containers, ensuring agents interact with real MCP servers and CLI tools. Scoring combines deterministic checks (verifying data access, Edge Function results) with LLM-as-a-judge evaluation for scenarios requiring semantic assessment.
Early results reveal encouraging findings about AI agent capabilities with Supabase. Agents perform surprisingly well even without Supabase-specific skills loaded—for example, Claude Opus 5 and Kimi K3 both achieved 100% on Build stage tasks without any skill. Skills did improve performance for other models, helping narrow gaps and encouraging agents to consult Supabase documentation more thoroughly. This suggests modern AI agents have strong foundational capabilities for Supabase development, though specialized skills continue providing value for edge cases and staying current with the latest APIs.
- The tool enables Supabase to identify agent failure modes, fix them systematically, and confidently ship new features without regressions
