AI Models Crack 'Baba Is You' Puzzle Game—But Cost Efficiency Reveals Industry Gaps
Key Takeaways
- ▸Claude Fable 5 and GPT-5.6 Sol successfully solved multiple 'Baba Is You' levels but with significantly longer solve times compared to human players
- ▸AI model cost-efficiency varies dramatically and unpredictably: Gemini 3.5 Flash was 2.4x more expensive than Fable 5; GPT-5.6 Terra cost 2.9x more than Sol
- ▸Researchers released Baba Is Harbor, an open-source benchmark framework that extracts game levels and logic for consistent, reproducible AI agent evaluation
Summary
Researchers evaluated multiple advanced AI language models including Claude Fable 5 and GPT-5.6 Sol on the acclaimed indie puzzle game 'Baba Is You,' finding that while current AI agents can solve game levels, they do so at significantly higher costs than humans and require longer solve times. The study uncovered dramatic cost disparities across models: Gemini 3.5 Flash proved 2.4x more expensive than Claude Fable 5 to solve the introduction stage, while the 'budget' GPT-5.6 Terra variant cost 2.9x more than the advanced Sol version. The researchers burned over $2,000 on experiments to produce these findings.
To enable systematic evaluation, researchers created an open-source framework called Baba Is Harbor, which extracts the game's levels and core logic into the Harbor agent evaluation system. This approach treats 'Baba Is You'—a game where players rewrite rules by rearranging text blocks—as a formal benchmark for abstract reasoning and agent planning, making it comparable across different models and evaluation harnesses using standardized tools and interfaces.
The research builds on prior work in this space, including the 2024 'Baba Is AI' paper and studies like 'Baba Is Eval' (where Claude Sonnet 3 could only complete the introductory level) and 'Baba Is Agent' (where Gemini 3.1 Pro and GPT-5.5 managed to beat stage one's 8 levels). This work positions puzzle-solving games as valuable benchmarks alongside formal reasoning tests like ARC-AGI-3.
- Over $2,000 in experimental costs highlights the practical affordability limitations of current AI agents despite their problem-solving capabilities
- This research extends a growing body of work treating puzzle games as valid AI benchmarks, complementing formal reasoning tests like ARC-AGI-3
Editorial Opinion
While AI agents can now solve complex reasoning puzzles like 'Baba Is You,' this research exposes a critical disconnect between capability and cost-efficiency. The fact that pricing fails to correlate with performance—sometimes cheaper models end up dramatically more expensive—reveals troubling opacity in AI economics. As capabilities plateau, the industry's competitive advantage will belong to models that deliver results affordably, not just technically. This work is a welcome corrective, shifting focus from 'can it be solved?' to 'should it be solved, and at what cost?'
