Music-Bench: New Open-Source Benchmark Reveals LLMs Struggle with Musical OCR
Key Takeaways
- ▸Current state-of-the-art LLMs (GPT-5.4, Claude Opus, Gemini-3-Flash) achieve poor performance on musical OCR tasks, with exact match rates well below acceptable levels
- ▸The benchmark deliberately starts simple—single notes on single staff—indicating that even basic musical notation reading exceeds current model capabilities
- ▸Music-Bench is open-source and configurable, supporting all major LLM providers and designed to scale to more complex musical elements once baseline performance improves
Summary
A newly released open-source benchmark called Music-Bench reveals that leading large language models—including OpenAI's GPT-5.4, Anthropic's Claude Opus, and Google's Gemini—perform poorly at reading printed musical notation. The benchmark tests LLMs' ability to interpret images of written music and identify individual notes, currently focusing on single-note recognition without chords, rests, or ties due to the models' poor baseline performance.
Created by jlebar and generated entirely with code synthesis, the benchmark includes public and private test splits with 48 examples. Initial results show concerning accuracy rates, with all tested models frequently misidentifying notes. For example, when shown four notes (F4, A4, F5, A4), GPT-5.4 returned five notes including an incorrect G4, while Claude Opus and Gemini produced significantly different sequences entirely.
The benchmark is designed with contrast pairs and comprehensive metadata to prevent model shortcutting and enable detailed error analysis. The creator has invited contributions from AI companies and researchers to improve model performance on this task, suggesting it represents a meaningful gap in current LLM multimodal capabilities.
- The benchmark uses contrast pairs and careful metadata collection to enable reproducible, contamination-resistant testing across models
Editorial Opinion
Music-Bench exposes a surprising blind spot in otherwise capable multimodal AI systems: the ability to read basic musical notation. While the benchmark acknowledges its current simplicity—testing only single notes—the poor performance across all tested models suggests this is a legitimate gap worth addressing. As AI systems become increasingly integrated into creative and professional workflows, the ability to reliably interpret musical notation could have meaningful applications in composition, music education, and accessibility tools. This benchmark provides a valuable public resource for tracking progress on this underexplored challenge.
