New Benchmark Reveals Significant Gaps in Multimodal LLM Chart Understanding
Key Takeaways
- ▸CharXiv benchmark contains 2,323 hand-curated charts from arXiv papers with descriptive and reasoning-focused questions
- ▸Significant performance gap: GPT-4o achieves 47.1% accuracy vs. InternVL Chat V1.5 at 29.2%, with humans reaching 80.5%
- ▸Models show fragile performance, with accuracy deteriorating up to 34.5% on slightly different charts or questions
Summary
Researchers have introduced CharXiv, a comprehensive evaluation suite designed to assess how well multimodal large language models (MLLMs) understand charts in realistic scenarios. The benchmark comprises 2,323 hand-curated charts sourced from arXiv papers, paired with both descriptive and reasoning-focused questions that require models to analyze complex visual elements. This new dataset addresses a critical limitation in existing benchmarks, which often rely on oversimplified, template-based charts and questions that don't reflect real-world complexity.
CharXiv evaluations reveal substantial gaps between leading models and human performance. OpenAI's GPT-4o emerges as the strongest proprietary model with 47.1% accuracy, while the best open-source model, InternVL Chat V1.5, achieves only 29.2%—a gap of nearly 18 percentage points. Both models significantly underperform against human evaluators, who achieve 80.5% accuracy. The research demonstrates that performance can deteriorate by up to 34.5% when models encounter slightly different chart styles or question formulations, exposing the fragility of existing evaluation methods.
These findings underscore previously underestimated weaknesses in MLLM chart understanding and suggest that existing benchmarks may provide overly optimistic assessments of model capabilities. The researchers hope CharXiv will drive improvements in how multimodal models handle complex visual information analysis, a critical capability for applications in scientific research, financial analysis, and data-driven reporting.
- Existing benchmarks likely overestimate MLLM capabilities by using oversimplified, template-based chart datasets
Editorial Opinion
The CharXiv benchmark is a timely reality check for the multimodal AI community. While leaderboard numbers often create a false sense of progress, this research demonstrates that real-world chart understanding remains a significant challenge for even the most advanced models. The large gap between GPT-4o and open-source models suggests that proprietary training advantages matter more than public benchmarks indicate—a finding that should prompt both commercial and open-source researchers to invest in more robust chart understanding capabilities.
