Llama 3.2 3B Achieves 85% on SimpleQA Benchmark, Matching Models 200x Larger
Key Takeaways
- ▸Llama 3.2 3B with retrieval achieved 85% on SimpleQA, just 3 points below a 671B parameter model
- ▸The performance gap was attributed to parameter count rather than retrieval quality or prompting techniques
- ▸Results suggest smaller models with effective retrieval may challenge larger models for question-answering tasks
Summary
An independent developer has demonstrated that Meta's Llama 3.2 3B model, paired with Keiro Research API for retrieval, achieved 85.0% accuracy on the SimpleQA benchmark's 4,326 questions. The result is notable for coming within just 3 percentage points of OpenDeepSearch's 671B parameter model (88.3%) and 0.8 points behind Sonar Pro (85.8%), despite using a model that is 100-200x smaller in parameter count.
The benchmark, run over a weekend using a local Llama 3.2 3B instance combined with Keiro's retrieval API, suggests that effective web-enabled retrieval can significantly narrow the performance gap between small and large language models for certain question-answering tasks. Only ROMA (357B) at 93.9% scored significantly higher. The developer noted that the systems ahead in the rankings achieved their advantage primarily through scale rather than superior retrieval or prompting strategies.
The results raise questions about the cost-effectiveness of massive models for retrieval-augmented generation tasks. The developer emphasized interest in exploring how small the "reader model" can become before model size becomes a limiting factor, suggesting that for many non-coding tasks, smaller models with web access may perform comparably to larger models. Full benchmark scripts and results have been made publicly available on GitHub, providing transparency into the methodology and enabling replication of the findings.
- The finding raises questions about compute cost-effectiveness for retrieval-augmented generation workloads
Editorial Opinion
This benchmark result is a compelling data point in the ongoing debate about model efficiency versus scale. The fact that a 3B parameter model can come within striking distance of systems hundreds of times larger suggests we may be approaching diminishing returns on pure scale for certain task categories. If retrieval quality proves to be the primary bottleneck rather than model size, it could accelerate a shift toward hybrid architectures that prioritize efficient reasoning over brute-force parameter counts, with significant implications for deployment costs and accessibility.
