NVIDIA-Backed Research Benchmarks 13 Local LLMs on Administrative Tasks—Gemma 4 Leads
Key Takeaways
- ▸Google's Gemma 4 12B achieved top-tier performance with the best resource efficiency among leading models
- ▸Model size is not a reliable performance predictor—task fit and structured output behavior matter more
- ▸Only Qwen 3.6 successfully completed PDF-to-calendar tasks, highlighting importance of model-specific strengths
Summary
A comprehensive benchmark study evaluated 13 locally-run language models on 21 administrative tasks, including calendar management, document parsing, financial calculations, email triage, and professional writing. Conducted on a single DGX Spark system with 128GB unified memory via Ollama, the study used both deterministic Python checks for structured outputs and a Llama 3.1 70B judge model for evaluating prose quality.
Google's Gemma 4 models ranked at the top, with the 31B and 12B variants statistically tied for strongest overall performance. Notably, the smaller 12B model achieved near-identical quality to its larger counterpart while requiring substantially less memory, making it the more practical default. Alibaba's Qwen 3.6 excelled at structured document work, notably being the only model to successfully pass PDF-to-calendar tasks.
A striking finding: model size proved an unreliable predictor of performance when runtime conditions were controlled. Several smaller models outperformed larger competitors, suggesting that instruction-following capability, output structure adherence, and task-specific optimization matter more than raw parameter count for administrative automation workflows. The benchmark separates these performance variables by controlling for GPU contention and using consistent evaluation criteria across all 13 models.
- Administrative automation requires more than fluent text generation—models must follow multi-step instructions and produce reliable structured outputs under realistic hardware constraints
Editorial Opinion
This benchmark addresses a critical gap in LLM evaluation: moving beyond raw capability metrics to practical, task-specific performance in constrained environments. For enterprise deployments considering local model execution, the finding that smaller models can match larger ones fundamentally challenges the "bigger is better" assumption. The emphasis on reproducible results—isolating GPU contention and using consistent judging—sets a higher standard for practical LLM benchmarking that should inform how the industry evaluates real-world fitness.
