Anthropic's Ultra-Short System Prompts Work for Small Models Too, Experiment Shows
Key Takeaways
- ▸Shorter system prompts maintain performance parity on smaller models; the short-prompt variant scored +2.3 points (within noise floor) on deepseek-v4-flash
- ▸32% reduction in input tokens for tasks with identical outcomes between long and short prompt runs, indicating significant cost savings potential
- ▸No performance cliff or task-category collapse observed; 20 tasks changed outcomes between runs (11 newly passed, 9 newly failed), but this reflects single-attempt variance, not systematic degradation
Summary
Anthropic cut Claude Code's system prompt by roughly 80% for their newest models, based on the principle that more capable models require less detailed instruction. A new experiment published by the Ante project tests whether this surprising finding holds for smaller models — specifically deepseek-v4-flash, a genuinely compact and cost-effective model far from frontier capabilities. The team ran an A/B test on terminal-bench 2.1 (89 tasks), comparing a full system prompt (~34k characters per request) against a heavily condensed version (~18k characters), with everything else held constant. The results challenge conventional wisdom: the shorter prompt achieved performance parity (+2.3 points, within noise), used 32% fewer input tokens in the same-outcome subset, and produced no capability cliff despite cutting the prompt roughly in half. The findings suggest that much accumulated prompt text may represent technical debt — legacy patches for failures in older model versions — rather than genuine instruction requirements for current systems.
- On tasks both variants solved, the short-prompt agent used slightly fewer steps (886 vs. 911), suggesting efficiency gains even in the overlap
- Challenges the assumption that small, fast, cheap models require verbose hand-holding; accumulated prompt text often reflects historical model limitations rather than current requirements
Editorial Opinion
This finding is particularly valuable because it suggests that prompt engineering has accumulated considerable technical debt — patches for failure modes in models two or three generations old, never revisited for deletion. If smaller, cheaper models can achieve performance parity with half the prompt overhead, the scaling implications for cost-sensitive deployments are significant. However, the authors' rigor in noting this is one experiment on one model with a deliberately selected subset (same-outcome tasks) and multi-attempt validation needed is essential — prompt engineering is nuanced, and what works for deepseek-v4-flash may not generalize. The broader lesson is that benchmarking and A/B testing prompt changes, rather than relying on intuition, can unlock substantial efficiency gains.
