LLM Councils Suffer Severe Groupthink, Losing 75% of Novel Ideas
Key Takeaways
- ▸LLM councils retain only 22-25% of unique ideas from individual models versus ~33% of consensus ideas—a systematic 50% penalty for novelty
- ▸Both simple answer blending and structured peer review yield nearly identical groupthink effects (24% vs 22% single-model idea retention)
- ▸Ideas rated as useful and non-obvious by independent judges are discarded simply because they don't appear in multiple model outputs
Summary
Independent research by Rohit Krishnan reveals that large language model 'councils'—ensembles where multiple models review and synthesize each other's responses—systematically suppress novel ideas in favor of consensus positions. Testing committee, peer-review, and direct-selection approaches across 16 open-ended prompts (strategy and writing tasks), Krishnan found that councils retain only 22-25% of high-quality ideas that originated from individual models, while preserving ~33% of ideas appearing in multiple responses—a substantial bias against novelty. The effect mirrors human committee dynamics: while outputs become more polished and readable, distinctive insights are eliminated.
Blind judges rated two-thirds of the suppressed ideas as useful and non-obvious, yet they failed to survive the council process. Examples include a retail observations noting salvaged scent cartridges as status symbols, an analysis arguing that logged-but-ignored risks are more dangerous than unknown ones, and a creative data-recovery approach using re-confirmation prompts as crowdsourcing. The research challenges the prevailing assumption that LLM diversity through councils automatically improves reasoning and problem-solving.
- The committee effect trades distinctive, unconventional insights for polish and consensus—potentially problematic for creative and strategic problem-solving
Editorial Opinion
This research exposes a critical blind spot in the LLM ensemble movement: diversity-by-consensus may suppress the exact kinds of novel thinking that make AI systems valuable for breakthrough insights. As organizations deploy LLM-powered decision-making systems, the findings suggest retaining access to individual model outputs alongside committee summaries may be essential. The suppression of non-consensus ideas is particularly damaging for innovation tasks where unconventional approaches often yield the highest payoff.
