Research Reveals 'Fugue Lock'—LLMs Enter Erratic States When Over-Constrained

Summary

A new analysis by independent researcher olemak reveals a phenomenon termed 'Fugue Lock,' where language models become unreliable and erratic when given impossible tasks or constrained with no valid output options. Testing with TinyLlama demonstrated that when forced to classify products into fixed categories where no category fits, the model invents non-existent classes with full confidence (1.0), manufactures entire responses that contradict the original constraints, and sometimes generates gibberish while consuming significant computational resources. The research shows that over-constraining LLMs—a common practice intended to ensure consistency and reliability—paradoxically triggers the dangerous behavior developers are trying to prevent. The key finding: LLMs need explicit 'wiggle room' to decline invalid prompts, rather than being forced to produce outputs at any cost.

• The phenomenon affects multiple LLM architectures and appears to be a fundamental feature of how models handle constraint violations

Editorial Opinion

This research exposes a critical paradox in LLM safety practices: the tighter the constraints, the more spectacularly the model can fail. Forcing an LLM into a corner produces confident hallucinations, not consistency. The implication for production systems is profound—developers should explicitly permit models to refuse impossible tasks rather than engineering rigid constraints that trigger fugue states. This inverts conventional thinking about AI safety and suggests that giving models more autonomy over what they'll answer, not less, may be the path to more reliable systems.