Hazy Research Reveals Transformer MLPs Are Natural Hebbian Memories—Enabling Instant Fact Storage Without Training
Key Takeaways
- ▸Closed-form MLP construction requires no gradient descent or training—facts can be instantly embedded into Transformer blocks using random matrices and Hebbian memory principles
- ▸Achieves information-theoretically optimal storage density of Θ(F log F) parameters for F facts, matching empirical performance of large language models
- ▸Enables practical applications including fact editing and knowledge injection in deployed models without retraining, and provides constructive interpretability of fact storage mechanisms
Summary
Researchers at Hazy Research have discovered that Transformer multi-layer perceptrons (MLPs) naturally function as Hebbian memories, enabling a closed-form mathematical construction for storing facts directly into models without any gradient descent or training. The breakthrough, led by Roberto Garcia, Jerry Liu, Ronny Junkins, and Chris Ré, provides the first practical method to build fact-storing MLPs that achieve information-theoretically optimal storage rates—matching empirical observations from prior work.
The construction is remarkably simple: given key and value embeddings representing facts (like mapping "France" to "Paris"), researchers sample random matrices and apply a gated MLP structure derived from classical Hebbian memory principles. The MLP then performs soft lookups using kernel similarity, collapsing to the correct fact when queried with a stored key. This approach stores F facts in just Θ(F log F) parameters—the theoretical optimum.
Beyond efficient fact storage, the research unlocks several critical capabilities: it explains why large language models store facts at optimal rates, provides constructive interpretability for how facts are represented inside Transformers, and most practically, enables seamless fact editing in deployed models without retraining. The closed-form construction means researchers can now predict exactly how facts will be encoded, moving beyond black-box mechanistic interpretability.
Editorial Opinion
This work elegantly bridges classical neuroscience (Hebbian learning) with modern deep learning architecture, offering both theoretical rigor and immediate practical applications. The ability to construct fact-storing MLPs in closed form represents a significant leap toward interpretable transformers and more efficient knowledge management in language models. If these results scale to production-scale models and diverse fact types, this could fundamentally reshape how we approach knowledge editing, retrieval-augmented generation, and model auditing.
