SuperBake: Direct Fact Installation in Transformer Weights Without Fine-Tuning
Key Takeaways
- ▸SuperBake installs verified facts by constructing precise neuron circuits directly in weights—eliminating fine-tuning, LoRA, and retrieval-augmented generation approaches
- ▸Achieves 91.3% verification on 1,000-fact benchmark while preserving prose perplexity, compared to SGD baselines that double perplexity damage
- ▸Solves the reversal curse by directly writing reverse mappings, reaching ~94% on reverse questions vs. 0.7% for gradient-trained approaches
Summary
A new research paper introduces SuperBake, a groundbreaking method for installing verified facts directly into transformer model weights through manual circuit construction rather than gradient-based fine-tuning. The system constructs precise neuron circuits at specific physical addresses within model MLP layers, with built-in behavioral verification that neutralizes any failed facts before deployment.
Testing on a 7,450-row benchmark with 1,000 facts across Llama-3.1-8B and Qwen2.5-7B models, SuperBake achieves 91.3% verification rates—significantly outperforming masked-SGD baselines at 76.9%—while maintaining baseline prose perplexity. The method proves particularly effective at solving the reversal curse problem, with constructed facts answering reverse questions at ~94% accuracy compared to 0.7% for trained baselines.
The research reveals that gradient-based fact cramming stores knowledge in a coherent sub-threshold population code within activation-function leakage, the same signal that damages prose quality. SuperBake rebuilds this mechanism as an addressable, gated, verifiable associative memory that remains transparent and inspectable. Deliverables are standard checkpoints compatible with existing inference code, accompanied by receipts documenting each fact's location and verification status.
- Processes facts at 14.8 facts/second with end-to-end construction in minutes on a single 80GB GPU
- Derives from mechanistic understanding of how SGD stores facts in activation-function leakage, enabling safer, transparent knowledge insertion
Editorial Opinion
SuperBake represents a paradigm shift: moving knowledge injection from opaque statistical fine-tuning to direct, inspectable weight construction. The verification-first approach and reversible fact construction could make knowledge grounding in LLMs dramatically safer and more trustworthy. If released and scaled, this technique could enable production-grade, auditable fact bases in language models—transforming knowledge editing from a black-box operation into architected, transparent engineering.
