Program Synthesis Enables Interpretable Explanations of Transformer Attention Mechanisms

Summary

Researchers have developed a novel approach to interpret and explain the behavior of attention heads in transformer language models using program synthesis. Rather than treating attention mechanisms as opaque neural computations, the team leverages a pre-trained language model to generate human-readable Python programs that can reproduce the patterns of attention heads given only raw text input.

The method analyzes attention matrices from trained models (GPT-2, TinyLlama-1.1B, and Llama-3B) on randomly selected training examples, then prompts a language model to synthesize executable Python programs that replicate these patterns. Generated programs achieve over 75% Intersection-over-Union similarity on held-out test data, with fewer than 1,000 programs needed to explain all attention heads across tested models.

Critically, the work demonstrates practical utility by replacing up to 25% of attention heads with their programmatic surrogates, incurring only a 16% average increase in perplexity while maintaining performance on downstream question-answering tasks. This scalable pipeline for reverse-engineering attention mechanisms advances the goal of symbolic transparency in neural networks.

• This work provides a practical pipeline for interpretability and reverse-engineering, advancing toward symbolic transparency in deep learning systems

Editorial Opinion

This research represents a significant step toward making transformer models more interpretable and trustworthy. Demonstrating that attention mechanisms can be replaced with human-readable code without substantial performance loss is a breakthrough for understanding what these models actually compute internally. However, the real test will be whether this approach scales efficiently to modern billion-parameter models and whether the generated programs provide insights that meaningfully inform model design and debugging.