Researchers Discover Internal Geometric "Addition Module" in Llama 3.1 8B

Summary

Researchers at Northeastern University and Stanford University have discovered a remarkable internal mechanism in Meta's Llama 3.1 8B language model that performs addition operations using circular geometric representations. Located in layer 18 of the network, this "addition module" uses Fourier features to represent numbers as points on circles in activation space, rather than on a linear number line as might be expected.

The discovery reveals that Llama represents numbers using multiple circles, each corresponding to the number modulo a different value—a system similar to a residue number system. For example, the number 17 is represented simultaneously as 1 on a mod-2 circle, 2 on a mod-5 circle, 7 on a mod-10 circle, and 17 on a mod-100 circle. This allows the model to balance precision across different scales and handle cyclic reasoning tasks.

What makes this finding particularly significant is that the same addition module is reused across diverse tasks with structurally similar problems—from arithmetic ("7 + 9?") to temporal reasoning ("what day comes two days after Friday?") to calendar calculations (determining which month comes six months after August). The researchers traced information flow across layers and validated the module's function using causal methods, demonstrating how neural networks efficiently repurpose computational machinery.

This work represents a major advance in mechanistic interpretability, the field focused on understanding how neural networks actually compute internally rather than just observing their outputs. By mapping the geometric structure of neural representations, researchers are gaining concrete insights into how language models reason and generalize—knowledge essential for debugging, controlling, and designing more robust AI systems.

• This discovery demonstrates the power of mechanistic interpretability—understanding the geometric structure of neural representations unlocks insights into how models compute and generalize

Editorial Opinion

This discovery is a watershed moment for mechanistic interpretability. For the first time, researchers have mapped a complete internal computational module with clear mathematical structure, showing how a neural network solves multiple problems using elegant geometric machinery. Understanding these hidden geometries isn't academic—it's essential for building AI systems we can debug, control, and trust. As models grow more powerful and opaque, this kind of foundational work on internal mechanisms will become increasingly critical.