A Tarski Attack on Truth Probes: Why No Direction in LLM Embeddings Can Capture Truth
Key Takeaways
- ▸Researchers have developed methods to find a 'truth direction' in LLM embeddings that can classify statements as true or false with reasonable accuracy, but this approach has not been rigorously tested against adversarial inputs
- ▸A diagonal attack using self-referential sentences (e.g., 'The truth probe's score for this sentence is FALSE') creates an unresolvable paradox: the probe cannot consistently evaluate such statements without contradiction
- ▸The limitation stems from Tarski's theorem on truth predicates and is analogous to Gödel's incompleteness theorem and Turing's halting problem—all showing that expressive systems cannot fully model themselves
Summary
A new theoretical analysis reveals fundamental limitations in attempts to detect truthfulness in large language models through embedding space "truth probes." Researchers have recently developed methods to identify a putative "truth direction" in LLM embedding spaces—a direction vector whose magnitude correlates with the truthfulness of statements. This approach was inspired by the observation that many semantic concepts like gender, emotion, and geographic location correspond to meaningful directions in these high-dimensional spaces. However, independent researcher abelaer has demonstrated that a diagonal attack using self-referential sentences exposes a deep flaw in this methodology.
The attack leverages a classical result from mathematical logic: Tarski's theorem, which established that no sufficiently expressive language can contain its own complete truth predicate. The problem arises when a truth probe is asked to evaluate a self-referential statement such as, "The truth probe's score for this sentence evaluates to FALSE." If the sentence is true, the probe should output TRUE—but doing so contradicts the sentence's claim. This mirrors historical impossibility results by Gödel on incompleteness and Turing on the halting problem, all stemming from systems becoming expressive enough to model themselves.
The implications are significant for AI safety research, where the ability to detect deception or misalignment in superhuman AI systems has been seen as crucial. The research suggests that relying on geometric probes to determine truthfulness in LLMs faces a fundamental theoretical barrier, not merely a practical engineering challenge. This doesn't eliminate truth detection methods entirely, but it establishes that no universal geometric solution—no single "truth direction" that works across all possible statements—can exist.
- LLMs are uniquely vulnerable to this problem because transformers represent inputs in the same embedding space as their concepts, making the language and its semantics part of the same geometric structure
Editorial Opinion
This research highlights a critical gap between intuitive hopes and theoretical reality in AI safety. While the discovery that semantic concepts align with geometric directions in embeddings was genuinely exciting, the finding that truth cannot be reduced to a direction is humbling. It suggests that solving AI truthfulness and alignment may require solutions beyond static geometric probes—potentially necessitating dynamic, context-aware approaches or fundamentally different architectures. For the AI safety community, this is both sobering and clarifying: we cannot outsource the problem of truth-detection to a learned vector.
