Researchers Question Whether LLMs' 'Human-Like' Attributes Are Actually Unique
Key Takeaways
- ▸Simple neural networks trained on non-linguistic substrates can exhibit properties similar to those attributed to LLMs, suggesting these are not uniquely human-like characteristics
- ▸Substrate matters: the same LLM behaviors might be interpreted differently depending on what system produces them, questioning whether the properties are intrinsic or observer-dependent
- ▸Current LLM research often lacks explicit measurement criteria, leaving interpretation vulnerable to anthropomorphic bias
Summary
A new arXiv paper challenges the widespread assumption that large language models possess uniquely human-like attributes such as understanding, morality, and reasoning. Researchers trained a simple neural network on the video game Age of Empires II and demonstrated that it exhibits similar properties attributed to LLMs, suggesting these perceived anthropomorphic characteristics may be substrate-independent artifacts rather than evidence of genuine cognition.
The authors argue that much discourse around LLMs incorrectly assumes or asserts human-like qualities without rigorous measurement criteria. By showing that a game engine can exhibit comparable behaviors, they demonstrate that the interpretation of LLM capabilities is heavily dependent on the observer's perspective and chosen measurement framework. The paper calls for explicit, measurable definitions before making claims about LLM anthropomorphism.
The research proposes a 'null assumption' approach where scientists assume LLM non-uniqueness rather than anthropomorphic attributes when designing experiments. The authors also prove that Age of Empires II exhibits functional and Turing completeness, supporting their thesis that any sufficiently complex substrate—from neural networks to LEGO to geography—could present similar emergent properties.
- A 'null assumption' framework could improve rigor by requiring researchers to explicitly test for anthropomorphic attributes rather than assuming they exist
Editorial Opinion
This paper makes a valuable contribution by pushing back against the field's growing tendency to anthropomorphize LLMs without rigorous justification. The Age of Empires II analogy effectively illustrates how easily we project human-like cognition onto systems that may simply be sophisticated pattern-matching. However, the work could strengthen its argument by more carefully distinguishing between the philosophical question of whether LLMs truly 'understand' versus the empirical question of how to rigorously test for understanding.
