LLMs Show 74% Success Rate on Real Matching Decompilation Tasks, Study Finds
Key Takeaways
- ▸LLMs successfully matched 74% of functions across 60 real decompilation tasks from active retro game projects
- ▸88% of successful decompilations demonstrated consistency, producing identical output across multiple runs
- ▸Mizuchi benchmarking pipeline combines programmatic and AI-powered tools, showing that LLMs augment rather than replace existing decompilation tooling
Summary
A comprehensive benchmark study tested large language models on 60 real functions from active retro game decompilation projects, revealing that LLM-powered decompilation pipelines successfully matched 74% of functions. The research, conducted using a custom benchmarking tool called Mizuchi, combined traditional programmatic decompilation tools with AI-powered approaches to test whether LLMs could effectively convert assembly code back into matching C source code. Matching decompilation is a critical technique in the retro gaming community for recreating original game source code, with notable projects like Super Mario 64 and The Legend of Zelda: Ocarina of Time fully decompiled using these methods.
The study addresses skepticism about LLM capabilities in decompilation by providing empirical data from real-world projects rather than synthetic benchmarks. Beyond the 74% match rate, the pipeline demonstrated strong consistency, with 88% of successful decompilations producing identical results across multiple runs. The research emphasizes that LLMs don't replace existing programmatic decompilation tools but rather augment them, working alongside established community tools to improve efficiency and accuracy. This work fills a critical gap in decompilation research by establishing standardized benchmarking methodologies that can guide future improvements to both AI-powered and traditional decompilation approaches.
- Study provides empirical data addressing skepticism about LLM decompilation capabilities and establishes standardized evaluation methodology for the community
Editorial Opinion
This benchmarking study provides much-needed empirical evidence that LLMs can meaningfully contribute to decompilation workflows, even if they don't solve the problem independently. The 74% match rate on real-world tasks is encouraging and suggests that hybrid approaches combining traditional tooling with AI will be the practical path forward for retro gaming preservation efforts. The emphasis on standardized evaluation metrics is particularly valuable, as it moves the field beyond anecdotal success stories toward reproducible, evidence-based improvements.
