Research Finds Symbolic Tools Outperform LLMs on Program Synthesis Tasks

Summary

A new research paper submitted to arXiv compares large language models against state-of-the-art symbolic tools across multiple program synthesis domains, including LTL reactive synthesis, syntax-guided synthesis, distributed protocol synthesis, and recursive function synthesis. The study evaluates Alibaba's open-source Qwen 32B model, OpenAI's frontier GPT-5, and established symbolic tools, with Qwen augmented by a symbolic verifier to improve performance.

The research reveals that symbolic tools consistently outperformed both LLMs across all tested domains. Symbolic tools solved more benchmarks than Qwen and either matched or exceeded GPT-5's performance. Most notably, symbolic tools demonstrated superior execution times compared to GPT-5 across all domains, and either matched or slightly outperformed Qwen despite the LLMs running on significantly more powerful hardware. These findings suggest that for structured program synthesis tasks, traditional symbolic approaches remain more reliable and efficient than current generative AI methods.

• Results suggest LLMs remain limited for structured, formally-verifiable synthesis tasks compared to traditional approaches

Editorial Opinion

This research provides valuable empirical evidence that despite recent LLM advances, specialized symbolic tools remain superior for formal program synthesis tasks. The findings underscore an important lesson: frontier LLMs excel at pattern matching and generation but struggle with domains requiring mathematical rigor and guaranteed correctness. Rather than viewing this as a limitation, it suggests a pragmatic future where LLMs and symbolic systems work in complementary fashion, with LLMs handling creative or approximate tasks while symbolic tools ensure formal correctness.