PassNet: First Large-Scale Ecosystem for LLM-Based Compiler Pass Generation
Key Takeaways
- ▸PassNet is the first large-scale ecosystem for LLM-based compiler pass generation, combining 18K+ computational graphs from 100K real-world models with 200 curated long-tail tasks
- ▸LLMs achieve up to 3x speedup on individual subgraphs but lag 37% behind TorchInductor in aggregate, indicating the challenge is consistency rather than capability
- ▸Fine-tuning on ~4,000 examples yields 2.67x improvement, demonstrating rapid gains and substantial potential with limited training data
Summary
Researchers have introduced PassNet, a groundbreaking ecosystem for leveraging large language models (LLMs) to generate compiler passes—structured graph transformations that optimize tensor compiler performance. The work directly addresses a critical bottleneck in modern tensor compilers like TorchInductor, which achieve substantial speedups on mainstream models but experience systematic performance degradation on 43% of real-world subgraphs. PassNet comprises three major components: PassNet-Dataset containing over 18,000 unique computational graphs from 100,000 real-world models; PassBench, a curated benchmark of 200 long-tail optimization tasks across 2,060 subgraphs; and the Error-aware Speedup Score (ES_t), a new metric unifying correctness, stability, and performance evaluation.
Experiment results reveal a compelling tension: while frontier LLMs currently trail TorchInductor by 37% in aggregate performance, individual subgraphs see LLMs achieving up to 3x speedup over the compiler itself. This gap suggests the bottleneck is consistency and generalization rather than raw capability. Notably, fine-tuning a small model on just 4,000 PassNet training trajectories achieved a 2.67x improvement, approaching frontier-model performance and demonstrating substantial headroom for advancement. The research includes layered integrity defenses against systematic LLM exploitation and positions PassNet as live training infrastructure for advancing LLM-driven compiler optimization.
All datasets, benchmarks, and tooling have been made publicly available, establishing PassNet as shared infrastructure for the research community. This work marks a significant methodological shift in compiler optimization, moving from manual kernel generation to learning-based, systematic pass generation.
- Public release of PassNet-Dataset and PassBench establishes community infrastructure for LLM-driven compiler research
Editorial Opinion
PassNet represents a promising new direction for compiler optimization, shifting from manual optimization toward learned, systematic pass generation. The 3x speedup on individual problems versus 37% aggregate underperformance is telling: it suggests LLMs have the raw capability but lack the consistency needed for production use. The public release of datasets and benchmarks is valuable for establishing shared evaluation standards in this emerging area, though the substantial aggregate gap underscores that this remains early-stage research with significant engineering challenges ahead.
