Zhipu AI Deploys Single-Rollout Asynchronous Optimization for More Stable LLM Training

Summary

Researchers have developed Single-rollout Asynchronous Optimization (SAO), a new technique for training large language models using reinforcement learning that addresses key stability challenges in agentic tasks. Unlike traditional batch-based RL approaches that are inefficient for long-horizon problems, SAO uses asynchronous training but replaces group-wise sampling with single-rollout sampling to reduce off-policy effects and improve generalization. The technique also introduces a strict double-side token-level clipping strategy to enhance optimization stability.

SAO was successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B) and consistently outperformed existing methods like GRPO on challenging benchmarks including SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. The researchers demonstrated that SAO can train stably for over one thousand steps and is particularly effective in online learning scenarios where models must adapt to changing environments. The technique's practical effectiveness in large-scale model training suggests a significant step forward in making RL post-training more efficient and reliable for production LLM systems.

• Particularly effective for online learning scenarios where models must continuously adapt to evolving environments

Editorial Opinion

This research represents meaningful progress in making RL-based LLM training both stable and scalable. The transition from batch-synchronized to genuinely asynchronous RL has been a long-standing challenge, and addressing both throughput and training stability simultaneously is valuable. That the technique successfully deployed in a large production model (GLM-5.2) adds credibility, though independent reproduction on other model families would strengthen confidence in its generalizability.