How OpenAI Routes Low-Latency Voice AI for 900M Weekly Users

Summary

OpenAI processes voice AI conversations for 900 million weekly active users, requiring a sophisticated infrastructure architecture to maintain the low latency that makes voice conversations feel natural rather than awkward. The engineering challenge stems from WebRTC's design assumptions (stable server IPs and ports) conflicting with Kubernetes' ephemeral infrastructure, where traditional Selective Forwarding Unit (SFU) architectures built for multiparty video calls are inefficient for OpenAI's predominantly one-to-one user-to-model traffic pattern.

OpenAI's solution splits the voice infrastructure into two distinct layers: a stateless relay service at the geographic edge that handles protocol-aware packet routing, and a stateful transceiver that manages the heavy WebRTC state machine. The architecture elegantly connects these layers by repurposing the ICE ufrag (a field exchanged during WebRTC protocol setup) as a routing key that the relay can read from the first packet of a new session, eliminating the need to maintain session state at the edge.

The implementation prioritizes three concrete requirements: reaching 900 million distributed users, completing connection setup rapidly so users can speak immediately, and maintaining consistently low round-trip latency for crisp conversational turn-taking. This technical deep-dive reveals the engineering sophistication required to serve voice AI at scale, including Global Relay deployment, userspace Go implementation, Redis caching, and socket-level optimizations.

• The architecture avoids traditional SFU (Selective Forwarding Unit) design because OpenAI's traffic is predominantly one-to-one, not multiparty
• Connection setup and round-trip latency must remain tight enough that conversation feels natural, not like a walkie-talkie with awkward delays

Editorial Opinion

This engineering case study demonstrates the depth of infrastructure investment required to make voice AI feel genuinely conversational at global scale. Rather than retrofitting existing WebRTC patterns, OpenAI's team (which includes original WebRTC architects) designed from first principles to solve the unique constraint of stateless Kubernetes infrastructure serving one-to-one conversations. The elegant use of protocol-level fields as routing keys shows how deep protocol knowledge can yield both elegance and efficiency—a reminder that voice AI's naturalness depends not just on model quality, but on the often-invisible infrastructure underneath.