Netflix Shares Architecture for In-House LLM Serving Platform
Key Takeaways
- ▸Netflix operates a fully in-house LLM serving platform integrated with existing production infrastructure rather than using hosted APIs
- ▸vLLM selected as the paved-path engine for better support of custom architectures, constrained decoding, and faster iteration cycles
- ▸Hybrid architecture routes CPU-bound models in-process and GPU-bound models to shared Model Scoring Service backed by NVIDIA Triton
Summary
Netflix has published technical details about its in-house LLM serving infrastructure, describing how the company built and operates a unified inference platform rather than relying on external APIs. The platform integrates with Netflix's existing JVM-based serving system and leverages NVIDIA Triton Inference Server, supporting both real-time and batch inference paths for diverse workloads including embeddings, ranking, and autoregressive decoding. The architecture separates concerns by running smaller CPU models in-process to avoid remote-call overhead, while delegating larger models to a shared remote inference backend called Model Scoring Service (MSS) that manages GPU scheduling and batching.
Key architectural decisions included selecting vLLM as the primary inference engine over TensorRT-LLM, citing better operational fit for custom model architectures, extensibility hooks for constrained decoding logic, improved debuggability, and familiarity among ML practitioners. Netflix's Java control plane handles deployment, versioning, health checking, autoscaling, and multi-region rollout across their infrastructure. The platform design reflects lessons learned from production usage, revealing trade-offs not anticipated during the design phase and emphasizing the company's commitment to ownership of the full ML stack rather than outsourcing inference to third-party vendors.
- Java-based control plane orchestrates deployment, versioning, autoscaling, and zero-downtime upgrades across multi-region rollout
- Platform designed to support diverse workload patterns including prefill-only inference, embeddings, ranking, and custom models with complex per-step logic
Editorial Opinion
Netflix's willingness to own its entire ML serving stack—rather than outsource to API providers—reflects a broader industry trend toward vertical integration of AI infrastructure at scale. For companies processing user interactions at Netflix's scale, the operational complexity of building in-house is worth the payoff in latency, cost control, and flexibility. This case study illustrates how architectural decisions compound: engine choice cascades into packaging strategy, which shapes API design and deployment patterns. Netflix's openness about both the 'why' and production trade-offs provides valuable guidance for other media and high-scale platforms considering similar bets.
