Netflix Engineers Full-Stack LLM Serving Infrastructure, Moves from TensorRT-LLM to vLLM
Key Takeaways
- ▸Netflix operates its own full-stack LLM serving platform instead of consuming third-party hosted APIs, achieving tighter integration with existing production infrastructure
- ▸The platform migrated from TensorRT-LLM to vLLM to support custom model architectures, extensible decoding logic for constrained outputs, and faster iteration cycles
- ▸Architecture supports multiple inference patterns: small CPU models running in-process, larger GPU-bound models routed to NVIDIA Triton via a shared Model Scoring Service, with both real-time and cached batch paths
Summary
Netflix has built and deployed a custom, full-stack LLM serving infrastructure rather than relying on external hosted APIs, integrating large language models directly into its existing production environment. The system is fronted by a unified JVM-based serving layer that handles routing, A/B testing, feature fetching, inference, and logging, with smaller models running in-process and larger GPU-bound models delegated to a shared Model Scoring Service (MSS) backed by NVIDIA Triton Inference Server. Netflix initially selected TensorRT-LLM but re-evaluated in summer 2025 and switched to vLLM as its primary inference engine after benchmarking against a broader workload mix that includes embedding generation, prefill-only inference, autoregressive decoding, and custom models with complex constraint logic.
The decision to move to vLLM was driven by three operational factors: the ability to load custom model architectures without multi-step compilation pipelines, extensibility hooks for custom decoding logic (required for constrained-generation work), and better debuggability compared to compiled inference engines. Netflix's approach integrates model deployment, versioning, health checking, autoscaling, and multi-region rollout through a Java control plane, allowing ML practitioners to package artifacts and have the platform automatically provision GPU instances and orchestrate zero-downtime upgrades.
- A Java control plane handles deployment orchestration, versioning, health checks, autoscaling, and multi-region rollout—abstracting operational complexity from ML practitioners
Editorial Opinion
Netflix's vertical integration of the ML inference stack reflects a mature engineering approach: building infrastructure tailored to their specific workload rather than forcing production needs into the constraints of generic hosted solutions. The shift to vLLM underscores how open-source inference engines have closed the performance gap with proprietary stacks while offering superior developer ergonomics—a trend that may drive other large-scale platforms toward similar custom-built approaches.
