Netflix Takes LLM Serving In-House: Architecture Decisions for Production-Scale Inference
Key Takeaways
- ▸Netflix built a custom in-house LLM serving platform deeply integrated into production infrastructure rather than using hosted APIs or separate ML silos
- ▸The company migrated from TensorRT-LLM to vLLM to gain extensibility for custom models, better debuggability, and support for constrained decoding logic
- ▸Architecture combines NVIDIA Triton Inference Server with a Java control plane for deployment, health checks, autoscaling, and multi-region orchestration
Summary
Netflix has published a detailed technical post about its in-house LLM serving infrastructure, revealing how the company chose to build and operate its own inference platform rather than relying on hosted APIs. The platform integrates deeply into Netflix's existing production environment, moving away from a separate ML silo model. Netflix's Model Runtime and Inference teams documented four key architectural decisions: engine selection, model packaging, API surface design, and deployment strategy.
The company originally built on TensorRT-LLM but migrated to vLLM by summer 2025 after open-source engines closed the performance gap and Netflix's workload expanded to include embeddings, ranking, and custom models with constraint logic. The platform uses NVIDIA Triton Inference Server as the underlying compute backend, with a Java control plane handling deployment, versioning, health checks, autoscaling, and multi-region rollouts. Small CPU models run in-process to avoid latency overhead, while larger models delegate inference to the remote Model Scoring Service (MSS).
Key considerations included balancing tight coupling between frontend and model artifacts, managing GPU scheduling, supporting diverse inference patterns (real-time, batch, embeddings, prefill-only), and enforcing output constraints during decoding. Netflix shared these decisions not as prescriptive guidance but as documentation of what worked for their scale and workload mix, acknowledging that production often reveals trade-offs the design phase didn't anticipate.
- Platform supports diverse inference patterns: real-time, cached batch, embeddings, prefill-only for ranking/retrieval, and constrained decoding
- Small CPU models run in-process to minimize latency; larger models use remote GPU inference through the centralized Model Scoring Service
Editorial Opinion
Netflix's in-house approach to LLM inference represents a significant shift in how large tech companies operationalize generative AI — moving from outsourced APIs to vertically integrated infrastructure. While the added complexity of managing GPUs, Triton, vLLM, and deployment orchestration is substantial, Netflix's scale and specific latency and customization requirements justify the investment. The decision to prioritize extensibility and debuggability over raw inference speed signals that production AI systems value flexibility and observability over benchmark numbers — a pragmatic lesson for other enterprises evaluating their AI infrastructure strategy.
