Datadog Launches LLM Routing Monitoring for Kubernetes Inference Extension
Serving large language models on Kubernetes just got smarter, and more observable. Datadog has published a detailed guide showing teams how to monitor LLM routing with the Kubernetes Inference Extension, a tool that routes AI inference traffic based on real backend state rather than blind round-robin load balancing.
The problem with standard HTTP load balancers is straightforward. They treat every model-serving pod as interchangeable. In reality, LLM backends are stateful. One pod may have a warm KV cache. Another may already hold the right LoRA adapter in memory. A third may have an empty queue. Routing without that context wastes GPU capacity and drives up latency.
The Kubernetes Gateway API’s Inference Extension addresses this directly. It gives the gateway access to backend serving state, including KV cache occupancy, LoRA adapter availability, and queue depth, so it can send each request to the most prepared pod. Additionally, the extension separates intelligent routing logic into a dedicated Endpoint Picker (EPP). The EPP evaluates telemetry from model servers before selecting a target. Importantly, this keeps core Kubernetes routing APIs clean while enabling advanced scheduling via tools like the CNCF llm-d inference scheduler.
Furthermore, the extension supports flow control. This adds a central request queue between the gateway and backend pods. Instead of committing a request to a busy pod immediately, flow control buffers it until a better pod is free. It also enforces priority tiers, interactive user traffic can skip ahead of background batch jobs. And if the pool is saturated, the EPP can shed low-priority requests entirely, returning 429 or 503 errors rather than letting congestion drag down the whole system.
However, knowing the routing system is in place is not enough. Teams still need to verify it is actually working in production. That is exactly where Datadog comes in.
Datadog’s monitoring approach spans three layers. First, the vLLM integration captures model server signals, KV cache utilization, running and waiting request counts, swap rates, and Time to First Token (TTFT). Second, the OpenMetrics integration scrapes Prometheus-compatible metrics from the Inference Extension, covering routing distribution, queue duration, and shedding events. Third, Datadog GPU Monitoring tracks VRAM pressure, a useful proxy for LoRA adapter saturation when a direct adapter-count metric is unavailable.
Together, these layers help teams answer a critical question: when TTFT rises, is the problem a routing misconfiguration or a capacity shortage? If GPU hardware metrics are healthy but TTFT is climbing alongside an uneven queue distribution, routing logic is likely the culprit. If pod restarts, OOMKills, and VRAM pressure are all elevated, the cluster is simply at its physical limit.
Datadog also recommends tracking Kubernetes CRD objects for the InferencePool and InferenceObjective resources. This confirms that the intended routing architecture is actually deployed and not drifting from its configuration.
For teams migrating from legacy ingress controllers, moving to the Gateway API is a prerequisite for inference-aware routing. The payoff, however, is significant: better GPU utilization, lower TTFT, and the ability to prioritize latency-sensitive traffic without overprovisioning hardware.
Full documentation for the Datadog vLLM integration, OpenMetrics integration, and Kubernetes Monitoring is available via Datadog’s developer portal.
