vLLM Gateway: Routing Self-Hosted Models Through a Production AI Gateway

vLLM ships with an OpenAI-compatible server that makes it trivial to plug into any AI gateway. But the real decision is which gateway layer you stack on top: a Kubernetes-native Inference Extension for model-aware routing, a proxy-style gateway like LiteLLM for virtual-key auth and multi-provider failover, or an enterprise API gateway like Kong for plugin-based governance. Each solves a different problem, and most production stacks end up combining two or three of them.

What vLLM Exposes Out of the Box

vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs. It provides an HTTP server that implements OpenAI's Completions API, Chat API, and more. The supported APIs include Chat Completions, Completions, Embeddings, Transcriptions, Translations, and the Responses API. Six API types, all speaking the protocol your application code already understands.

You start vLLM with a single command and an optional API key:

vllm serve NousResearch/Meta-Llama-3-8B-Instruct --dtype auto --api-key token-abc123

That command gives you a production-capable inference endpoint. Point your existing OpenAI client at it and you are serving a model. The key word is "a model." Singular. One team. One endpoint. No authentication beyond a single shared key, no rate limiting, no budget tracking, no logging beyond what the server emits. For prototyping, that is fine. For production, it is not enough.

Why a Raw vLLM Endpoint Is Not Enough

A single vLLM endpoint handles one model and one team. Once you add a second model, a second team, or a second provider, you need a management layer.

An AI gateway is a proxy layer that sits between your application and one or more LLM backends. It handles authentication via virtual keys, rate limiting, per-team budget enforcement, request logging, and automatic failover across providers. Five specific problems each get their own solution:

• Virtual keys, not API keys. You issue a virtual key per team or app. The real provider key lives only inside the gateway. If a team member leaves or a key leaks, you revoke one virtual key without touching the underlying credential.
• Per-team, per-model rate limiting. The marketing team's chatbot and the engineering team's code assistant share the same GPU fleet. Without rate limiting, one team exhausts capacity and the other times out.
• Hard budget caps. Set a monthly dollar limit per virtual key. When the budget hits zero, the gateway returns 429s instead of silently running up a bill.
• Full request logging. Every request is logged with model name, token counts, latency, cost, and the virtual key that made it. Your compliance team needs this for audit trails. Your ops team needs it for capacity planning.
• Automatic failover. If your primary vLLM instance returns 503, the gateway retries on OpenAI or another fallback without any change to client code. The application never knows the backend switched.

The Kubernetes-Native Path: Gateway API Inference Extension

The Gateway API Inference Extension is an official Kubernetes project that adds inference-specific routing to the standard Gateway API. It addresses a fundamental mismatch: traditional load balancers use HTTP path or round-robin, but LLM inference workloads are long-running, resource-intensive, and partially stateful. They also do not account for model identity or request criticality, such as distinguishing an interactive chat from a batch job.

The Inference Extension introduces two CRDs. InferencePool is managed by platform admins and defines a pool of model server pods deployed on shared GPU nodes, with configuration for scaling and balancing. InferenceModel is managed by AI/ML owners and maps a public name like "gpt-4-chat" to the actual model within an InferencePool. The split is deliberate: platform teams own the infrastructure, model teams own the routing.

The Endpoint Selection Extension, the routing component inside the Inference Extension, showed significantly lower p90 latency at higher QPS (500+) when tested with vLLM on H100 GPUs running 10 Llama2 model replicas. Standard Kubernetes Service spreads requests evenly across pods without knowing which ones are queueing. The ESE knows which pods have capacity, and routes accordingly.

The vLLM Production Stack provides three primary deployment options and supports round-robin, session-based, prefix-aware, KV-aware, and disaggregated-prefill routing strategies. It also spins up vLLM servers with traffic routing to different models, service discovery, and fault tolerance through the Kubernetes API.

NGINX Gateway Fabric can become an Inference Gateway with the Gateway API Inference Extension, adding model-aware routing, serving priority for models, and model rollouts. For teams already running NGINX in production, this is the lowest-friction path to inference-aware routing.

The Proxy-Gateway Path: LiteLLM, Envoy AI Gateway, and the vLLM Semantic Router

If Kubernetes-native routing is the infrastructure layer, proxy-style gateways are the application layer. They sit in front of vLLM and handle the five problems listed above: auth, rate limiting, budgets, logging, and failover.

LiteLLM is the most widely deployed open-source AI gateway. It has 40,000-plus GitHub stars and 100-plus provider integrations. It speaks the OpenAI protocol natively, so your existing application code needs zero changes. LiteLLM is CPU-bound and can run on any small instance alongside the vLLM backends it fronts. It typically adds single-digit millisecond overhead per request.

The vLLM Semantic Router operates as an Envoy External Processor that semantically routes OpenAI API-compatible requests to the most suitable backend model. It uses BERT-based or decoder-only LoRA classification to decide whether a query is math, creative writing, code, or general chat, and routes accordingly. It also provides PII detection, prompt guard, and a semantic cache to cut latency and token usage. The router has dual Go and Rust implementations with tight Envoy ExtProc integration.

Pairing the vLLM Semantic Router with Envoy AI Gateway adds enterprise-grade capabilities: token rate limiting with input, output, total, and time-based windows, model and provider failover, traffic splitting with canary testing, and OpenTelemetry-based observability. Envoy AI Gateway supports 20-plus LLM providers including OpenAI, AWS Bedrock, Azure OpenAI, Google Gemini, Anthropic, Groq, and self-hosted models. We covered this gateway in detail in how Envoy AI Gateway works and where it falls short.

For teams that want gateway capabilities without running Kubernetes, ngrok's AI Gateway offers a managed alternative. You can connect a vLLM server as a custom provider, adding credentials, rate limiting, and observability without Kubernetes infrastructure. The setup is three steps: start vLLM with vllm serve, expose it via an ngrok internal endpoint, and register it as a custom provider with the OpenAI Chat Completions API format. The AI Gateway adds API keys to upstream requests server-side, so the key never leaves the gateway.

The Enterprise API Gateway Path: Kong

Kong AI Gateway supports vLLM as a provider through the AI Proxy and AI Proxy Advanced plugins. It routes chat completions to /v1/chat/completions on the vLLM upstream. The integration requires Kong Gateway version 3.14 minimum and supports chat completions with streaming.

Kong is enterprise-focused with a strong plugin ecosystem and SSO, but it is heavier to operate and requires Kong infrastructure. For organizations that already run Kong for API management, adding vLLM as a provider is a natural extension of existing governance. For teams starting from scratch, the operational weight of Kong is harder to justify against lighter alternatives, a tradeoff we cover in more depth in Kong AI Gateway: LLM routing, pricing, and limits.

How These Layers Stack

The layers do not compete. They complement each other. vLLM replicas sit behind NGINX, that cluster gets fronted by an inference router that classifies query complexity, and the AI gateway sits above the router handling what NGINX cannot: virtual key auth, per-team spend, and failover to cloud APIs when your self-hosted capacity is full.

The full production topology looks like this:

1. Load balancer (NGINX) distributes traffic across vLLM replicas. It handles health checks, basic HTTP routing, and connection pooling.
2. Inference router (Gateway API Inference Extension or vLLM Semantic Router) classifies query complexity and routes to the appropriate model or pool. It reduces tail latency through model-aware endpoint selection.
3. AI gateway (LiteLLM, Kong, Envoy AI Gateway, or SHIM) handles virtual key auth, budget enforcement, request logging, PII redaction, semantic caching, and provider failover.

SHIM supports OpenAI-compatible endpoints including vLLM, adding semantic caching and PII redaction that vLLM alone cannot provide. When a raw vLLM endpoint processes a request containing a customer's email address or credit card number, that data passes through unchanged. A gateway layer with PII redaction catches and scrubs that data before it reaches the model, which matters for GDPR, KVKK, and any regulated industry. For a broader look at the gateway landscape and where each layer fits, see AI firewall vs. AI gateway vs. control platform.

The cost case for stacking these layers is straightforward. By routing traffic to your self-hosted GPU fleet first and falling back to cloud APIs only when capacity is full, you can cut inference costs by 60-80%, a routing pattern we break down further in AI FinOps for engineering teams shipping LLM features. The gateway is the layer that makes that routing decision.

When You Should Not Add a Gateway

Not every deployment needs a gateway. Skip the gateway if you use a single LLM provider with no plans to change, if your workload has ultra-low-latency requirements where even a few milliseconds of overhead matters, or if your team is too small to justify the ops overhead of another service.

A direct vLLM endpoint is simpler and faster when you have one model and one team. High-frequency trading systems, real-time voice AI under 200ms time-to-first-token, and internal prototype tools do not need a gateway. Start with the direct endpoint, and add a gateway the moment you add a second model, a second team, or a second provider.

The vLLM gateway decision is not about whether to add a layer. It is about how many layers you need, and which ones solve the problems you actually have.