Envoy AI Gateway routes LLM traffic across 16 providers on Kubernetes. Learn what its v1.0 ships, what it lacks, and who should use it.
Envoy AI Gateway hit v1.0 on June 23, 2026, sixteen months after v0.1. That milestone matters because the project is now committing to API stability: the CRDs you write today will not break under you without a documented migration path.
But stable does not mean complete. Envoy AI Gateway is a data-plane proxy for LLM traffic. It routes requests, counts tokens, and emits telemetry. It does not manage budgets per team, offer a dashboard, cache semantically similar prompts, or enforce prompt guardrails. That split is deliberate, and understanding it saves you from adopting the wrong tool for your problem.
Envoy AI Gateway is an open-source project built on Envoy Proxy to handle request traffic from applications to Generative AI services. It lives inside the CNCF ecosystem, which means it inherits Envoy's extension model, its community governance, and its operational assumptions: you run Kubernetes, you configure via YAML, and you are comfortable with proxy internals.
Its project objectives include automatic failover, end-to-end security with upstream authorization, a policy framework for usage limiting, and extensible architecture via Envoy's plugin framework. These are infrastructure concerns. The project does not aim to be a developer platform with API key management, spend dashboards, or prompt engineering tools.
This distinction shapes every evaluation. If you need an AI proxy that sits between your code and your providers at the network layer, Envoy AI Gateway is built for that. If you need a platform that also governs who spends how much on which model with what guardrails, you are looking at a different category of tool entirely.
The gateway uses a two-tier pattern. The Tier One Gateway is your centralized entry point. It handles authentication, top-level routing, and global rate limiting. The Tier Two Gateway handles ingress traffic to self-hosted model serving clusters, with endpoint picker support for LLM inference optimization.
The practical result is model virtualization. Your application points to a single OpenAI-compatible endpoint. The gateway handles provider-specific routing underneath. You can switch or mix providers without touching application code, run A/B tests between models, migrate gradually from one vendor to another, or maintain multi-provider strategies for redundancy and cost.
A Medium introduction to the project describes the core value proposition: platform teams get cost controls and observability, while developers never touch provider-specific SDKs. One endpoint, many backends, zero application-level awareness of which provider is serving a given request.
The overhead is competitive. The gateway adds roughly 1 to 3ms of latency per request, which is negligible against typical LLM response times of hundreds of milliseconds to several seconds.
The v1.0 release is worth examining feature by feature, because the gap between "supports 16 providers" and "is a complete AI gateway platform" is where most evaluation mistakes happen.
Stable CRDs. The v1 API includes AIGatewayRoute, AIServiceBackend, BackendSecurityPolicy, GatewayConfig, MCPRoute, and MCPRouteSecurityPolicy. The stability commitment is strict: these resources will never break without a critical security reason, and any change requiring user action ships with a documented migration path.
16 providers behind one endpoint. The full list: OpenAI, Azure OpenAI, Google Gemini, Google Vertex AI, AWS Bedrock, Anthropic, Mistral, Cohere, Groq, Together AI, DeepInfra, DeepSeek, Hunyuan, SambaNova, Grok, and the Tetrate Agent Router Service. All accessible through a single OpenAI-compatible interface.
Centralized credential management. BackendSecurityPolicy keeps provider credentials out of application code. It supports API keys plus AWS, Azure, and GCP cloud-native identity, including Workload Identity.
Token-aware rate limiting. Version 1.0 attributes cost separately for input, output, cached, and reasoning tokens, scopes those costs per route with fleet-wide defaults, and adds the QuotaPolicy API for quota-aware routing around rate-limited upstreams.
Prompt caching. A provider-agnostic unified cache_control API for prompt caching. This is distinct from semantic caching.
OpenTelemetry observability. Every request emits traces using GenAI semantic conventions, with OpenInference compatibility for evaluation tools like Arize Phoenix, across chat, embeddings, image generation, audio, MCP, and reasoning endpoints.
MCP gateway. The v1 release includes a production-grade Model Context Protocol gateway with server multiplexing, per-tool routing via include/exclude rules, OAuth 2.0 JWT claim forwarding, and CEL-based fine-grained authorization.
Provider fallback. Automatic failover between AI providers when an upstream goes down or hits rate limits.
That is a solid infrastructure feature set. The gaps are on the application side.
A feature comparison across open-source AI gateways surfaces the missing pieces clearly:
| Capability | Envoy AI Gateway | LiteLLM | Bifrost |
|---|---|---|---|
| Semantic cache | No | No | Yes (Weaviate) |
| Virtual keys | No | Yes | Yes |
| Budget control | No | Basic | Yes (4-tier) |
| Web UI | No | Yes | Yes |
Four gaps, each with different implications.
No web UI. All configuration happens through Kubernetes CRDs and YAML. There is no dashboard to inspect traffic, no visual interface for setting up routes, no click-to-configure provider backends. For platform engineering teams comfortable with GitOps workflows, this is fine. For teams where product managers or finance need visibility into AI spend, it is a blocker.
No semantic caching. The gateway supports provider-level prompt caching through a unified API, but it does not cache semantically similar queries. If two users ask the same question with slightly different wording, both requests hit the provider and both consume tokens. For high-volume consumer applications with repetitive query patterns, this gap translates directly into higher costs.
No virtual keys or budget hierarchy. You cannot issue virtual API keys to different teams or projects, set per-team spending limits, or build tiered budget controls. The gateway's rate limiting is token-aware and route-scoped, but it operates at the infrastructure level, not the organizational level. If your finance team asks "how much did the marketing team spend on Claude last month," the gateway's telemetry data can help you calculate an answer, but it will not enforce a cap or alert when a budget threshold is crossed.
No built-in prompt guardrails. Content filtering, PII redaction, and prompt injection detection are not part of the gateway's feature set. You would need to layer those on separately, either through custom Envoy filters or through an additional middleware layer.
Beyond features, there is the learning curve. The Envoy xDS configuration model is powerful but dense. Teams outside the Envoy ecosystem will spend meaningful time learning CRD patterns, understanding the extension model, and debugging routing rules expressed in YAML rather than through a GUI.
You run Kubernetes with Istio already. Envoy is in your stack. You need stable, vendor-agnostic LLM routing at the infrastructure layer with strong observability and credential management. Your platform engineering team is comfortable with CRDs and GitOps. You want CNCF-backed open source with a strict API stability commitment. The 1 to 3ms overhead and 16-provider support give you the routing substrate you need, and you will build (or buy) application-layer governance separately.
Your team does not run Kubernetes, or does not want to. You need a web UI for non-engineering stakeholders to see spend and usage. You need virtual keys with per-team budget enforcement out of the box. You want semantic caching to reduce costs on consumer-facing applications with repetitive queries. You need prompt guardrails without bolting on another system.
The distinction is architectural. Envoy AI Gateway solves the data-plane problem: getting requests from your applications to the right model, reliably, with telemetry. It does not solve the control-plane problem: governing who can spend how much, what content is allowed, and how to visualize all of it without writing queries against raw telemetry data.
For teams that need both layers handled in a single system, a managed AI gateway covers the routing and the governance in one stack, without requiring Kubernetes or xDS expertise to operate.