// the find
envoyproxy/ai-gateway
Manages Unified Access to Generative AI Services built on Envoy Gateway
Envoy AI Gateway is a Kubernetes-native API gateway for LLM traffic, built on top of Envoy Gateway rather than implementing its own proxy. It normalizes requests across providers, handles backend auth for each cloud's credential flow, and enforces token-based rate limits. The target audience is platform teams running AI workloads on Kubernetes who need centralized control across multiple providers.
The decision to build on Envoy Gateway instead of rolling a new proxy is the right call — you get battle-tested connection management, TLS termination, and observability for free, and the team focuses on the AI-specific translation layer. Per-provider schema translation in the data plane (internal/apischema) means you can send an OpenAI-format request and have it rewritten to Bedrock or Anthropic's wire format at the gateway, without touching application code. Token-based rate limiting is genuinely harder to implement correctly than request-count limits and more relevant for LLM cost control; the fact that it's a first-class feature rather than an afterthought is notable. Backend auth handlers cover the annoying credential flows for AWS (IRSA, pod identity), GCP (Workload Identity), and Azure rather than punting to the application layer.
Both v1alpha1 and v1beta1 CRDs exist simultaneously in the codebase, which signals an unstable API surface — anyone adopting this now is signing up for migration work when things stabilize. The two-tier gateway architecture (Tier 1 for routing, Tier 2 for self-hosted inference) adds real operational complexity; you're running two gateway layers, each with its own config, rollout, and failure modes. The hand-rolled provider schema implementations will drift as providers update their APIs — if Anthropic ships a new parameter or changes a response field, you're waiting on a PR to get merged before it works. Kubernetes is a hard prerequisite for any real deployment; the aigw CLI downloads Envoy binaries as a workaround, which is fine for demos but not a supported production path.