// the find
BerriAI/litellm
Python SDK, Proxy Server (AI Gateway) to call 100+ LLM APIs in OpenAI (or native) format, with cost tracking, guardrails, loadbalancing and logging. [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, VLLM, NVIDIA NIM]
LiteLLM is a Python library and proxy server that puts a unified OpenAI-compatible interface in front of 100+ LLM providers. You point your existing OpenAI client at it, set a base URL, and swap models without touching your application code. It targets teams that need centralized LLM management: cost tracking, virtual keys, rate limiting, and provider fallbacks.
The provider coverage is genuinely impressive and actively maintained — new providers show up within days of launch, and the compatibility matrix is honest about which endpoints each provider actually supports. The router's fallback logic is practical: you define a list of deployments in priority order and it handles retries, rate-limit errors, and failover automatically. The proxy runs at 8ms P95 at 1k RPS with a real benchmark linked, not a made-up claim. Docker images tagged `-stable` have gone through 12-hour load tests before publishing, which is more release discipline than most OSS projects bother with.
The codebase is enormous and the abstractions leak badly — provider-specific quirks surface as undocumented edge cases that you only discover in production. The enterprise tier gates legitimately useful features (SSO, fine-grained RBAC, audit logs) behind a paid license with no public pricing, which makes it hard to evaluate for regulated environments. The Prisma ORM dependency for the proxy means you're pulling in a Node-based toolchain into what is otherwise a Python project, which is a real pain for deployment and CI. Token usage normalization across providers is best-effort; some providers return null for prompt tokens and LiteLLM silently passes that through, breaking any cost tracking that depends on it.