// the find
ai-dynamo/dynamo
A Datacenter Scale Distributed Inference Serving Framework
Dynamo is NVIDIA's orchestration layer for running LLM inference across multi-GPU and multi-node clusters. It sits above existing engines like vLLM, SGLang, and TensorRT-LLM, adding disaggregated prefill/decode, KV-aware routing, and autoscaling. This is infrastructure for AI teams running serious inference workloads — if you're on a single GPU, there's nothing here for you.
The disaggregated prefill/decode architecture is the real differentiator: separating the two phases into independently scalable GPU pools lets you tune hardware to workload shape rather than overprovisioning both. KV-aware routing — sending requests to workers that already hold the relevant KV cache blocks — is a concrete, measurable optimization (the 2x TTFT improvement from Baseten is plausible and the mechanism makes sense). The Rust core for the hot path is the right call; Python extensibility on top keeps the integration surface manageable. The zero-config DynamoGraphDeploymentRequest YAML is genuinely useful: specifying a model and SLA target and having the system profile and deploy is a real operational improvement over manual tuning.
The NVIDIA hardware dependency is load-bearing, not incidental — the benchmarks that matter (GB200 NVL72, GB300) are on hardware almost no one outside hyperscalers and well-funded labs can touch, so the headline numbers aren't reproducible for most readers. The project is backed by NVIDIA, which means the OSS version will always lag the internal one and priorities will follow NVIDIA's roadmap, not the community's. The multi-layer dependency stack (Dynamo + vLLM/SGLang/TRT-LLM + Kubernetes + optional etcd/NATS) means a production incident can be extremely hard to isolate — the fault tolerance story covers worker crashes but not the coordination layer going wrong. Local development without GPUs is effectively not supported despite the file-based discovery mode existing.