finds.dev

NVIDIA-NeMo/Switchyard

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

View on GitHub

★ 3,467RustApache-2.0updated Oct 2026

// the take

Switchyard is NVIDIA NeMo's Rust library (libsy) for deciding which LLM should handle a given request inside an agent harness or gateway, offering several routing strategies from simple classifiers to execution-trace-based escalation. It's aimed at teams building or operating their own agent gateway who want to mix cheap and expensive models without writing routing logic from scratch, not at someone wanting a drop-in hosted proxy.

Routing is decoupled from model calls and credentials — you keep your own HTTP clients and retry logic, the library just returns a decision, which keeps it embeddable rather than another service to run. Several routing strategies (stage/execution-based, LLM-classifier, composite, escalation, plan/execute) are backed by actual benchmark numbers on Terminal-Bench 2.1 and DeepSWE instead of just claims. It has real multi-language reach — Go, TypeScript and Python bindings plus a C FFI crate — not just a Rust-only library. It's already wired into existing infra via an OpenRouter hosted router, a LiteLLM plugin, and a NeMo Relay plugin, so there's more than one path to adoption without building your own gateway.

Pre-1.0 and it shows: of the five components, only libsy is Beta, the rest are Alpha or Demo-only, and the standalone server explicitly isn't a supported production deployment path — there's no stable artifact to actually run in production yet. The easiest integration route, LiteLLM, is experimental and checkout-only and can't make the intermediate model calls that classifier or escalation algorithms need, so most users are limited to the two simplest strategies. There's a documented bug where the NeMo Relay plugin can lose upstream error status and details for models outside its own routes, meaning enabling it can break error handling for unrelated traffic. The default Auto mode is just a fixed preset (Stage router, 0.5 confidence threshold) rather than something adaptive, and the README itself admits routing to a cheaper model doesn't guarantee a cheaper successfully completed task — real savings require running your own evaluation loop on top.

We read the README, the directory tree and a key source file in November 2026. Nobody pays to be listed. How we judge →

// at a glance

Stars
3,467
Forks
348
Language
Rust
License
Apache-2.0
Last commit
31 Oct 2026

// nearby

If Switchyard isn't quite it.

All repos →