// the find
Michael-A-Kuykendall/shimmy
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.
Shimmy is a single-binary Rust LLM inference server that runs GGUF models through a custom WebGPU engine called Airframe, exposing an OpenAI-compatible API with no C++ toolchain required. The main pitch is that WGSL compute shaders work on any GPU that speaks Vulkan, D3D12, or Metal — no CUDA, no llama.cpp, nothing to compile at install time. It targets developers who want to point an existing OpenAI client at local models without wrestling with Python environments.
The WebGPU/WGSL approach is the real differentiator here. Running on Vulkan, D3D12, and Metal from a single code path means you get actual cross-vendor GPU acceleration — NVIDIA, AMD, Intel, and Apple Silicon — without pinning to CUDA or shipping multiple backends. That's harder to pull off than it sounds. The TurboShimmy INT4 KV cache compression runs entirely in WGSL with no CPU roundtrips, delivering a genuine ~7x VRAM reduction for the KV cache; fitting a 3B model comfortably on a 4GB GPU is a concrete win for consumer hardware. Model architecture is auto-derived from GGUF metadata rather than hardcoded per-model constants, which means new quantized models don't require code changes to load. The VRAM sizing math is documented explicitly with the formula and worked examples — you can actually plan your deployment instead of guessing.
GPU-verified model coverage tops out at 3B parameters in the stable track; the 7B models are listed as 'pending remote GPU validation', which is a notable gap for the primary differentiator. The one-model-per-instance limitation is a real operational cost — serving multiple models means running multiple processes and managing multiple ports, something Ollama handles natively. The install story is genuinely confusing: `cargo install shimmy` gives you the huggingface (CPU) engine, not Airframe, and you need the pre-built GitHub Releases binary for GPU support — this is buried in footnotes and will silently mislead people who just reach for cargo. The MoE/70B support is featured prominently in the README but is explicitly labeled as roadmap-only and not implemented in this version.