// the find
evilsocket/cake
Distributed inference for mobile, desktop and server.
Cake is a Rust inference server that shards transformer models across heterogeneous devices — your old iPhone, a Linux box, a MacBook — so you can run models that don't fit in any single machine's VRAM. It handles text, image (Stable Diffusion, FLUX), and TTS, with an OpenAI-compatible API. For someone sitting on a pile of old hardware who wants to run a 70B+ model without buying a $10k GPU, this is the most practical approach I've seen.
The layer-sharding over mDNS is genuinely clever: workers don't need the model files, the master streams only the tensor shards each worker needs over zstd-compressed connections. Backend coverage is real — CUDA, Metal, Vulkan, ROCm, and CPU all in one binary, with custom kernels (WGSL, MSL, CUDA) rather than just wrapping candle defaults. The benchmark suite and autoresearch directory suggest the author is actually measuring things rather than hoping they're fast. Architecture auto-detection from HuggingFace config.json means you can pull any supported model family without manual wiring.
The FAIR license is a hidden trap for anyone evaluating this for a company: any business use requires a signed commercial agreement, which means legal review before you can even experiment with it in a work context. The README itself warns this is 'experimental code being actively developed and changed very quickly' — that's honest, but the API surface for distributed inference has no stability guarantees. Mobile support (the iOS/Android app) appears skeletal from the directory structure; the Kotlin side is two files and the Swift side is a bridge, so don't expect to actually run inference on your phone today. There's no documentation on failure handling in the cluster — what happens when a worker drops mid-generation is unclear from the README.