// the find
lyogavin/airllm
AirLLM 70B inference with single 4GB GPU
AirLLM runs giant models on consumer GPUs by streaming one transformer layer at a time from disk, keeping only a single layer in VRAM. A 671B DeepSeek model on 12GB of VRAM sounds absurd until you realize the bottleneck shifts from memory to disk I/O — which is exactly the tradeoff this repo makes. The audience is researchers and hobbyists who want to run large models locally and can tolerate very slow generation.
The core insight — layer-by-layer streaming from disk — is genuinely clever and requires no model modifications. The `AutoModel` abstraction handles a wide range of architectures (Llama, Qwen, Mistral, ChatGLM, Mixtral MoE) through a single entry point, so you don't need to know which class to instantiate. Block-wise weight-only quantization is the right call here: since the bottleneck is disk load, you only need to compress weights, not activations, which preserves accuracy better than full quantization. MacOS MLX support is a real addition, not an afterthought.
Inference is extremely slow — streaming layers from disk on every forward pass means throughput measured in tokens per minute, not per second, which makes it unsuitable for anything interactive. The repo has accumulated three years of cruft: training scripts, RLHF notebooks, a Chinese dataset, and fine-tuning code that have nothing to do with inference and aren't clearly separated from it. There's no streaming/token-by-token generation in the public API — you wait for the whole output. The test suite is thin (two unit test files and some notebooks) for a library that relies on correct layer splitting and reassembly.