// the find
dusty-nv/NanoLLM
Optimized local inference for LLMs with HuggingFace-like APIs for quantization, vision/language models, multimodal agents, speech, vector DB, and RAG.
NanoLLM is an inference library for running LLMs on NVIDIA Jetson edge devices, with a plugin-based architecture that wires together speech, vision, RAG, and agent components. It targets people building embedded AI pipelines on constrained hardware — think robotics, home automation, or kiosk deployments — not cloud servers. The HuggingFace-like API is a thin wrapper over MLC, AWQ, and AutoGPTQ backends.
The plugin graph architecture (each component is a node with typed inputs/outputs) is a clean way to compose multimodal pipelines without writing glue code. Supporting three quantization backends (MLC, AWQ, AutoGPTQ) and letting the `auto` loader pick based on available hardware is practical for Jetson's varied memory constraints. The web studio for visually wiring plugins together is genuinely useful for iterating on agent topologies without restarting. Speech support covers both Whisper and NVIDIA Riva ASR/TTS, so you can pick latency vs. accuracy depending on the task.
Last commit was October 2024 and the latest release is 24.7 — this is already going stale as Jetson JetPack versions move on; no indication of active maintenance. The whole thing is Jetson-first and the Docker container dependency is real: running this outside the `dustynv/nano_llm` image is not a supported path, which makes local dev painful. The robotics dataset modules (RLDS, DROID, robomimic) are half-baked stubs that look like research scaffolding rather than production utilities. With 381 stars and no tests beyond a handful of manual scripts, the surface area is large relative to the community actually stress-testing it.