// the find
ggml-org/llama.cpp
LLM inference in C/C++
llama.cpp is the dependency-free C/C++ inference engine that made running LLMs and VLMs locally practical on everything from Raspberry Pis to multi-GPU servers. It's for anyone who needs to serve GGUF-quantized models without pulling in PyTorch or a Python runtime, and it underpins a huge chunk of the local-inference ecosystem (Ollama, LM Studio, etc. all sit on top of it or its ideas).
Backend coverage is genuinely wide - CUDA, Metal, Vulkan, SYCL, HIP, CANN, RPC, WebGPU, even RISC-V vector extensions and IBM zDNN, and CI actually builds all of them. The quantization scheme (1.5-bit through 8-bit) plus the ggml tensor library underneath is the reference implementation everyone else copies. The conversion/ directory handles an enormous number of model architectures individually, and llama-server ships an OpenAI-compatible API plus a built-in web UI, so it's not just a CLI toy.
The codebase is sprawling and the core library API is still not stable/documented in a normal way - there's literally a pinned issue asking what the public API should even be. Chat templates, tool calling, and the various parser/grammar layers (jinja, peg-parser, chat-auto-parser) have grown organically and are hard to trace through. Model support lags upstream transformers by days-to-weeks for novel architectures, and quantization quality/perplexity tradeoffs aren't always well documented per-model, so you're often just trying quant levels until output looks reasonable.