// the find
getumbrel/llama-gpt
A self-hosted, offline, ChatGPT-like chatbot. Powered by Llama 2. 100% private, with no data leaving your device. New: Code Llama support!
A Docker-compose wrapper around llama.cpp and a forked Chatbot UI that lets you run a handful of pre-selected Llama 2 / Code Llama GGUF models locally with an OpenAI-compatible API. It's aimed at people who want a one-click local ChatGPT clone on a home server (built as an Umbrel app) rather than developers who want to swap models or fine-tune anything.
The OpenAI-compatible API via llama-cpp-python is the actual useful part — point any app built against the OpenAI SDK at localhost:3001 and it just works. CUDA, Metal (M1/M2), and Kubernetes deployment paths are all covered with separate compose files, so it scales from a laptop to a GPU box to a cluster without you writing that glue yourself. The benchmark table across real hardware (Raspberry Pi to M1 Max) is a genuinely useful data point most self-hosted LLM repos don't bother publishing.
Dead since April 2024 — in local-LLM time that's several generations behind; GGML support alone (half the compose files) is for a format llama.cpp dropped over a year ago. Model selection is hardcoded to a fixed list; 'load custom models' and 'switch between models' are still unchecked on the roadmap, meaning you're stuck with whatever Nous Hermes/Phind quantizations the maintainers picked. The UI is a fork of Chatbot UI rather than upstream-tracked, so UI fixes and feature additions from the original project don't flow through. 70B model needs 41GB+ RAM just to load, which the README is upfront about but makes the 'runs anywhere' pitch misleading for anything beyond the 7B tier.