// the find
zylon-ai/private-gpt
Complete API layer for private AI applications on local models: RAG, skills, tools, MCP, text-to-sql, and more. Works with any OpenAI-compatible inference server.
PrivateGPT is an open-source API layer that sits between an OpenAI-compatible inference server (Ollama, vLLM, llama.cpp) and your application, exposing a Claude-API-shaped interface for messages, RAG, tools, MCP, and structured data access. It's aimed at developers building local/private AI apps who want Claude-API ergonomics without the API-compatibility work of gluing raw completions endpoints into an agent loop themselves.
Modeling the API surface on the Claude Messages spec (streaming, tool use, token counting, citations) is a genuinely useful choice — code written against it should port to the real Claude API with minimal changes if you ever want to switch backends. The scope is broad and mostly implemented, not just promised: built-in code execution sandboxing, database/CSV tooling, and MCP support all have real modules in the tree, not just docs pages. The codebase separates concerns cleanly at the component level (embedding factories, context stack builder, chat interceptor chain) which suggests this survived more than one refactor rather than being bolted together.
There are two parallel task-queue implementations (arq and celery) both present in the tree with their own task registries, which either means a migration is mid-flight or two subsystems are maintained in parallel — worth checking git history before betting on either. The README leans hard on cross-selling Zylon (the paid platform), and the compatibility table admits real gaps: no prompt caching, no OAuth/org support, and only 'basic' skills, so parity with the Claude API it's imitating is partial. Running it for real means standing up a separate LLM server and a separate embedding server plus whatever broker Celery/arq needs, which is a heavier operational footprint than the single-binary pitch of the original 2023 version this project is a ground-up rewrite of.