// the find
karpathy/llm-council
LLM Council works together to answer your hardest questions
A local web app from Karpathy that fans a query out to several LLMs via OpenRouter, has them anonymously critique each other's answers, then has a 'Chairman' model synthesize a final response. It's for people who want quick side-by-side model comparison and are willing to pay per-query for the privilege of asking 2N+1 LLM calls instead of one.
The three-stage design (independent answers, anonymized cross-review, chairman synthesis) is a legitimately useful pattern, not just a gimmick — anonymizing identities before the review stage specifically guards against models rubber-stamping their own vendor's output. The codebase is small and readable: five backend files (config, council, openrouter, storage, main) with clear separation of concerns, so it's easy to fork and bend to your own needs. Routing everything through OpenRouter means swapping council members is a one-line edit in config.py rather than juggling separate SDKs and API keys per provider.
The author states outright he won't maintain or support it — there's no test suite in the tree, so any breakage from OpenRouter API changes or frontend dependency drift is yours to fix. Storage is flat JSON files under data/conversations with no locking or schema, which is fine for one person on localhost and will fall over immediately under concurrent use. Every query costs 2N+1 LLM calls (N first opinions, N reviews, 1 chairman) with no cost estimate or budget guard in the README, so a council of four models turns one question into nine paid API calls. There's no auth on the local server, so it's not something you'd expose beyond your own machine without adding that yourself.