// the find
zhudotexe/kani
kani (カニ) is a highly hackable microframework for tool-calling language models. (NLP-OSS @ EMNLP 2023)
kani is a Python microframework for building LLM-powered applications with tool/function calling. It sits between raw API clients and heavy orchestration frameworks like LangChain — you get automatic context window management, a clean `@ai_function` decorator for tool exposure, and async-first design, without opinionated prompt templating or hidden abstractions. Aimed at NLP researchers and developers who want control over the prompt layer without writing all the boilerplate themselves.
The `@ai_function` decorator with `Annotated` type hints is genuinely well-designed — your function's docstring and type annotations become the tool schema automatically, no separate JSON spec to maintain. Async-first from day one means running parallel chat sessions is natural, not bolted on. The engine abstraction is clean enough that swapping OpenAI for Anthropic or a local llama.cpp model is a one-line change. MCP server support (both local and remote) is a real addition — you can expose or consume MCP tools without leaving the framework.
604 stars and 30 forks after two-plus years suggests limited real-world adoption outside the Penn NLP lab — the community extensions page exists but the ecosystem is thin compared to alternatives. No built-in persistence or session storage; chat history lives in memory and you're on your own for anything that survives a process restart (there's a save/load utility but it's manual). The token counting for context management relies on each engine implementing it correctly, and the HuggingFace engine's token counting accuracy depends on whether the model's tokenizer is loaded — add a huge model and you might get surprises at context boundaries. The `model_specific` directory is a collection of per-model prompt-format hacks that will rot as model vendors iterate.