// the find
codefuse-ai/ModelCache
A LLM semantic caching system aiming to enhance user experience by reducing response time via cached query-result pairs.
ModelCache is a semantic cache layer for LLM applications — you embed it between your app and the LLM, and repeated or semantically similar queries return cached answers instead of hitting the model again. It's a fork/evolution of GPTCache from the Alibaba/Ant Group CodeFuse team, aimed at teams running LLM products at scale where inference cost is a real line item.
Multi-tenant support with per-model data isolation is genuinely useful for SaaS products where you're serving multiple customers or models from one deployment. The long/short text differentiation with separate similarity thresholds is a practical fix for a real problem — short queries like 'what is 2+2' need a tight threshold, long prompts need a looser one. Redis-based vector search integration gets cache lookup down to ~10ms, which is actually competitive with the overhead you'd save. The pluggable embedding backend (HuggingFace, ONNX, PaddleNLP, FastText, local Milvus) means you're not forced into a specific model or cloud dependency.
The setup story is rough — the 'standard' service requires MySQL plus Milvus, and you have to manually download model weights from HuggingFace and place them in a specific directory. That's three external dependencies before you write a line of application code. The last meaningful activity was mid-2024 (the 2025 push is ambiguous), and the TODO list is long with no indication of when items ship. Similarity threshold tuning is entirely on you — there's no tooling to help you figure out whether your threshold is causing false hits (wrong cached answers served) or false misses (cache never used), which is the hardest operational problem with semantic caching. The codebase has two near-duplicate module trees (modelcache and modelcache_mm) that share a lot of structure but are maintained separately, which suggests the abstractions haven't quite converged.