// the find
yale-sys/prompt-cache
Modular and structured prompt caching for low-latency LLM inference
This is the reference implementation for the 'Prompt Cache' paper (arXiv 2311.04934), which precomputes and reuses KV-cache attention states for reusable chunks of a prompt (system instructions, schemas, personas) so repeated structure doesn't get recomputed on every request. It's aimed at researchers studying LLM inference latency, not at anyone trying to drop caching into a production serving stack — it works by monkey-patching specific `transformers` model internals (Llama2, Falcon, MPT).
The core idea is genuinely useful and the numbers back it up: TTFT drops from 286ms to 78ms on GPU in their demo by skipping recomputation of cached segments. PML, the XML schema language for marking which prompt modules are cacheable vs. dynamic (`cache="false"`), is a clean way to express prompt structure explicitly rather than hoping a KV-cache implementation figures it out implicitly. It ships with a full LongBench-based accuracy and latency benchmark harness (`eval.py`, `eval_acc.py`), so the claims are reproducible, not just asserted.
It only supports three specific older model families by patching their `transformers` forward passes directly — there's no path to vLLM, TGI, or any API-served model, so the technique can't actually be applied to how most people run inference today. Setup is fragile: it needs git submodules (LongBench, a forked bleurt built from source) and pinned older model checkpoints. No CI configuration and only two thin test files, and the last push was November 2024, so this reads as a paper snapshot rather than a maintained project — don't expect fixes if something breaks on current `transformers` versions.