// the find
AutoArk/TinyEngram
Research of DeepSeek Engram Architecture based on Qwen-3 and Stable Diffusion series.
TinyEngram is a research repo implementing DeepSeek's Engram architecture — an N-gram memory module grafted into transformer layers — on top of Qwen-3 and Stable Diffusion. The core claim is that Engram beats LoRA on catastrophic forgetting while matching its adaptation quality, and the same mechanism transfers to vision models. Aimed at ML researchers who want a working, reproducible baseline for the Engram approach without building everything from scratch.
The experiments are actually reproducible: pinned requirements, full training scripts, DeepSpeed configs, and detailed hyperparameter ablation docs are all present. The vision extension is a genuinely interesting idea — treating visual concepts as N-gram-triggered memory injections in the CLIP text encoder lets you stack thousands of concepts without interference, because exact N-gram matching means unrelated triggers never collide. The catastrophic forgetting comparison against LoRA is well-designed: they deliberately overfit LoRA to show the tradeoff and document exactly what hyperparameters they used. There is a paper on arXiv with the theory behind the vision findings, not just a README with pretty images.
The benchmark numbers are tiny-model territory (Qwen-0.6B) and the improvements over baseline are modest enough that it's hard to tell how much transfers to anything production-scale. The vision results use a five-image dataset of a fictional character — the composability claim about 'thousands of engrams' is stated but not demonstrated at scale. The core `engram.py` implementation is a single file with no inline documentation, so understanding the gated retrieval mechanism requires reading the original DeepSeek paper alongside. Training speed is measurably slower than LoRA and the repo doesn't give a concrete throughput comparison, just 'LoRA converges faster.'