// the find
GokuMohandas/fast-weights
🏃 Implementation of Using Fast Weights to Attend to the Recent Past.
A 2016-era TensorFlow implementation of the fast weights paper from Ba, Hinton et al., which augments RNNs with a rapidly-decaying weight matrix for short-term associative memory. It reproduces the toy key-value recall task from the paper and compares four model variants: plain RNN, layer-normalized RNN, GRU, and the full fast-weights version. For anyone trying to understand the paper through code rather than math.
The README explains the efficient reformulation of A(t) as a sum of outer products — skipping explicit matrix construction — which is the genuinely interesting part of the paper and often glossed over. Four comparable model variants let you see the contribution of layer norm and fast weights independently, which is more honest than just showing the best result. The bag-of-tricks section (identity init for slow weights, unit-variance init) is useful practical knowledge that doesn't appear in the paper.
Frozen in TensorFlow 0.x syntax — this won't run on anything installed in the last six years without significant porting work. The toy task is the only evaluation; there's no sequence-to-sequence or language modeling experiment despite the README promising one in a 'sep repo' that never materialized. No requirements.txt with pinned versions, so reproducing the original results is basically impossible now. The efficient implementation described in the README isn't actually what the code uses, so the most interesting computational property goes untested.