Related Work

The KV cache is the memory of a decoder-only transformer, and compressing it is a crowded field. This page organizes the papers surfaced and verified in a 2026-07 literature pass (see Positioning for what they imply about the mechanisms studied here). Sixteen papers, grouped by what decides which memories survive.

Two-tier memories: exact-recent + compressed-old

The architecture family Tiered Memory belongs to.

• Compressive Transformer (arXiv:1911.05507, 2019) — the original two-tier design: a FIFO of exact recent activations, evictions compressed into a coarser long-term memory. Trained from scratch.
• Infini-attention (arXiv:2404.07143, 2024) — local exact attention plus a compressive global memory, retrofit onto a pretrained model by fine-tuning — the closest precedent for training with the memory installed.
• CentroidKV (arXiv:2506.11418, 2025) — the closest single paper to the tiered read: attention sink + the 64 most recent tokens kept exact, the rest clustered by key similarity into centroids scored as \exp(q\cdot\hat k/\sqrt d + \log n) — the same structure and the same log-mass score term, training-free.

Eviction: choose what to drop

• StreamingLLM (arXiv:2309.17453, 2023) — attention sinks + a recent window; the eviction baseline. Re-implemented in this repo as the sink:s,w attention and measured against the cluster memories on the stream harness.
• SnapKV (arXiv:2404.14469, 2024) — an observation window of recent queries decides which past entries are retained — a probe-buffer cousin of Nerode merging, but used for selection, not merging.
• Expected Attention (arXiv:2510.00636, 2025) — ranks evictions by expected attention weight under a modeled distribution of future queries.
• CriticalKV (arXiv:2502.03805, 2025) — ranks by a worst-case output-perturbation bound; notably shows value states matter beyond attention weights (echoed by the value-mean condition in the Nerode merge test).

Merging by similarity: fuse nearby keys

• ClusterKV (arXiv:2412.03213, 2024) — cosine key-clustering, used at cluster granularity for recall/selection.
• LOOK-M (arXiv:2406.18139, EMNLP-F 2024) — key-only cosine merging for multimodal caches.
• WeightedKV (arXiv:2503.01330, 2025) — drops keys but merges their values, attention-weighted, into neighbors.
• FlowMM (arXiv:2511.05534, 2025) — similarity plus sensitivity-gated merging; a good survey of the merging subfield.

Merging by output: preserve what readers see

The family closest in spirit to reader-indistinguishability, though none implements a pairwise congruence test:

• KeepKV (arXiv:2504.09936, 2025) — merges, then compensates so inference is unperturbed, with an "Electoral Votes" mass-correction term (≈ the log-mass score). Compensation instead of a merge test: it fixes up any merge rather than refusing incompatible ones.
• KVSlimmer (arXiv:2603.00907, 2026) — Hessian-based closed-form asymmetric K/V merging.
• GRKV (arXiv:2605.31105, 2026) — output-matching merge via global ridge regression over a static final-prompt query window — the nearest published thing to a reader-based criterion, but not dynamic and not pairwise.
• Attention Matching (arXiv:2602.16284, 2026) — optimizes a compact K/V set to reproduce attention outputs (up to 50× compaction) by global latent optimization.

Learned compression

• DMC (arXiv:2403.09636, ICML 2024) — a learned per-token append-vs-accumulate decision, trained by continued pretraining — the trained cousin of the surprise-gated insertion studied here.

Not yet primary-verified

CaM, KVMerger, D2O, and Squeezed Attention were surfaced but not verified against their full texts; whether any contains a probe-query merge test closer than GRKV's static window is an open question tracked in Positioning.