KV Cache

A decoder-only transformer generates one token at a time, and each new token attends over every previous position. To avoid recomputing the past, the model caches each position's key and value vectors per layer and head — the KV cache. It is the model's working memory: everything the model can remember about the sequence lives there.

Its cost is the problem. The cache grows linearly with context — for long streams it dwarfs the weights — and attention must scan all of it per token, so per-token latency grows with context too. An unbounded stream therefore needs a finite replacement, and every scheme answers the same question: which memories survive, and in what form?

• Eviction drops entries — by recency (sliding window), by learned or heuristic importance, or keeping "attention sink" positions that soak up probability mass.
• Merging fuses entries — by key similarity, or by preserving what future queries would read (see K-Means Clustering for the cluster-read view).
• Learned compression trains the model itself to decide what to keep.

The trade-off is measurable: exactness while the cache is below capacity, fidelity to full attention just above it, and long-stream health far beyond it. Finite Memory for Unbounded Streams explores this design space experimentally, and Related Work maps the published field.