Tiered Memory

The topline architecture: a semantic/episodic split in the lineage of the Compressive Transformer and Infini-attention (see Related Work), plus a merging criterion borrowed from automata theory — merge memories the readers cannot distinguish (a Nerode congruence over keys), not merely nearby keys.

Mechanics: an exact episodic ring of the last e raw (k, v) pairs per (layer, kv-head); evictions demote into a K-Means Memory-style centroid bank; one softmax reads the union (ring entries at mass 1). Every nerode tokens, centroid pairs merge when their score profiles against a probe buffer of recent queries agree within tau at every probe and their value means agree within tv RMS — keys differing only in directions queries never probe merge here, but never under Euclidean distance.

Findings

1. The ring fixes short-range fidelity. On the standard probe the standing-best configuration agrees with causal attention on 68 of 70 positions — the best of any compressing configuration measured (full table in K-Means Memory) — and merging causes no short-range damage.
2. Ring size is a copy-loop dial: sharp recency attention over a long window lets generation lock onto its own tail. A long ring (e=64) produced seed-dependent repetition collapses on 2048-token streams; e=16 never collapsed. The episodic channel must never search its own output.
3. Aggressive merging reproduces the reindex lesson from K-Means Memory: pooled mass's log-mass score advantage outcompetes fresh content. tau=0.2, tv=0.05 is the working regime. A merged-mass suppression score (max instead of sum on merge) confirmed the collapse mechanism and widens the safe window, but doesn't advance the standing best.
4. Surprise-gated insertion (g>0) spends capacity on novelty, not arrival order: a demoted token births a centroid only if its distance to the nearest centroid exceeds g× the running mean assignment distance. (A training-free cousin of DMC's learned append-vs-accumulate policy — see Positioning.) The gate only works behind the ring: gating all inserts from token 0 anchors the distance EMA on garbage, which is why the same idea failed as a k-means birth policy.
5. Gate and Nerode merging are substitutes, not complements — a full 2×2 ablation confirmed each mechanism alone rescues the baseline's worst seed, but "both" is numerically identical to gate-only wherever the gate is active: gating prevents duplicate centroids, so merges drop to ~0. Pick one — nerode-only for the analysis story (and the novelty claim), gate-only for cheapest operation.
6. The gate carries a 4× smaller bank: at m=32 (64× compression at 2048 tokens) the gated configuration holds a stream floor of 0.40/0.35/0.38 where ungated k-means at the same budget collapsed in the original experiment runs.

Against the eviction baseline

The literature's canonical streaming baseline is attention sinks + a recent window (StreamingLLM), re-implemented here as sink:s,w:

Standing best

Merging frees slots, so merge aggression converts directly into birth capacity for new content — 64 centroids that act like 128. Frozen, this retrofit already beats every prior configuration on validation CE (2.0644 vs frozen exact's 2.6080 at 2× context): the exact ring supplies the sharp local attention that cluster reads blur, without the RoPE-extrapolation penalty. Training in the Loop takes it further.