K-Means Memory

Motivated by the Lattice Memory findings (the no-dynamics ablation was the fidelity ceiling; the best spring constant trended to zero): keep the mass-weighted centroid read, drop the topology entirely, and let the partition be data-adaptive streaming k-means over keys.

Mechanics: greedy fill — while a slot is free, every token births its own centroid, so short contexts are exact attention. Once full, tokens merge into the Euclidean-nearest centroid; a deferred scale factor makes the leak O(1) per token; centroids dying below a mass floor free their slots so births resume as old content fades. The leak doubles as a linear recency bias through the \beta\log(\text{mass}) score term.

Findings

1. Dropping the lattice topology lost nothing and (attested) doubled streaming speed, with equal-or-better lexical stats at 16× compression (2048 tokens in 128 centroids per head).
2. Birth policy matters more than capacity. A DP-means-style relative threshold self-throttled — the mean-distance EMA anchors high and almost nothing births, making capacity settings indistinguishable. Greedy fill fixed it instantly. (The idea returns, rehabilitated, as the surprise gate.)
3. Deep-stream degradation (~position 1900) persists and is not explained by memory blur alone or RoPE extrapolation alone: position-free scoring (rope=none) made it worse, so positional scoring is protective.
4. Re-indexed relative RoPE is implemented, verified, and — frozen — refuted. Each centroid tracks a decaying mean insertion position; at read time its key is rotated by its clamped integer age, so scoring uses relative positions that never leave the trained range — StreamingLLM's rolling-position trick generalized to clusters. The read path is exact to float noise (0.000014 max |Δlogit| at zero leak), but deep streams got worse. Diagnosis: raw phase interference is a feature — a position-incoherent cluster's rotated keys partially cancel, automatically suppressing blurry blobs; re-indexing restores their full magnitude at one fake age and lets stale mush outcompete fresh tokens. Position handling and forgetting are coupled. The story reverses with training in the loop — see Training in the Loop.

Measured behaviour

Short-range fidelity on the standard probe, and 3-seed stream health at 2048 tokens (8× trained context):

kmeans:m=128 holds the distinct-token ratio roughly flat across the stream; at half that budget (m=64) the worst seed sags — the problem Tiered Memory fixes at half the m=128 budget. Frozen (no tuning), the compressed memory pays a real perplexity tax at 2× context: validation CE 4.2283 for raw k-means vs 2.6080 for exact attention with RoPE extrapolation (see Training in the Loop for how tuning closes this).