18 Attention sinks and concentration
Where we are. In Ch. 13 we met the sink (that bright column over the first token). Now we look at it in depth —why it forms— and present one of our clean results: concentration (the sink) and positional decay (γ) are two independent things, not the same one. It matters because the recent literature often conflates them, and separating them avoids mistaken conclusions.
18.1 The idea in one sentence
That a model concentrates a lot of attention on a few tokens (sink) and that its attention decays with distance (γ) are distinct mechanisms: you can move one without touching the other.
18.2 Key concepts and their role in the transformer
Before getting into detail, we define this chapter’s terms and what each one is for inside a transformer:
- Attention sink. Definition: the huge amount of attention that almost all tokens dump onto a few (typically the first), even though they carry no meaning. In the transformer: it’s the softmax “relief valve” —which forces each row to sum to 1—; without a place to dump the leftover weight, attention would destabilize.
- Massive activations. Definition: a few enormous hidden-state values, far above the rest. In the transformer: they’re the cause of the sink —they steer attention toward their token— and are tied to how the model compresses information.
- Sink mass. Definition: the fraction of attention accumulated on those few tokens, a number between 0 and 1. In the transformer: it measures how concentrated attention is; it’s a phenomenon of magnitude, not of position.
- Positional decay (γ). Definition: the exponent by which attention falls off with distance (
A(d)∝d^−γ). In the transformer: it predicts the model’s effective reach and the compressibility of its KV (Ch. 20); it’s a positional phenomenon. - RoPE base θ. Definition: the constant that sets the frequencies of the rotary position encoding. In the transformer: it’s the lever we rescale in the experiment; moving θ changes γ but, as we’ll see, doesn’t touch the sink.
- Within-model control. Definition: comparing the same model with itself by changing one single thing. In the transformer: it’s what lets us isolate the cause (θ) without the cross-factors of comparing different models.
- Independence (⊥). Definition: two mechanisms are independent if moving one doesn’t move the other. In the transformer: our central result —concentration and decay are two separate axes—.
With this clear, let’s get to why the sink forms and to the proof that γ and the sink aren’t the same thing.
18.3 What a sink is (and why it forms)
Recall (Ch. 13): the sink is the huge attention that almost all tokens dump onto the first token, even though it means nothing. The cause is structural: since softmax forces each row to sum to 1, a token that doesn’t need to look at anything in particular has to dump its weight somewhere —and models learn to “park” it on the first tokens—.
🧩 Analogy — the softmax overflow drain. Softmax forces each row to sum to 1: it’s a sink that’s always full. A token that needs to look at nothing has a leftover of “water” (weight) that must go somewhere —so it opens the drain and pours it into the first token—. The sink isn’t garbage: it’s the valve that keeps that leftover from distorting what does matter.
Recent research has sharpened this:
- It’s universal and forms during pre-training, it’s not a quirk of the BOS token (Gu et al. 2025): it arises from the optimization dynamics and the data.
- It’s caused by massive activations: a few enormous hidden-state values (Sun et al. 2024) that steer attention toward their token. And it’s been shown that sink and compression of representations are “two sides of the same coin” (Queipo-de-Llano et al. 2025).
In a sentence: the sink is a norm/magnitude phenomenon (how much attention piles up at one point), not a positional one (how it spreads with distance).
The nice thing about 2025-26: the sink has gone from a phenomenon you observe to a component you design. Since the model “needs” somewhere to dump its leftover attention (so as not to force a distribution that distorts the softmax), several frontier models now build in an explicit sink: gpt-oss (OpenAI 2025) adds a per-head learned sink —an extra term in the softmax denominator that absorbs that mass— instead of letting the model improvise it over the first token. It’s the clearest sign that the sink isn’t a defect but a functional piece: once you understand what it’s for, you put it in on purpose.
18.4 Our result: γ ⊥ sink
Here’s the contribution. (The symbol ⊥ means “independent”: moving one doesn’t move the other.) If concentration and decay were the same thing, moving one would move the other. We ran the clean experiment —the one that controls everything else—: take one and the same model (pythia-1b) and rescale only its RoPE base θ, by a factor of 256×, without touching anything else.
figures/make_fig17_orthogonality.py over data/gap1_ntk_rescale.csv.
The result is crisp: γ traverses almost its entire range (0.75 → 0.999) while the sink mass barely moves (0.371 → 0.387). Same lever; one mechanism changes enormously, the other not at all. Conclusion:
Positional decay (γ) and structural concentration (sink) are independent mechanisms. They’re not two names for the same thing, nor does one follow the other.
Check it yourself: move θ and you’ll see the tail’s slope change (γ from 0.75 to 1.0) while the sink’s pillar doesn’t budge —two knobs that don’t touch:
18.5 Why this matters (and why within-model)
Two reasons, and here it links with the honesty of Ch. 16:
- It adjudicates a confusion in the 2026 literature. Many works speak of “concentration” of attention as if it were the same as its decay with the distance. It’s not: our experiment separates them with data. When measuring or comparing attention, you have to treat them as two distinct axes.
- The clean control is within the same model. In Ch. 16 we warned that comparing raw γ across different models mixes factors. Here we avoid that problem: same model, same everything, only θ changes → any effect is caused by θ. That’s the difference between describing (atlas) and isolating a cause (controlled experiment).
There’s a tempting thermodynamic analogy (our Part III). Since softmax conserves probability mass (each row sums to 1), under RoPE with sharp decay (γ>1) the excess mass “has to condense” into the ground state —the first token—, just as in a Bose-Einstein condensation particles pile up in the ground level. The theory even gives a closed form for the condensed fraction as a function of γ:
\[ N_{\text{cond}} \;=\; 1 \;-\; \frac{1}{\zeta(\gamma)}\int_{1}^{L} d^{-\gamma}\,\mathrm{d}d \]
where \(\zeta\) is the Riemann zeta function and \(L\) the context length.
But let’s be surgical about what the data do and don’t say. The formula predicts the condensate depends on γ, and that condensation onsets at the Hagedorn point (γ=1): in Phase A (γ<1) there would be little condensate. Now, our within-model experiment (Figure 18.1) sweeps γ from 0.75 to 0.999 —all of it Phase A— and there it finds a large, flat sink (≈0.38). Carefully, this means:
- What the data do show: a robust sink in Phase A, where the condensation picture would predict little → this is in tension with identifying the sink with the BEC condensate. And, consistent with this chapter’s thesis (γ⊥sink), its mass isn’t set by γ.
- What we cannot claim: the clean sweep never enters Phase B (γ>1), which is where the BEC’s γ-dependent prediction lives. We do have γ>1 points in the atlas (e.g. Mistral-7B γ≈1.06, Llama-3-8B γ≈1.05, BLOOM-7B γ≈1.22 on text), but they’re cross-model: they change architecture, data and scale all at once, so they confound the effect of γ with everything else —they’re not a controlled sweep within the same model, the way the γ<1 one is—.
Honest conclusion: BEC is a valid conceptual lens —it explains why a sink must exist (mass conservation)— but its quantitative, γ-dependent prediction is unsupported and open, neither confirmed nor cleanly refuted. We use it as mechanistic intuition, not as a number. ⚠ In dispute / open question — a within-model test at γ>1 is still missing.
18.6 The practical implication
When you analyze a model’s attention, don’t mix two questions:
- How concentrated is it? → the sink, the mass on a few tokens (a magnitude phenomenon; it’s managed, e.g., by keeping a few initial tokens as in StreamingLLM).
- How does it decay with distance? → γ (a positional phenomenon; it’s what predicts the reach and the compressibility of the KV, Ch. 20).
🧩 Analogy — volume and tuning. On an old radio, volume and tuning are two separate knobs: turning up the volume doesn’t change the station. The sink is the volume (how much attention concentrates); γ is the tuning (where, at what distance). Turning θ moves the tuning and doesn’t touch the volume. Whoever confuses them tries to turn up the volume by spinning the dial —and wonders why it isn’t louder—.
Confusing them leads to “fixing” one while believing you’re fixing the other.
Do secondary sinks appear or disappear when sweeping θ? It’s an open question that the literature itself flags (large-θ models sometimes lack them, and “the underlying cause remains an open question”, (On the Discrepancy of Secondary Attention Sinks in Large-Theta Models 2025)). We have the apparatus to study it; we flag it as pending work, not as resolved.
Four separate literatures describe suspiciously similar anomalies: the attention sinks of this chapter; the massive activations we already saw as their magnitude cause (Sun et al. 2024); the artifact tokens of ViTs —patches of very high norm that accumulate attention and global compute, the ones registers came to remedy— (Darcet et al. 2023) (Chapter 34); and the activation outliers that break naive quantization from ~6.7B parameters onward (Dettmers et al. 2022) (Chapter 36). All four rhyme: a few tokens/dimensions with enormous values that distort the rest. But each literature studies them on its own, and nobody has done the within-model experiment that measures all four at once in the same model to see whether they’re a single thing or four. The θ sweep of this very chapter (γ ⊥ sink) is the natural template for that test: one lever, one model, and measure what moves together and what doesn’t. We leave it as an open question, not a result.
tafagent reports signals tied to concentration (e.g. peak_max_share and the η-regime, which detects when behavior is sink/SWA-like) in addition to γ. You’ll see on your model that they’re distinct axes: a model can have high γ and little sink, or the other way around.
18.7 Summary
- The sink is structural (softmax sums to 1) and of magnitude: caused by massive activations; universal, forms during pre-training (Gu et al. 2025; Sun et al. 2024).
- Clean result (ours): γ ⊥ sink-mass — rescaling θ by 256× within the same model moves γ (0.75→1.0) and leaves the sink flat (~0.38) → independent mechanisms.
- Why it matters: it separates a common confusion (concentration ≠ decay); and the within-model control isolates the cause (versus the cross-model confounds of Ch. 16).
- Open and honest: secondary sinks under θ-rescale are an unresolved question. And the big one: are sinks, massive activations, ViT artifact tokens and quantization outliers four phenomena or one? Nobody has yet done the within-model test (box above).
Next (Chapter 18): we’ve talked about “normal” (dense) attention. But there’s a whole taxonomy of attention mechanisms —sparse, local, linear, GQA/MQA, MoE—, each with its cost and its when. The comparative map nobody has.
18.8 Exercises
- Independence. If concentration and decay were the same phenomenon, what would have happened to the sink mass when γ rose from 0.75 to 1.0? What actually happened?
- The control. Why does rescaling θ within the same model isolate the cause better than comparing two different models?
- Two axes. Give an example of a practical decision that depends on the sink and another that depends on γ.
- Honesty. Which question about sinks do we explicitly leave unresolved?
The within-model θ-rescale experiment (γ ⊥ sink-mass) and its data are open: Predicting How Transformers Attend (Zenodo).