47  R4 · Measuring your own γ (and reproducing the atlas)

What it is. A laboratory turning real attention into a reproducible decay measurement. First decide what is being measured, then fit the curve. The book’s promise remains: give you a way to check a claim rather than ask you to believe it.

After repeating the measurement, report a protocol mismatch, differing curves or apparently stable repeats. Record it in resolver follow-up to adapt the next check. This is your declaration; it does not modify the initial analysis.

47.1 The idea in one sentence

γ needs a surname: model, layer, head, texts, positions and fitting rule. Two numbers called γ may answer different questions when their protocols differ.

🧩 Analogy — measuring rain. A garden rain gauge, a city average and a storm maximum all measure rain, but are not interchangeable. Likewise, the last attention query and an average over all queries observe similar distances from different situations.

47.2 The procedure, step by step

  1. Freeze the measurement conditions. Model identifier and revision, tokenizer, template, corpus and seed; effective length, precision, engine and RoPE settings. Use independent, representative texts. Random tokens are a separate control.
  2. Collect real attention per layer and head. Run a backend exposing attention probabilities. output_attentions=True does not guarantee that every kernel returns them. Record the actual backend and version; disable dropout and document the cost of materializing large matrices.
  3. Declare query positions. For instance, only the final position of each fixed-length text. Averaging every row is a different protocol: it mixes different normalizers and available contexts. Do not interchange the results.
  4. Separate sink, recency and tail. Declare how many initial tokens are excluded and the distance range. Measure sink mass separately. Do not silently renormalize rows after exclusion: that changes each query’s relative contribution.
  5. Average by distance, then fit. First take the arithmetic mean of valid weights at each distance; then take logs and fit a line. Fitting all individual pairs in log-log space is another estimator: it overweights distances with more pairs and averages logs.
  6. Deliver the curve and uncertainty. γ, log-log R², residuals, pair counts per distance and fit range. Repeat per head and layer before summarizing the model.

47.3 The formula, with every choice visible

Imagine pouring measurements taken at the same distance into one glass. Its average level is one point; the line summarizes how those levels change with distance:

\[\bar A(d)=\frac{1}{|\mathcal P_d|}\sum_{(b,i,j)\in\mathcal P_d} A^{(b)}_{ij}, \qquad \log\bar A(d)=c-\gamma\log d+r_d.\]

  • \(b\) identifies a text; \(i,j\), query and key. Fix one head and one layer.
  • \(\mathcal P_d\) contains only protocol-admitted pairs with \(i-j=d\); \(|\mathcal P_d|\) counts the observations entering that mean.
  • \(c\) sets the line’s height; \(\gamma\) is its sign-reversed slope. \(r_d\) is the residual: what the line misses at that distance.

Worked example. Two samples contribute 0.04 and 0.16 at one distance. Their mean is 0.10. Averaging their logs instead corresponds to \(\sqrt{0.04\times0.16}=0.08\). Changing the order of averaging and taking logs has already changed the data before fitting γ.

47.4 Minimal code that really averages by distance

This example receives already extracted probabilities from one head and layer, shaped [samples, query, key]. It requires equal-length texts without padding and dense causal attention. It does not load weights or run inference. For another mask or varying lengths, adapt and version the protocol rather than padding with zeros.

Download the complete estimator.

import numpy as np
from gamma_measurement import measure_gamma

# attention.npy: real probabilities, one layer/head, without dropout or padding.
A = np.load("attention.npy")
n = A.shape[-1]
result = measure_gamma(
    A, queries=[n - 1], sink_tokens=4,
    d_min=16, d_max=min(512, n - 5),
)
print(result["gamma"], result["r2_log"])

The numbers 4, 16 and 512 are illustrative choices, not universal constants: they need a sufficiently long sequence and a range specified before inspecting the fit. The estimator rejects ranges with too few distances and zero means; it does not add an artificial ε to conceal them. Its central operation is:

# q: declared query positions; A: samples × query × key.
for d in range(d_min, d_max + 1):
    valid_q = q[q - d >= sink_tokens]
    if valid_q.size:
        values = A[:, valid_q, valid_q - d]
        distances.append(d)
        means.append(values.mean())  # arithmetic mean BEFORE taking logs
        counts.append(values.size)

47.5 Controls that make the result interpretable

R² measures fit, not probabilistic confidence. A high R² says a line describes this curve well on these axes and this interval; it neither proves a universal law, quantifies γ uncertainty nor guarantees answer quality. A nearly flat curve can make R² undefined; the code returns None in that case.

Resample complete texts with replacement and repeat the pipeline to build a bootstrap interval; do not treat token pairs as independent observations. When several chunks come from one document, resample documents. One sample cannot estimate between-text uncertainty. Nor does the interval automatically include corpus or protocol bias.

Inspect residuals and repeat with declared neighboring ranges. Compare an exponential and a cutoff power law over the same distances and error scale; if adding parameters, use held-out distances or texts and disclose selection. A high R² alone does not decide which best describes the tail. Publish curves that do not support a single exponent too.

Table 47.1: Minimum receipt for a γ measurement.
Deliverable Purpose
Model/revision, corpus, seed, tokenizer and backend Repeat the execution
Layer, head, queries, mask, length and exclusions Identify the measured distribution
Curve, pair counts, range and residuals Detect mixtures, gaps and shape failures
Per-head/layer γ, interval and resampling unit Separate variation from an aggregate summary
Protocol identifier and code Compare compatible measurements

From a notice to the next check. In Solve my problem → Attention, paste one or two reports containing γ and open Show protocol details. The panel lists 19 known conditions in four groups: model/data, sampling, execution and aggregation/fit. It identifies missing documentation and differences from the reference. Custom fields count, but their names and values are not exported.

What to check first prioritizes context conflicts or limitations, then protocol differences and missing documentation. If all checked conditions are documented and match, it proposes inspecting the curve and repeating the run. Recover settings from the actual experiment; never invent them to make reports match. A documented field does not establish accuracy or validate the theory. The optional route demo uses synthetic uniform attention to illustrate missing conditions; no model is run. The guide lets you return to the same form.

47.6 Compare with the geometric prediction

Compute γ_Padé at the same length and effective RoPE settings. Call the difference \(\delta=\gamma_{\rm observed}-\gamma_{\rm geometric}\) a description of the residual, not an automatic causal attribution to training. Rule out corpus, position, mask and fit differences before interpreting δ. The decomposition in Chapter 16 proposes an explanation; a common protocol makes it testable. Uncertainty in γ does not turn prediction error into certainty.

47.7 Open data to reproduce (or refute)

  • The γ atlas (79 measurements across 42 unique models from 4 families) and the within-model experiments (θ-rescale, γ⊥sink) are published as open data alongside Paper I (Zenodo 20314038) — downloadable, reproducible, criticizable.
  • The decay measurements are also published as the TAF Attention-Decay Dataset (CC-BY-4.0) on Hugging Face, under the identifier karlexmarin/taf-attention-decay; if the identifier has changed, search for it by name in the Hub’s datasets search.
  • The tafagent model panel (23 models: config and predictions, not attention measurements) and the tool’s formal apparatus are open at github.com/karlesmarin/tafagent-registry and github.com/karlesmarin/lean-taf (formal proofs).
Warning⚠ What this laboratory checks

A fit and its controls assess \(A(d)\propto d^{-\gamma}\) under a specified protocol. They do not by themselves validate D_f, output quality, context extension or a causal mechanism. For those steps, return to the design in Section 41.9.

Note🧪 Try it — tafagent

From an identifier or config.json, tafagent computes configuration-based predictions and can display previously registered measurements; the browser does not execute the model to measure attention. Your own measurements require external inference, for example the project’s diagnostic CLI, followed by viewing/importing supported output. The CLI and this example may aggregate differently: check their protocols before comparing. R5 explains the tool.

47.8 Summary

  • Declare the estimator: averaging weights and averaging their logs are different operations.
  • Keep the data’s layers: head, layer, query, sink, recency and tail.
  • R² is neither an interval nor a quality guarantee. Publish residuals, uncertainty and protocol.
  • Prediction and measurement have different provenance: a config.json contains no executed attention.

47.9 Exercises

  1. Two means. Verify the 0.04 and 0.16 example. Why need fitting individual pairs not reproduce the slope of the mean curve?
  2. Two queries. Measure the last row and all rows of the same matrices. Which part of the protocol changed, despite using the identical model?
  3. Experimental unit. You have ten documents, each split into twenty chunks. What do you resample to avoid pretending you have two hundred independent documents?
  4. A perfect R². Construct a synthetic \(d^{-0.8}\) tail. What does recovering it with the code validate, and what does it not establish about a real transformer?

Solutions: ?sec-soluciones-r4. Next reference (R5): the tafagent manual.