43 The global workspace: reading the model’s whiteboard
Where we are. We close Part VII with a story that comes full circle. Back in Chapter 4 we called the residual stream “the shared whiteboard” — an analogy to understand how layers communicate. In July 2026, Anthropic showed the analogy was more literal than we knew: inside that whiteboard there is a part that works as a genuine workspace — you can read it, you can edit it, and it is where the model holds its reasoning (Gurnee et al. 2026). This chapter tells how it was discovered, what it proves (and what it doesn’t), and how you can measure it yourself.
43.1 The idea in one sentence
On the big whiteboard of the residual stream there is a corner written in clean letters — a small subspace (~10 %) whose content is in “token language”, readable — and that corner is the model’s working memory: it holds the concepts of deliberate reasoning while the rest of the network scribbles in automatic shorthand.
43.2 Key concepts and their role in the transformer
- Jacobian lens (J-lens). Definition: the logit lens (Chapter 38) with “per-layer prescription glasses”: it transports the activation with the average Jacobian \(J_\ell = \mathbb{E}[\partial h_{final}/\partial h_\ell]\) before projecting onto the vocabulary. In the transformer: it reads content at intermediate layers where the logit lens only sees noise.
- Workspace (J-space). Definition: the part of the residual stream the J-lens can read — sparse combinations of “one-token” vectors. In the transformer: the clean corner of the whiteboard: ~dozens of active concepts at a time, working memory (not a store of facts).
- Depth band. Definition: the stretch of layers where that space is populated (in the Claude models, ~38 %–92 %). In the transformer: before it, content is not yet readable; after it, everything is already committed to the output.
- Automatic/deliberate dissociation. Definition: erasing the workspace crushes flexible reasoning and leaves automatic tasks untouched. In the transformer: the causal proof that it is a system, not an artifact of the lens.
- Ignition. Definition: an ambiguous mixture of two concepts snaps abruptly to one of them right at the band’s entrance. In the transformer: that is how the model resolves ambiguity — with a threshold, not gradually.
- Externalization. Definition: writing the steps out (the chain-of-thought of
- offloads the whiteboard onto paper. In the transformer: that is why CoT survives when the whiteboard is erased: what is written down no longer needs to be held.
The whole journey of this chapter, at a glance:
43.3 The whiteboard had a corner in clean letters
Let’s start with the question that triggers everything. If the residual stream is a whiteboard where layers take notes — could we read it ourselves?
You already know the first attempt from Chapter 38: the logit lens projects any layer’s activation onto the vocabulary — “what would the model say if it stopped here?”. It works… but only near the end. At early and intermediate layers it returns noise.
Now think: why does it fail there? Is there really nothing written across half the network — or are we reading with the wrong glasses?
The answer turned out to be the latter. Each layer writes on the whiteboard in its own “coordinate system”, and the logit lens reads all of them with the last layer’s coordinates. The fix is conceptually simple: measure, averaging over ~1,000 texts, how on average what is written at layer ℓ translates into the final layer’s language — that is exactly an average Jacobian — and translate before reading:
\[\text{lens}_\ell(h) = \text{unembed}\big(\,\text{norm}(J_\ell\, h)\,\big)\]
🧩 Analogy — prescription glasses for each layer. The logit lens tries to read every line of the whiteboard with the same glasses (the final line’s). The Jacobian lens is fitted for each line: it measures how much the path from that line to the end “defocuses”, and compensates before reading. With the right glasses, lines that looked like scribbles turn out to be full of words.
One important warning before we continue: this lens does not predict the next token (the model’s own output already does that). It reads which concepts the model has on the table — the ingredients, not the served dish. That difference is the key to the whole chapter.
And a clarification worth keeping in mind as you read everything that follows: since the residual stream is additive, \(J_\ell = I + \Delta_\ell\) — the J-lens is the logit lens plus an average linear correction. Its “advantage” therefore measures how much that correction helps; that this advantage coincides with a functional working memory is something the causal interventions below (editing, erasing) sustain, not the lens’s advantage on its own.
43.4 Watch it work
How many tokens before saying “Paris” do you think the model has already settled on it? Watch — then switch to “logit lens” to understand why nobody had seen this before:
What you just watched is the paper’s central finding: “Paris” lives on the whiteboard positions before being said, is held there while the sentence advances, and collapses toward the output at the end. The model doesn’t improvise token by token as much as it seems: it plans on the whiteboard.
43.5 And how do we know it isn’t a mirage?
Good question — it is exactly the one Chapter 39 demands. A lens can “see” patterns that do nothing (correlation). What evidence would turn the whiteboard into a causal fact? Think about what you would ask for… The paper brings three pieces, each stronger than the last:
- It can be edited. Ask the model to “think of a sport”: the J-lens shows Soccer before it answers. Now subtract the Soccer vector and add Rugby — the model answers “Rugby” (~88 % of the time). We are not reading an echo; we are editing what the model thinks.
- It mediates reasoning. In “the number of legs of the animal that weaves webs”, the word spider is neither in the question nor in the answer (“8”) — but it is on the whiteboard. Swap it for ant and the model answers “6” (~70 %). The invisible intermediate step of reasoning lives there, and manipulating it redirects the conclusion.
- Double dissociation. The decisive experiment: erase the ~10 most active directions of the whiteboard. Automatic tasks (classifying, parsing, MMLU) barely flinch; flexible reasoning (multi-hop, mental arithmetic, trivia) collapses. Same model, same erasure, two fates — the classic signature neuroscience uses to tell real systems from correlations.
The double dissociation is best understood by doing it. Erase the whiteboard yourself and watch which family of tasks falls and which doesn’t even notice:
(Illustrative bars showing the qualitative pattern — the exact per-task, per-model numbers are in the paper (Gurnee et al. 2026), Fig. 24.)
And a fourth piece with a practical moral: chain-of-thought survives the erasure. Why? Answer before reading on… Because writing the steps externalizes working memory: the paper acts as the whiteboard. Keep this idea — we will come back to it.
- Nothing about consciousness: the paper studies the functional analogue of “conscious access”; the authors take no position on subjective experience — the press headlines do. (2) The lens only sees concepts that are one token: a bias toward what is nameable in a single word. (3) The “broadcast” happens layer by layer, not through recurrence as in the brain. (4) Nanda’s review (Nanda 2026) classifies it as a hypothesis-generation tool: expect false positives. Anything a lens suggests needs causal confirmation — the rule of Chapter 39 never sleeps.
43.6 Where is the whiteboard — and why there?
The whiteboard does not fill the whole network: in the Claude models it lives in a band between ~38 % and ~92 % of the depth. Before the band, nothing readable; after it, everything is already aligned with the imminent output.
And something with a physics flavor happens at the band’s entrance: ignition. If you inject an ambiguous mixture of two concepts (60 % France + 40 % China), early layers treat it as what it is — a mixture. But the band does not tolerate ambiguity: it snaps to one or the other when a threshold is crossed. Try it:
Here Anthropic stops — they measure where, they don’t explain why. We measured it too, on the Pythia ladder with a pre-registered protocol (predictions frozen before measuring). What the data show:
- The readability knee — the layer where the logit lens starts to read, i.e. where content aligns with the output — sits at a practically constant depth fraction, ≈ 0.69, in all three measured models (0.67 / 0.71 / 0.69), even though they differ 16× in size and 4× in head dimension.
- pythia-410m additionally shows a clear advantage band (layers 8–13, fraction 0.33–0.54) where the J-lens reads future content 3–6× better than the logit lens. In pythia-1b that band does not appear — which is also a data point.
And you don’t have to take our word for it: these are the measured curves, model by model. Pick one and compare the two lenses (the violet shade is its advantage band; the ring, the knee where the logit lens wakes up) — note how different the six stories are:
(Real data from the M1′ protocol — 100 held-out sequences, median word-like bestRR; full CSVs and lenses in the research repository.)
Why the band sits where it sits remains an open question. We had a candidate: the critical layer \(L_{crit}\) from our own work (Marín 2026b) — the depth where RoPE’s positional scaffolding runs out (the three scales of Chapter 15), computable from the hyperparameters \(\theta\), \(T\) and \(d_{head}\). We put it to the test with predictions frozen before measuring, and it failed on the model that discriminated. The numbers:
We tested H1 on the Pythia ladder with a pre-registered protocol. Result:
- In 70m and 410m (both with \(L_{crit}/N = 0.63\)), the logit lens knee fell on the prediction within ≤2 layers — but there \(L_{crit}\) coincides with the typical fraction, so it does not discriminate.
- The discriminating model was pythia-1b (\(d_{head}=256\) → \(L_{crit}/N = 0.94\)): its knee came out at L11/16 = 0.69, in the null-hypothesis range (fixed fraction), not in H1’s. Its J-lens advantage band also does not exist (the J-lens never doubles the logit lens at any layer).
- The first three models suggested a constant ≈ 0.69 — but the full ladder disproved it: 0.667 / 0.750 / 0.708 / 0.688 / 0.542 / 0.656 across the six Pythias, and 0.81–0.84 on two Llamas. The knee turns out to be a family property with internal scatter, neither a universal constant nor a clean function of hyperparameters.
The failure localizes to the \((d_{head}/64)^{1/\sqrt{12}}\) factor, which had been validated with n=2. One lead remains open: the fraction without that factor, \(\log_\theta(T/2\pi) = 0.628\), predicts all three knees within ≤2 layers — but on Pythia (identical θ and T across the ladder) it is indistinguishable from a universal constant. The test that separates them (a model with a different θ, e.g. Llama-3-8B → prediction 0.55 vs ~0.69) is registered as H1b and pending execution. Data, figure and pre-registration: research repo.
43.7 And what is the band? The operator, not the content
We’ve just seen where the band lives, and that no hyperparameter predicts why it falls there. Before giving up on the “why,” let’s ask something more basic —and more uncomfortable—: when the J-lens reads a layer well, is that the layer that writes the answer? It sounds like it should be: if the lens sees “Paris” there, “Paris” must be stored there. The measured answer is no. And grasping that changes what the band means.
Let’s run the experiment the paper (Marín 2026a) does with numbers. At each layer we measure two different things:
- Transport (violet): how much the rest of the network, downstream, depends on what’s in this layer. It’s exactly the J-lens advantage —if the lens reads future content here, it’s because this line transports cleanly to the end.
- Writing / injection (amber): how much this layer, itself, stamps onto the answer direction. It’s direct logit attribution (DLA) — we unpack it in the next section.
Look at them layer by layer —below, in 3D— and watch the two envelopes. They don’t go together: they avoid each other. Where transport swells (in the band, mid-network), writing barely exists; where writing rules (at the top, near the output), transport dies down.
This isn’t a fluke of these models. It’s forced, and the reason fits in one line. Because the residual stream only adds (\(h_{\ell+1}=h_\ell+w_\ell\)), the Jacobian the J-lens uses is not a property of layer ℓ: it’s the product of everything that happens after it —
\[J_\ell=\prod_{k\ge\ell}(I+w_k').\]
Read it slowly, because here’s the key to the chapter. The lens reading layer ℓ well means its content travels easily through all the following layers — not that anything gets decided there. That’s why the widget’s violet beam climbs up the future of the layer you pick: that’s what the lens averages. The band marks on-ramps, not printing presses.
🧩 Analogy — the on-ramp and the printing press. A highway on-ramp carries enormous traffic: everything headed to the city passes through it. But nothing is made on the ramp. The printing press, at the end of the journey, stamps the page — and hardly any traffic flows through it. Lots of throughput ≠ where it’s written. The J-lens, by averaging the Jacobian, lights up the busiest on-ramps; it’s tempting to mistake them for the workshop, but they’re opposite things — which is why transport and writing come out anticorrelated (ρ = −0.80).
And here’s the chapter’s honest moral, the one that sharpens —without toppling— everything above: a J-lens advantage, on its own, does not prove there’s a workspace holding stored content. In large part it’s the average operator supplying a generic transport the raw residual lacks. How do we know? Because that readout saturates with ~20 texts of fitting, and a small fixed set of targets already beats the best single-text readout: if it were prompt-specific content, it couldn’t be averaged like that — it’s denoising, not stored memory.
Mind what this doesn’t say. It doesn’t undo Anthropic’s causal proofs on the Claude models: editing, erasing and the double dissociation (above) remain causal. It respects them and puts them in their place — the lens points at where to look; only intervention proves there’s content there. It’s, again, the rule of Chapter 39, now with a theorem behind why a lens can mislead.
Theorem (transport factorization). For \(h_{\ell+1}=h_\ell+w_\ell(h_\ell)\), the input-output Jacobian factorizes as \(J_\ell=\prod_{k\ge\ell}(I+w_k')\), so that \(J_\ell-I=\sum_{k\ge\ell}w_k'+(\text{cross terms})\). The Jacobian at ℓ is a property of the future of ℓ, not of ℓ. (Stated here; machine-checked in Lean 4 / Mathlib in the formalization paper.)
The measurement. \(\mathrm{Spearman}(A_\ell,\ \mu_\ell)=-0.80\) on pythia-410m (\(-0.85\) for gross circulation), and it replicates: \(-0.86\) on pythia-2.8b, \(-0.79\) on qwen3-1.7b. The mean-Jacobian estimator saturates at \(N\approx20\) texts and a small fixed target rank (\(K^\star=5\)–\(20\)) already beats the per-prompt oracle.
Honest scope: one model per family, correlational. No swap/ablation here — the advantage is a hypothesis, not a proof (Chapter 39).
43.8 Three clocks, not one
In the previous section we separated transport (what the lens reads) from writing (what the layer injects). Those are two of three clocks that, seen carelessly, look like the same one. The naive question “is the answer here yet?” actually merges three distinct questions, each answered at a different depth:
- Can it be read? — the residual is already lens-decodable. Marks the clock \(\tau_\mathrm{decode}\) (transport onset).
- Has it been written? — some layer has injected it into the answer direction (DLA). Marks \(\tau_\mathrm{write}\).
- Has it locked? — the running decision no longer changes. Marks \(\tau_\mathrm{commit}\).
(And a fourth, in between: the logit lens knee — when content aligns with the output.) The measured surprise: they don’t coincide. On pythia-410m they run in a row —
\[\underbrace{\tau_\mathrm{decode}=0.17}_{\text{can be read}} \;<\; \underbrace{\text{band }0.25\text{–}0.54}_{\text{transport advantage}} \;<\; \underbrace{\text{knee}\approx\tau_\mathrm{write}=0.66}_{\text{gets written}} \;<\; \underbrace{\tau_\mathrm{commit}=0.96}_{\text{locks}}\]
— they’re not three readings of the same instant: they’re a front sweeping across depth. The residual becomes readable long before it’s written, and it’s written long before the decision closes. Watch it — and switch families to discover that the order is universal but the spacing is not:
43.8.1 The answer is, literally, a sum
Why can we talk about “what each layer writes” so precisely? Because there’s an exact identity, not an approximation. If you freeze the final normalization at the value taken by the full residual, the unembedding becomes affine and the output logit is the sum of what each layer contributed:
\[\boxed{\ \mathrm{logit}(t)\;=\;\beta_t\;+\;\sum_\ell U(\Delta h_\ell)\cdot e_t\ }\]
No trick and no remainder: validated to \(7.6\times10^{-6}\). It’s norm-aware — LayerNorm (Pythia, GPT-2) centers and scales; RMSNorm (Qwen, Llama) only scales— and you must use the right \(U\) in each case. This exact sum is what we call injection (DLA): the slice of the logit that layer ℓ put there.
And writing has a sign. Since it’s a sum, one layer can add evidence… and another subtract it. On pythia-410m, ~7 of 24 layers write against the correct answer before another fixes it. So “the answer is written at layer ℓ” is not well-posed without saying with what sign — one more way the naive reading conflates distinct events.
43.8.2 Two families, two clocks
If the order of the three clocks is universal, is their spacing too? No — and that’s the lovely part. A single number, \(\Delta_\mathrm{prep}=\tau_\mathrm{write}-\tau_\mathrm{decode}\) (how long content is readable before it’s written), separates families with no overlap:
- Type A (early decode, \(\Delta_\mathrm{prep}\ge0.5\)): Pythia, Qwen2.5 — they read early and prepare long.
- Type B (late decode, \(\Delta_\mathrm{prep}\le0.3\)): Qwen3, GPT-2 — they read late; in GPT-2 the order even inverts (it writes before it’s readable).
The third clock, \(\tau_\mathrm{commit}\), is architectural, not a law: it always falls in the last 1–3 layers because the final layer dominates the write to the vocabulary (in pythia-160m it carries 93%). We don’t treat it as a deep invariant.
The conceptual moral (🟡 few models per family). There is no universal chronology, but there are universal observables. As in a dynamical system: the state variables are the same for everyone (transport, injection, commitment), but the trajectories —when each thing happens— are each family’s own (Marín 2026a). Interpretability gains a set of thermometers that distinguish how each family spreads computation across depth.
43.9 And what places the band there?
Knowing now that the band is a property of the transport operator —not a store of content—, the “why does it fall there?” question sharpens: what places that transport advantage in depth? The positional \(L_{crit}\) failed; let’s try the book’s other thermometers, the attention ones: the decay exponent γ (Chapter 17) and the dictionary \(T=1/\gamma\), \(\chi=1/|\gamma-1|\) of Chapter 23. The intuition is pretty: subcritical models (γ≪1, delocalized attention from early layers) should have transportable content early → early bands. And with one γ protocol, it fits.
Here comes an honest correction, the kind this book doesn’t hide. We measure γ two ways and they don’t agree. The Part I atlas γ (Chapter 17) and the “depth-batch” γ disagree by up to 0.25 on the same models (qwen2.5-3b: 0.77 vs 0.94; pythia-410m: 1.02 vs 0.94) — and under the second protocol the sign of the correlation flips. That is: raw γ does not discriminate where the band falls; what looked like a trend was an artifact of how we measure γ. We leave it as inconclusive, not a result (Marín 2026a).
What does survive —only as a candidate— is subtler: not raw γ, but its trained displacement, \[\delta=\gamma_\mathrm{obs}-\gamma_\mathrm{geom},\] how far training moved attention from RoPE’s pure geometry (Part II’s learned-imprint axis). With δ the pattern lines up: δ<0 —training delocalized attention beyond geometry— goes with early bands (4/4); δ≈0 with absence of a band; and the only δ≫0 with the only mid band. The band’s strength, by contrast, doesn’t follow δ but the model’s size. The case that decides it: two models with nearly identical raw γ (0.748 vs 0.772) sit at opposite extremes, and δ separates them (\(+0.001\) vs \(-0.183\)).
It fits too well to keep quiet about, and too little to believe: it’s n=7 (the models with γ and a band measured at once), the cross is confounded with size and with \(d_{head}\) (pythia-1b, no band, is also the \(d_{head}=256\) one), and it is not frozen as a prediction. In our registry it’s hypothesis H16, in progress: raw γ came out inconclusive, and δ is the live lead. The clean test is to measure δ of the new models blind to their bands and check the rule out of sample.
43.10 Measure your own whiteboard
The code is public and works with open HuggingFace models (github.com/anthropics/jacobian-lens, Apache-2.0). On a 16 GB consumer GPU: minutes for a 70M model, hours for a 2.8B one (the cost is fitting the lens — one forward + \(d_{model}/b\) backwards per text):
import transformers, jlens
hf = transformers.AutoModelForCausalLM.from_pretrained("EleutherAI/pythia-410m").cuda()
tok = transformers.AutoTokenizer.from_pretrained("EleutherAI/pythia-410m")
model = jlens.from_hf(hf, tok)
lens = jlens.fit(model, prompts=pile_texts) # ~100-1000 texts of 128 tokens
lens_logits, model_logits, _ = lens.apply(model, "The capital of France is the city of",
positions=[-1])One methodological tip that cost us a day to learn: always compare against the logit lens. The real band is where the J-lens reads future word-like content at least 2× better than its control — without that contrast, any lens will show you mirages. The already-fitted lenses for the Pythia ladder will be available for direct download, so that locating the band costs you not a single backward pass.
43.11 And what is knowing where it is good for?
The chapter ends the way it began: with a question. If you knew exactly which part of a small model thinks and which part just types — what would you do differently? Four answers, from most mature to most speculative:
- Read it (today): monitor which concepts are loaded — debugging and auditing. In the paper, the whiteboard contained leverage, panic or fake that never surfaced in the output.
- Write it (experimental): hold a concept by clamping its vector — so a small model doesn’t drop the needle in a long context.
- Protect it (open hypothesis): asymmetric compression — quantize and prune the 90 % automatic part aggressively (Chapter 36) and protect the corner that reasons.
- Size it (out of reach today): placing the band at will would require knowing what places it — and our candidate (\(L_{crit}\)) was refuted when measured. Without a verified theory of the band’s position, this is an aspiration, not a lever.
And the cheapest lever of all you already knew without knowing it: if a small model’s whiteboard is small, don’t ask it to think silently — give it paper (1). Chain-of-thought is not a prompt trick: it is a working-memory prosthesis.
🧩 Analogy — the operating room with X-rays. Without the lens, optimizing a small model is blind surgery: you cut where you can and pray. With the whiteboard located, you know which part of the patient thinks and which part just types — and compressing, distilling or shoring up can finally respect the difference.
43.12 Summary
- The “shared whiteboard” of Chapter 4 has a corner in clean letters: a subspace (~10 %) readable in token language — the model’s workspace.
- It is read with the Jacobian lens (logit lens + per-layer prescription glasses); it reads concepts on the table, not the next token.
- Causal evidence in three steps: editable (concept swaps), mediates reasoning (the invisible intermediate), double dissociation (erasing the whiteboard crushes the flexible and spares the automatic).
- It lives in a depth band; why there remains open. In our measurement (8 models, 2 families), the readability knee is a family property (Pythia 0.54–0.75, Llama 0.81–0.84); the candidate \(L_{crit}\) was refuted and no hyperparameter-based predictor survived.
- What the band is (Section 43.7): a property of the transport operator (\(J_\ell=\prod_{k\ge\ell}(I+w_k')\), the Jacobian looks at the layer’s future), not a store of content — which is why transport and injection are anticorrelated (\(-0.80\)) and a lens advantage doesn’t prove a workspace (it sharpens, without toppling, Anthropic’s causal result).
- What places it (Section 43.9): raw γ came out inconclusive (two measurement protocols and the sign flips); the live lead is the trained displacement \(\delta=\gamma_\mathrm{obs}-\gamma_\mathrm{geom}\) (H16, 🟡 n=7, not frozen).
- Externalizing (CoT) is giving paper to a small whiteboard — the immediate practical lever for modest models.
- Golden rule (Chapter 39): the lens generates hypotheses; only causal intervention confirms them.
tafagent has a 🪜 Depth map mode that reads this chapter’s depth axis straight from config.json: the two independent geometric axes (distance γ ⟂ depth L_crit), the layer where the attention regime changes (the \(L_{crit}\) formula of Exercise 5), the three RoPE scales, and — for atlas models — the δ band-position lead of Section 43.9 (H16). And it says so honestly: L_crit is not the commitment knee, and transport / writing / commitment are measured, not derived from config.
Next (Part VIII): the reference tools — formula sheet, cookbook, and how to measure every quantity in this book yourself. Now including the whiteboard.
43.13 Exercises
- The glasses. Why does the logit lens fail at intermediate layers, and what exactly does the average Jacobian correct? Why is it fitted averaging over a corpus?
- The invisible intermediate. In “the number of legs of the animal that weaves webs is”, what is on the whiteboard that is neither in the question nor in the answer? How is it shown to be causal rather than decorative?
- Double dissociation. Which two families of tasks does erasing the whiteboard separate? Why is that separation better evidence than any correlation?
- Externalization. Why does chain-of-thought survive the erasure of the whiteboard, and what practical recipe follows for small models?
- The test that failed. The formula \(L_{crit} = N_L\log_\theta(T/2\pi)(d_{head}/64)^{1/\sqrt{12}}\) predicted pythia-1b’s boundary (\(N_L=16\), \(\theta=10^4\), \(T=2048\), \(d_{head}=256\)) at layer 15; the measured knee came out at layer 11 (fraction 0.69, as in the other two models). Which factor of the formula do the data point to as responsible for the failure? And why can’t the Pythia ladder distinguish a “universal constant fraction” from plain \(\log_\theta(T/2\pi)\)?
- Skepticism. Give two reasons why a J-lens readout could be a false positive, and design the experiment that would rule it out.