Blueprint
How Transformers Attend — Part I📱 5 short screens

📱 The paper in five short screens, for a phone or a first look: the result, what it says in words, how it is proved, what has been checked, and where to try it. Nothing here loads Python.

🎯 The result 1 / 5

.

At distance \(d_k\), a probe averages allowed attention entries using weights \(\alpha _{kr}\ge 0\), \(\sum _r\alpha _{kr}=1\). If row \(r\) has \(n_{kr}\ge 1\) allowed keys, its uniform-attention null is

\[ A_{0,k}=\sum _r\frac{\alpha _{kr}}{n_{kr}},\qquad G_w(A_0)=-\frac{\operatorname {Cov}_w(x,\log A_0)}{V_x}. \]

For every positive profile on the same grid and regression weights,

\begin{equation} G_w(A/A_0)=G_w(A)-G_w(A_0). \end{equation}
98

A constant null has zero slope. Keeping the same queries, support sizes and averaging weights at each distance is sufficient for a constant null.

💬 In plain words 2 / 5

La pendiente medida depende de la sonda y de los pesos de la regresión. Una fila uniforme puede producir un perfil aparente; restar la pendiente de un nulo construido con el mismo instrumento cancela esa contribución de forma algebraica. La prueba no describe cómo se comporta un modelo entrenado.

🛠️ How it is proved 3 / 5

La Parte I reúne cinco teoremas y dos proposiciones del cuerpo, ocho teoremas de calibración y cotas en el apéndice K, seis definiciones y una conjetura empírica. Las pruebas Isabelle cubren ocho grupos del apéndice; Lean verifica partes de cuatro resultados del cuerpo. Ninguna de esas pruebas establece cómo se entrena un modelo.

It rests on:

🛡️ What is checked 4 / 5

🟢 3 formal · 🟡 9 formal in part · 🔴 3 by hand · ⚪ 7 cited / no proof here.

Cite: Definition 7.3, p. 33, doi:10.5281/zenodo.24379538

🔎 What a referee has to check by hand →

🧪 Try it 5 / 5

Labs in your browser (they load Python the first time):

📕 The paper (PDF)📘 The full blueprint

📎 Cite: doi:10.5281/zenodo.24379538