- Boxes
- definitions
- Ellipses
- theorems and lemmas
- Blue border
- the statement of this result is ready to be formalized; all prerequisites are done
- Orange border
- the statement of this result is not ready to be formalized; the blueprint needs more work
- Blue background
- the proof of this result is ready to be formalized; all prerequisites are done
- Green border
- the statement of this result is formalized
- Green background
- the proof of this result is formalized
- Dark green background
- the proof of this result and all its ancestors are formalized
- Dark green border
- this is in Mathlib
On a prespecified model and task panel, an inverse-coordinate score might be associated with factual-error frequency after controlling for context length, model and protocol. This association and its calibration remain untested here.
Let \(d_{\rm rot}=2m\) be the rotary dimension of a head. In the inspected NeoX basis, plane \(k\) contains coordinates \((k,k+m)\), \(0\leq k{\lt}m\); coordinates \(d_{\rm rot},\ldots ,d_{\rm head}-1\) bypass rotation. A complete-cycle classification at \(T\) uses the actual frequencies and the condition \(T\omega _k\geq 2\pi \). It is a geometric count, not a measure of information, usefulness or semantic function. The words zero_dead and zero_alive below identify archived conditions only.
For real distance \(d{\gt}0\), call plane \(i\) cycled when \(d\omega _i\ge 2\pi \). The legacy symbols \(n_\mathrm {dead}\) and \(n_\mathrm {active}\) denote cycled and uncycled counts, respectively; they do not classify functional usefulness.
For a specified checkpoint, precision, attention implementation, distance grid and length/query rule, define
For \(\theta {\gt}1\), \(T{\gt}0\), \(N_L\) layers and head dimension \(d_{\rm head}{\gt}0\), define
This is the historical configuration proposal previously denoted \(L_{\rm crit}\). Its full-head factor also ignores partial RoPE: Pythia-1B has \(d_{\rm head}=256\) but \(d_{\rm rot}=64\). Replacing one by the other would define a new candidate, not validate the old formula. Its value need not be an integer or lie in \([0,N_L]\); rounding and any clipping must be stated before comparing it to a measured layer index.
At \(\lambda =0,\gamma =1\), \(Z_N=H_N\sim \log N\). The other cases are
Assume each upstream head output at query \(t\) depends only on input positions at or before \(t\). The displayed gate followed by a per-query output projection preserves that prefix causality. Because it uses no sequence aggregate or mutable state, partitioning the same head-output tensor into query chunks does not change the result.
A finite cosine sum of the form
is not automatically a nonnegative normalized attention distribution. In particular, the proposed ratio of logarithms cannot supply the Padé exponent:
For \(A_0(d)=c/(ad+b)\) with \(a,b,c{\gt}0\), on the nondegenerate grid of Group 1,
With the grid and weights fixed, this slope strictly decreases with \(b\). Replacing all distances by \(\lambda d_k\), \(\lambda {\gt}0\), makes the slope strictly increase with \(\lambda \); its limits as \(\lambda \downarrow 0\) and \(\lambda \to \infty \) are respectively zero and one. The amplitude \(c\) has no effect.
For integer \(N\ge 1\) and real \(g,h\), let \(\psi (g)=\log Z_N(g)\), \(U(g)={\mathbb {E}}_{p_g}\log d\), \(I(g)=\operatorname {Var}_{p_g}(\log d)\) and \(S(g)=-\sum _dp_g(d)\log p_g(d)\). All these functions are smooth to every order for finite \(g\), and
Writing \(L=\log N\),
Moreover,
For \(N\ge 2\), \(I(g){\gt}0\), and \(S\) has its unique maximum at \(g=0\).
Let \(J\ge 1\), \(a_j{\gt}0\), \(g_j\in \mathbb R\), and \(A(d)=\sum _{j=1}^{J}a_jd^{-g_j}\) for \(d{\gt}0\), with coefficients and exponents independent of distance. For \(x=\log d\), put \(\pi _j(x)=a_je^{-g_jx}/\sum _\ell a_\ell e^{-g_\ell x}\). Then
The finite-grid exponent satisfies
If two component exponents differ, the local slope strictly decreases, and the mixture cannot be an exact power on any open interval.
At distance \(d_k\), a probe averages allowed attention entries using weights \(\alpha _{kr}\ge 0\), \(\sum _r\alpha _{kr}=1\). If row \(r\) has \(n_{kr}\ge 1\) allowed keys, its uniform-attention null is
For every positive profile on the same grid and regression weights,
A constant null has zero slope. Keeping the same queries, support sizes and averaging weights at each distance is sufficient for a constant null.
Let \(N\ge 2\) be an integer and \(p_g(d)=d^{-g}/Z_N(g)\) on \(\{ 1,\ldots ,N\} \), with \(g\in \mathbb R\). For \(1\le k{\lt}N\), the prefix mass \(F_g(k)=\sum _{d\le k}p_g(d)\) is strictly increasing in \(g\). For \(0{\lt}f{\lt}1\), the well-defined integer quantile
is nonincreasing in \(g\) and, for fixed \(g,f\), nondecreasing in \(N\). These conclusions include negative exponents, zero and one.
For integer \(m\ge 1\), \(\theta {\gt}1\), \(L=\log \theta \) and real \(d{\gt}0\), define
where \(\operatorname {Ci}(x)=-\int _x^\infty \cos t/t\, dt\) for \(x{\gt}0\). Then
with Euler’s constant \(\gamma _E\). At fixed \(d,\theta \), the normalized Riemann approximation has error at most \(d(1-1/\theta )/m\), tending to zero as \(m\to \infty \).
On a finite nonempty support, let \(q=\operatorname {softmax}(s)\) and \(p=\operatorname {softmax}(s+r)\) for finite real scores. With \(\omega =\max _jr_j-\min _jr_j\),
The bound is optimal over positive distributions and errors with this oscillation. If \(|r_j|\le \eta \), then TV is at most \(\tanh (\eta /2)\). For fixed values,
Let \(p\) be a probability on a finite nonempty key set and retain a subset \(K\) with \(p(K)=1-\epsilon {\gt}0\). Set \(q_j=p_j/(1-\epsilon )\) in \(K\) and \(q_j=0\) outside. Then
If \(\epsilon {\gt}0\), then \(\operatorname {KL}(p\Vert q)=+\infty \). For fixed values \(v_j\), \(o=\sum _jp_jv_j\) and \(o_K=\sum _jq_jv_j\) satisfy
For \(\epsilon {\gt}0\), if \(o_R\) is the conditional mean over removed keys, \(o-o_K=\epsilon (o_R-o_K)\). The constants one and two are optimal in general.
For integer \(N{\gt}1\) and the discrete family \(p_\gamma \), define \(S_\gamma (N)=-\sum p_\gamma \log p_\gamma \).
For every real \(\gamma \), \(0\le S_\gamma (N)\le \log N\), with equality in the upper bound exactly at \(\gamma =0\). For \(N=1\) the entropy is zero for every exponent.
For fixed \(0{\lt}\gamma {\lt}1\), as \(N\to \infty \),
\begin{equation} S_\gamma (N)=\log N-\log (1-\gamma )-\frac{\gamma }{1-\gamma } +O(N^{\gamma -1}\log N). \end{equation}12At \(\gamma =1\), \(S_1(N)=\tfrac 12\log N+\log \log N-\gamma _E/2+o(1)\). This is a separate asymptotic regime, not substitution into the preceding fixed-exponent formula.
For fixed \(\gamma {\gt}1\), \(S_\gamma (N)\to \log \zeta (\gamma )-\gamma \zeta '(\gamma )/\zeta (\gamma ){\lt}\infty \).
For density proportional to \(d^{-g}\) on \([1,N]\), \(N{\gt}1\), and \(0{\lt}f{\lt}1\), the unique \(f\)-mass quantile is
For token distances \(\{ 1,\ldots ,N\} \) with counting measure, the corresponding exact quantile is instead \(D_{f,N}=\min \{ k:\sum _{d=1}^k d^{-g}\ge f\sum _{d=1}^N d^{-g}\} \). For the discrete case \(N\) is an integer. Both quantiles are defined for every finite real \(g\); they do not include a self key at \(d=0\) or a separately modelled attention sink.
For integer \(N{\gt}1\) and \(0{\lt}\lambda {\lt}\log N\), consider
There is a unique real exponent and a unique maximizing distribution:
The exponent is positive, zero or negative according as \(\lambda \) is below, equal to or above \(\log (N!)/N\). At the endpoints \(\lambda =0\) and \(\lambda =\log N\), the feasible distribution is the point mass at \(1\) or \(N\), respectively, obtained as an exponent limit. For \(N=1\) the distribution is unique but the exponent is unidentifiable.
Let \(N\ge 1\) be an integer and \(\gamma \in \mathbb R\). Define
Among all probability vectors \(p\) on this support satisfying \(\mathbb E_p[\log d]=\mathbb E_{p_\gamma }[\log d]\), the unique entropy maximizer is \(p_\gamma \). We use counting measure, natural logarithms and \(0\log 0=0\).