41 Evaluation: measuring without fooling yourself
Ready to move from reading to a test? Choose a task and open TAF with a guided route.
Where we are. We have trained (Part IV), used (Part V), made cheaper (Part VI) and audited from the inside (Chs. 37-39) a transformer. What’s left is the question that decides everything else: how do we know a model is good? Evaluation is as hard —and as easy to get wrong— as training. And it belongs here, in Part VII, because it shares its backbone: almost everything you measure is an imperfect proxy, and the biggest danger is not measuring too little, but measuring wrong and believing it.
Control formatting before interpreting a score. The template resolver distinguishes missing templates in the inspected files, several choices without an unambiguous default, and a present template. See the verification steps asks you to inspect the selection and rendered conversation while holding model, prompt and generation settings fixed. Finding a template does not establish that the server applies it correctly or explain an incorrect answer by itself. Three optional examples show these states without executing Jinja or inference.
After repeating, report differing controls, a gap that persists after alignment or a gap that disappeared. Record it in resolver follow-up to adapt the next check. This is your declaration; it does not modify the initial analysis.
41.1 The idea in one sentence
Evaluating an LLM means estimating its capability with metrics that are proxies —none measures it directly— so knowing how to evaluate is, above all, knowing the traps: data contamination, the judge that gets fooled, and Goodhart’s law.
41.2 Key concepts and their role in the transformer
Before the details, we define this chapter’s terms and what each is for:
- Perplexity. Definition: \(e^{\text{loss}}\); “how many tokens the model is hesitating between” (Ch. 11). In the transformer: the cheapest internal metric —but only comparable with the same tokenizer, and it doesn’t tell you whether the model is useful—.
- Benchmark. Definition: a fixed set of tasks with known answers. In the transformer: the standard way to put a number on a capability (knowledge, code, math, long context).
- LLM-as-judge. Definition: using another LLM to score open-ended answers. In the transformer: it scales human evaluation, but inherits biases (position, verbosity, self-preference).
- Data contamination. Definition: the test leaked into the training data. In the transformer: it inflates the score without the model being any better —the number-one threat to validity—.
- Goodhart’s law. Definition: “when a measure becomes a target, it stops being a good measure.” In the transformer: optimizing for a benchmark degrades what the benchmark was meant to measure.
- Construct validity. Definition: whether the metric really measures the capability you claim, or just a correlated shortcut. In the transformer: the underlying question of the whole chapter.
- Statistical rigor. Definition: error bars, several seeds, significance. In the transformer: without them, a “+2%” can be noise.
41.3 Why a whole chapter
We already saw text metrics in Ch. 29 (BLEU, ROUGE, the LLM judge). Here we go up a level: evaluating the whole model. And we put it in Part VII on purpose, because evaluation is where the book’s honesty is really tested: a high number does not prove capability. The rest of the chapter is a tour of the metrics and of the ways each one can fool you.
41.4 Perplexity: the internal metric (and its limits)
The cheapest metric is the one you already have from training (Ch. 11): perplexity, \(\text{PPL}=e^{\text{loss}}\). It measures the model’s average surprise on a text: a PPL of 8 means it hesitates, on average, “as if among 8 tokens”. Lower = predicts better.
PPL depends on how you split the text: a model with a large vocabulary cuts the same text into fewer tokens and comes out with a PPL that is not comparable to another model’s. That’s why, to really compare, we use bits-per-byte (bpb), which normalizes by bytes of text, not tokens. And even so: a low PPL doesn’t say the model is useful —only that it predicts the next token well—.
41.5 Capability benchmarks: putting a number on what it can do
To measure specific capabilities we use benchmarks: batteries of tasks with known answers. The big axes of 2024-2026:
| Capability | Reference benchmarks | What they measure |
|---|---|---|
| Knowledge / reasoning | MMLU (Hendrycks et al. 2021), MMLU-Pro, GPQA, BBH | exam questions across 57 subjects; GPQA at PhD level |
| Code | HumanEval (Chen et al. 2021), MBPP, SWE-bench (Jimenez et al. 2024) | writing functions that pass tests; SWE-bench, solving real GitHub issues |
| Math | GSM8K (Cobbe et al. 2021), MATH, AIME | word problems (GSM8K) and competition math (MATH/AIME) |
| Long context | NIAH (needle-in-a-haystack), RULER (Hsieh et al. 2024) | finding/using information hidden in a huge context |
| Chat / preference | Chatbot Arena (Elo), MT-Bench | which answer humans prefer, head to head |
Benchmarks have an expiry date: MMLU, brutal in 2020, is now beaten by frontier models by over 92% —a ceiling—. The community’s answer is much harder successors: ARC-AGI-2 (Chollet et al. 2025) (fluid abstract reasoning, with a human baseline), Humanity’s Last Exam (Phan et al. 2025) (expert-level questions, created precisely because MMLU got easy) and FrontierMath (Glazer et al. 2024) (research mathematics, where the best models solve <2%). The lesson: a saturated benchmark no longer distinguishes models —you have to raise the bar—.
🧩 Analogy — the exam and the student. A benchmark is an exam; the score is a proxy for what the student knows. A well-made exam correlates with real knowledge… until the student gets the exam in advance (contamination) or studies only for that exam (Goodhart). Then the score rises and the knowledge doesn’t.
41.6 The benchmark mirage: contamination
Here is the number-one threat, and the reason this chapter lives in Part VII. Contamination is when the test questions appeared in the training data (web data is enormous and benchmarks are on the web). The model then remembers the answer instead of reasoning it out: the score goes up, real capability doesn’t move.
Move it yourself. Turn up the contamination dial and watch the score shoot up while real ability stays flat —the gap is pure mirage:
It’s not an abstract fear: there are methods that detect contamination (e.g. checking whether the model completes a test word for word, or comparing performance before/after the cutoff date (Golchin and Surdeanu 2024)). A concrete example: OpenAI’s audit of SWE-bench (OpenAI 2026) found that ~59% of the reviewed tasks had defective tests or signs of contamination —so much so that they stopped reporting that metric—. The honest rule: a high score on a public benchmark, with no contamination control, proves nothing. Serious benchmarks renew themselves (new versions, private sets) precisely because of this.
And its close cousin, Goodhart’s law: as soon as a benchmark becomes the target (what drives headlines and funding), teams optimize for it —and it stops measuring what it measured—. That’s why no single number is enough.
41.7 The judge is an LLM (and it can be fooled)
For open-ended answers (an essay, a chat reply) there is no single correct solution, so we use another LLM as judge —or the head-to-head human vote of the Chatbot Arena (Elo ranking)—. It’s powerful and it scales, but the judge has biases worth knowing:
- Position bias: it tends to prefer the first (or the second) answer because of its order. Fixed by averaging both orders.
- Verbosity bias: it rewards the longer answer, even when it isn’t better.
- Self-preference: a judge tends to score its own model family’s style higher.
- Sycophancy (Sharma et al. 2023): models tend to agree with you; a judge can reward the answer that sounds confident and agreeable over the correct one.
🧩 Analogy — the tired panel. An LLM judge is like a panel that, without meaning to, scores the one who speaks first higher, the one who goes on longer, and the one who resembles it. Useful for high volume, but you have to control those biases, not trust it blindly.
41.8 Statistical rigor: is that “+2%” real?
One last, quieter trap: almost no model number comes with an error bar. But changing the seed, the order of the few-shot examples, or the prompt template moves the score by several points. A “+2% on MMLU” without a confidence interval can be pure noise. The honest move: report several seeds, error bars and significance before claiming victory.
41.9 A shared laboratory for context and compression
An accuracy number is like a race time: without distance and terrain, comparing it can mislead. To connect context in ?sec-contexto-evidencia with cache in Section 21.6, publish the race conditions, not just the score.
- Freeze model and task. Weight revision, tokenizer, chat template, engine, precision, generation parameters and question set. Hold out examples that are not used to choose the cache budget.
- Vary one difficulty at a time. Length, position, filler, evidence scope and dispersion, or reasoning steps. State what stayed fixed. LURCH and CRE motivate these controls (Qin et al. 2026; Zhang et al. 2026).
- Establish a full-cache reference. Then compare compression policies at matched effective memory. Separate prefill, decode, KV bytes and total memory.
- Score by task and example. Retrieval, integration and reasoning need their own evidence. Use the same questions on both sides; keep per-example outcomes and estimate uncertainty by resampling complete questions.
- Report what did not run. OOM, unexecuted tasks and unsupported configurations differ from zero accuracy. Preserve actual failures so an average does not hide lost behavior.
If the full cache answers 80 of 100 questions correctly and compression answers 72 of the same 100, the difference is \(72\%-80\%=-8\) percentage points, while relative retention is \(72/80=90\%\). These are distinct summaries of an invented example. To assess stability, you need correctness pairs per question: 80 and 72 alone cannot determine uncertainty in the paired difference.
Connection to TAF. Add observed γ, its interval, fit range and protocol version as descriptors, following Chapter 47. Correlation with accuracy can motivate a hypothesis; a useful predictor needs evaluation on models and tasks not used for fitting. Neither R² nor an evidence-coverage score is a calibrated probability that an answer is correct.
41.9.1 Following up an evaluation difference
The TAF evaluation-drift resolver accepts two scores on a 0–100 scale and declared settings. Its comparability checkbox is a user declaration: it does not verify task versions, splits, examples or metrics. Its numerical band is heuristic, not a confidence interval or significance test.
See the verification steps prioritizes unknown template application, different application states, changed framework/precision/batch, then repetition or uncertainty estimation. This order applies even if scores agree. Two “applied” states do not prove identical templates or rendered conversations.
Establish a common baseline, change one setting at a time and keep per-example results where available. Record model/tokenizer revisions, task version, examples, seeds, generation settings and rendered inputs. A smaller gap after a change is evidence to investigate; it does not certify a fix or by itself attribute the cause to a kernel or template.
Optional demo: five situations illustrate these priorities in Explore another example. Scores are synthetic, not executed evaluations. The lm-evaluation-harness interface documents configuration, seed and sample logging; check the options in your version. TAF’s aggregate results do not replace those records or metric-appropriate uncertainty analysis.
41.10 Bridge to our theme (brief and honest)
Here evaluation touches the book directly. Our headline engineering claim —the D_f window derived from γ (Ch. 20)— is unvalidated (the avenue-2 attempt crashed for lack of memory, Ch. 19). The place where it would stop being a hypothesis is exactly this one: a long-context benchmark (NIAH / RULER (Hsieh et al. 2024)) comparing KV retention under D_f against the field’s heuristics (Ada-KV, LAVa). Put plainly: evaluation is the home where our own thesis gets tested —it’s not just one more topic—.
tafagent ships two evaluation diagnostics that attack exactly this chapter’s traps: the Contamination Prior (estimates the risk that a model has seen a benchmark) and LongScore (measures real long context with RULER, not the advertised length). Before you believe a results table, run it through here.
41.11 Summary
Laboratory: declared conditions, a full-cache reference, paired outcomes and uncertainty; an unexecuted task is not a model failure.
Evaluating = measuring with imperfect proxies. No metric measures capability directly; knowing how to evaluate is knowing the traps.
Perplexity (\(e^{\text{loss}}\)): cheap and internal, but not comparable across tokenizers (use bits-per-byte) and it doesn’t say whether the model is useful.
Benchmarks by capability: knowledge (MMLU/GPQA), code (HumanEval/SWE-bench), math (GSM8K/MATH), long context (NIAH/RULER), chat (Arena/Elo).
Contamination: the test leaked into training → the score rises, capability doesn’t (mirage). With Goodhart’s law: optimizing for a benchmark spoils it.
LLM-as-judge: it scales, but with biases of position, verbosity, self-preference and sycophancy —control them—.
Rigor: without error bars and several seeds, a “+2%” can be noise.
Bridge: long-context evaluation (RULER/NIAH) is where our D_f window must be validated —still pending—.
Next (Chapter 41): we close the book with what no metric fully captures —ethics, safety and limitations—: biases, hallucination, responsible use and, above all, what we still don’t know.
41.12 Exercises
- Perplexity. Why can’t two models with different tokenizers be compared by perplexity? What metric fixes it, and why?
- The mirage. Explain, using this chapter’s widget, why a high score on a public benchmark does not prove capability. What is contamination?
- Goodhart. Give an example of how optimizing for a benchmark could worsen the model on the capability the benchmark was meant to measure.
- The judge. Name three biases of the LLM-as-judge and how each is mitigated.
- Rigor. A paper reports “+1.5% on MMLU” with no error bars. What do you ask before believing it?
- Long context. Why is RULER/NIAH the natural place to validate (or refute) our D_f window from Ch. 20?