41 Evaluation: measuring without fooling yourself
Where we are. We have trained (Part IV), used (Part V), made cheaper (Part VI) and audited from the inside (Chs. 37-39) a transformer. What’s left is the question that decides everything else: how do we know a model is good? Evaluation is as hard —and as easy to get wrong— as training. And it belongs here, in Part VII, because it shares its backbone: almost everything you measure is an imperfect proxy, and the biggest danger is not measuring too little, but measuring wrong and believing it.
41.1 The idea in one sentence
Evaluating an LLM means estimating its capability with metrics that are proxies —none measures it directly— so knowing how to evaluate is, above all, knowing the traps: data contamination, the judge that gets fooled, and Goodhart’s law.
41.2 Key concepts and their role in the transformer
Before the details, we define this chapter’s terms and what each is for:
- Perplexity. Definition: \(e^{\text{loss}}\); “how many tokens the model is hesitating between” (Ch. 11). In the transformer: the cheapest internal metric —but only comparable with the same tokenizer, and it doesn’t tell you whether the model is useful—.
- Benchmark. Definition: a fixed set of tasks with known answers. In the transformer: the standard way to put a number on a capability (knowledge, code, math, long context).
- LLM-as-judge. Definition: using another LLM to score open-ended answers. In the transformer: it scales human evaluation, but inherits biases (position, verbosity, self-preference).
- Data contamination. Definition: the test leaked into the training data. In the transformer: it inflates the score without the model being any better —the number-one threat to validity—.
- Goodhart’s law. Definition: “when a measure becomes a target, it stops being a good measure.” In the transformer: optimizing for a benchmark degrades what the benchmark was meant to measure.
- Construct validity. Definition: whether the metric really measures the capability you claim, or just a correlated shortcut. In the transformer: the underlying question of the whole chapter.
- Statistical rigor. Definition: error bars, several seeds, significance. In the transformer: without them, a “+2%” can be noise.
41.3 Why a whole chapter
We already saw text metrics in Ch. 29 (BLEU, ROUGE, the LLM judge). Here we go up a level: evaluating the whole model. And we put it in Part VII on purpose, because evaluation is where the book’s honesty is really tested: a high number does not prove capability. The rest of the chapter is a tour of the metrics and of the ways each one can fool you.
41.4 Perplexity: the internal metric (and its limits)
The cheapest metric is the one you already have from training (Ch. 11): perplexity, \(\text{PPL}=e^{\text{loss}}\). It measures the model’s average surprise on a text: a PPL of 8 means it hesitates, on average, “as if among 8 tokens”. Lower = predicts better.
PPL depends on how you split the text: a model with a large vocabulary cuts the same text into fewer tokens and comes out with a PPL that is not comparable to another model’s. That’s why, to really compare, we use bits-per-byte (bpb), which normalizes by bytes of text, not tokens. And even so: a low PPL doesn’t say the model is useful —only that it predicts the next token well—.
41.5 Capability benchmarks: putting a number on what it can do
To measure specific capabilities we use benchmarks: batteries of tasks with known answers. The big axes of 2024-2026:
| Capability | Reference benchmarks | What they measure |
|---|---|---|
| Knowledge / reasoning | MMLU (Hendrycks et al. 2021), MMLU-Pro, GPQA, BBH | exam questions across 57 subjects; GPQA at PhD level |
| Code | HumanEval (Chen et al. 2021), MBPP, SWE-bench (Jimenez et al. 2024) | writing functions that pass tests; SWE-bench, solving real GitHub issues |
| Math | GSM8K (Cobbe et al. 2021), MATH, AIME | word problems (GSM8K) and competition math (MATH/AIME) |
| Long context | NIAH (needle-in-a-haystack), RULER (Hsieh et al. 2024) | finding/using information hidden in a huge context |
| Chat / preference | Chatbot Arena (Elo), MT-Bench | which answer humans prefer, head to head |
Benchmarks have an expiry date: MMLU, brutal in 2020, is now beaten by frontier models by over 92% —a ceiling—. The community’s answer is much harder successors: ARC-AGI-2 (Chollet et al. 2025) (fluid abstract reasoning, with a human baseline), Humanity’s Last Exam (Phan et al. 2025) (expert-level questions, created precisely because MMLU got easy) and FrontierMath (Glazer et al. 2024) (research mathematics, where the best models solve <2%). The lesson: a saturated benchmark no longer distinguishes models —you have to raise the bar—.
🧩 Analogy — the exam and the student. A benchmark is an exam; the score is a proxy for what the student knows. A well-made exam correlates with real knowledge… until the student gets the exam in advance (contamination) or studies only for that exam (Goodhart). Then the score rises and the knowledge doesn’t.
41.6 The benchmark mirage: contamination
Here is the number-one threat, and the reason this chapter lives in Part VII. Contamination is when the test questions appeared in the training data (web data is enormous and benchmarks are on the web). The model then remembers the answer instead of reasoning it out: the score goes up, real capability doesn’t move.
Move it yourself. Turn up the contamination dial and watch the score shoot up while real ability stays flat —the gap is pure mirage:
It’s not an abstract fear: there are methods that detect contamination (e.g. checking whether the model completes a test word for word, or comparing performance before/after the cutoff date (Golchin and Surdeanu 2024)). A concrete example: OpenAI’s audit of SWE-bench (OpenAI 2026) found that ~59% of the reviewed tasks had defective tests or signs of contamination —so much so that they stopped reporting that metric—. The honest rule: a high score on a public benchmark, with no contamination control, proves nothing. Serious benchmarks renew themselves (new versions, private sets) precisely because of this.
And its close cousin, Goodhart’s law: as soon as a benchmark becomes the target (what drives headlines and funding), teams optimize for it —and it stops measuring what it measured—. That’s why no single number is enough.
41.7 The judge is an LLM (and it can be fooled)
For open-ended answers (an essay, a chat reply) there is no single correct solution, so we use another LLM as judge —or the head-to-head human vote of the Chatbot Arena (Elo ranking)—. It’s powerful and it scales, but the judge has biases worth knowing:
- Position bias: it tends to prefer the first (or the second) answer because of its order. Fixed by averaging both orders.
- Verbosity bias: it rewards the longer answer, even when it isn’t better.
- Self-preference: a judge tends to score its own model family’s style higher.
- Sycophancy (Sharma et al. 2023): models tend to agree with you; a judge can reward the answer that sounds confident and agreeable over the correct one.
🧩 Analogy — the tired panel. An LLM judge is like a panel that, without meaning to, scores the one who speaks first higher, the one who goes on longer, and the one who resembles it. Useful for high volume, but you have to control those biases, not trust it blindly.
41.8 Statistical rigor: is that “+2%” real?
One last, quieter trap: almost no model number comes with an error bar. But changing the seed, the order of the few-shot examples, or the prompt template moves the score by several points. A “+2% on MMLU” without a confidence interval can be pure noise. The honest move: report several seeds, error bars and significance before claiming victory.
41.9 Bridge to our theme (brief and honest)
Here evaluation touches the book directly. Our headline engineering claim —the D_f window derived from γ (Ch. 20)— is unvalidated (the avenue-2 attempt crashed for lack of memory, Ch. 19). The place where it would stop being a hypothesis is exactly this one: a long-context benchmark (NIAH / RULER (Hsieh et al. 2024)) comparing KV retention under D_f against the field’s heuristics (Ada-KV, LAVa). Put plainly: evaluation is the home where our own thesis gets tested —it’s not just one more topic—.
tafagent ships two evaluation diagnostics that attack exactly this chapter’s traps: the Contamination Prior (estimates the risk that a model has seen a benchmark) and LongScore (measures real long context with RULER, not the advertised length). Before you believe a results table, run it through here.
41.10 Summary
- Evaluating = measuring with imperfect proxies. No metric measures capability directly; knowing how to evaluate is knowing the traps.
- Perplexity (\(e^{\text{loss}}\)): cheap and internal, but not comparable across tokenizers (use bits-per-byte) and it doesn’t say whether the model is useful.
- Benchmarks by capability: knowledge (MMLU/GPQA), code (HumanEval/SWE-bench), math (GSM8K/MATH), long context (NIAH/RULER), chat (Arena/Elo).
- Contamination: the test leaked into training → the score rises, capability doesn’t (mirage). With Goodhart’s law: optimizing for a benchmark spoils it.
- LLM-as-judge: it scales, but with biases of position, verbosity, self-preference and sycophancy —control them—.
- Rigor: without error bars and several seeds, a “+2%” can be noise.
- Bridge: long-context evaluation (RULER/NIAH) is where our D_f window must be validated —still pending—.
Next (Chapter 41): we close the book with what no metric fully captures —ethics, safety and limitations—: biases, hallucination, responsible use and, above all, what we still don’t know.
41.11 Exercises
- Perplexity. Why can’t two models with different tokenizers be compared by perplexity? What metric fixes it, and why?
- The mirage. Explain, using this chapter’s widget, why a high score on a public benchmark does not prove capability. What is contamination?
- Goodhart. Give an example of how optimizing for a benchmark could worsen the model on the capability the benchmark was meant to measure.
- The judge. Name three biases of the LLM-as-judge and how each is mitigated.
- Rigor. A paper reports “+1.5% on MMLU” with no error bars. What do you ask before believing it?
- Long context. Why is RULER/NIAH the natural place to validate (or refute) our D_f window from Ch. 20?