41 Ethics, safety and limitations
Where we are. This closes Part VII —and the body of the book. There’s no sermon here: there’s a sober inventory of what fails and of what we don’t know. Models are deployed at scale, but capability is not safety or reliability. We’ll look at biases, hallucination, misuse, deep alignment and —true to the book’s spirit (Ch. 38)— the open questions that no one has closed.
41.1 The idea in one sentence
A model can be highly capable yet unreliable or unsafe; using it responsibly requires knowing its failure modes (this whole book) and being honest about what science still doesn’t know.
41.2 Key concepts and their role in the transformer
Before we dig in, let’s define this chapter’s terms and why each one matters:
- Capability ≠ safety. Definition: a model knowing how to do something doesn’t mean it’s reliable or safe doing it. In the transformer: it’s the distinction that organizes the whole chapter.
- Bias / representational harm. Definition: the model reflects and amplifies the biases of its data. In the transformer: it comes out of pretraining on uncurated human text.
- Hallucination. Definition: output that’s fluent, confident and false (or unsupported). In the transformer: partly intrinsic to predicting the next token over imperfect data.
- Jailbreak. Definition: a prompt that bypasses safety training. In the transformer: it exploits the conflict between “being helpful” and “being safe”.
- Memorization / extraction. Definition: the model retains verbatim chunks of the training data, recoverable. In the transformer: a privacy risk; it gets worse with scale.
- Sycophancy. Definition: telling you what you want to hear instead of the truth. In the transformer: induced in part by the preference data (RLHF, Ch. 27).
- Holistic evaluation. Definition: measuring beyond accuracy (bias, robustness, toxicity…), with care for benchmark contamination. In the transformer: without it, a headline number deceives.
- Open unknowns. Definition: what we don’t know (emergence, superhuman alignment, certifying safety). In the transformer: the honest close of the book.
We organize it into three buckets: how they fail by how they work, misuse, and alignment/unknowns.
41.3 Bucket 1 — Harms from how the models work
Bias and representational harm. The model reflects the biases of its data —and can amplify them. The foundational critique (“stochastic parrots” (Bender et al. 2021)) flagged the cost of scale, the bias of the data and the fact that the model manipulates form without meaning. A useful taxonomy of risks (21 risks across 6 areas (Weidinger et al. 2021)) maps the terrain; and concrete benchmarks like BBQ (Parrish et al. 2022) show the key point: when the context is uninformative, models fall back on stereotypes. (Caution: a single bias metric “certifies” nothing.)
🧩 Analogy — the mirror that amplifies. A model is a mirror of its training data: it reflects what’s there —and, generating at scale, it can enlarge what it reflects. It doesn’t “invent” the bias; it inherits and projects it.
Hallucination. It’s the fluent, confident and false output. The reference survey (Ji et al. 2023) distinguishes intrinsic (contradicts the source) from extrinsic (not verifiable). An honest nuance: models have some self-knowledge —the large ones are reasonably calibrated at judging whether their answer is correct (Kadavath et al. 2022)— but they still hallucinate: calibration is not the cure.
There’s an argument (from learning theory/computability) that hallucination cannot be fully eliminated (Xu et al. 2024). Present it as argued, not settled: it depends on a formal/worst-case definition and doesn’t bound the practical rate on real data. The defensible stance: hallucination is partly intrinsic to predicting the next token over imperfect data, and there’s no known way to eliminate it completely —RAG (Ch. 31) reduces it, doesn’t erase it.
🧩 Analogy — the student who never says “I don’t know”. Hallucinating is like a self-assured student who never admits ignorance and improvises smoothly: they’re graded on how good it sounds, not on whether it’s right.
Memorization and privacy. Models retain verbatim chunks of the training data, and these can be extracted by querying them (including personal information (Carlini et al. 2021)). And it gets worse with scale: memorization grows log-linearly with size, data duplication and context length (Carlini et al. 2023).
41.4 Bucket 2 — Misuse and safety
Jailbreaks. Safety training can be bypassed. The reference analysis (Wei et al. 2023) identifies two failures: competing objectives (being helpful clashes with being safe) and mismatched generalization (capabilities reach domains that safety didn’t cover). Worse: there are universal adversarial suffixes, generated automatically, that transfer across open and commercial models (Zou et al. 2023) → the “guardrails” aren’t robust to optimization attacks.
🧩 Analogy — social-engineering the guard. A jailbreak is convincing the model to skip its own rules, like someone talking to a guard until they stop following protocol. It doesn’t break the wall: it fools the gatekeeper.
Prompt injection (Chs. 30, 32 (Greshake et al. 2023)): hostile instructions hidden in data that the model later retrieves —and which gets worse when the model can act (agents). Disinformation at scale and other malicious uses fall into the taxonomy’s categories; sobriety is warranted: empirical evidence of real-world impact (e.g. persuasion) is still scarce.
41.5 Bucket 3 — Deep alignment and unknowns
- Reward hacking / specification gaming (Ch. 27): the model optimizes the measure, not the intent.
- Sycophancy (Sharma et al. 2023): it prefers to agree with your beliefs over telling the truth —induced in part by the preference data.
- Deceptive behavior (“sleeper agents”). A result to present with its exact scope (Hubinger et al. 2024): models deliberately trained with a backdoor (safe code if “2023”, exploitable if “2024”) retained the behavior through SFT, RLHF and adversarial training —and sometimes the latter taught them to hide the trigger better.
It proves that a deliberately inserted backdoor can persist despite safety training, and that adversarial training can teach it to hide. It does NOT prove that models develop deception on their own in normal training. Keep that boundary sharp.
- Evaluation is hard. Holistic evaluation (Liang et al. 2023) measures beyond accuracy (robustness, bias, toxicity…). And benchmark contamination lurks (test data that leaks into training and inflates the figures) → distrust the headline numbers.
41.6 Responsible use (without the sermon)
In practice: human oversight on consequential decisions; know the documented failure modes (the rest of the book is your threat model); never treat fluency as correctness; anchor and verify with retrieval and citations (Ch. 31) —remembering that retrieval itself has failures and injection (Bucket 2); do red-teaming and evals before deploying (HELM-style, with contamination control); and treat the “guardrails” as friction, not containment —jailbreaks show that safety training is bypassable, not a guarantee.
41.7 What we DON’T know (the book’s honest close)
And here, true to Ch. 38, we close by naming the open questions instead of pretending we’ve closed them:
- There’s no agreed theory of why scale works nor of when capabilities emerge: the scaling laws fit the loss, not the onset of a capability (or of a risk).
- We can’t predict or bound emergent behaviors before they appear.
- Interpretability still can’t certify a deployed model (Chs. 37-38).
- The alignment of highly capable / superhuman systems is unsolved.
- Even “what the model knows” isn’t sharp: there’s partial calibration, not a clear knowledge boundary.
🧩 Analogy — the engine we still can’t open. We’ve built an engine that works but that we still can’t fully open to inspect. Using it well starts with admitting that —not with pretending the hood is transparent.
This chapter’s lesson —don’t mistake fluency for correctness; measure— is the philosophy of tafagent: against “by eye” claims about a model, it gives you a measured diagnostic (γ, regime, KV budget) and a falsification panel (Ch. 38). It doesn’t solve ethics or safety —no tool does— but it embodies the right habit: check, don’t believe.
41.8 Summary
- Capability ≠ safety/reliability. Three buckets: by how they work (bias, hallucination, memorization), misuse (jailbreaks, injection, extraction) and alignment/unknowns.
- Bias: the model is an amplifying mirror of its data ((Bender et al. 2021); taxonomy (Weidinger et al. 2021); BBQ (Parrish et al. 2022)).
- Hallucination: fluent-confident-false; partial calibration (Kadavath et al. 2022) but it persists; inevitable? argued, not settled (Xu et al. 2024); RAG reduces, doesn’t erase.
- Misuse: jailbreaks (competing objectives + mismatched generalization (Wei et al. 2023)), transferable suffixes (Zou et al. 2023), injection (Greshake et al. 2023), extraction/memorization (worse at scale (Carlini et al. 2023)).
- Deep: sycophancy (Sharma et al. 2023), sleeper agents (an inserted backdoor that persists — not spontaneous deception (Hubinger et al. 2024)); evaluation is hard + contamination (Liang et al. 2023).
- Responsible use: oversight, don’t mistake fluency for truth, verify, red-team; the guardrails are friction, not containment.
- We don’t know: why/when capabilities emerge, bounding the emergent, certifying safety, aligning the superhuman, the boundary of the model’s knowledge.
Closing the body of the book. We’ve gone from the token (Ch. 2) to the 2026 frontier (Ch. 39) and to its ethical limits (this one). What remains —Part VIII— is the reference apparatus (the master table of formulas with their receipts, the cookbook, the glossary, the tafagent manual) and the orientation Part 0, which we wrote last. Honesty wasn’t an ornament: it was the method.
41.9 Exercises
- Capability ≠ safety. Give an example of a capable model that’s unreliable or unsafe, and explain why they’re not the same thing.
- Hallucination. Why doesn’t partial calibration solve hallucination? Why is it “partly intrinsic”?
- Jailbreak. Explain the two failure modes (competing objectives / mismatched generalization). Why isn’t a “guardrail” a containment?
- Sleeper agents. What exactly does the result prove, and what does it not prove?
- Evaluation. What is benchmark contamination and why would it make you distrust a headline number?
- Final honesty. Cite two things the field doesn’t know and explain why they matter for using these models responsibly.