Why LLM Evaluation Is Hard — and Non-Negotiable

Most engineering teams can ship a web service they can reason about: the same input yields the same output, a failing assertion is unambiguous, and a green test suite means the behavior is locked in. LLM-powered systems break every one of those assumptions. The model is stochastic, the space of acceptable outputs is enormous and fuzzy, and a change that looks harmless — swapping a model version, editing one line of the system prompt, reordering few-shot examples — can silently shift quality across thousands of requests. Evaluation is the discipline that turns "it feels better" into "it is measurably better on the cases we care about, with no regression on safety." Without it you are flying blind, and in production that blindness is expensive.

This lesson frames the whole course. It explains why the familiar testing playbook collapses when the unit under test is a language model, what a real evaluation harness looks like at its core, and the maturity ladder that separates a demo from a dependable product. Everything that follows — metrics, judges, datasets, RAG and agent evals, CI and monitoring — is an elaboration of the ideas introduced here.

Why the usual testing playbook fails

Traditional software testing rests on two properties: determinism and exact correctness. A function that adds two numbers has exactly one right answer, and a unit test pins it forever. LLM outputs violate both properties at once, and they violate them in ways that are not fixable with a cleverer assertion.

• Non-determinism. With any temperature above zero the same prompt produces different completions. Even at temperature zero you cannot rely on byte-identical output: providers batch requests, floating-point reductions on GPUs are not associative, and the vendor may silently update the served checkpoint. "Reproducible" for an LLM means "statistically stable," not "identical."
• No single correct answer. For "summarize this support ticket" there are thousands of good summaries and thousands of bad ones. Correctness is a fuzzy region in output space, not a point. An exact-match assertion against one reference rejects perfectly good outputs and accepts nothing useful.
• Semantic, not lexical, equivalence. "The payment failed because the card expired" and "Charge declined: expired card" mean the same thing and share almost no tokens. Any metric that compares surface strings scores them as wildly different, which is exactly backwards.
• Emergent, high-dimensional failure modes. LLMs fail in ways ordinary functions cannot: hallucinated facts, subtle instruction-following drift, format violations, tone problems, prompt-injected content, refusal on benign requests, and sycophancy. A single pass/fail bit cannot represent this surface.

The practical consequence is blunt: you cannot assert output == expected. You need graded, multi-dimensional evaluation that tolerates paraphrase while still catching real regressions. The rest of the discipline is about building measurements that behave well under exactly these conditions.