Every LLM project we inherit arrives with the same confident claim: "it works in the demo." Of course it does — the demo was built around the inputs the model handles well. The uncomfortable truth is that a demo is a sample size of one, hand-picked. Production is a sample size of millions, adversarial, and indifferent to how proud you are of your prompt. The gap between those two worlds is where evaluation lives, and it is the single discipline that most separates teams who ship reliable AI from teams who ship anxiety.
This piece is about the metrics we actually track when we take an LLM application to production — not the ones that look good in a research paper, but the ones that predict whether a user got value and whether tomorrow's deploy quietly broke something.
Start with the decision, not the metric
The most common evaluation mistake is measuring what is easy rather than what matters. BLEU and ROUGE are trivial to compute and almost never correlate with whether a summary was useful. Before choosing a single metric, we force ourselves to answer one question: what decision will this number change? If a metric moving up or down would not alter a ship/no-ship call, a rollback, or a roadmap priority, it is decoration.
We split evaluation into two regimes that answer different questions:
- Offline evals answer "is this candidate better than what we have?" — run against a fixed dataset, before deploy, gating the release.
- Online evals answer "is the live system healthy right now?" — computed on real traffic, feeding dashboards and alerts.
Confusing the two is a classic failure: teams obsess over a golden test set and never instrument production, so the first sign of a regression is a support ticket.
Build a golden set before you touch the model
An offline eval is only as honest as its dataset. We assemble a golden set of 100–300 representative cases per critical task, drawn from three sources: real historical queries, edge cases engineers know are hard, and failure cases harvested from earlier production incidents. Each case carries a reference answer or, more often, a rubric — because for open-ended generation there is rarely one correct output, only a range of acceptable ones.
A golden set that never grows is a golden set that is slowly lying to you. Every production incident should end with a new row in the dataset, so the same failure can never ship twice.
Crucially, freeze this set and version it. When someone claims "the new prompt is better," they should mean better on a dataset that has not moved under their feet.
The metrics we actually track
For a retrieval-augmented or generative feature, our default scorecard has four families:
- Task success — did the output accomplish what the user asked? For structured tasks this is exact-match or schema validity; for open tasks it is a rubric score.
- Faithfulness / groundedness — is every claim in the answer supported by the retrieved context? This is the single most important safety metric for RAG, and the one that catches hallucination.
- Relevance — does the answer address the actual question, not an adjacent one the model found easier?
- Operational health — p50/p95 latency, cost per request, token consumption, and error/timeout rate. Quality that arrives in nine seconds at ten cents a call is a different product than quality in one second at one cent.
LLM-as-judge, used responsibly
For anything open-ended, human grading does not scale to every deploy. We use an LLM judge — a separate, stronger model prompted with an explicit rubric — to score faithfulness and relevance. It works, but only under discipline: the judge prompt is itself a versioned artifact, we calibrate it against a few hundred human labels to measure agreement, and we prefer pairwise comparisons ("is A better than B?") over absolute 1–10 scores, because models are far more reliable at ranking than at calibrated scoring.
def grade_faithfulness(question, context, answer, judge):
prompt = f"""You are a strict grader. A claim is FAITHFUL only if it is
fully supported by the CONTEXT. Do not use outside knowledge.
Question: {question}
Context: {context}
Answer: {answer}
Return JSON: {{"verdict": "faithful|unfaithful",
"unsupported_claims": [ ... ]}}"""
result = judge.complete(prompt, response_format="json")
return result["verdict"] == "faithful"Two guardrails on the judge: never let a model grade its own family of outputs without a human calibration check, and always log the judge's reasoning so a human can audit disagreements. An unaudited judge is just a second model you also don't understand.
Aggregate honestly — the mean hides the failures
A 92% average task-success score feels great until you notice the 8% failures are concentrated in your highest-value customer segment. We never report a single mean. Instead we slice: by query type, by document source, by customer tier, by input length. Regressions almost always hide in a slice. The dashboard that matters shows the worst slice, not the average, because that is where trust erodes first.
Close the loop with online signals
Offline evals catch what you anticipated; production reveals what you didn't. We instrument every LLM response with lightweight online signals that need no ground truth:
- Implicit feedback — did the user copy the answer, retry, rephrase, or abandon? A retry rate creeping upward is an early smoke alarm.
- Explicit feedback — thumbs up/down, but treated as a weak, biased signal, not gospel.
- Guardrail trip rate — how often safety filters, schema validators, or refusal detectors fire.
- Drift indicators — shifts in input distribution and output length that often precede a quality drop.
The highest-leverage practice we deploy is running the offline eval suite in CI on every prompt or model change, and shadow-scoring a sample of live traffic daily. When a provider silently updates a model behind a stable API name — which happens more than vendors admit — that daily shadow score is often the first thing to move.
What good looks like
A team with evaluation under control can answer three questions in under a minute: Is the current system better than last week's? Which slice is worst right now? If we ship this change, what breaks? If those answers require a meeting, you don't have an eval system — you have opinions.
Evaluation is not the glamorous part of applied AI, but it is the part that lets you sleep. It converts "the model feels better" into a number you can defend, a gate you can automate, and an alarm that wakes you before your users do. Build it early, version everything, and never trust a mean.