How to Actually Evaluate LLM Output

Teams ship language-model features with less testing than they would accept for a date-formatting function. The reason is understandable — output is open-ended and there is no obvious assertion to write — but the consequence is that quality regressions reach users unnoticed.

Start with an evaluation set, not a framework

Twenty to fifty real examples, with the characteristics you care about written down, beat an elaborate harness built on invented cases. Draw them from actual usage where possible, and deliberately include the hard ones: ambiguous questions, missing information, adversarial phrasing, and the long tail that broke things before.

This set is an asset. It grows every time something goes wrong in production, and it is what lets you change a prompt or a model without guessing.

Grade what you can grade cheaply

Not everything needs a judge. A surprising amount can be checked deterministically:

• Does the output parse as the schema you asked for?
• Are required fields present and within range?
• Does every claim carry a citation, where citations are required?
• Is it within length bounds?
• Does it avoid terms you have forbidden?

These catch real regressions and cost nothing per run. Build them first.

Model-as-judge, used carefully

For qualities that resist assertions — faithfulness to source, tone, whether the question was actually answered — a model can grade output, with caveats.

Judges are biased toward longer answers, toward their own family's style, and toward confident phrasing. They are more reliable comparing two candidates than scoring one in isolation, and more reliable against a rubric with concrete criteria than against "rate this out of ten".

Validate the judge against human judgement on a sample before trusting it. A judge that disagrees with your team is measuring something, but not what you wanted.

Measure faithfulness separately from correctness

In a retrieval system these are different failures with different fixes. An answer can be faithful to the retrieved context and still wrong, because retrieval returned the wrong passage. It can be correct and unfaithful, because the model used prior knowledge rather than the source.

Both matter. Conflating them sends you optimising the wrong half of the pipeline.

Run evaluations in CI

A prompt is code. Changing it should trigger the same evaluation suite as changing a function, and a meaningful drop should fail the build. Without this, prompt changes accumulate untested and quality drifts in a direction nobody chose.

Pin model versions explicitly. A provider updating a model underneath you is a dependency change, and it will move your numbers.

Watch production, not just the suite

Offline evaluation tells you about the cases you thought of. Production tells you about the rest. Log inputs and outputs, sample them for review, and track the signals that correlate with failure — retries, regenerations, abandoned sessions, users rephrasing the same question.

The question a user asked three times in a row is a better test case than anything you would have invented.