Validate against human labels

Validating a judge means comparing its verdicts against human labels on examples the judge has not seen.

The process is simple:

1. Start with traces that humans have labeled PASS or FAIL.
2. Keep a held-out set that is not used in the judge prompt.
3. Run the judge on that held-out set.
4. Compare the judge’s verdicts with the human labels.
5. Report the confusion matrix, precision, recall, and F1 for the class that matters.

The held-out set is the key. If an example appears in the judge’s few-shot prompt or was used to develop the judge, it cannot also be used to validate the judge. The judge has already seen it.

Accuracy hides what matters

Suppose the faithfulness judge is tested on 100 human-labeled traces: 80 PASS and 20 FAIL.

It agrees with human labels on 83 of 100 traces, so the headline looks good: 83% accuracy.

But the confusion matrix tells a different story:

The judge correctly passes 77 of 80 faithful answers, but catches only 6 of 20 failures.

Accuracy              = (77 + 6) / 100        = 83%
Precision (P) on FAIL = 6 / (6 + 3)           = 67%
Recall (R) on FAIL    = 6 / (6 + 14)          = 30%
F1 on FAIL            = (2 * P * R) / (P + R) = 41%

So the judge is not good enough for faithfulness evaluation. It misses most of the unfaithful answers.

Hold out a real test set

Do not validate a judge on examples that appear in its prompt or were used to tune it. Split your labeled traces into: few-shot examples that go inside the judge prompt, dev set for iterating on the judge, and held-out test set for final validation only.

On the dev set, the most useful step is reading disagreements. For example:

• User question: “How long do trial-account records stay available?”
• Retrieved context: standard accounts keep records for 30 days; trial accounts keep them for 90 days.
• Agent answer: “Records are retained for 30 days after closure.”
• Judge verdict: PASS (found a matching sentence in the context).
• Human label: FAIL (the answer used the wrong account type).

That disagreement tells you what example to add: faithfulness requires support for the specific entity and condition in the user’s question, not just any matching sentence in the retrieved documents.

Once the judge performs well on dev, run it once on the held-out test set and report the result.

When calibration is poor

Reading the disagreements usually reveals one of a few patterns:

• Missing failure subtype. The judge catches outright hallucinations but misses subtle contradictions, dropped caveats, or unsupported generalizations.
• Over-flagging. The judge calls everything FAIL when the agent paraphrases instead of quoting verbatim.
• Unclear judging standard. The judge says some answers are unfaithful that the humans accept as fine.
• Right for the wrong reason. The verdict is correct but the critique points at the wrong thing.
• Too-small validation set. Wide error bars on metrics; unreliable conclusions (Eval scores are samples, not truth covers when this matters).

The fix progression is examples → criterion → model.

Examples almost always move first: if the judge is wrong on five cases of subtle contradiction, consider adding five contradiction examples to the few-shot block. If multiple disagreements trace to ambiguity in what counts as a failure rather than missing examples, the adjudicator sharpens the criterion. Model swaps come last. Most calibration problems live in the prompt, not the model, and switching models is expensive in cost, latency, and stability.

Revalidate when things change

The judge agrees with humans on the held-out set today. That doesn’t mean it will agree in six months. Foundation models update, the agent’s failure distribution shifts, and new failure types appear. Calibration is a snapshot: true at measurement time, increasingly stale afterward.

Your eval system also drifts covers what to do about it: re-validate against fresh labels on a schedule and monitor agreement rates over time. Large drift in those agreement rates is usually a signal that the failure distribution itself has changed.

Every deployed judge should carry a short record of what it was validated on: the domain, the trace types, the test-set size, the date, and what it’s allowed to be used for. Without that record, judges get re-used on data the original validation doesn’t cover, and the team trusts a number that no longer applies.

The judge is part of the system

An LLM judge is not an oracle. It is a model component that needs the same discipline as any other classifier: held-out labels, a confusion matrix, precision and recall on the class that matters, and periodic revalidation.

Once the judge is validated, there is still one more question: how much should we trust movement in the score? A judge may report that v2 improved by two points, but that movement may be real or may be noise. The next article covers that next layer: uncertainty, sample size, and when an eval result is large enough to act on.

Start with the failure mode

The wrong question is “Should we use an LLM judge here?” The right question is “What exactly are we trying to detect?”

• Code check when the failure has a structural shape: a missing required field, an invalid category, a prohibited phrase in the response.
• LLM judge when the failure requires reading language in context: tone mismatch, redundant clarifying questions, subtle policy violations.

Use code for structural failures

A code-based evaluator is a function. Take a trace, return a score with a reason.

def check_handoff_schema(trace) -> Score:
    required = {"case_id", "user_email", "issue_category", "summary"}
    handoff = trace.outputs["handoff_record"]
    missing = required - handoff.keys()
    if missing:
        return Score(passed=False, reason=f"Missing fields: {missing}")
    return Score(passed=True, reason="Handoff record well-formed")

Five lines. Runs in milliseconds. Deterministic: same input, same output, every time. Debuggable: when it fails, the reason tells you exactly what to look at. Free: no API calls, no token costs, no rate limits. Runs on every trace, on every CI build, on every production trace if you want.

Code and judges aren’t always alternatives. Sometimes they’re layers. A regex check for refund-promise language catches explicit cases (“Your refund has been approved”); it misses subtler implications (“We’ll make sure you get your money back”). The fix isn’t to abandon the regex for a judge. Run the cheap regex on every trace; run a more expensive judge only when the trace involves a refund request and the regex didn’t fire. The principle: run cheap checks broadly, run judges where they add value.

Use LLM judges for semantic failures

Some failure modes don’t have a structural shape. For instance, the user is frustrated but the agent responds in an upbeat and causal manner.

A judge worth trusting is binary (pass or fail, not 1-5) and grounded in labeled examples from the dataset the previous article produced. The next article covers the prompt design and the calibration step: comparing the judge’s verdicts to human labels before treating its numbers as measurement. Until then, the discipline is simple: don’t deploy a judge whose agreement with humans you haven’t measured.

Keep each evaluator narrow

The mistake is building one big judge called quality_judge_v3 and asking it to score everything: tone, helpfulness, schema validity, routing. That judge produces a number. It does not produce a diagnosis.

It’s the same reason you don’t write a single function called is_correct() in production code. Composite signals are harder to act on, harder to debug, and harder to improve.

Suppose the intake agent’s “overall quality” judge drops from 87% to 81% over a release. What changed? Tone got worse? Helpfulness got worse? Grounding got worse? Some combination? The composite verdict can’t tell you. You read failures by hand, you guess, and you propose a fix without knowing which dimension you’re targeting.

Specialized evaluators fix this. The tone judge tracks tone. The redundant-question judge tracks redundancy. When tone scores drop and redundancy scores hold steady, you know what changed. When you fix the tone issue, the tone judge tells you whether the fix worked.

The cost is more API calls per trace, one per judge instead of one total. In practice this is cheap, because most evaluation runs sample only a fraction of production traffic. The per-trace cost matters less than signal quality.

A reasonable evaluator stack for the intake agent might look like:

Seven evaluators, each answering one question. When the dashboard shows a regression, you can identify which question’s answer changed.

One related discipline: when a regression case enters the suite, tag it by the layer where the failure originated, not the layer where it became visible. A malformed handoff at turn 5 caused by a classifier misread at turn 3 should be tagged as a classifier failure.

Code first, judges when you need them

Your LLM judge is a classifier covers the next layer: making judges trustworthy when you do need them. A judge that produces verdicts in the right format isn’t the same as a judge that produces correct verdicts.

The next time someone proposes a judge, first ask: could code answer this?

Start with traces, not imagined categories

A common mistake is to start with abstract qualities like accurate, helpful, concise, safe, professional. They describe how the team imagines quality before seeing how the system actually fails.

For a scheduling assistant, the real failures are more specific: choosing the wrong Alex, writing to the wrong calendar, confirming before the calendar API succeeds, losing information from earlier turns, or mishandling a forwarded email thread. These failures are concrete enough to test and fix, and they only emerge when humans read real traces.

So start with traces: label what happened, write short critiques, and let the failure modes emerge from the examples. When labeling reveals an obvious bug, fix it instead of building a judge to detect it.

Keep labels consistent

Reviewers will disagree on edge cases. Designate one adjudicator to own the labeling guide, make the call on contested cases, and update examples as the standard becomes clearer.

For domain-heavy products, the adjudicator should be whoever is closest to the product’s real standard of correctness, not whoever happens to be available. In a healthcare project I led, anchoring medical-billing-level labels to a domain expert’s judgment rather than engineers’ guesses gave the dataset real ground truth.

Keep the labels themselves simple: pass or fail. Likert scales feel more nuanced but are harder to act on and invite false precision. The nuance lives in the critique.

Design for coverage and difficulty

Once labels are consistent, the next question is whether the dataset is representative.

A raw production sample usually overrepresents easy traffic. For a scheduling assistant, that means lots of simple booking requests and too few high-risk cases: ambiguous attendees, wrong-calendar writes, permission errors, forwarded email threads, time-zone mistakes.

Design the dataset intentionally:

• Main tasks: booking, rescheduling, cancellation.
• Common ambiguity: unclear attendee, unclear event, vague time phrase, forwarded email thread.
• High-risk failures: wrong calendar, premature confirmation, missing conflict check, permission error.

Set target counts for the slices that matter most (30 ambiguous-attendee cases, 25 wrong-calendar cases). The numbers are illustrative; the point is deciding the targets before collecting.

For rare but important slices, do a cheap pre-filtering pass before full labeling. If wrong-calendar writes or permission errors are rare in raw traffic, find the candidate traces first, then label those carefully. In a clinical-note project I led, sections like allergies and labs appeared sparsely across clinician-patient transcripts; a quick yes/no pass identified which transcripts contained them.

Balance difficulty too:

• Easy cases: the system should already pass these. They protect against regressions.
• Medium cases: the system is mixed. These show whether the product is improving.
• Hard cases: the system often fails. These show the frontier.
• Safety cases: the system must refuse, escalate, or avoid forbidden behavior.

Too easy and the dataset saturates; too hard and it doesn’t show progress. Aim for a visible mix so the team can tell what actually improved.

Eval scores are samples, not truth returns to sample size and uncertainty. If a slice matters, include it deliberately and track it separately.

When you don’t have production data

Before launch, the dataset has to come from somewhere other than production. The key rule: generate inputs, not outputs. Let the real system produce the responses, and label the resulting traces like any other trace.

Two sources work well in combination.

Persona-based generation. Define three to five user personas that match the product (for the scheduling assistant: a time-zone-confused product manager, an executive with overlapping calendars, an IC handling forwarded meeting threads). For each persona, generate eight to twelve plausible queries.

Expert elicitation. Sit with someone who has done this product’s work before: a customer success lead, a power user from a prior product, a domain expert. Ask: “What is the worst question someone could ask this?” and “Where do you expect this to fail?” The answers encode failure intuition the team doesn’t yet have from data, including the high-risk cases the coverage section calls out.

The dataset is the product’s memory

An eval dataset is the product’s memory of what has gone wrong, what the team decided “good” means, and what must not break again.

That memory has to keep growing. The first version will be incomplete; failure modes will get sharper as production reveals new ones; the adjudicator will tighten the examples annotation guide as the team’s standard becomes clearer. The danger is pretending the dataset doesn’t need to evolve.

What eval is for

A score is useful only if it changes what the team does next. Eval supports different decisions at different stages: during development, whether a new prompt is better; before merge, whether old failures came back; in production, whether new failure patterns are emerging.

The operational test for any metric: When this number moves, what decision changes? If the answer is merge, hold, investigate, roll back, or add a regression case, the metric is useful. If the answer is “nothing,” it does not belong on the dashboard.

The minimum viable eval loop

A useful eval system starts with traces and ends with decisions. The first version needs six pieces: traces, labels, failure modes, checks and judges, regression cases, and a regular review.

1. Traces

A trace records one agent run: state, retrieved context, actions, tool calls, verification, and the final state it saved. It shows how the agent got there, not just where it ended up.

An agent’s final message often hides the real failure. The intake agent might say: “Thanks, I captured your request and will route it to our support team for review.” The trace may show that it classified the issue as “billing” instead of “refund,” failed to copy the order ID into the handoff, and marked the user’s frustration as “low” despite the message “I’ve asked about this three times already.”

Reading only the final answer misses the first bad move. A malformed handoff at turn five may trace back to a misclassification at turn three.

4. Checks and judges

Each important failure mode should become either a check or a judge.

• A check is a function that examines a trace and returns pass or fail without an LLM call. Use checks for structural failures: refund language appears, a required field is missing, the handoff schema is invalid, or the order ID was dropped.
• A judge is a separate LLM that examines a trace and returns a verdict on something a function cannot reliably decide. Use judges for semantic failures: the clarifying question was redundant, the tone was wrong, or the agent answered the wrong question.

For example, “the agent must never promise a refund” should be a code check:

def evaluate_no_refund_promise(trace) -> Score:
    response = trace.outputs.get("agent_response", "")
    matches = find_matches(response, REFUND_PROMISE_PATTERNS)

    if matches:
        return Score(
            passed=False,
            reason=f"Refund-promise language: {matches}",
        )

    return Score(passed=True, reason="No refund-promise language")

Checks are cheap and deterministic. Run them on every important change.

Judges need calibration: run the judge on traces humans have already labeled, then look at where it agrees, where it misses failures humans caught, and where it flags cases humans considered acceptable. Calibration tells you whether the judge is reliable enough to support the decision you want it to support.

Don’t ask an LLM judge what code can check focuses on when a failure should become a code check versus an LLM judge. Your LLM judge is a classifier goes deeper on judge calibration.

The smallest useful promise

Having a minimal eval loop in place is enough to change how the team builds. “Does v2 feel better?” becomes “Which failure modes improved, which regressed, and do we trust the measurement?” Eval stops being a side research exercise and becomes part of the engineering loop.

The harness made the agent’s behavior observable, the eval system turns that behavior into evidence, and the hardening loop turns evidence into a system harder to break.

The next article goes one layer deeper: the eval dataset.