Evaluating LLM and agent systems

Generated by Claude through GigAI 0.1.11.10 on 2026-10-11.

Concept: Model evaluation · Step 20 of 34 in AI forward deployed engineer · version 1

Practice this

On this page

Evaluation turns a vague sense that the system seems fine into a number tied to defined success criteria, so prompts, models and architectures can be changed with evidence. A working eval practice has three layers: fast assertions in code, model-graded and human-graded quality checks, and measurement on real usage. For agents the unit under test is the whole trajectory and the final state of the world, not only the last message.

Read more

How it is done in practice

On a customer engagement the eval set is agreed with the customer early, because it doubles as the acceptance definition for the pilot. The team collects a few dozen real cases with the customer's domain expert, writes code assertions first, adds a calibrated model judge for the qualities code cannot check, and wires both into CI so every prompt or tool change shows a before and after. During the pilot someone reviews a sample of traces each week with the expert, adds new failure cases to the set, and reports pass rates by failure category instead of one headline score.

Technologies

OpenAI EvalsAnthropic Console evaluation toolpytestLangSmithBraintrustpromptfooInspect AI

Code

python
import json, statistics

def grade(case, trace):
    calls = [s["tool"] for s in trace["steps"] if s.get("tool")]
    return {
        "outcome": trace["final_state"] == case["expected_state"],
        "no_forbidden": not set(calls) & set(case["forbidden_tools"]),
        "within_budget": len(trace["steps"]) <= case["max_steps"],
    }

def run_suite(cases, run_agent, trials=3):
    report = {}
    for case in cases:
        passes = [all(grade(case, run_agent(case["input"])).values())
                  for _ in range(trials)]
        report[case["id"]] = statistics.mean(passes)
    return report

print(json.dumps(run_suite([], lambda x: {}), indent=2))

Subtopics

Success criteria and test sets

What goes beyond a gold set

Criteria should be specific, measurable and tied to the customer's task, usually several at once: task accuracy, format validity, tone, latency, cost per case. Build the set from the real task distribution plus deliberate edge cases such as ambiguous input, irrelevant input and very long input. Volume with automated grading generally beats a small hand-graded set. Models can generate additional cases from a seed set, which a person then spot-checks for realism.

Code, model and human graders

Choosing a grader

Use the fastest reliable grader that fits. Code-based grading, such as exact match, string or regex checks and schema validation, is cheap and deterministic. Model-based grading handles nuance: give the judge a detailed rubric, ask it to reason before deciding, and force a constrained verdict such as pass or fail or a short scale. Human grading is the most flexible and the slowest, so reserve it for calibration.

Keeping a judge honest

A model judge is itself a system that needs evaluation. Have a domain expert label a sample with pass or fail and a written critique, run the judge on the same sample, and measure agreement; revise the rubric until the two align, then re-check periodically. Binary judgements with reasons are easier to align than numeric scores. Where possible use a different model from the one being graded, and keep judge prompts in version control with the eval results they produced.

Evaluating agent trajectories

What to measure

An agent can reach a right answer by a wasteful or unsafe path, or a wrong answer after correct steps, so grade at several levels. Outcome checks inspect final state: was the ticket created with the right fields, did the record change as intended. Trajectory checks inspect the trace: were the expected tools called with valid arguments, were forbidden tools avoided, how many turns and tokens were used. Component checks isolate one step, such as routing accuracy or retrieval quality. Prefer outcome checks where several valid paths exist, since strict step matching penalizes legitimate alternatives.

Harness needs

Agent evals need a controlled environment: seeded test data, sandboxed or mocked tools with realistic responses, and reset between cases so runs are independent. Because agent behavior varies between runs, repeat each case several times and report a pass rate instead of a single result. Multi-turn tasks need a simulated user, often another model following a scripted persona and goal. Store full traces with each result so a failing case can be opened and read step by step.

Error analysis and evals in the delivery loop

From traces to tests

The highest-value activity is reading traces. Sample real or pilot interactions, write a short note on what went wrong in each, group the notes into failure categories, and count them; the largest category sets the next piece of work. Each confirmed failure becomes a regression case. Unit-style assertions run on every change in CI, model-graded suites run on a schedule or before release, and a sample of production traffic is scored continuously to catch drift. A simple viewer that shows the trace beside the grade removes friction and gets domain experts involved.

Sources

Related lessons

Practice this
Agent orchestration and durable executionResponsible AI controls and guardrails