Build the Evaluator Before the Agent

If you ship the agent before the evaluator, you are guessing. Accuracy jumps I trust never came from a warmer system prompt. They came from a second component whose only job is to disagree with structured evidence.

This spoke sits under the Agentic Systems Operating Manual. Read that for the full stack. Here we only do evaluation: what it is, how to test AI agents without fooling yourself, and how Spurlock Studios wires it into a five-day pilot.

Why self-grading fails

A model asked to check its own output is continuing a story in which it already “finished.” The verdict is correlated with the work. In production that looks like green runs where the tool errored, the summary soft-pedaled the error, and nobody looked at the trace.

Self-grading is useful as a hint inside a worker. It is not your metric. Your metric is an independent judge with a narrower context: acceptance criteria + artifact (+ retrieval evidence if the job requires citations). Not the worker’s private reasoning transcript.

What an evaluator is

An evaluator is a component — code, model, or both — that returns a structured verdict:

{
  "verdict": "fail",
  "failures": [
    {
      "criterion": "severity_present_or_explicit_no_match",
      "evidence": "summary claims policy X with zero citation URLs"
    }
  ],
  "next": "re-retrieve with query focused on refund policy; rewrite summary"
}

Rules that make it real:

1. Independent context. Do not feed the worker’s chain of thought.
2. Mechanical first. Schema, enums, regex allowlists, test suites, arithmetic checks — assert in code.
3. Model only for judgement. Tone, omission of material risk, “did this answer the question asked.”
4. Evidence, not vibes. “Not good enough” is useless. “Criterion 3 failed because…” is actionable.
5. Revision ceiling. Usually three. Then escalate with the full package.

How to test AI agents (a practical harness)

1. Write criteria before tools

Sit with the buyer. Translate “good” into pass/fail lines. If you cannot, you are not ready to build. Ambiguous taste is a product workshop, not an agent ticket.

2. Build a golden set

Thirty to one hundred real jobs beat a thousand synthetic toys. Include traps: empty retrieval, contradictory docs, hostile or weird inputs, partial tool failures. Label expected severity of failure (hard fail vs soft fail).

3. Split offline and online

Offline: run the suite on every prompt, model, or tool change that could affect behavior. Online: sample production, score with the same evaluator, alert on drift. Offline without online is academic. Online without offline is firefighting.

4. Track the right numbers

• Pass rate on the golden set
• Average revisions to pass
• Cost per passing run
• Escalate rate
• Silent-fail rate (pass online, human later marks wrong) — expensive to measure, worth sampling

Do not celebrate latency alone. Fast wrong is still wrong.

5. Version the evaluator

When criteria change, version them. A jump in pass rate after you loosened criteria is not a model win. Treat evaluator changes as carefully as worker changes.

Mechanical vs model judges

Check type	Examples	Owner
Schema	JSON matches Zod/JSON Schema	Code
Business rules	Totals equal line items; status in enum	Code
Retrieval contract	Citation required or explicit no-match	Code + light model
Quality judgement	Summary completeness; email tone	Model evaluator
Safety	Banned promises; PII patterns	Code + allowlists

Bias toward code. Models are for the residue.

Wiring evaluation into the loop

Control flow that works:

1. Worker produces artifact in a non-final state.
2. Evaluator runs with criteria + artifact.
3. Pass → allow write-back / terminal success.
4. Fail → attach evidence → worker revises.
5. Ceiling hit → escalate to human with trace, cost, and failures.

Never let the worker mark the run done. Terminal success is an evaluator privilege (or a human override).

Common evaluation mistakes

Rubrics that are essays. If a criterion needs a page of interpretation, split it or delete it.

One number for everything. A single “quality score 0–100” hides which contract broke. Prefer binary criteria plus optional severity.

Training on the golden set in prompts. Do not paste the answers into the worker prompt and call it improvement. That is memorization.

Evaluator with write tools. The judge must not mutate production systems. Read-only at most.

No adversarial cases. If every golden item is sunny-path, your suite will green-light fragile agents.

What this looks like in a Spurlock Studios pilot

The $1,500 · 5-day agentic pilot starts with criteria, not with a toy chatbot. Day one is job contract and evaluator shape. Mid-week is worker + sandbox against a small golden set. End of week is a passing path on your data, with escalate wired, and a quote for hardening.

You keep the agent either way. Details: /agentic. Start: /contact?intent=agentic-pilot.

Relationship to the rest of the stack

Evaluators without sandboxes still let bad actions through between grades. Evaluators without state machines still loop forever. Evaluators without cost caps still burn money failing. Evaluation is first in design order, not the only layer. The parent manual maps the rest.

If you only remember one sentence: the evaluator is how you know whether anything else worked.

Building the first golden set without boiling the ocean

Teams stall on evaluation because they imagine needing a thousand labeled cases before writing code. You need enough cases to catch the failure modes you already know, plus a few you do not.

Start with twenty. For each case record: raw input (ticket text, lead payload, transcript excerpt), the artifact you expect (or the properties it must have), and the traps (empty knowledge base, contradictory policy, tool timeout). Label three as “must never auto-pass.” Those become regression anchors.

Expand to fifty once the worker exists and you see new failure classes in the wild. Retire cases that no longer reflect the job. A golden set is a living product artifact, not a museum.

Who labels? A domain reviewer who will feel the pain of wrong outputs — not only an engineer who owns the prompts. Disagreement between reviewers is useful: it means your criteria are still ambiguous. Resolve ambiguity in the criteria document before you ask models to guess.

Calibrating model judges

When a criterion needs a model judge, calibrate it like a sensor. Take thirty artifacts with human labels. Run the judge. Measure agreement. Where it disagrees, either tighten the criterion into something code can check, or add few-shot exemplars of pass and fail to the evaluator only — never paste the full golden answers into the worker.

Watch for leniency drift: judges that learn to pass because the worker’s prose is confident. Periodically inject known-bad artifacts. If they pass, freeze deploys until the judge is fixed.

Organizational adoption

Evaluation fails socially when leadership rewards demos over scores. Make pass rate and cost per pass visible in the same meeting as feature launches. Refuse to widen tool allowlists when scores regress. Treat evaluator version bumps as release notes.

Spurlock Studios carries this discipline into every agentic engagement because without it the rest of the operating manual is decoration. If you want the five-day proof on your data, the pilot at $1,500 exists to install criteria first — details on /agentic.

Worked micro-example: invoice line extraction

Job: extract line items from supplier PDFs into JSON for AP.

Mechanical criteria: schema valid; quantities positive; currency in allowlist; sum of lines equals stated total within one cent; vendor_id present in ERP or flagged unknown_vendor.

Model criteria (narrow): description fields are not empty boilerplate; if PDF is handwritten and unreadable, verdict is fail with unreadable_source rather than inventing lines.

Revision policy: one re-parse with a different OCR path, then escalate. No third fantasy pass.

That evaluator can be half code. The agent (or even a simpler pipeline) becomes measurable overnight. Most “we need a bigger model” requests on this job die when the totals check is enforced.

Scoring rubrics that stay binary

Resist 1–5 stars as your primary system. Stars feel nuanced and hide which contract broke. Prefer a list of binary criteria with optional severity tags (blocking vs soft). Soft failures can pass the run but open a ticket for human style review; blocking failures force revise or escalate.

Example soft: “summary longer than 120 words.” Example blocking: “refund promise without policy citation.”

Publish the rubric where domain reviewers can edit it through a change process. Shadow edits in private docs recreate self-grading at the organizational layer.

Adversarial cases worth including

• Instructions inside user content attempting to override criteria
• Empty or whitespace-only inputs
• Mixed languages
• Partial tool success (one of three calls failed)
• Out-of-policy requests (“just approve the refund”)
• Near-duplicate jobs that should be idempotent

If your suite is only happy paths, your pass rate is a vanity metric.

Shipping cadence tied to scores

Define release trains: worker changes cannot ship if golden-set pass rate drops more than an agreed epsilon or cost per pass exceeds band. Emergency hotfixes must add a case within 48 hours. This is how evaluation becomes the product rather than a slide.

Connect to sandboxes and state machines next; then prove on a $1,500 pilot — /agentic, /contact?intent=agentic-pilot.

FAQ

What is AI agent evaluation?

AI agent evaluation is the practice of judging agent outputs against explicit acceptance criteria using an independent process — preferably mechanical checks plus a separate judge — and measuring pass rates on fixed suites and sampled production runs. It is not the agent saying “looks good.”

How do you test AI agents before production?

Build a golden set of real jobs with pass/fail criteria, implement an independent evaluator, run the suite on every meaningful change, and keep a human escalate path. Soft-launch with write gates until online scores match offline expectations.

Should the evaluator be another LLM?

Often partly. Use code for anything assertable. Use a model evaluator when judgement is required — and still force structured output with per-criterion verdicts and evidence. Same-vendor or cross-vendor both work; independence of context matters more than logo diversity.

How many revision attempts should an agent get?

Three is a strong default for pilots. Some jobs warrant one (high cost side effects). Some warrant five (cheap drafts). The number must be explicit and enforced by the state machine, not left to the model’s optimism.

When is evaluation “good enough” to widen autonomy?

When golden-set pass rate, cost per pass, and escalate rate meet the bar you set with the buyer — and online samples hold for a defined window. Autonomy is earned by scores, not by demo applause.

How does this fit Spurlock Studios engagements?

Every agentic pilot and build assumes an evaluator harness. If a prospect wants agents without criteria, we push them toward automation or a scoping workshop first. See the operating manual and /agentic.