Observability for Agents: Traces, Scores, and the Dashboard Ops Actually Reads

If your only view into an agent is a model-provider chart, you will miss the expensive failure mode: runs that look healthy, spend money, and ship wrong work. Observability for agents is traces, scores, and an ops cadence — not a graveyard of JSON in a bucket.

This spoke belongs to the Agentic Systems Operating Manual. It assumes evaluators, state machines, and cost controls exist so there is something meaningful to observe.

What AI agent observability must show

For each run, a human should answer without spelunking:

1. What job was this?
2. Which state did it die in?
3. Which tools ran, with what redacted args/results?
4. What did the evaluator say, with evidence?
5. How much did it cost, and how many revisions?
6. Did it done, escalate, or abort — why?

If any answer requires downloading a raw prompt dump by default, the UX failed.

Tracing LLM tool calls

A useful trace is a tree or span list:

• run (job_id, tenant, job_type)
  • state:intake
  • state:plan → model span (model id, tokens, latency)
  • state:act → tool:crm.get → tool:enrichment.lookup
  • state:evaluate → mechanical checks + model judge spans
  • state:revise …
  • terminal span with reason code

Include:

• Stable ids for run, parent, and tool call
• Model name/version and token usage when provided
• Tool name, side-effect class, latency, error codes
• Pointers to artifacts (URIs), not always full payloads
• Redaction policy applied at write time

Do not mark a run successful because the HTTP layer returned 200 if the evaluator failed. Terminal status is evaluator/human authority.

Scores: offline and online

Offline: golden-set pass rate, cost per pass, revision depth — on every meaningful change.

Online: sample production runs through the same evaluator. Chart:

• Pass / fail / escalate rates
• Silent-fail samples (human overrides)
• Cost bands by job_type
• Drift after deploys

Alert when online pass rate drops versus the trailing baseline, or when cost per pass spikes. That is how you catch prompt regressions and bad retrieval indexes.

The dashboard ops actually reads

Keep one screen for the weekly meeting:

Panel	Purpose
Runs by terminal state	Are we escalating more?
Pass rate (online sample)	Quality
Cost per pass	Unit economics
Top evaluator failure codes	Where to fix
Kill-switch / budget events	Control plane health
p95 latency	SLO, secondary

Anything that needs a data scientist to interpret will not get read. Link out to full traces for incidents.

Logging hygiene

• Never log secrets or raw credentials
• Redact PII by default; allow break-glass access with audit
• Separate debug verbosity from production verbosity
• Retain traces long enough for disputes and model comparisons; document retention

Observability that leaks customer data is a security incident with charts.

Tracing across multi-agent handoffs

Propagate trace_id / run_id through handoff packages. Each agent adds spans under the same run. If Agent B starts a new unrelated id, you will never reconstruct the story.

What vendor “LLM observability” gets wrong for business agents

Many tools stop at prompt/response capture. Necessary, insufficient. Business agents need state names, tool side-effect classes, evaluator verdicts, and budget events in the same timeline. Buy or build toward that model; do not confuse token charts with ops readiness.

Incident response using traces

When something bad ships:

1. Find run ids in the write system’s audit (CRM, email, tickets)
2. Open the trace; identify first bad tool call or failed criterion ignored
3. Freeze writes if pattern is broad (kill switch)
4. Patch evaluator or sandbox; add a golden case
5. Replay the suite before re-enabling autonomy

Blameless for humans; ruthless for missing criteria.

Pilot minimum

Even a $1,500 · 5-day Spurlock Studios pilot ships a thin observability slice: structured run logs, evaluator verdicts, cost, and terminal reason. Full dashboards can wait for a build; blindness should not.

/agentic · /contact?intent=agentic-pilot

Anti-patterns

Logs only on error. You need successes too for baselines.

Storing full prompts forever in Slack. Wrong system, wrong retention.

Metrics without owners. Every panel needs a human who acts on it.

Tracing only the model, not tools. Most business damage is a tool call.

Sampling strategies that do not lie

Tracing 100% of runs is ideal at low volume and expensive at high volume. When you sample:

• Always keep 100% of escalate, abort, budget trips, and irreversible writes
• Sample passes, but stratify by job_type and tenant size
• Upsample after deploys for 48 hours

Sampling only successes to “save money” hides the story.

Correlating with business systems

Store external receipt ids (CRM note id, ticket comment id) on the trace. When a salesperson says “the agent wrote nonsense,” you jump to the run in seconds. Without correlation ids, observability is a museum.

Privacy reviews

Before enabling full prompt capture in prod, run a privacy review: what PII appears, who can access, retention, export paths. Prefer artifact URIs + hashes over duplicating sensitive payloads into a third-party SaaS by default.

Weekly ops ritual (30 minutes)

1. Glance terminal-state mix
2. Open top three failure codes — decide fix owner
3. Check cost per pass vs band
4. Review one escalate package end-to-end
5. Note any kill-switch events

Ritual > giant platform. Spurlock Studios installs the thin version during the $1,500 pilot so the ritual has data: /agentic. Broader context: operating manual.

Reason codes catalog

Standardize terminal reason codes: eval_pass, eval_fail_exhausted, budget_exhausted, tool_auth_error, policy_violation, human_reject, timeout, out_of_scope. Dashboards group on these. Free-text reasons make trends impossible.

Comparing prompts scientifically

When testing prompt A vs B, run the same golden set, same tool stubs, same budget. Report pass rate, cost, latency. Store the trace batch under an experiment id. Intuition-only prompt merges are how regressions ship.

On-call primer

On-call owns kill switches, credential rotations, and “freeze writes” decisions. Model quality tweaks wait for business hours unless an active incident is ongoing. Write that sentence into the runbook before launch.

Pilot installs the minimum signal — /agentic · manual.

Tracing LLM tool calls across vendors

Normalize provider-specific usage fields into your span schema. You will switch models; your dashboards should not require a rewrite each time. Store raw provider payloads as optional debug attachments with stricter retention.

User-reported wrongness loop

Add a “report wrong output” control that captures run_id. That button is worth more than three vanity charts. Route reports into a weekly review and into golden-set candidates.

SLOs for agents

Example: 99% of runs reach a terminal state within 15 minutes; <1% abort for unknown errors; online pass rate ≥ offline − 5 points. SLOs make AI agent observability actionable.

Minimum viable traces ship in the pilot — /agentic.

Field workbook: standing up traces in a week

Day 1: define the span schema (run, state, model, tool, eval, terminal). Day 2: emit spans from the runner with redaction. Day 3: build a single dashboard with six panels. Day 4: wire alerts for budget trips and pass-rate drops. Day 5: rehearse an incident using a deliberate bad deploy in staging.

This workbook mirrors how Spurlock Studios approaches thin observability inside an agentic pilot. You do not need a perfect platform to start; you need correlated run ids and evaluator scores beside tool calls.

What “good enough” looks like for tracing LLM tool calls

For each tool span, store name, side-effect class, duration_ms, error_code, arg_hash, result_hash, and a redacted preview limited to a few hundred characters. Hashes let you prove two runs touched the same payload shape without retaining PII forever. Previews let humans debug without opening cold storage.

Model spans store provider, model id, input_tokens, output_tokens, latency_ms, and cache_hit if known. If the provider omits usage, estimate with a documented formula and mark usage_estimated=true so finance does not treat it as gospel.

Quiet failures and how observability catches them

The dangerous agent run returns 200, writes a plausible CRM note, and fails a soft criterion nobody watches. Online sampling through the evaluator is the countermeasure. Pair it with a business-side audit: once a week, a domain reviewer rates twenty random notes. Disagreement between human and evaluator is itself a signal — either criteria drift or judge drift.

Dashboard copy for non-engineers

Label panels in business language: “Jobs finished cleanly,” “Jobs sent to a human,” “Jobs stopped for budget/policy,” “Average cost when successful,” “Top reasons for human handoff.” Avoid model jargon on the primary screen. Deep links remain for engineers.

AI agent observability is how trust scales. Without it you are scaling hope. Continue with cost controls and the operating manual. When you want this wired on a real job in five days, use the $1,500 pilot on /agentic or /contact?intent=agentic-pilot.

Closing note on the weekly read

AI agent observability fails when nobody looks. Put a thirty-minute ritual on the calendar with named owners. Tracing LLM tool calls matters only if a human opens a failing span and changes criteria, tools, or budgets. Install the minimum in the pilot, then grow the dashboard as volume grows — /agentic · operating manual.

One more operating rule

Correlate CRM write ids to run ids on day one — even before pretty charts. When sales forwards a bad note, you should open the trace in under a minute.

If a panel has no owner, delete it. Orphan metrics create false comfort and burn attention.

FAQ

What is AI agent observability?

It is the practice of recording and reviewing runs with enough structure — states, model calls, tool calls, evaluator scores, cost, and terminal reasons — to debug, govern, and improve agents in production.

How do you trace LLM tool calls well?

Create spans for each model and tool invocation under a stable run id, record redacted I/O, latency, errors, and side-effect class, and align terminal status with evaluator outcomes rather than HTTP success alone.

Which metrics matter most week to week?

Online pass rate, escalate rate, cost per passing run, top failure codes, and budget/kill-switch events. Latency matters, but after correctness and cost.

Do we need a special vendor on day one?

Not always. Structured logs plus a simple dashboard can cover a pilot. As fleets grow, specialized tracing tools help — if they ingest tool and evaluator events, not only prompts.

How does Spurlock Studios handle observability?

Thin traces and scores ship in the pilot; richer operator dashboards land in fuller builds. The stack is described in the operating manual.

How does observability relate to evaluators?

Evaluators produce the quality signal; observability stores and surfaces it beside cost and tools. Without evaluators you are tracing activity, not correctness.