Idempotent Agent Tool Writes: Retries Without Double Emails or Double Charges

When an agent tool write times out, the dangerous question is not “did the model fail?” — it is “did the side effect already land?” Make writes safe by minting a stable idempotency key in the runtime before any retry layer can fire, then reusing that same key for model retries, harness retries, and HTTP client retries.

This spoke sits inside the Agentic Systems Operating Manual. The n8n workflow pattern lives in Idempotency Keys in n8n; this post owns the agent case — stacked retry layers and keys the model must never invent.

The short answer

• Timeouts are ambiguous: the upstream may have committed while your agent saw a network error.
• Agents stack retries (model loop + harness + HTTP). One timed-out write can become two charges or two emails.
• Birth the key in the runtime from (run_id, tool_name, intent_fingerprint), not in the prompt.
• Prefer native Idempotency-Key headers when the API supports them; otherwise use a local ledger + dedupe gate before the write.
• Test duplicate delivery in staging with forced timeouts before you grant write autonomy.

What is the agent-specific idempotency failure mode?

Workflow automation usually has one retry owner (the workflow engine). Agents have three:

Layer	What retries	Typical trigger
Model loop	“Tool failed, try again”	Tool error text in the next turn
Harness / runner	Re-invoke the act step	Timeout, crash, checkpoint resume
HTTP client	Same request again	408/429/5xx, connection reset

If each layer invents its own “try again” without a shared key, a single ambiguous timeout becomes stacked side effects. That is the agent-specific failure — not “webhooks can fire twice,” but “three systems each think they are being helpful.”

Why “only call once” in the prompt fails

Prompts do not control TCP. A model that obediently “calls send_email once” still loses when:

1. The HTTP call hangs past the client timeout after the provider accepted the message.
2. The harness resumes the run after a deploy and re-enters state:act.
3. The model sees a generic timeout string and emits a second tool call with slightly different arguments.

Instructional discipline is not a transport guarantee. Treat “call once” as documentation for humans, not as a safety control.

How do I make agent tool writes safe when the call times out?

Procedure that holds up in production:

1. Classify the tool as read, write_idempotent, or write_irreversible before registration.
2. Mint a key in the runtime when the act step decides to call a write tool — before the HTTP request starts.
3. Persist key → status (pending | succeeded | failed_poison) in a ledger keyed by tenant.
4. Pass the same key into every retry of that logical write: harness replay, HTTP retry, and any model re-emit for the same intent.
5. On timeout: leave status pending (or unknown), do not mint a new key, and either poll for receipt or escalate — never “just send again” with a fresh identity.
6. On success: store upstream receipt id beside the key; mark succeeded.
7. On definitive failure (4xx that will not succeed on retry): mark failed_poison so retries stop.

Timeout means unknown. Unknown means reuse the key or escalate — never invent a second write identity.

How do I generate stable keys across retry layers?

Key material should be stable for the business intent, not for the HTTP attempt:

key = hash(tenant_id + run_id + tool_name + intent_fingerprint)

intent_fingerprint is a canonical hash of the fields that define the side effect (to, template_id, invoice_id, amount_cents) — not of ephemeral fields like requested_at or random UUIDs the model invents.

Source of key	Safe?	Why
Model-generated UUID in tool args	No	New UUID on every re-emit
HTTP attempt id	No	New per transport retry
Runtime: run_id + tool + intent hash	Yes	Survives all three layers
Upstream event id (when writing because of an event)	Yes	Aligns with business identity

Store the key on the tool span so observability can prove which retries shared identity.

Where the key is born — model vs runtime

Birthplace	Outcome
Model fills idempotency_key	Model invents a new key after timeout; duplicates ship
Runtime injects key into tool call envelope	Retries reuse identity even if the model rephrases args
Runtime + schema forbids model override	Strongest: model cannot “helpfully” rotate the key

Default: the harness owns the field. If the tool schema exposes idempotency_key, strip or overwrite model-supplied values before dispatch.

What if the upstream API has no Idempotency-Key header?

Many CRM, email, and internal APIs do not speak Stripe-style idempotency. Options, in order of preference:

1. Native unique constraint — if the API accepts a client-supplied external id (external_id, reference, client_ref), use your key there.
2. Pre-write ledger gate — before calling the API, claim the key in your DB with a unique index. If claim fails because status is succeeded, return the stored receipt and skip the call. If pending and younger than TTL, wait/poll; if older than TTL, escalate.
3. Read-before-write with stable lookup — only when the domain has a natural unique query (invoice already paid, ticket already has comment hash X). Fragile; document the race.
4. Outbox + single worker — enqueue the write once; a single consumer performs the HTTP call. Agent retries enqueue the same outbox id.

Do not pretend a header exists. Build the ledger. The n8n spoke covers workflow dedupe storage patterns; agents need the same idea on the tool boundary.

Read tools vs write tools — retry rules

Tool class	Retry on timeout?	Key required?
Read / search	Yes, usually safe	Optional (cache key helps)
Write with server idempotency	Yes, same key	Required
Write without server idempotency	Only after ledger claim or escalate	Required locally
Irreversible external (wire, legal notice)	Human or outbox only	Required + approval

Checklist before marking a tool write_idempotent in the registry:

• Side-effect class documented
• Key birthplace = runtime
• Ledger or native unique field wired
• Timeout path leaves status unknown/pending, not failed
• Forced duplicate-delivery test exists

Compensating actions that stay idempotent

Compensations (void charge, send apology, delete draft) are also writes. They need their own keys, derived from the original:

compensate_key = hash(original_key + ":compensate:" + action)

Rules:

1. Never compensate twice for the same original key.
2. Never compensate if the original write status is still pending — resolve unknown first.
3. Log compensation under the same run_id with a distinct tool span.

Blind “undo” loops are how you get a charge, a void, and a second charge.

Failure example: double invoice email

Job: Agent drafts and sends invoice reminder for inv_8841.

What happened:

1. Runtime minted no key; model called email.send.
2. Provider accepted the message; client timed out at 30s.
3. Harness retried state:act. Model called email.send again with a new message_id it invented.
4. Customer received two reminders; support spent a day on “your system is broken.”

Cost: trust, not just SMTP fees.

Fix:

• Runtime key: hash(tenant + run + email.send + inv_8841 + template_reminder_v2)
• Ledger claim before SMTP
• On timeout, poll provider by key/metadata or escalate — do not re-emit with a new message id

How do I test duplicate delivery before production?

Staging drills that catch the stacked-retry bug:

1. Inject latency past the HTTP timeout after the mock server records the write.
2. Confirm harness retry reuses the same key and the mock sees one logical commit.
3. Force model re-emit by returning a fake timeout string once; assert second tool call carries the injected key (or is blocked).
4. Crash mid-pending and resume from checkpoint; assert no second charge.
5. Poison 409/duplicate from upstream; assert agent treats as success-with-receipt, not endless retry.

Drill	Pass criterion
Slow success + client timeout	Exactly one side effect
Double harness resume	Ledger blocks second HTTP
Model invents new args, same intent	Same key; one effect
Upstream duplicate error	Maps to succeeded

If you have not run the timeout drill, you have not tested agent writes.

Ledger fields that belong on the trace

Put these on the tool span and in the ledger row:

Field	Purpose
idempotency_key	Shared identity across retries
intent_fingerprint	Prove which args defined the write
status	pending / succeeded / failed_poison / unknown
attempt	Transport attempt count (not a new key)
upstream_receipt_id	Correlate to CRM/email/payment
first_seen_at / succeeded_at	Dispute timeline
run_id / tool_call_id	Join to agent trace

Without receipt correlation, ops cannot answer “which run sent the second email?”

Interaction with state machines and durable runners

If you use explicit states (state machines for agent loops), store the key on the act transition. Checkpoint resume must reload pending keys — a durable runner that forgets them is a double-write machine with extra steps.

Anti-patterns

UUID in the prompt template. Guarantees uniqueness per emit — the opposite of idempotency.

Retrying irreversible tools on any error string. Distinguish timeout/unknown from validation_failed.

Per-layer keys. Model key ≠ harness key ≠ HTTP key means three charges.

Deleting ledger rows on failure. If the write may have landed, keep the key until you know.

Treating HTTP 200 as the only success. Some APIs return errors after committing; prefer receipt ids.

Decision list: ship write autonomy?

Ship autonomous writes only when all are true:

1. Tool is classified and keyed in the runtime.
2. Upstream supports idempotency or local ledger gate is live.
3. Timeout drill passed in staging.
4. Evaluator or policy gate can block high-risk tools (operating manual).
5. Kill switch can freeze the write tool class without redeploying prompts.

If any box is open, keep the tool behind human approval or an outbox.

Worked ledger claim (pseudo)

claim(key):
  insert ledger(key, status=pending) on conflict do nothing
  if conflict and status=succeeded: return cached_receipt
  if conflict and status=pending and age < TTL: wait or escalate
  if conflict and status=pending and age >= TTL: escalate unknown
  if inserted: call upstream with Idempotency-Key=key (or external_id=key)
  on success: status=succeeded, store receipt
  on timeout: leave pending, schedule resolve job
  on hard 4xx: status=failed_poison

Agents call claim through the tool adapter — never raw HTTP from the model. Hash intent fields (to, template_id, invoice_id); do not put raw customer bodies into the key string.

Pilot minimum

A Spurlock Studios $1,500 · 5-day agentic pilot that includes write tools ships: runtime key injection, a thin ledger, timeout classification, and one forced-duplicate drill in staging — not a promise that “the model will be careful.”

/agentic · /contact?intent=agentic-pilot

FAQ

How is this different from n8n idempotency keys?

n8n idempotency keys dedupe workflow executions and webhook redeliveries inside an automation graph — see Idempotency Keys in n8n. Agent idempotency keys dedupe tool writes across model loops, harness resumes, and HTTP clients. Same idea, different boundary: the tool adapter, not the workflow trigger.

Read tools vs write tools — retry rules?

Reads can usually retry freely; writes need a stable key and a ledger or native idempotency before any retry. Irreversible writes should escalate or use a single-consumer outbox when status is unknown after timeout.

Where should the key be born — model or runtime?

Runtime. Keys born in the model get rotated on every re-emit after a timeout, which causes the duplicates you are trying to prevent. Inject and overwrite at the harness boundary.

What if the upstream API has no Idempotency-Key header?

Use a client-supplied unique field if the API has one, or claim the key in your own ledger before the call and skip/replay from stored receipts. Do not invent a header the vendor ignores.

How do compensating actions stay idempotent?

Derive a compensation key from the original key plus action name, refuse to compensate while the original is still pending, and record compensation on the same run trace so you never void twice.

What ledger fields belong on the trace?

At minimum: idempotency_key, intent_fingerprint, status, attempt, upstream_receipt_id, timestamps, and run_id. Those fields let ops prove one logical write across stacked retries.

CTA

Timeouts without keys are how agents earn a reputation for double-billing. Wire runtime idempotency before you widen write autonomy — /agentic · /contact?intent=agentic-pilot.