Why Your Automation Broke on a Tuesday

Nobody’s automation fails during the demo. It fails eleven weeks later, on a Tuesday, when a third-party API starts returning null in a field that has always been a string, and your pipeline cheerfully writes eight hundred empty records into a CRM that a salesperson is about to open.

I have built roughly six hundred of these. The failures are not creative. They are the same four failures, and they have the same four fixes.

1. The duplicate that ran twice

Webhooks are delivered at least once. Not exactly once. Every provider you integrate with will, eventually, deliver the same event twice — usually because their first delivery attempt timed out on your end after you had already processed it.

If that event charges a card, sends an email, or increments a counter, you now have a support ticket.

The fix is an idempotency key computed from the payload itself, checked against a store before anything irreversible happens:

const key = createHash("sha256")
  .update(`${payload.id}:${payload.updated_at}`)
  .digest("hex");

if (await seen.has(key)) return { status: 200, note: "duplicate" };
await seen.set(key, true, { ttl: 60 * 60 * 24 * 7 });

Two nodes. It prevents the entire class.

2. The retry that made it worse

The default instinct is to wrap the failing step in a retry. This is correct roughly half the time, and actively harmful the other half, because retrying a partially-applied multi-step operation re-applies the steps that already succeeded.

Failures belong in a dead-letter queue, not a retry loop. Route the exception out of the main thread with three things attached: the original input, the execution ID, and the error. Then notify a human with a one-click replay link. You keep the data, you keep the ability to fix and re-run, and you stop the pipeline from thrashing against an API that is down.

3. The schema that changed under you

This is the Tuesday failure. An upstream provider ships a change, a field goes from string to null, and because most automation tools are permissive by default, the bad value propagates all the way to your database.

Put a validator immediately after every external call. Not a big one — a shape check:

const Contact = z.object({
  id: z.string(),
  email: z.string().email(),
  company: z.string().min(1),
});

When validation fails, the item goes to the review queue and the run pauses. A paused pipeline is a five-minute inconvenience. A poisoned database is a weekend.

4. The autonomy nobody asked for

The last failure is a design failure rather than an engineering one: the workflow was allowed to do something irreversible without anyone agreeing to it.

My rule is that anything which spends money, contacts a customer, or deletes a record gets a human gate by default. Not forever — you move the gate once the numbers earn it. But the first version of every pipeline proposes and waits.

The uncomfortable part

None of this is interesting. It is four structures, they add maybe fifteen percent to the build, and they are the entire difference between an automation you trust and one you check every morning.

If you are evaluating someone to build these for you, ask them to describe their error path. If the answer is “it retries,” keep looking.