Engineering Leadership Software Development

Published Jul 18, 2026

Idempotency Is the Unsung Hero of Enterprise Software

Idempotency decides whether "twice" is safe. Production Notes #06 on making duplicates harmless before retries, webhooks, and humans force the question.

Quick Brown Fox9 min

Production Notes #06 · Part of Binary and Beyond. LinkedIn newsletter edition follows.

No one puts idempotency on a roadmap.

It never appears in a client brief. It does not demo well. You cannot screenshot it for a board deck. It has an awkward name that half the team pronounces differently.

And yet, after years of building enterprise systems, it is one of the few properties I now treat as non-negotiable on any write path that touches money, inventory, or a customer promise.

Because idempotency is the difference between a system that tolerates the real world and one that quietly corrupts itself the first time something happens twice.

The word that hides a business decision

Idempotency has a clean technical definition: an operation is idempotent if performing it multiple times produces the same result as performing it once.

Read a record — idempotent. Set a value to true — idempotent. Add 10 to a balance — not idempotent.

That definition is correct and almost useless in a planning meeting, because it sounds like an API implementation detail.

It is not.

Idempotency is a business decision wearing a mathematical costume. It answers a question every enterprise process eventually asks under pressure:

If this happened twice, do we charge the customer twice, ship twice, invoice twice — or exactly once?

That is not a question for the HTTP client.

That is a question for whoever owns the revenue.

Why "twice" is the normal case, not the edge case

Production Notes #05 argued that retries frequently make bugs worse — because a retry is just a deliberate way of making something happen twice. But retries are only one source of duplication.

In production, "twice" arrives from everywhere:

A webhook is delivered twice because the first acknowledgement timed out.
A user double-clicks the submit button on a slow connection.
A mobile client resends a request after the app resumes from background.
A queue guarantees at-least-once delivery, which is a polite way of saying sometimes twice.
An operator manually re-runs a batch job because they were not sure the first run finished.
A partner replays a day of events after fixing an outage on their side.

None of these are exotic. They are Tuesday.

Production Notes #02 made the case that every integration is a distributed system. Duplication is one of the first properties distributed systems force on you. Once two systems exchange messages over an unreliable network, "exactly once delivery" becomes a marketing phrase, not an engineering guarantee.

So the real question is not how do we prevent duplicates?

It is how do we make duplicates harmless?

That is what idempotency buys you.

The illusion of exactly-once

Teams spend enormous effort trying to guarantee that a message is delivered exactly once. Better brokers, stronger acknowledgements, distributed transactions, two-phase commits.

Most of that effort is misdirected.

You cannot reliably achieve exactly-once delivery across independent systems. What you can achieve is exactly-once effect — by making the receiver idempotent.

The distinction matters because it moves the responsibility to the place that can actually enforce it. The network cannot promise you one delivery. The receiver can promise that the tenth delivery changes nothing the first did not.

This is why mature systems stop chasing perfect delivery and start designing honest receivers:

A payment service that treats a repeated charge request with the same key as the same charge, not a new one.
An order endpoint that returns the original order when the same request arrives again, instead of creating a duplicate.
A fulfilment consumer that records which message IDs it has already processed and silently ignores repeats.

The network stays unreliable. The business stays correct anyway.

Idempotency keys are a contract, not a column

The common implementation is an idempotency key — a client-generated identifier attached to a request so the server can recognise a repeat.

But the key is the easy part. The hard part is deciding what the key means.

Who generates it? If the client generates it, a retry reuses it and dedup works. If the server generates it after receiving the request, two retries look like two distinct operations and dedup fails. This single decision determines whether the whole scheme works.
What is its scope? Is the key unique per customer, per endpoint, per day, forever? A key that collides across customers is a data leak. A key that expires too soon reopens the duplication window.
What does a repeat return? The original result? A conflict error? Silent success? Each choice changes how every downstream caller must behave.
How long do you remember it? Storing keys forever is a storage problem. Forgetting them too quickly is a correctness problem. The retention window is a real design parameter, not a default.

None of these are answered by adding a nullable idempotency_key column and hoping. They are a contract between the caller and the system — and like every contract in Production Notes #03, it only works if both sides agree on what it means.

Idempotency is easier before you need it

Here is the uncomfortable timing problem.

Idempotency is cheap to design in and expensive to retrofit.

Designed in, it is a key, a uniqueness constraint, and a decision about what a repeat returns. A few days of thought at the right moment.

Retrofitted, it is a forensic project. You are now trying to answer "did this already happen?" against a schema that never recorded intent, only outcomes. You reconstruct idempotency from timestamps, amounts, and hope. You write reconciliation scripts that guess whether two similar-looking orders are a duplicate or a coincidence.

I have watched teams add idempotency after the first duplicate-charge incident. It is always more expensive than the meeting they skipped six months earlier — because now there is production data shaped by its absence.

The lesson is not "always build it everywhere." The lesson is: decide deliberately, on every write path that matters, before the first duplicate arrives.

Where idempotency quietly earns its keep

The reason it is an unsung hero is that when it works, nothing happens. There is no incident, no war room, no dramatic recovery.

A payment provider retries a webhook — the order is not duplicated, because the handler recognised the event ID.

A dealer double-submits an order to a live portal integrated with production vehicle systems — the second submission returns the first order instead of creating a phantom one, because the write path was keyed and deduplicated rather than trusting the network to behave. That is the difference between an integration that survives real users and one that generates pricing disputes and reconciliation tickets. We designed for that on dealer kits and VISTA ordering — duplicate submits and sync retries had to resolve to one kit order, not two, or the accessories portal would invent inventory and pricing disputes against the vehicle system of record.

An operator re-runs last night's batch after a crash — the second run processes only what the first missed, because each record carried a processed marker.

None of these become stories. That is precisely the point. Idempotency converts would-be incidents into non-events. It is insurance that pays out silently, which is exactly why it is undervalued in planning and overvalued the morning after it was missing.

Idempotency in AI workflows

AI pipelines make this more urgent, not less.

An agent that calls tools — charge a card, create a ticket, send an email, update a CRM — is a system issuing side effects based on probabilistic reasoning. If the orchestration retries a step because a model call timed out, or a human re-runs a flow to "try again," those tool calls can fire twice.

A non-idempotent tool call behind an AI agent is a duplicate charge waiting for a plausible-sounding prompt.

The fix is not a better model. It is the same discipline enterprise systems have always needed: side effects must be idempotent, because the layer above them — human, queue, or model — will eventually ask for them twice.

Questions to ask before shipping a write path

I use this checklist whenever a feature creates, charges, sends, or mutates something a customer or auditor would notice.

If this exact request arrives twice, what should happen? Name the correct outcome in one sentence — one charge, one order, one email.
Who generates the idempotency key, and does a retry reuse it? If the answer is "the server," you probably do not have dedup.
What is the uniqueness scope and retention window for that key? Per what, for how long, enforced by which constraint?
What does the system return on a duplicate? The original result, a conflict, or silent success — and can every caller handle that?
Is the dedup enforced at the data layer, with a uniqueness constraint, or only in application code that a race condition can slip past?
How do we detect a duplicate that slipped through before finance or a customer does?

If you cannot answer the first question in one sentence, the write path is not ready — no matter how clean the happy path looks in staging.

A different standard for "done"

A write operation is not done when it succeeds once in testing.

It is done when you can state what happens the second, third, and tenth time the same request arrives — and prove the business outcome is identical.

That standard feels excessive for a single endpoint.

It feels obvious the first time a customer is charged twice because a phone lost signal for four seconds.

The teams I trust in production are not the ones with the most sophisticated messaging infrastructure. They are the ones who assumed every important operation would eventually happen twice — and decided, in advance, that twice would be safe.

Idempotency is not a feature. It is a promise that the second time will not hurt.

The systems that keep that promise are the ones nobody has to think about — which is the highest praise enterprise software ever earns.