Production Notes #05 · Part of Binary and Beyond. LinkedIn newsletter edition follows.
The first fix for a flaky integration is almost always the same.
Add a retry.
Wrap the call in a policy. Back off exponentially. Maybe jitter the delay. Ship it Friday. Feel responsible.
Retries are seductive because they look like reliability without a design meeting. The dependency failed once; the client tried again; the request succeeded. Green dashboard. Case closed.
Except the bug you fixed on Friday is often the duplicate order finance discovers on Monday.
The retry that looked successful
Picture a checkout flow on an ordinary afternoon.
The customer submits payment.
Your service calls the payment gateway.
The gateway authorises the charge.
Your service sends the response back to the browser.
The browser never receives it — mobile network blip, load balancer timeout, client closed the tab.
From the customer's perspective, the payment might have failed.
From the gateway's perspective, the payment succeeded.
Your retry policy does exactly what you told it to do.
It tries again.
Sometimes the second attempt is rejected as a duplicate — good.
Sometimes it isn't — the gateway treats it as a new authorisation because the idempotency key was missing or scoped wrong.
Sometimes your service retries not the payment call but the downstream side effect — create order, reserve inventory, notify warehouse — while the first attempt actually completed but the acknowledgement was lost.
Now you have two orders, one charge, and three systems that each believe they handled the situation correctly.
Nothing threw an exception worth paging.
The retry worked.
That is the failure mode.
Production Notes #04 was about brownouts — systems that fail in pieces while the business keeps moving. Retries are how brownouts become duplicate truth: the same operation executed twice, recorded once in one place and twice in another.
Retries solve transport. They do not solve state.
Most retry logic is implemented at the wrong layer.
It answers: will we call again if the HTTP status is ugly or the socket times out?
It does not answer:
- What happened if the server processed the request but the response never arrived?
- Is this operation safe to run twice?
- Which system owns the outcome if attempt one and attempt two disagree?
- How will we discover duplication before a customer or auditor does?
Those are state questions.
Production Notes #01 argued that the expensive failures are divergent state, not raw load. Retries are one of the fastest ways to create divergent state — especially when teams add them before they define idempotency, boundaries, and reconciliation.
Production Notes #03 covered authority when systems disagree. Retries often manufacture that disagreement: two writes with the same intent, different timestamps, and no shared key that says they are the same event.
Transport reliability and state correctness are related.
They are not the same problem.
Why teams reach for retries first
Retries are cheap in code and expensive in production — but the invoice arrives later.
They require no product decision about degraded modes.
They require no new durable state like payment_pending or fulfilment_unknown.
They require no conversation with the partner about idempotency keys.
They fit neatly in middleware, SDK defaults, and queue consumers.
They also match how engineers experience failure: I called it; it failed; I called it again; it worked.
That experience is true for read-only operations and for idempotent writes with a stable key.
It is frequently false for multi-step business processes where each hop has its own timeout, retry policy, and definition of success.
When five systems each retry independently, you do not get five chances at reliability.
You get five chances to race.
When retries help — and when they hide the real bug
Retries are appropriate when all of these are true:
- The operation is read-only, or
- The write is provably idempotent with a key the downstream system honours, and
- Duplication is detectable and harmless, or
- The retry targets a single bounded step with a clear before/after state machine
Retries are a liability when:
- Success is inferred from the first hop only while side effects continue asynchronously
- Different services retry different steps of the same business action on different schedules
- The client, the API gateway, the worker, and the partner each implement their own retry policy with no shared idempotency contract
- "Success" means best effort rather than verified outcome
- The team has no reconciliation job to find duplicates — only a metric that says
retry_countwent down
The uncomfortable pattern: retries mask missing idempotency long enough for the feature to ship, then turn intermittent timeouts into permanent data problems.
The amplification loop
Retries do not only duplicate work.
They amplify timing bugs.
A slow dependency under load triggers more retries.
More retries increase load.
Increased load increases timeouts.
Timeouts trigger more retries.
The system enters a polite DDoS against itself while dashboards show elevated latency and heroic autoscaling.
Circuit breakers help at the transport layer.
They do not, by themselves, answer what state the business is in when the circuit opens after half the side effects ran.
That is why Production Notes #02 treated integrations as negotiations between observers of the same process. Each retry is another observer making another claim about what happened.
We saw this pattern on live dealer platforms where vehicle order sync, kit ordering, and workflow state had to stay aligned across VISTA and a production accessories portal — not because retries are forbidden, but because retry without reconciliation would have turned intermittent API blips into duplicate kit orders and pricing disputes. The dealer kits and VISTA ordering rollout paired sync handling, explicit error recovery, and audit trails with the HTTP client — the difference between transport resilience and production correctness.
Questions to ask before you add retry logic
Use this before wrapping another call in a policy — especially on a write path that touches money, inventory, or customer commitments.
- If this request ran twice, what would break? Name the duplicate — order, charge, email, ticket, ledger entry.
- Does the downstream system honour an idempotency key? If not, a retry is a gamble, not a policy.
- Who generates the key — client, gateway, worker, or partner? If the answer is "whoever got there first," you will eventually duplicate.
- What state do we persist before the call, after acceptance, and after verified completion? If there is only "success" and "failure," partial success has nowhere to live.
- Which retries are safe at the edge vs only safe in a orchestrated worker? Browser retries and server retries are not interchangeable.
- What happens when the circuit opens mid-flow? Name the business state: cancelled, queued, unknown, or falsely confirmed.
- How will we detect duplicates without waiting for finance or support? If the answer is "we'll reconcile manually," estimate the cost honestly.
If you cannot answer the first question in one sentence, the retry is not hardening.
It is deferring a state design conversation until after the incident.
A different default
I still use retries in production systems.
I use them where they belong — bounded, keyed, observable, and paired with reconciliation.
The default I question is not "should we ever retry?"
It is "did we add retries instead of defining what twice means?"
The teams I trust treat idempotency and reconciliation as part of the feature, not as cleanup after launch. They design for the request that succeeded on the server and failed on the wire — because that request happens every week, not only during disasters.
Retries are a tool.
They are not a substitute for knowing which version of the truth you are willing to tolerate when two systems disagree.
The next essay in this thread goes further on the antidote: idempotency — not as a buzzword, but as a business contract about what "once" means.
Production Notes #05 · Part of Binary and Beyond. LinkedIn newsletter edition follows. Untangling retry logic and integration state on a live platform? Start a conversation.