Production Notes #12 · Part of Binary and Beyond. LinkedIn newsletter edition follows.
The first suggestion in the war room is almost always the same.
"Restart the service."
Sometimes that is correct. A process is wedged. Memory is leaking. A stuck lock will not release until the JVM dies. Restart is a legitimate tool.
Often it is a category error.
Because the failure was not a dead process. It was a partial outcome: money captured, inventory reserved, email sent, warehouse not notified. Restart clears the process. It does not unwind the promise. It does not finish the half-done write. It does not tell finance which of the three "paid" rows is true.
Restart is optimism about clean state.
Recovery is a designed path through dirty state.
The Tuesday that restart made worse
Picture a mid-market fulfilment stack.
Checkout calls payments. Payments authorises. The order service writes paid and enqueues a fulfilment message. The message worker crashes mid-handler: stock is reserved in the WMS API, the local reservation row never commits, the customer already has a confirmation email from an earlier step that "succeeded."
Dashboard for the API process: red, then green after a restart.
Orders still sitting in paid with no warehouse acknowledgement.
Support sees paid. Warehouse sees nothing. Customer sees a tracking promise that will not arrive.
Nobody needed a smarter health check. They needed a recovery path that assumes the world keeps the parts of the work that already escaped the process boundary.
Production Notes #04 argued that production fails partially, not completely. Restart is what you reach for when you still wish failure were total: wipe the slate, boot clean, pretend Tuesday did not leave residue.
Why restart feels like engineering
Restart is visible. It has a button. It shows up in runbooks. On-call can do it at 2am without reading the domain model.
Recovery is invisible until you design it. It needs answers that sound like product work:
- What already happened outside this process?
- What is still safe to retry (Production Notes #05, Production Notes #06)?
- What must a human decide because the machine cannot know?
- How do we leave an audit trail that next week's dispute team can trust?
Those questions do not fit in a Kubernetes rolling restart. So teams ship the button and call the architecture "resilient."
Resilience without recovery is just faster ways to return to a confused green.
Recovery is a first-class product surface
Mature systems treat recovery the way they treat checkout: named, tested, owned.
That usually means:
Compensating actions, not hope. If stock was reserved and the local write failed, there is a path that releases the reservation or completes the local record against the remote truth. "We'll notice in reconciliation" is not a path. It is a backlog item wearing confidence.
Replay with intent. Queues and outboxes that can be replayed safely because every write was designed for twice. Restarting a consumer without idempotency is how you mint duplicate shipments.
Explicit "stuck" states. Orders, payments, and tickets that can sit in needs_attention with a reason code beat silent paid rows that look healthy and are not. Operators need a queue, not a legend about which log line means what.
Bounded human overrides. Someone will need to mark an exception. That person needs a command with preconditions, not edit access to the status column (Production Notes #07).
Time budgets on holds. A reservation that never expires, or a "pending" payment that never times out, turns every partial failure into permanent scarcity. Recovery includes expiry, not only repair.
None of this requires a particular framework.
It requires admitting that the happy path is half the product. The other half is what happens when the happy path stops halfway.
What recovery looks like when it is real
Take the fulfilment crash again, but with a recovery path.
On restart, the worker does not blindly re-run "create reservation." It asks the outbox: which order ids were mid-flight? For each, it checks the WMS for an existing reservation key. If the remote side already has it, the local row is completed to match. If the remote side does not, the worker either retries the reservation under the same idempotency key or moves the order to needs_attention with a reason a human can act on.
Support does not invent a status. They run ForceCompleteFulfilment or ReleaseReservationAndRefund, each of which writes an audit event.
That is slower to design than a restart button.
It is also the difference between an incident that ends in an hour and one that leaks into month-end.
The same pattern shows up in payments: a capture that timed out after the processor succeeded needs a reconcile-against-gateway step, not another capture. In CRM sync: a half-pushed contact needs a keyed upsert, not a second create. In appointment booking: a held slot after a crash needs expiry plus a clear claim, or you recreate the race you thought you fixed (Production Notes #11).
Recovery is not a single library. It is the habit of designing the "from here" path for every irreversible step — and testing that path the same way you test checkout.
The restart tax
Every time a team uses restart as the primary recovery mechanism, they pay a tax:
- Lost context. In-memory progress dies. So does the informal knowledge of "we were on step three."
- Amplified duplicates. Clients retry. Webhooks retry. Humans click again. Restart does not cancel the outside world.
- False confidence. Green pods hide unpaid debts to other systems.
- Training debt. On-call learns to reboot instead of reading state. The next hire inherits the ritual.
I have watched incidents where three restarts "fixed" the error rate and six weeks of finance cleanup followed. The process was healthy. The business was not.
The same pattern shows up in agency delivery systems that "recover" by redeploying a stuck admin tool while partner portals still hold half-applied price lists. The button felt decisive. The downstream systems kept the lie.
The teams I trust write recovery runbooks that start with observe external side effects, not bounce the box. When we modernise platforms that grew a restart culture, the expensive work is rarely the orchestrator. It is naming the incomplete states operators were never given a screen for — then wiring commands that finish those states without rewriting history by hand.
Questions to ask before the next "just restart it"
Use this when a design review ends with "if it fails, we'll restart and retry."
- What can already be true outside this process after a crash? Name the remote writes, emails, and ledger entries.
- Which of those are safe to do twice? If you cannot answer, you do not have a retry strategy. You have a restart fantasy.
- What does "stuck" look like in the UI and the data model? If stuck is invisible, recovery will be tribal knowledge.
- Who is allowed to force a completion or a rollback? Name the role and the command.
- How long can a hold or pending state live before it becomes a new failure? Put a clock on it.
- What does reconciliation need that restart erases? Logs, correlation ids, and before/after snapshots are recovery infrastructure.
- Are we calling the system resilient because pods come back, or because promises get finished?
If the answers collapse to "Kubernetes will reschedule it," you have process recovery. You do not yet have business recovery.
A different standard for "resilient"
A service is not resilient because it restarts cleanly.
It is resilient when an interrupted workflow can be observed, classified, and driven to a correct terminal state without inventing history.
That standard feels heavy next to a health-check dashboard.
It feels obvious the first time a green deploy sits on top of a pile of half-paid orders.
Restart clears a process. Recovery repairs a promise. Build for the second one, and the first one becomes a tool again instead of a religion.
Green after bounce is not the same as done.
If the work already escaped the process, the only honest next step is a path that knows what "finished" means from here.
Related reading
- Designing Software for Partial Failure: green hops are not an intact promise
- Why Retries Make Bugs Worse: deliberate twice without a contract
- Idempotency Is the Unsung Hero of Enterprise Software: recovery's prerequisite on write paths
- Legacy application modernization: when restart rituals outgrow the systems that invented them
Production Notes #12 · Part of Binary and Beyond. LinkedIn newsletter edition follows. Building production systems that recover through partial state instead of hoping a restart erases Tuesday? Start a conversation.