Production Notes #04 · Part of Binary and Beyond. LinkedIn newsletter edition follows.
Most architecture reviews still treat failure as a binary event.
The system is up.
Or the system is down.
We draw redundancy boxes. We talk about failover. We rehearse the disaster recovery runbook for the day the data centre disappears.
Those exercises matter.
They also miss how enterprise software actually breaks.
It rarely fails all at once.
It fails in pieces.
A payment gateway times out while the commerce platform stays healthy.
A warehouse API returns 503 while the ERP keeps accepting orders.
A recommendation model degrades while checkout continues to work.
A CRM sync stalls for three hours while support agents keep answering tickets with yesterday's customer profile.
That is partial failure — and it is the normal operating mode of connected software, not an edge case you schedule for Q4.
When "mostly working" is the dangerous state
Picture a mid-market retailer on a busy afternoon.
Checkout is green.
Payments are authorising.
Inventory looks fine on the storefront.
Then the warehouse fulfilment API starts rejecting requests — rate limit, maintenance window, or a bad deploy on their side. Nobody on the retailer's team owns that system. They only own the integration.
Orders keep landing.
Customers get confirmation emails.
Finance sees revenue.
Fulfilment does not see work.
For ninety minutes the business is partially succeeding. That is worse than a clean outage. A clean outage stops intake. Partial failure keeps intake while silently breaking the promise you just made to the customer.
Support discovers it when someone asks where their order is.
Engineering discovers it when the reconciliation job fails at midnight.
Leadership discovers it when the NPS survey arrives.
This is not a rare story. It is the default story once you have more than one system in the path — which is every enterprise path that matters.
Production Notes #02 argued that every integration is a distributed system. Partial failure is what that fact feels like in production: some nodes answer, some do not, and the business still has to decide what "success" means in between.
Production Notes #03 covered authority when systems disagree. Partial failure is often how that disagreement is born — one write lands, the next does not, and both sides keep their own version of events.
We design for total failure. We ship for partial failure.
Most teams are better prepared for catastrophe than for degradation.
Catastrophe has a name. A status page. A war room. A clear owner.
Degradation is quieter.
One dependency is slow.
One field is stale.
One webhook queue is backing up.
One AI enrichment step is timing out, so the workflow skips it and continues with a thinner payload.
The dashboard still shows green because the homepage loads and the primary API returns 200.
Green dashboards are not the same as intact business operations.
The gap between those two is where expensive incidents live.
I have watched teams celebrate "five nines" on a service that was returning success while dropping a critical side effect on the floor. Availability of the HTTP layer is not availability of the business promise. Partial failure exploits exactly that confusion.
Timeouts are a product decision
When engineers say "we added a three-second timeout," they often think they have finished the reliability work.
They have only finished the transport work.
A timeout answers: how long will we wait?
It does not answer: what is true afterward?
Did we create the order locally and fail to notify the warehouse?
Did we charge the card and fail to reserve inventory?
Did we show the customer a success screen based on the first hop alone?
Without an explicit policy, every timeout becomes an improvised story told differently by engineering, support, and the customer.
Timeouts belong in the same design conversation as user messaging and durable state — not only in the HTTP client configuration.
What partial failure actually requires
Designing for partial failure is not the same as adding retries — Production Notes #05 will go further on why retries often amplify the damage.
It is a set of product and architecture decisions made before the dependency misbehaves:
Which capabilities are allowed to degrade? Checkout without personalisation is usually acceptable. Checkout without payment authorisation is not.
What does the user see when a dependency is unavailable? A honest delay, a reduced feature, a queued action — or a silent lie that everything worked?
What does the system record when a side effect did not complete? An order that is "paid but not fulfilled" needs a first-class state, not a log line someone might read.
How long may disagreement persist? Production Notes #01 was about divergent state. Partial failure is how that divergence arrives: one hop succeeds, the next does not, and both systems believe they are correct.
Who is paged for a brownout? If only total outages create tickets, partial failure will train your organisation to ignore the real risk.
These are not infrastructure questions alone.
They are product questions wearing operational clothing.
The human layer prefers clean stories
Partial failure is also organisationally inconvenient.
Executives want a binary: Are we up?
Vendors want a binary: Was it our SLA?
Agencies want a binary: Did we deliver the integration?
Partial failure refuses the binary. It forces sentences like: Payments worked. Fulfilment did not. Customers were told otherwise for seventy minutes.
That sentence creates ownership arguments.
So teams unconsciously design for the failure mode that is easiest to narrate — the big outage — and under-invest in the failure mode they live with every week.
Incident reviews reward drama. Brownouts reward discipline. Only one of those shows up naturally in a slide deck.
The fix is not better storytelling after the incident.
It is making degraded modes explicit in the design so the organisation has language for them before the war room — including which team is allowed to say "we are degraded" without waiting for a total outage.
AI workflows fail the same way — only quieter
An AI-assisted support flow that calls the CRM, a knowledge base, and a policy service will not usually fail as a blank screen.
It will fail as a confident answer built from two good sources and one missing one.
The model fills gaps. That is its job.
Your job is to decide whether a missing tool call should block, degrade, or refuse — not whether the prompt was clever enough.
Production AI delivery that ignores partial failure is just integration theatre with a chat UI. The plumbing still has to say what happens when one tool is down.
Questions to ask before the next dependency goes live
I use this checklist when a feature depends on anything you do not fully control — partner API, internal service, model call, or batch job.
- If this dependency is unavailable for fifteen minutes, what should still work?
- If it is unavailable for four hours, what must stop accepting new work?
- What does the customer see in each case — and is that message true?
- What durable state do we write when a side effect fails after a successful upstream step?
- How do we detect partial success without waiting for a customer complaint?
- Who owns the brownout — the team that calls the dependency, or the team that runs it?
- What is the reconciliation path when we discover divergence after the fact?
If the answers are "we'll retry" and "we'll monitor the API," you have not designed for partial failure.
You have designed for hope.
A different standard for "done"
A feature is not done when the happy path works in staging.
It is done when you can name the degraded modes you accept, the ones you refuse, and the signals that tell you which mode you are in.
That standard feels heavy for a single webhook.
It feels obvious after the afternoon when orders kept flowing and fulfilment did not.
The teams I trust in production are not the ones with the most dramatic failover demos.
They are the ones who assumed something would be broken on an ordinary Tuesday — and decided, in advance, which brokenness the business could survive.
Enterprise software does not fail cleanly.
It fails in pieces.
The question is not whether something will fail. It is which parts are allowed to fail without taking the promise to the customer with them.
Production Notes #04 · Part of Binary and Beyond. LinkedIn newsletter edition follows. Building production-grade delivery that survives brownouts? Start a conversation.