Production Notes #09 · Part of Binary and Beyond. LinkedIn newsletter edition follows.
Most architecture reviews treat consistency like a moral category.
Strong is good.
Eventual is a compromise you make when you cannot afford good.
That framing sounds rigorous. It is usually backwards.
In connected enterprise software, eventual consistency is not the degraded mode. It is the normal operating mode once more than one system owns a slice of the truth. Strong consistency across those systems is the expensive fiction: a wish that every write lands everywhere before anyone is allowed to proceed.
The teams that stay calm do not pretend every dashboard shows the same number at the same millisecond.
They decide which disagreements are allowed, for how long, and who is authoritative while they last.
That is eventual consistency with a product contract, not hand-waving.
The meeting that demanded "real-time everywhere"
Picture a mid-market retailer reviewing a new order console.
Product wants the screen to show payment, inventory reservation, warehouse acceptance, and finance recognition as one green state the moment the customer clicks.
Engineering draws four systems on the whiteboard.
Someone says: "We need strong consistency across all of them."
Nobody defines what that means operationally.
Do we block checkout until the warehouse ACK arrives?
Do we hide the order from support until the ERP posts it?
Do we refuse the payment webhook until search indexing finishes?
In the demo, every hop is local and fast. The green state appears instantly.
In production, the warehouse API slows on Monday mornings. The ERP batch runs every fifteen minutes. Search indexes on a delay. Payments authorise in hundreds of milliseconds and fulfil on a different clock.
The "real-time everywhere" requirement does not create consistency.
It creates waiting, timeouts, and retries (Production Notes #05, Production Notes #08).
Or it creates a lie: the UI shows done while half the path is still pending.
Production Notes #04 called that partial failure. Eventual consistency is how you design for that reality instead of pretending the path is a single transaction.
What "eventual" actually means in a business system
In distributed systems literature, eventual consistency says replicas will converge if writes stop.
In enterprise delivery, the useful version is narrower:
Given this business event, which systems may temporarily disagree, what is the maximum acceptable disagreement window, and which system's answer wins for which decision during that window?
That sentence is an authority map (Production Notes #03) plus a clock.
Without the clock, "eventual" means "we hope."
Without the authority map, "eventual" means "everyone is right until finance notices."
Production Notes #01 argued that divergent state is harder than scale. Eventual consistency is not the cause of divergence. Unbounded, unnamed divergence is.
Why strong consistency feels safer (and often isn't)
Strong consistency appeals because it matches how humans tell stories about money and inventory.
There is one number.
It is true now.
Everyone sees it.
Across a single database with a well-designed transaction, that story can be true.
Across commerce + payments + warehouse + ERP + CRM + analytics, it cannot be true without turning every user action into a distributed transaction with a political SLA.
Teams then invent workarounds that look strong and behave eventual:
- Long synchronous calls that wait "until everything is done"
- Global locks that serialize the business into a queue
- Nightly "reconcile everything" jobs that quietly admit the day was eventual
- Read-your-writes UX that hides pending states behind a spinner
The spinner is not strong consistency.
It is eventual consistency with worse latency and a less honest UI.
The cost of refusing "eventually"
When a product forbids temporary disagreement, three bills arrive:
1. Availability collapses into the slowest dependency.
If checkout cannot complete until every side effect confirms, partner brownouts become your outage.
2. Retries multiply side effects.
Timeouts look like failure. Clients try again. Without idempotency, "wait for consistency" becomes duplicate truth.
3. Support invents a parallel truth.
Agents learn tribal knowledge: "Ignore the ERP for the first hour." That is an undocumented consistency model. It still governs the business.
I have watched programmes spend months chasing "single real-time view" when the real need was simpler: show pending honestly, converge on a deadline, and name who to trust before the deadline.
When strong consistency is the right call
Eventual is usually correct. It is not always correct.
Prefer strong (or single-writer transactional) consistency when:
- Two outcomes cannot both be true even briefly (unique inventory unit, seat, coupon single-use)
- The business cannot accept a compensating action later (some regulated ledgers, some payment capture rules)
- One system is clearly the only writer and everyone else is a reader of its committed state
Prefer eventual (with bounds) when:
- Multiple systems must act on the same event and have independent failure domains (Production Notes #02)
- The user can understand accepted / processing / complete / failed
- Reconciliation and compensation are cheaper than global blocking
- "Real-time" actually means "fast enough acknowledgement," not "every replica updated"
The mistake is treating these as ideology instead of product design.
A design checklist before you ban "eventual"
I use these questions when a stakeholder asks for strong consistency across systems:
- Which decision needs the same answer in two places at the same moment? Name the decision, not the dashboard.
- What is the maximum disagreement window the business can tolerate? Minutes, hours, next batch?
- Who is authoritative during the window? Commerce for customer-facing status? Warehouse for pickability? ERP for recognition?
- What does the user see while systems disagree? Pending is a feature. Silent lying is not.
- What converges the state: sync call, queue, batch, human? If the answer is "hope," you do not have a model.
- What happens if convergence fails after the window? Compensation, alert, freeze intake, or manual ops?
- Are writes idempotent so retries do not invent extra truth while we wait?
If the answers are vague, "we need strong consistency" is not a requirement.
It is anxiety wearing an architecture word.
AI workflows are eventual too
Tool-calling agents that update CRM, tickets, and knowledge bases in one turn rarely commit all tools in one atomic business transaction.
They commit in pieces.
One tool succeeds. One times out. The model narrates confidence anyway.
Production AI delivery that ignores eventual outcomes will ship the same lie as a checkout spinner: a finished story over unfinished state.
Design the agent like an integration path. Name pending. Name authority. Name the reconciliation when tools disagree.
A different standard for "consistent enough"
Consistency is not a badge you earn by adding another distributed lock.
It is a promise about which truths may lag, and which decisions may not.
Eventual consistency is usually correct because enterprise software is made of systems that fail and succeed on different clocks. The honest architecture admits that, then puts bounds and owners around the lag.
The dangerous architecture pretends every screen is a single transaction across the company.
Until Monday morning, when the warehouse is slow, the ERP has not posted, and someone asks why the green dashboard disagrees with the pick list.
What good looks like in practice
On programmes where inventory, ordering, and fulfilment already live in different systems, the durable pattern is rarely "make them one database."
It is explicit pending states, bounded sync windows, and reconciliation that treats lag as expected rather than as a defect to hide.
We have seen that shape on live dealer platforms where vehicle and kit workflows had to stay aligned across VISTA and a production portal without pretending every screen was atomically current. The work was less about eliminating disagreement and more about making disagreement operable: who wins for which decision, how long lag may last, and how ops notices when convergence fails. That is eventual consistency as delivery discipline, not as an excuse for messy data. The dealer kits and VISTA ordering rollout is one concrete version of that discipline.
Related reading
- The Myth of a Single Source of Truth: authority during disagreement
- Designing Software for Partial Failure: green hops are not an intact promise
- The Hidden Cost of Synchronous APIs: waiting for every replica is still a consistency choice
- Legacy application modernization: multi-system paths where lag is a product constraint
Production Notes #09 · Part of Binary and Beyond. LinkedIn newsletter edition follows. Building production-grade delivery that names disagreement windows instead of hiding them? Start a conversation.