Production Notes #15 · Part of Binary and Beyond. LinkedIn newsletter edition follows.
The war room has three clocks and none of them agree.
Support sees the order as paid at 14:02. The payment provider shows authorised at 14:01:47. The warehouse never received a reservation. Finance will not see the capture until the evening batch. The customer got an email that said "confirmed" because a timeout on the warehouse call was treated as success in a "fail open for conversion" hotfix from last quarter.
Someone asks the obvious question: "What is the real time of this order?"
There is no real time.
There are four systems, four clocks, four definitions of "done," and a product that pretended they shared one now.
That is why time is the hardest dependency in enterprise software. You can vendor a database. You can retry a webhook. You can scale a queue. You cannot buy a shared present tense across organisations that do not share a process, a ledger, or a wall clock.
The SLA that was really a wish
Picture a mid-market retailer writing acceptance criteria for a new fulfilment path.
Product wants "real-time inventory." Ops wants "orders in the warehouse within five minutes." Finance wants "recognition same day." The agency partner wants a green demo before the steering committee.
Engineering writes: sync call to warehouse on checkout, webhook from payments, ERP export every fifteen minutes, search index "near real time."
Nobody asks which clock owns which promise.
On launch week the warehouse API p95 climbs. Checkout starts timing out. The team raises the timeout from two seconds to eight. Conversion improves. Support tickets about "paid but not reserved" climb with it. The timeout did not buy reliability. It bought a longer window in which the system invents a story about what happened.
Production Notes #08 covered the cost of waiting on someone else's latency. This essay is about the quieter cost underneath: waiting is a claim about time, and most products make that claim without naming whose clock they trust.
Why time is not like other dependencies
Most hard dependencies can be treated as services.
A payment API has a contract, a status page, and a support channel. A message bus has throughput and a dead-letter policy. A third-party OCR vendor can be swapped when the bills get silly.
Time does not behave like that.
Clocks disagree. NTP keeps machines close enough for logs. It does not make a warehouse ACK and a payment capture share a business instant. "Same second" across two companies is a political agreement, not a physics fact.
Batches lag by design. Overnight ERP posts, fifteen-minute stock feeds, daily settlement files are not bugs. They are the business buying cheaper truth on a slower clock. Treating them as "eventually real-time" is how programmes inherit silent debt.
Timeouts invent truth. A timeout is not "the other system failed." It is "we stopped waiting." If your code maps that to success, cancelled, or retry-forever, you have authored a business event from the absence of evidence. Production Notes #05 is what happens when that invented event gets multiplied. Production Notes #04 is what happens when you refuse to admit the path was only partly true.
SLAs encode politics. "Four-hour sync" and "real-time dashboard" often mean different owners protecting different risks. The number in the contract is rarely the number the software can defend under load.
"Real-time" is usually a wish about other people's clocks. Customers hear "instant." Engineering hears "our p95." Ops hears "before the van leaves." Finance hears "before close." Same word. Four deadlines.
I have seen architecture reviews spend hours on vendor selection and ten minutes on which clock owns the customer-visible state. The vendor was never the hard part.
What teams ask vs what production needs
| What the brief asks for | What production needs |
|---|---|
| Real-time everywhere | Named clocks per decision, with explicit lag |
| Raise the timeout | Pending state, not a longer invented truth |
| Same dashboard for all teams | Authority map while systems disagree (#09) |
| SLA of N seconds end to end | Bound per hop, plus what happens when a hop misses |
| Sync until everything confirms | Accept, process, converge: three times, three owners |
| "Just make the batch faster" | Admit the batch is the clock for that domain |
The left column sounds decisive in a steering pack.
The right column is what keeps Tuesday operable when one partner is brownout and the batch still owns recognition.
Timeouts are product decisions wearing ops clothing
Teams treat timeouts as tuning.
Two seconds feels aggressive. Eight feels safe. Thirty feels "enterprise."
Those numbers are not neutral. They decide how long the product is willing to stay honest about not knowing.
Short timeouts without pending states create false negatives and retry storms.
Long timeouts without isolation turn your availability into whoever is slowest on the path (Production Notes #08 again).
Either way, the timeout is a product boundary: after this duration, what may we claim?
Mature paths answer in language a war room can use:
- We may claim accepted when payment authorises.
- We may claim reserved only when the warehouse ACK arrives, or when a bounded compensation path opens.
- We may claim recognised only after ERP posts, even if the customer already has a parcel.
That is not slower software. It is software that refuses to launder delay into certainty.
Batches, feeds, and the quiet clock everyone ignores
Enterprise programmes love to draw the interactive path and forget the batch path.
The UI is "modern." The money still posts overnight. Stock still arrives as a file. Compliance still closes on a calendar, not on a websocket.
Production Notes #09 argued that eventual consistency with named bounds is usually correct. The bound is a clock. If you never name the batch as the clock for finance truth, product will keep asking why the console disagrees with the ledger at 3pm.
I use a simple rule on multi-system programmes:
Every customer-visible claim must point at the clock that can make it false.
If "in stock" can be falsified by the next feed, the UI should not speak as if the warehouse API was the last word for the next six hours. If "paid" can be reversed by settlement, support macros should not treat authorisation as permanent history.
This is especially visible in legacy application modernisation work, where a new API facade sits in front of systems that still think in nightly cycles. The hard dependency is rarely the language rewrite. It is teaching the new surface to respect the old clocks instead of papering over them with optimistic copy.
Recovery has a clock too
When something breaks, teams reach for restart because restart feels like clearing time.
It does not.
A restart clears a process. It does not rewind a payment, unsend an email, or unsay "confirmed" to a customer. Production Notes #12 is about repairing partial state. Repair only works if you know when each side effect became true on which clock.
Without that, ops replays messages "from this morning" and discovers that this morning meant three different things in three systems.
Pending is not a UI consolation prize. It is the only honest state while clocks have not converged.
A design checklist before you promise "real-time"
I use these questions when a brief says real-time, immediate, or sync end to end:
- Which decision needs the same answer at the same moment? Name the decision (can we ship, can we capture, can support refund), not the dashboard.
- Which clock is authoritative for that decision? Payment provider, warehouse, ERP, commerce, human ops? Pick one primary. Name the others as lagging views.
- What is the maximum acceptable delay before the business considers the promise broken? Minutes, hours, next batch, next business day?
- What does the user see while we wait? Pending with a reason beats a spinner that resolves into a lie.
- What does a timeout authorise us to claim? Success, failure, unknown, retry? "Unknown" must be a first-class outcome.
- What converges the lagging systems, and who notices if convergence misses the bound? Queue, batch, reconcile job, human. If the answer is hope, you do not have a model.
- Are writes idempotent so a late ACK cannot invent a second truth? Delay without idempotency is how duplicates become policy.
- Does any SLA number encode a political compromise rather than a measured hop? If yes, design for the measured hop and negotiate the politics in the open.
If the answers collapse to "make it faster," you have not designed for time. You have wished for better weather on the network.
AI paths inherit every clock they touch
Tool-calling agents that update CRM, tickets, and ledgers in one turn do not get a shared now for free.
One tool returns in 200ms. One times out. One writes into a system that will only surface the row after a search index catches up. The model still narrates a finished story because language models are trained to complete, not to wait.
Production AI delivery that ignores clocks ships the same lie as the checkout timeout: confidence over unfinished time.
Design the agent like an integration path. Name pending. Name which tool's clock owns which claim. Refuse to let fluent prose collapse four timestamps into one "done."
A different standard for "fast enough"
Speed matters. Customers feel latency. Ops feel backlog. Finance feels close.
But "fast enough" is not a single p95 across the company. It is a set of bounded delays attached to named decisions, with honest pending between them.
Time is the dependency you cannot vendor, retry, or scale away. Clocks disagree. Batches lag. Timeouts invent truth. SLAs encode politics. "Real-time" is usually a wish about other people's clocks. Design for bounded delay, named clocks, and honest pending, not for pretending every hop shares one now.
The teams I trust in production are not the ones with the lowest timeout values. They are the ones who can say, in one sentence, which clock owns the promise the customer just received, and what the product is allowed to claim before that clock has spoken.
Related reading
- The Hidden Cost of Synchronous APIs: waiting is still a claim about someone else's clock
- Why Eventual Consistency Is Usually Correct: disagreement windows need bounds and owners
- Designing Software for Partial Failure: green hops are not an intact promise
- Building Systems That Recover Instead of Restart: repair needs timestamps that mean something
- Legacy application modernization: new surfaces that still sit on old clocks
Production Notes #15 · Part of Binary and Beyond. LinkedIn newsletter edition follows. Building production-grade delivery that names clocks instead of pretending every hop shares one now? Start a conversation.