Production Notes #11 · Part of Binary and Beyond. LinkedIn newsletter edition follows.
Engineers call them race conditions.
Businesses call them overselling, double booking, phantom inventory, and "how did two people get the same appointment?"
Same failure. Different vocabulary.
A race condition is what happens when two correct operations run at the same time and produce an incorrect business outcome. Each request looked fine in isolation. The dashboard stayed green. The code review passed. And still you sold the last unit twice, allocated the same kit to two dealers, or credited a wallet that had already been emptied.
This is not an academic concurrency puzzle. It is a revenue and trust problem wearing a computer science name.
What a race looks like on a Tuesday
Two customers open the last unit of a SKU at the same moment.
Both see "1 in stock."
Both click buy.
Both get confirmation emails.
Warehouse has one box.
Or: two dealer portals submit kit orders against the same vehicle configuration before either write has committed. Or two support agents claim the same escalation ticket. Or a refund and a capture both land on a payment that was only authorised once.
In each case the individual action was reasonable. The collision was the bug.
Production Notes #01 argued that state is harder than scale. Race conditions are one of the sharpest proofs of that claim. Throughput is irrelevant if two writers can disagree about what is still available.
Why teams keep shipping them
Race conditions survive for the same reason idempotency stays off the roadmap: they are invisible in the happy path.
Staging has one tester. Staging does not click "buy" from two browsers at the same millisecond. Staging does not run two partner APIs into the same inventory row during a flash sale.
So the feature ships.
Then production adds concurrency: real users, real retries (Production Notes #05), real mobile clients that pause and resume, real ops tools that "just update the row."
The race was always possible. Traffic only made it frequent.
The business invariant is the missing design
The technical fix people reach for first is a lock.
Sometimes a lock is correct. Often it is a bandage on an unstated rule.
Before you lock anything, write the business invariant in one sentence:
- There is never more reserved stock than physical stock.
- A vehicle kit is owned by at most one open order.
- A payment capture and a refund cannot both succeed for the same authorised amount.
- An appointment slot is held by at most one confirmed booking.
If you cannot write that sentence, you do not know what "correct" means when two writes collide. A mutex will not invent the rule for you. It will only serialise confusion.
The teams that stay out of reconciliation hell treat the invariant as a product decision, then enforce it at the write path: atomic check-and-set, unique constraints, reservation records with clear lifetimes, or a single writer for that slice of state.
Locks are a tool. The invariant is the design.
Reservations need a lifetime
Even a correct atomic claim can become a race if the hold never expires.
A cart that reserves stock for three days, then never releases it, starves every other buyer. An appointment hold that dies only when support clears it by hand creates phantom scarcity. A "pending" kit order that sits forever blocks the next dealer.
So the invariant is not only at most one owner. It is at most one owner, for a defined window, with a clear release path.
Design the timeout and the release with the same seriousness as the claim. Otherwise you trade overselling for artificial shortages, and ops invents a second process to unblock the first.
Check-then-act is where money leaks
Most production races are a boring pattern:
- Read the current value.
- Decide in application code.
- Write the new value.
Between step 1 and step 3, another request did the same thing.
That gap is where overselling lives.
"Check stock, then create order" without an atomic reservation is not a stock system. It is a rumour system with a checkout button.
The same pattern appears in wallet debits, seat maps, coupon redemptions, and approval workflows. Anywhere the code says "if available, then claim" without making claim and availability one operation, you have a race waiting for a busy hour.
Shared databases make races communal
Architecture Files #10 covered how a shared schema couples ownership. It also couples concurrency.
When three products write the same inventory row, each team's "safe" check-then-act assumes it is the only writer. It is not. The race is no longer inside one service. It is across release trains, ORMs, and people who never sat in the same planning meeting.
If you must share state, you need a single authority for mutations that matter, or a reservation protocol everyone obeys. Hoping that "our transaction is quick enough" is not a protocol.
Retries turn rare races into regular incidents
A race that happens once under load becomes a cascade when retries and duplicate deliveries enter the picture.
First request reserved the last unit but the client timed out.
Retry arrives. Second reservation path runs. Or the retry creates a second order because the write was not idempotent.
Now you have a race and a duplicate. Finance meets warehouse meets support on Monday.
Concurrency bugs and duplication bugs compound. Design for both, or you will spend your life reconciling the interaction between them.
On dealer kits and VISTA ordering we treated double-submits and concurrent reservations as the same class of failure: two writers claiming one kit. The write path had to resolve to one owner, not two "successful" orders that disagreed with the vehicle system of record. That is the difference between a portal that survives a busy morning and one that invents inventory disputes.
Questions before you ship a claim path
Use this when a feature reserves, allocates, debits, books, or otherwise claims something scarce:
- What is the business invariant in one sentence? What must still be true after two concurrent claims?
- Is the check and the claim one atomic operation, or a read in app code followed by a later write?
- Who else can mutate this state: other services, admin tools, batch jobs, partner APIs?
- What happens if the client retries after a timeout mid-claim? Same key, same reservation, or a second one?
- How do we detect a broken invariant before a customer or auditor does?
- What is the recovery path when the invariant fails: cancel, compensate, or manual reconciliation?
If question 1 has no answer, the feature is not ready. If question 2 is "read then write," you are shipping a race with a UI.
A different standard for "done"
A claim path is not done when one user can complete it in staging.
It is done when two concurrent claims cannot both succeed against one unit of scarce state, and when a retry of a successful claim does not invent a second one.
That standard sounds heavy for a simple "buy" button.
It feels obvious the first time you apologise for selling something you did not have.
Race conditions are not niche. They are what scarce resources look like when software meets more than one person at a time. Name the business rule. Enforce it where the write happens. Everything else is hoping the Tuesday stays quiet.
Race conditions are business problems when they break a promise the customer already believes you kept. Treat them that way in design, not only in the postmortem.
Related reading
- Why Retries Make Bugs Worse — retries multiply collision windows
- Idempotency Is the Unsung Hero of Enterprise Software — same claim must not become two claims
- The Problem with Shared Databases — shared writers make races communal
- Dealer kits and VISTA ordering — concurrent reservations resolving to one kit owner
- Legacy application modernization — write-path contracts in systems that already ship