AI Reality Checks #16 · Part of Binary and Beyond. LinkedIn newsletter edition follows.
The demo always looks like subtraction.
A form that took twenty minutes becomes a prompt. A triage queue that needed three people becomes a tool-calling agent. A weekly report that ate Friday afternoon becomes a generated brief in thirty seconds.
Someone in the room says: "We eliminated that work."
They are half right.
The human steps that used to absorb the mess are gone. The mess is not. It moved into places the slide deck does not show: verification, data contracts, ownership, failure modes, and the operational cost of keeping the new path honest.
AI does not eliminate complexity.
It relocates it.
The Friday that looked finished
Picture a mid-market ops team shipping an "AI intake" for support tickets.
Before: an agent read the email, classified the issue, looked up the account in the CRM, checked open orders, and wrote a first reply. Slow. Expensive. Visible.
After: a model classifies the ticket, calls three tools, drafts the reply, and posts a confidence score. The dashboard shows tickets closed without a human touch. Leadership celebrates the deleted queue time.
Then Monday arrives.
Two tickets get the same refund because the model retried a timed-out tool call. One VIP is told their order shipped when warehouse state was still pending. A classification that looked correct in the eval set maps a regulated complaint into the wrong queue. Nobody owns the bad reply because "the AI did it," which is not an owner.
The deleted work was reading and typing.
The new work is judging model output, bounding tool side effects, naming who is authoritative when tools disagree, and paying for retries that invent duplicate truth (Production Notes #05, Production Notes #06).
That is not a failed AI project.
It is a successful deletion of one kind of labour, followed by an unpaid invoice for another.
What "moved" actually means
When teams say complexity decreased, they usually mean visible human effort decreased.
That is real. Drafting, searching, summarizing, and first-pass classification are genuine savings when the path is designed honestly.
The ledger is incomplete if you stop there.
Complexity that used to live in a person's judgment now lives in:
Verification. Someone still has to decide whether the output is safe to act on. Sampling, human review, eval suites, and escalation rules are not optional polish. They are the new queue.
Data contracts. The model only looks smart when inputs are shaped. Schema drift, missing fields, and "close enough" CRM notes become production incidents (Production Notes #10). The contract work did not vanish when the prompt arrived. It became load-bearing.
Ownership. Who is accountable when the agent updates the wrong record? Product? Ops? The vendor? Without a named owner, every failure becomes a meeting about blame instead of a change to the path.
Failure modes. Partial tool success, hallucinated certainty, silent degradation when a dependency slows: these replace the old failure mode of "the agent was out sick." Production Notes #09 already argued that AI workflows are eventual. Tool calls commit in pieces. Narratives arrive before state converges.
Operational cost. Tokens, eval runs, review time, incident load, and the engineering needed to make tool calls idempotent. Dashboards that only show "tickets auto-closed" hide the bill.
The complexity did not leave the system.
It left the spreadsheet column labelled "FTE hours."
Why dashboards lie about the move
Most AI adoption metrics are subtraction metrics.
Minutes saved. Tickets auto-handled. Drafts generated. Cost per interaction down.
Those numbers answer one question: did we remove a human step?
They do not answer: did we create a verification step that is slower, costlier, or more politically expensive than the step we removed?
A classic pattern:
- Automate first-pass classification.
- Celebrate the drop in queue time.
- Discover that false confidence creates expensive corrections downstream.
- Add a review layer that was never budgeted.
- Call the review layer "temporary" for eighteen months.
The temporary review is the complexity, relocated.
Another pattern: the model writes a correct-looking summary of three systems that disagree. The UI shows one paragraph. The systems still disagree. You have not solved state hardness. You have papered over it with fluent prose.
Fluent prose is not convergence.
A third pattern is quieter. The team ships a prompt that works on last month's tickets. Product changes a field name in the CRM. Nobody updates the tool schema. Accuracy falls for two weeks while dashboards still report "AI handled volume." The failure is not intelligence. It is an unowned contract.
Subtraction metrics cannot see unowned contracts.
The mental model: deletion vs deposit
Treat every AI introduction as a transfer, not a deletion.
Human step removed
↓
Work deposited into:
verification · contracts · ownership · failure design · run cost
If you cannot name the deposit, you have not finished the design. You have finished the demo.
Mature teams ask a different question than "how much labour did we remove?"
They ask: where did the judgment go, and is that place staffed, instrumented, and owned?
Immature teams celebrate the empty chair and are surprised when the incident channel fills up.
Where the deposit usually lands (and what to budget)
| Deleted human step | Typical deposit | Budget for |
|---|---|---|
| Reading and classifying | Eval set + drift monitoring | Ongoing eval ownership, not a one-off golden set |
| Looking up records across tools | Data contracts + authority map | Schema owners, pending states, who wins on conflict |
| Writing the first reply | Human review policy | Sampling rates, escalation criteria, audit trail |
| Executing a side effect (refund, update, email) | Idempotency + compensation | Idempotency keys, replay safety, clear undo paths |
| Noticing something is wrong | Observability for model + tools | Latency, tool error rates, disagreement windows |
None of that is anti-AI.
It is how you keep the savings from turning into a different kind of overtime.
Teams that treat AI app delivery as production engineering already know this shape. The model is a component. The path around it is the product.
AI workflows inherit every distributed systems bill
Once an agent calls tools that mutate business state, you are no longer "using AI."
You are running a distributed write path with a probabilistic planner in the middle.
That path still needs:
- Named pending states when tools disagree
- Retries that do not double-charge or double-update
- Contracts so the model is not guessing field meanings
- An owner when the path invents a confident wrong answer
The industry language of "agents" can hide this. Agents sound autonomous. Production still needs boundaries.
I have seen programmes where the model layer was impressive and the surrounding path was a soft CRUD status field with no legal transitions. The AI looked modern. The workflow was still a dropdown. Complexity had moved from typing into unverifiable side effects.
If the non-AI version of the path could not survive partial failure, the AI version will not either. It will fail faster, with better grammar.
A checklist before you call complexity "eliminated"
Use this when a stakeholder says AI removed a process:
- Which human judgment was deleted? Name the decision, not the tool.
- Where does that judgment live now? Model, reviewer, rule engine, or "nowhere"?
- What is the verification budget? Sampling rate, eval cadence, who signs off on changes to prompts or tools.
- What are the data contracts for every tool input and output? Who owns schema when CRM fields drift?
- What happens on partial tool success? One write lands, one times out, the model narrates done. What is true?
- Are side-effecting tool calls idempotent? Retries are normal. Duplicates are a design choice.
- Who is paged when the path is confidently wrong? A vendor SLA is not an owner.
- Which metric would show the deposit, not only the deletion? Review hours, correction rate, disagreement window breaches, cost per successful safe action.
If answers 2 through 8 are vague, the complexity was not eliminated.
It was deferred until the first ugly Monday.
What maturity looks like
Mature AI engineering does not mean bigger models.
It means treating relocated complexity as first-class work.
Evals are not a science fair project. They are regression tests for judgment. When the prompt, tools, or upstream schemas change, the suite runs again. A one-time golden set is a souvenir.
Human review is not a failure of automation. It is a control plane for irreversible actions. Reversible drafts can run hot. Refunds, legal notices, and CRM mutations should not.
State design still matters more than prompt cleverness. An agent that updates three systems without an authority map is an integration programme with a stochastic router. The hard part remains the same as every other multi-writer path: who may write, what may lag, and how you notice when lag becomes a lie.
Operational cost is part of the product. If token spend and review load are invisible, your "savings" are an accounting error. Include them in the same conversation as "minutes saved," or you are comparing a numerator to a missing denominator.
The calm teams I trust do not argue about whether AI works.
They argue about which complexity they are willing to hold, and where.
That is the same discipline that makes eventual consistency operable, retries safe, and data contracts enforceable. AI did not invent those problems. It made them louder by removing the humans who used to absorb them quietly.
Related reading
- State Is Harder Than Scale: divergence outlives the demo
- Why Eventual Consistency Is Usually Correct: AI tool paths commit in pieces
- Data Contracts Matter More Than APIs: schemas are load-bearing for models too
- Why Retries Make Bugs Worse: timeouts plus side effects without idempotency
The honest accounting
AI is good at deleting steps that were mostly pattern matching under fatigue.
It is bad at deleting the need for truth, ownership, and recovery.
When a programme claims it eliminated complexity, ask to see the deposit: the evals, the contracts, the review policy, the idempotent tools, the named owner for confident failure.
If those are missing, the complexity is still in the building.
It just stopped showing up on the old dashboard.
AI doesn't eliminate complexity. It moves it into the parts of the system you only notice after the demo.