AI Reality Checks #20 · Part of Binary and Beyond. LinkedIn newsletter edition follows.
The demo asks a quiet question and gets a fluent answer.
The model cites a CRM note, drafts a reply, and updates a ticket. Someone in the room says the hard part is done. The model is smart enough. Trust will follow.
It does not.
Trust is not a property of the weights. It is a property of the path around them: who may act, what they may change, how disagreements resolve, and whether a human can reconstruct the decision after the fact.
Teams that chase smarter models while skipping those boundaries keep shipping systems people use carefully, then abandon the first time something looks confident and wrong.
Building AI systems people can trust is not a branding exercise.
It is production engineering with a stochastic component in the middle.
The pilot that earned applause and lost the floor
Picture a mid-market support org with an "AI first reply" programme.
Week one is magic. Response time drops. Drafts look senior. Leadership screenshots the dashboard.
Week four is quieter. A VIP gets a refund tone that legal never approved. Two accounts receive the same commitment because a tool call timed out and retried. A compliance lead asks for the trail of one reply and gets a model completion log with no tool payloads, no policy version, and no named reviewer.
The model did not become less capable.
The organisation discovered it could not defend the path.
That is the trust failure. Not low BLEU. Not a wrong citation in a lab eval. An inability to answer: who authorised this, against which rules, with which systems, and what happens when those systems disagree.
Trust is not accuracy
Accuracy matters. It is also incomplete.
A system can be accurate in the eval set and untrustworthy in production because:
Authority is fuzzy. The model acts as "the AI" instead of a named role with scoped permissions.
Writes are unbounded. Drafts and mutations share one path. Retries invent duplicate truth (Production Notes #05, Production Notes #06).
Contracts are tribal. Field meanings live in Slack. The model narrates disagreement as settled fact (AI Reality Checks #17, AI Reality Checks #18).
Review is theatre. A checkbox that nobody owns is not oversight. It is a liability magnet.
Observability stops at fluency. Dashboards show tokens and thumbs-up. They do not show safe mutations per hour with audit.
People trust systems that are boring under scrutiny.
Fluent systems that cannot explain themselves get demoted to "assistive only" the first time an exec gets a call from legal.
What trust looks like in the stack
A useful way to read an AI feature:
Prompt / model turn
↓
Policy gate (what this role may ask for)
↓
Tool / retrieval path (contracts, auth, idempotency)
↓
Business systems (CRM, warehouse, ticketing)
↓
Human review or auto-commit (matched to blast radius)
↓
Audit record (inputs, tools, policy version, actor)
Most trust incidents start below the model.
MCP and tool protocols make the middle layers cheaper to wire (AI Reality Checks #19). They do not invent ownership. They make missing ownership visible faster.
Immature programmes measure demo wow.
Mature programmes measure defensible actions.
The four surfaces that earn trust
I use these as a design checklist when someone asks whether an AI path is "production ready."
1. Named authority
Who is the model acting as?
User-delegated identity, service account, or impersonation are different products. "The bot" is not an identity. If CRM and warehouse disagree, which result is canonical while the path is pending (Production Notes #09)?
Without a named authority map, every incident becomes a meeting about blame.
2. Bounded side effects
Separate read tools from mutating tools in review policy.
Drafts can be automatic. Refunds, deletes, and legal language should not share that lane. Every mutating tool needs an owner, an idempotency key, and a failure story when only half the path commits.
If the model needs context, give it a read model or a side table with an explicit merge contract. Do not grant "the agent needs the main tables" as a second product living inside your order schema (Architecture Files #10).
3. Visible review matched to risk
Review is not a moral stance. It is a rate limiter on blast radius.
Low-risk drafts can ship with sampling. Irreversible mutations need a human gate with a name on the approval. Temporary review layers that last eighteen months are not temporary. They are the product. Budget them.
4. Reconstructable audit
After the fact, a reviewer should be able to answer:
- What inputs and tool results did the model see?
- Which policy and prompt versions were live?
- Who approved the mutation, if any?
- What was written, where, and under which key?
If the answer is "we have the chat transcript," you have a demo log.
You do not have an audit trail.
A contrast teams blur
| What teams ask for | What trust needs |
|---|---|
| A better model | Owned paths around any model |
| Higher auto-resolve rate | Bounded mutations with recoverable failure |
| "AI handled it" | Named actor + policy version on the record |
| One green confidence score | Pending states when systems disagree |
| Prompt polish | Contracts, auth, and evals for partial failure |
Prompt work is real. It does not replace the table on the right.
How this shows up in delivery
When we build AI app delivery for agencies and product teams, the first credible milestone is rarely a flashier completion.
It is a boring one: a path that legal can walk, ops can operate, and engineering can page.
On the Arkreach PR analytics build, the hard trust problem was not generating sentiment scores. It was making article-level measurement defensible for agency clients who would quote those numbers in client reviews. The architecture had to survive scrutiny, not just a demo day (case study).
Government and regulated briefs make the same point louder. On-premise screening, review queues, and audit are not features you bolt on after the model looks smart. They are the product envelope the model sits inside.
Trust is the envelope.
A checklist after "the model is good enough"
Use this when a stakeholder says the remaining work is polish:
- List every mutating action. Reads are cheap. Writes need owners and keys.
- Map authority. User, service, or impersonation — and what each may touch.
- Name the canonical system when tools disagree. Pending is allowed. Silent lying is not.
- Match review to blast radius. Drafts and irreversible acts do not share one gate.
- Version prompts and policies the same way you version APIs.
- Log tool payloads, not only model text.
- Write evals for partial failure. Timeout mid-flow, stale read, schema drift, duplicate write.
- Name on-call for the path, not "the AI vendor."
If items 2–8 are TBD, the model may be good enough.
The system is not yet trustworthy.
What good looks like six months in
Calm teams treat AI features like any other production surface:
- Mutations go through the same change control as REST endpoints
- Data contracts are tested when upstream schemas move
- Review policy is written down and staffed
- Dashboards show tool reliability and audit completeness, not only fluency
- Incidents produce path changes, not model blame
They do not argue about whether the model is smart enough to trust.
They argue about which production boundaries each tool is allowed to cross — and whether the programme budgeted for that work after the demo looked finished.
Related reading
- AI Doesn't Eliminate Complexity. It Moves It.: deleted human steps deposit verification and ownership work
- MCP Is Just the Beginning: connectors do not replace auth, contracts, or audit
- Garbage In, Garbage Out Is Now a Million-Dollar Problem: fluent automation scales disagreement
- AI app development: production AI paths with governance, not demo glue
The honest close
People do not trust AI systems because the answer sounded right in a meeting.
They trust systems that stay boring when someone asks hard questions: who acted, under what authority, against which contracts, with which recoverable failure modes.
Smarter models help.
Owned boundaries are what make those models safe to leave running on Monday.
Trust is not a model upgrade. It is an operating system for action.