AI Reality Checks #17 · Part of Binary and Beyond. LinkedIn newsletter edition follows.
Garbage in, garbage out was never a slogan about storage.
It was a warning about decisions.
If the inputs are wrong, inconsistent, or owned by nobody, the output will be wrong with confidence. Spreadsheets taught that lesson slowly. Batch jobs taught it on Monday mornings. Human review caught some of it because someone still had to stare at a row and feel uneasy.
LLMs and AI workflows change the economics.
They do not invent clean definitions. They scale whatever you already have: bad taxonomy, stale CRM fields, three meanings of "lead," and tribal knowledge that never made it into a field description. Then they write emails, update records, score opportunities, and summarise accounts as if the underlying vocabulary were settled.
The million-dollar problem is not the model.
It is treating messy enterprise data as if a prompt can heal ownership and contracts.
The pilot that scored the wrong pipeline
Picture a mid-market B2B company with a CRM that has grown for a decade.
Sales, marketing, CS, and finance each use Account, Contact, and Opportunity. The fields look shared. The meanings are not.
Marketing's "MQL" is a form fill plus a score threshold that changed twice last year. Sales marks "SQL" when a discovery call happens, or when a demo is booked, depending on the region. CS uses Customer for anyone with an active seat. Finance uses Customer for anyone who has been invoiced this fiscal year. A partner account can be "active" in the portal and "prospect" in the CRM because nobody owns the bridge.
Leadership funds an AI workflow: summarise accounts, draft outreach, and recommend next best actions from CRM plus email history.
The demo is impressive. The model writes fluent notes. It ranks accounts. It suggests who to call.
Three months later (this is a labeled fictional scenario, not a case study), the programme has spent roughly $180k on model access, integration work, and change management. Pipeline reviews start citing AI-ranked opportunities. One quarter later, a reconciliation shows that a material share of "high priority" accounts were partners, churned seats still marked active, or leads whose stage names had drifted after a taxonomy rename that never updated historical rows.
The model did what it was asked.
It automated disagreement.
Nobody had failed at prompting. They had failed at deciding what a row was allowed to mean before they asked software to act on it at scale.
Why AI makes GIGO expensive
Before generation and tool-calling agents, bad data mostly produced bad reports and awkward meetings.
People argued. Someone opened five tabs. Someone said "ignore that field, we use the other one." The organisation paid in time and political friction.
AI workflows change three things at once.
1. Volume.
A human misreads twenty accounts a day. An agent can touch thousands of records, draft thousands of messages, and leave thousands of confidence-shaped summaries in places executives will quote.
2. Authority theatre.
Fluent language reads like judgment. A wrong status in a CRM field is a data problem. The same wrong status inside a polished paragraph feels like insight. Teams override their own unease because the prose sounds finished.
3. Side effects.
Modern AI delivery is not only chat. It writes back: updates fields, creates tasks, routes tickets, triggers sequences. Garbage stops being a display issue and becomes a write path. AI Reality Checks #16 argued that AI moves complexity rather than deleting it. GIGO is where that moved complexity lands: on ownership, validation, and recovery when the agent was confidently wrong.
The old GIGO joke assumed you could spot the garbage on the way out.
Generation makes the garbage look curated.
The real inputs are definitions, not documents
Teams often treat "better data for AI" as a retrieval problem.
Index the wiki. Embed the tickets. Connect the CRM. Add more context.
Context helps. It does not replace a definition.
If five systems disagree about whether a record is a customer, more context gives the model five confident stories. If "open opportunity" includes parked deals that sales never closes, the model will treat inertia as pipeline. If stage names were renamed in the UI but not backfilled, historical reasoning becomes archaeology with a temperature setting.
Architecture Files #13 made the case that an API without a data contract is a suggestion. AI workflows are the most aggressive consumers those contracts ever meet. They do not politely ignore ambiguous fields. They invent a reading and proceed.
Production Notes #03 argued that truth is owned, not shared. AI does not create a single source of truth by summarising everything into one paragraph. It creates a single narrative over contested ownership. Narratives travel farther than schemas. That is the risk.
What has to be true before generation is useful:
- Named meaning for the fields that trigger action (customer, lead, stage, amount, churn risk, open ticket).
- Named owners who can change that meaning, and a process that reaches consumers before the change ships.
- Freshness rules: how stale is too stale for an automated decision.
- Authority during conflict: which system wins for which decision when CRM, billing, and product analytics disagree.
- Validation before write-back: what must be checked before an agent is allowed to update a record or send external language.
Without those, "RAG over the enterprise" is a polite name for amplifying undocumented folklore.
Where the money actually goes
Model invoices are visible. They are rarely the large bill.
The expensive failure modes look like this (again, illustrative programme costs, not industry averages):
Wrong outreach at scale.
An agent drafts sequences from stale "interested" flags. Sales burns cycles. Prospects learn you do not know their account. Recovery is relationship work, not a config change.
Pipeline fiction.
Leadership plans headcount and spend on AI-ranked opportunities that were never opportunities under finance's definition. The correction arrives as a bad quarter, not a model eval score.
Silent write corruption.
An agent "normalises" fields: maps stages, fills empty owners, merges notes. It spreads one team's vocabulary into another team's records. Months later, nobody trusts the CRM enough to automate anything else.
Re-platform tax.
After the pilot, the organisation finally does the taxonomy and ownership work it skipped. That work was always required. Doing it after you have trained people to trust fluent wrongness costs more than doing it first.
I have watched teams blame the model for "hallucinating" when the CRM already contained three incompatible truths. The model did not invent the conflict. It published it.
Humans papered over what models will not
People are terrible databases and excellent diplomats.
A senior AE knows that "Enterprise - West" means something different from the dropdown label. A CS lead knows which "active" accounts are really waiting on a renewal conversation. Finance knows which "closed-won" rows still need a credit memo.
That knowledge lives in Slack, call notes, and memory. It never becomes a contract. The organisation still functions because humans reconcile at the edges.
AI workflows remove the diplomat and keep the database.
They will not call the person who "just knows." They will call the field. If the field is a compromise from 2019, the compromise becomes policy at machine speed.
This is why "we'll clean the data after the pilot" is usually backwards.
The pilot is the moment you teach the organisation that messy inputs are acceptable because the output sounds good. That habit is harder to unwind than a prompt template.
What to fix before you scale generation
Use this checklist when a programme wants to put LLMs on CRM, support, or internal knowledge with write-back or executive visibility.
- List the nouns that trigger money or promises. Customer, lead, opportunity, invoice, entitlement, severity. If two teams define one noun differently, stop. Resolve the noun before you generate language about it.
- Name an owner per noun (a role, not a Slack channel). Who may change meaning, and who must be notified?
- Write one sentence per critical field: what it means, what
nullmeans, how fresh it must be, and which system is authoritative when copies diverge. - Separate read assistants from write agents. Summaries that cannot update records are a different risk class from agents that can. Do not promote one into the other without validation gates.
- Define illegal transitions for stages and statuses. If humans should not jump
prospect→closed-wonwithout evidence, an agent must not either. (Workflow discipline still applies when the actor is a model.) - Require evidence citations for actions that spend attention or reputation. Which fields, which tickets, which events? Fluent prose without provenance is a report, not an operational input.
- Run a disagreement drill before launch. Pick ten accounts where CRM, billing, and product analytics conflict. Decide what the agent is allowed to say and do in each case. If the room cannot agree, the model will not save you.
- Budget for remediation ownership. Who watches false positives, poisoned fields, and silent taxonomy drift after week two?
If the checklist feels heavier than the model integration, that is the point. The integration was never the hard part.
A different standard for "AI-ready data"
"AI-ready" is often sold as volume, embeddings, and connectors.
Operationally, AI-ready means something narrower:
The organisation can defend, in one sentence each, the meanings an automated system will treat as true, who owns those meanings, and what happens when two systems disagree.
That standard does not require perfect data.
It requires bounded data: known gaps, named authority, and validation before side effects. Perfect cleanliness is a fantasy. Owned ambiguity is engineering.
Teams that skip this do not fail because the model is weak. They fail because they asked generation to paper over a governance debt that humans used to absorb in private.
When we help enterprises put AI into production workflows, the durable work is rarely the cleverest prompt. It is the same unfashionable work as any serious integration: contracts, ownership, and honest pending states when the inputs are not yet trustworthy enough to act.
The prompt will not heal the contract
Prompts are instructions.
Contracts are commitments about meaning under change and disagreement.
You can prompt a model to "be careful with customer status." You cannot prompt it into a definition your company refused to own. You cannot prompt away a decade of overloaded fields. You cannot prompt a write path into safety if nothing validates the write against an authority map.
GIGO was always true.
What changed is the blast radius. Bad inputs used to waste analyst hours. Now they waste quarters of automated decisions that look like progress until reconciliation arrives.
Fix definitions, authority, and validation before you scale generation.
The model can wait. The million-dollar mess cannot.
Garbage in, garbage out was always an engineering problem.
LLMs only made the invoice match the metaphor.
Related reading
- Data Contracts Matter More Than APIs: meaning, compatibility, and who may change a field
- The Myth of a Single Source of Truth: owned authority instead of shared wishful thinking
- AI Doesn't Eliminate Complexity. It Moves It.: where the hard problems go after the demo
- Why Every Workflow Is Really a State Machine: illegal transitions still matter when the actor is an agent
AI Reality Checks #17 · Part of Binary and Beyond. LinkedIn newsletter edition follows. Building AI workflows where definitions and validation come before scaled generation? Start a conversation.