What breaks in real AI implementations

The fastest way to build false confidence in an AI system is to test it with the cleanest input, on the shortest path, while the project team is watching.

Demos reward model capability. Production rewards system behaviour.

That distinction has become more important as powerful models turn sophisticated features into a few lines of code. Summarization, extraction, search, classification, voice interaction, and tool use can look almost finished before the surrounding product has learned how to handle volume, ambiguity, timeouts, permissions, human review, or failure.

In production, the AI is rarely the whole system. It is one uncertain component inside a very certain business promise.

My work with AI agents, retrieval systems, voice workflows, and quality automation keeps returning to the same lesson: real implementations break at the seams. The interesting work starts after the first successful answer.

Representative inputs break polished demos

In one anonymized implementation, the upload flow worked with a small synthetic file. The team had evidence that the endpoint, queue, and processor could communicate. Then a representative 12.8 MB archive entered the same path and failed.

The product was designed to accept files much larger than that. The application code was not the immediate constraint. An intermediary proxy had a 10 MB request-body limit.

Nothing about the model changed. The failure lived in the path around it.

This is not an exotic edge case. Real inputs are long, irregular, multilingual, duplicated, partially corrupt, badly named, and larger than the examples used during development. Voice calls contain silence, interruptions, background noise, and transfers. Knowledge bases contain conflicting versions. Customer forms contain unexpected characters and missing fields.

The important principle is simple: test the contract, not merely the code path. If the product promises a certain file size, response time, language, or recovery behaviour, the end-to-end system has to prove that promise under representative conditions.

Retries are a business rule

Another implementation exposed a less visible seam. An external provider returned malformed JSON. The worker did the sensible thing and scheduled a retry with backoff. The problem was timing.

The retry became available just after the worker's 30-second idle window ended. The job was correctly queued but was not reclaimed in the expected operating cycle. Each component behaved plausibly on its own. Their timing assumptions did not form a reliable system.

The defect was fixed and a deterministic retry test proved the corrected behaviour. The more useful lesson is that “we retry on failure” is not an implementation detail. It is an operating promise with unanswered questions:

• Which failures are safe to retry?
• How many times, over what period, and at what cost?
• Can a repeated request create duplicate actions?
• Who is alerted when retries are exhausted?
• What state does the customer see while recovery is happening?
• Can an operator safely resume or replay the work?

For a lead-scoring workflow, a delayed retry may be inconvenient. For a customer communication, payment action, or decision affecting a person, duplicated or silently abandoned work can be material.

Retries, idempotency, dead-letter handling, and operator recovery belong in the business design because they determine what actually happens to people when the system is imperfect.

External providers change the reliability equation

AI systems often depend on models, transcription services, vector stores, email platforms, telephony, identity providers, and data sources outside the company's direct control. The application may be healthy while one dependency is slow, rate limited, unavailable, or returning a response that satisfies the transport protocol but violates the expected schema.

Vendor selection therefore cannot stop at benchmark quality or price per token. Leaders need to understand:

• availability and rate limits;
• data handling and retention;
• version changes and model deprecation;
• regional hosting and contractual obligations;
• fallback behaviour and portability;
• how incidents are communicated;
• which part of the service the company can operate without the vendor.

A fallback does not always mean a second model. It may be a deterministic report, a human queue, a reduced-function experience, or an honest message that preserves the user's work. Reliability is the ability to keep the business promise at an acceptable level, not the ability to pretend every dependency is available.

The workflow breaks before the model does

A model can produce a good answer and still fail the business.

Consider an agent that qualifies a sales lead. If nobody owns the next action, the classification creates no value. If the CRM fields are inconsistent, the agent may route correctly into a broken process. If representatives do not understand the rationale, they may ignore the recommendation. If the system sends a message without respecting consent or regional requirements, technical success creates commercial risk.

This is why process mapping precedes automation. Teams need to see the current decision, actors, handoffs, exceptions, controls, and measures. AI changes one part of that system. The rest must be redesigned around the new capability.

Observability has to include the business

Traditional monitoring asks whether the service is up, how long requests take, and how many errors occur. AI adds another layer. A system can be available and returning valid responses while becoming less useful, less accurate, or less aligned with the workflow.

Operational trust requires visibility across four dimensions:

• Technical: latency, availability, error types, retries, cost, and dependency health.
• Model: task-relevant quality, refusals, drift, uncertainty, and recurring failure patterns.
• Human: adoption, overrides, corrections, workarounds, and overreliance.
• Business: outcome, cycle time, service quality, loss, conversion, and downstream exceptions.

The metrics need to connect. A falling override rate might mean the model improved. It might also mean users stopped checking. Higher usage might indicate value, or that a mandatory tool added steps. A lower cost per request may be irrelevant if more requests need manual repair.

The owner of the business outcome must interpret these signals with the technical team. Otherwise, the system is optimized for what is easy to measure.

Human review must be designed, not announced

“A human remains in the loop” sounds reassuring. It says very little about the actual control.

Who is the human? What information do they see? Do they have time, authority, and expertise to disagree? Does the interface make uncertainty visible? Are their corrections captured? What happens during high volume? Can the organization prove that review occurred?

A review step can become ceremonial when people accept outputs by default or when the queue exceeds capacity. It can also destroy value if every low-consequence answer requires manual approval.

The right design depends on consequence. Low-risk drafting may need sampling and feedback. Customer-facing recommendations may require confirmation before action. High-impact decisions may require independent evidence, explanation, recourse, and strict limits on automation.

The operating owner matters more than the project sponsor

Projects have sponsors. Services need owners.

After launch, someone must decide when to update prompts, models, thresholds, knowledge, or workflows. Someone must review incidents and recurring errors. Someone must manage provider changes, cost, and user training. Someone must decide when the system no longer deserves to exist.

If that responsibility lives only with the delivery team or vendor, the business has outsourced control of a capability it depends on.

A credible operating model names:

• the business owner for value and acceptable behaviour;
• the technical owner for service reliability and change;
• the data owner for access, quality, and lifecycle;
• the risk and privacy roles for oversight and escalation;
• the frontline role for feedback and exception handling.

This engineering debt has been visible for years

The foundational paper “Hidden Technical Debt in Machine Learning Systems” warned that live ML systems accumulate complexity through data dependencies, glue code, pipeline jungles, external changes, monitoring gaps, and weak recovery tooling. The model code is only a small part of the production system.

Generative AI changes the interface and expands what can be built. It does not repeal systems engineering. In some ways, it raises the standard because model outputs are probabilistic, provider behaviour changes, and natural-language interfaces invite users to explore paths the product team did not anticipate.

The answer is not to wait until every uncertainty disappears. It is to design for uncertainty as a normal operating condition.

What to demand before calling it production

1. Representative end-to-end tests, not only model evaluations.
2. Named owners for outcome, reliability, data, and risk.
3. Explicit handling for uncertainty, exceptions, and human escalation.
4. Observability that connects technical events to business consequences.
5. Safe retries, duplicate prevention, and operator recovery.
6. Dependency and fallback plans proportionate to the business promise.
7. A controlled release, feedback loop, and ability to stop.

None of this is glamorous. That is precisely why it differentiates builders from demonstrators.

A reliable AI implementation is not the one that never fails. It is the one that knows where failure can occur, detects it quickly, contains the consequence, preserves the user's trust, and becomes better because the failure happened.