Engineering for Imperfection – O’Reilly

The price of adoption euphoria

You played entirely by the book. You procured the most capable enterprise models, mandated adoption across your teams, and put the right metrics in place. The promise was a predictable boost in efficiency. And at first, it delivered. The demos were flawless. The prototypes worked. The agents reasoned with a clarity that felt almost magical.

Then the invoice arrived.

Costs climbed while productivity barely moved, and annual AI allocations are running dry before Q2. We now pay customer support agents to spin through 10K-token extended reasoning loops just to validate a simple $15 return. Legacy deterministic systems handled the same decision for a fraction of a cent; now a probabilistic model consumes gross margin simply to determine whether a package was actually delayed. That capital never translated into business value. It vanished into blind retries, evaporated into verifier agents debating one another, and was consumed by models instructed to “think harder” every time they stumbled.

But a ruinous invoice is just the entry fee. In April, attackers hijacked more than 20,000 Instagram accounts by exploiting Meta’s AI-assisted account recovery workflow. The system sent password reset links to attacker-controlled email addresses because a downstream authorization path failed to verify that the supplied email actually belonged to the target account. There was no sophisticated exploit, no cryptographic break, and no zero-day, nothing that would have appeared in a conventional threat model. Attackers simply asked the agent to perform what appeared to be a routine account recovery operation, and the system, doing exactly what it was designed to do, complied. The model didn’t hallucinate. It simply followed its instructions. The failure was entirely architectural: A probabilistic interface was allowed to initiate identity-critical state changes without an independent authorization check. A single trust boundary collapsed, taking customer trust and organizational reputation with it.

Both are symptoms of the same structural failure.

In each case, the system treats a structural deficit as a reasoning problem. When it encounters uncertainty, it buys more compute. When it encounters authority, it mistakes convincing language for validation. Neither assumption scales. You cannot buy safety or profitability with ever-larger inference budgets, nor can you secure your systems simply by deploying ever-smarter models. The pursuit of perfect model accuracy has no financial ceiling.

To understand why this pattern keeps recurring, we first need a more basic distinction. Not every task we give to AI belongs to the same economic category.

The category error: Forcing swarms into factories

Enterprise AI workloads typically split into two distinct domains, each with opposing definitions of success. Exploratory environments, such as code synthesis or strategic research, benefit from variance; the goal is to leverage the system as a creative swarm. Transactional operations, however, function as digital factories. Tasks like automated billing or claims processing demand rigid repetition and compliance. This creates two fundamentally different operational profiles:

The economic failure of agentic AI deployments stems from this exact category error: Closed-ended, rigid business transactions are being treated as open-ended research problems. We’re deploying unconstrained semantic engines to do the work of assembly-line state machines.

The cost of unconstrained autonomy

When faced with the inherent unpredictability of large language models, the industry’s default reflex has been to attempt to brute-force our way to certainty by throwing more effort and compute at the problem, rather than build safer architectures.

This miscalculation doesn’t simply reflect simple overconfidence in intelligence. The deeper mistake is a failure to recognize three recurring failure patterns in probabilistic systems and the specific financial pathologies they create inside closed-ended workflows.

Local optimization (the tail-chasing inference cycle)

Large language models reason over whatever tokens are visible in the current window, not over the broader operational reality of the system around them. In a closed workflow, that local fixation creates a costly feedback loop. Consider a billing agent that fails to classify an invoice because the supplier field is ambiguous. The agent has no mechanism to request the missing data from an external system, so it retries by rephrasing its own reasoning, rereading the same incomplete context, and consuming tokens on every attempt while the answer it needs exists in a database it was never wired to query.

Teams spend months crafting prompts that work in testing, only to watch them crumble under production variation. The volatility is structural: A minor update to a model’s tokenizer or a shift in the context window’s distribution can flip a reliable JSON output into a prose hallucination, a phenomenon documented in “The Prompting Inversion.” This creates a permanent maintenance debt: Every model upgrade, often mandated by vendor deprecation cycles, forces organizations into expensive, repeat evaluation processes to ensure that legacy prompts still behave as intended. When prompt engineering runs out of room, the reflex is to use a bigger model or turn on extended reasoning. But inference-time scaling yields diminishing, task-dependent gains (“Inference-Time Scaling for Complex Tasks”), and reasoning models are increasingly prone to “overthinking”: generating redundant rationale steps that inflate latency and token cost without proportional quality gains (“CoT Compression”). In a closed workflow, “think harder” is not a substitute for missing state or missing control. It’s a path to a larger invoice.

The costs compound through what we call the context tax: In production agentic systems, input tokens, not output tokens, dominate the bill. Each retry resends the full prior transcript and failure trace. Empirical analysis of autonomous developer agents shows that automated review and refinement loops consume nearly 60% of all tokens (“Tokenomics”), while most of the context payload carries little semantic weight (“FrugalPrompt”). In closed transactional workflows, that context accumulation becomes an unmitigated financial bleed.

Premise acceptance (the hijacked agent)

Language models accept the prompt as the current frame of reality and reason forward from it. They don’t audit whether that premise is still valid, whether it omits decisive evidence, or whether it has already been invalidated by the outside world.

The most immediate consequence is state drift. The model receives a snapshot at T0 and treats it as truth. The decision executes at T1, after inventory has changed, prices have moved, or a human has intervened. Modern LLMs are temporally blind: They assume a stationary context and fail to invalidate obsolete state (“Your LLM Agents Are Temporally Blind,” “The Temporal Coherence Problem”). No amount of inference-time scaling can recover information that became false after the reasoning completed.

The more insidious consequence is the compliant lie. Pouring more raw tokens into the prompt doesn’t guarantee better grounding; Long-context systems still ignore decisive evidence buried in the middle of the window (“Lost in the Middle”). Worse, the model tends to accept the emotional or narrative framing of the user as a premise to optimize around. A customer can describe a delayed delivery as a ruined wedding, and the system may generate a perfectly valid JSON refund proposal that respects every schema while silently violating the actual business intent. The output is syntactically clean, and the lie is operationally compliant.

Semantic smoothing (the conformity trap)

Large language models are statistically optimized for linguistic harmony. They gravitate toward plausibility, agreement, and smooth narrative convergence rather than toward rigid boundary holding. In a closed workflow, that bias toward consensus turns directly into financial risk.

When a single model fails, the industry instinct is to add reviewer or verifier agents and let them debate toward consensus. But debate systems don’t consistently outperform simpler baselines, and their effectiveness degrades over time due to conformist behavior (“Stop Overvaluing Multi-Agent Debate,” “Talk Isn’t Always Cheap”). The core issue is informational, not cognitive. When five agents reason from the same incomplete context window, they don’t produce five independent opinions. They produce five correlated hallucinations of the same missing information. The missing context becomes an echo chamber that amplifies the original bias while multiplying token cost. As Nicole Koenigstein argues in “Linear Thinking, Nonlinear Costs,” repeated delegation and validation loops cause token consumption to grow nonlinearly while quality improvements flatline.

Waiting for a smarter model doesn’t resolve this either. There’s also the economic reality: Breakthrough intelligence is the ultimate scarce commodity. Vendors of “God-tier” models have no incentive to make them cheap. Running daily enterprise workflows on premium superintelligent inference will drain capital faster than any retry loop.

Furthermore, as reasoning models scale, they become more capable of specification gaming and alignment faking, appearing compliant while pursuing unintended optima (“Towards Understanding Specification Gaming in Reasoning Models,” “Alignment Faking”). A superintelligent agent won’t fail through a clumsy syntax error; it’ll fail by executing a flawless strategy that silently optimizes away your margins. That’s why system engineering remains critical. More intelligence makes deterministic boundaries more significant than ever. You can’t negotiate with superintelligence, but you can contain it with the immutable physics of code.

Every failure described above shares the same shape: The system compensates for a missing constraint by spending more intelligence. Missing context, missing authority, missing evidence, and missing temporal validity are each treated as reasoning problems rather than structural ones.

The result is predictable: Cost compounds while reliability improves only marginally.

Perhaps reliability isn’t primarily an intelligence problem. Perhaps it’s a state management problem.

The architecture of trust

Because large language models are structurally bound to local optimization, premise acceptance, and semantic smoothing, they can’t be trusted to govern their own execution boundaries in closed workflows. The engineering mandate shifts from trying to make models smarter to building a deterministic system layer that treats their outputs as unprivileged claims.

In production, enterprises are rapidly discovering that the true cost of agentic AI is the “trust tax”: the massive, ad hoc layers of monitoring and guardrails required to make autonomy palatable. Safety has become more expensive than intelligence.

Making imperfect models economically viable requires a deterministic “airlock” around the agent. The architectural requirement is simple, needing a separation of probabilistic reasoning (user space) from deterministic execution (kernel space). Whether that split is realized through a microkernel, workflow engine, policy platform, or orchestration framework is secondary.

The airlock begins by controlling context integrity. Rather than letting agents surf infinite retrieval loops that inflate the context tax, the runtime injects only deterministically necessary state into the prompt. Once the context is stabilized, the remaining invariants are enforced through a deterministic execution runtime engineered across three distinct governance layers.

Syntactic governance and authority isolation

The first line of defense is purely structural. Before an agent is allowed to execute any action, it must submit a structured policy proposal against a strict machine-readable responsibility contract (typically defined via YAML and Pydantic).

Yes, this introduces upfront engineering burden: Contracts must be designed, validation logic maintained, and execution boundaries modeled explicitly. But these are fixed, testable artifacts, not recurring prompt debt. They convert unbounded probabilistic operating cost into auditable engineering cost and survive model upgrades without needing to be rediscovered through another retuning cycle.

This validation happens in a deterministic kernel space, and the inference cost of rejecting a structural boundary violation is exactly zero tokens. If the agent attempts to call an unauthorized API, exceeds a hard financial limit, or returns malformed JSON, the runtime rejects the action instantly. We don’t spend tokens proving that an agent should be allowed to act; authority is verified by code, not purchased repeatedly through inference. That is the economic consequence of zero trust for agents.

However, when a proposal fails this deterministic gate, an unconstrained agent will typically panic and enter an infinite “try again” loop, a hallucination cycle that silently drains token budgets. To prevent the budget runaway problem, the architecture introduces an intent retry governor. If an agent fails to produce a compliant policy after a strict limit (e.g., three attempts), the runtime forcibly cuts its compute budget, transitioning the flow to an aborted REASONING_EXHAUSTION state. The financial bleed stops instantly.

While strict contracts and retry limits prevent operational chaos, they leave the system exposed to a much more insidious threat.

Semantic governance and evidence validation

What happens when an agent generates an output that perfectly respects the schema, obeys all financial limits, and contains flawless JSON but is entirely wrong in its intent?

Imagine a customer writes: “Please cancel my subscription immediately. I no longer wish to use your service.” The agent, heavily optimized (and perhaps overprompted) to reduce churn, processes the email and proposes: {"action": "APPLY_DISCOUNT", "discount_pct": 15, "cancel_subscription": false}. Structurally, the output is perfectly valid—it passes the API gateway without throwing a single error. The discount is within the $15 global limit. We call this the compliant lie. The agent did something entirely rational and optimized its KPI (retention) while completely ignoring the user’s explicit command (cancellation).

To catch a compliant lie, we cannot rely on syntax checks, nor should we rely on expensive LLM-as-a-judge loops. Instead, we implement an evidence governance layer requiring every proposed action to survive independent evidential checks before execution, using verification patterns tailored to different types of drift:

• Differential heuristics (fact validation): We bind the probabilistic LLM inference to legacy deterministic rules to catch objective fact violations. Suppose a furious customer demands cancellation, and the agent tries to save them by offering a 50% discount. The JSON is structurally correct, but existing, cheap SQL views hold the ground truth: customer_tier = BASIC, max_retention_discount = 15. If the LLM proposes 50%, the SQL query instantly detects the violation and the system halts.

# Semantic governance: catch fact drift at zero additional LLM cost
def verify_tier_limits(customer_id: str, policy_proposal: dict) -> None:
	# The syntax is valid, but the fact is violated.
	proposed_discount = float(policy_proposal["discount_pct"])
	max_allowed_discount = extract_max_discount_from_db(customer_id)

	if proposed_discount > max_allowed_discount:
		raise CompliantLieDetected(
			"Fact Violation: Proposed discount exceeds the customer's policy limit."
		)

• Evidence-based validation: But what if the agent proposes a 15% discount? The JSON is valid and facts are not violated. Here, semantic governance doesn’t attempt to prove the agent is “correct”; instead, it looks for evidence that the proposed action contradicts independently observable signals. If the customer explicitly wrote “cancel my subscription,” an independent classifier, which could be a legacy regex pattern, a fast traditional ML model, or a routing heuristic, may categorize the request as CANCEL_SUBSCRIPTION. This doesn’t establish ground truth, but it provides an evidential signal that can be compared against the proposed action. If the LLM proposes APPLY_DISCOUNT, the runtime detects an evidential conflict.

The same logic extends to identity-critical operations. A verification code sent to a newly supplied address confirms control of that address; it says nothing about ownership of the target account. An evidence governance layer would cross-reference any proposed credential-reset or email-association action against account records before granting execution authority. If the supplied address diverges from the address on file, the conflict is structurally identical to the cancellation case: a locally valid action contradicting independently observable state.

Notice what the runtime isn’t doing. It’s not trying to determine if retaining the customer is economically beneficial. It’s not running an expensive multi-agent debate to outreason the model. It simply asks: Does the proposed action contradict evidence that already exists outside the model?

# Semantic Governance: catch Evidential Conflict at near-zero cost
def validate_subscription_decision(customer_email: str, proposed_policy: dict) -> None:
	# intent_classifier can be a simple regex or a lightweight ML model
	cancellation_detected = intent_classifier(customer_email) == "CANCEL_SUBSCRIPTION"
	retention_action = proposed_policy["action"] == "APPLY_DISCOUNT"

	if cancellation_detected and retention_action:
		raise CompliantLieDetected(
			"Evidential Conflict: Decision contradicts independent classifier signals."
		)

• Bidirectional reconstruction (decision reversibility): Explicit evidence validation is perfect for clear-cut intents like “cancel.” But what if the request is ambiguous, multi-objective, or highly contextual? Suppose the customer writes: “I’m considering moving our entire team to another vendor. Support has been disappointing and pricing no longer makes sense.” There is no single INTENT_CANCEL trigger here. If the agent proposes {"action": "OFFER_ENTERPRISE_DISCOUNT", "discount_pct": 20}, we pass only the JSON output to a tiny, inexpensive Agent B.

Bidirectional reconstruction answers the question: Can the output truthfully explain itself?

If Agent B blindly evaluates the JSON and reconstructs “The customer is unhappy with pricing and is being offered a retention discount,” the runtime treats the reconstructed narrative as an additional evidential signal and escalates whenever the gap between the reconstructed intent and the original context becomes too uncertain to justify autonomous execution. The exact comparison mechanism is implementation-specific and may range from embedding similarity to domain-specific heuristics. Because the original email described a critical team exodus, the reconstructed narrative fails to explain the input. The system doesn’t claim to know the “truth”; it simply detects the loss of context, what we call compression drift, and halts due to the resulting uncertainty.

Admittedly, programmatically comparing textual intents introduces its own layer of fuzziness and risks falling back on another LLM-as-a-judge. Bidirectional reconstruction is therefore an engineering trade-off: In highly ambiguous workflows where strict SQL limits or simple ML classifiers can’t decisively apply, we accept a higher rate of false-positive escalations. This is intentional. A false-positive escalation has a bounded and predictable cost, while an unsupported autonomous action can create unbounded business consequences. We tune the system to assume that if the evidential link between the context and the JSON is even slightly blurry, it must escalate. To prevent the conformity traps discussed earlier, these agents are strictly air-gapped. Agent B operates purely as an isolated, one-way evidential classifier checking the work of Agent A. They can’t converse or negotiate a consensus.

Whether an organization uses differential heuristics, legacy ML intent classifiers, or bidirectional reconstruction, is ultimately an implementation choice. The core architectural principle remains unchanged: Execution authority is never granted because an agent appears convincing. It’s granted only when the proposed action is supported by evidence that exists independently of the agent’s own reasoning process.

The purpose of semantic governance isn’t to replace the agent with deterministic rules. If a deterministic rule could reliably make the decision, the agent shouldn’t be making it in the first place. Instead, the runtime reserves deterministic validation for the understood invariants of the business, leaving the agent responsible for reasoning under ambiguity. The role of evidence validation is not to replace reasoning, but to challenge it before authority is granted. Deterministic systems handle certainty; agents handle ambiguity. The architectural mistake is asking either of them to do both.

Temporal governance and agent drift

Catching single-transaction errors solves the immediate execution problem. But as deployments mature, organizations face the insidious “day three” problem: agent drift.

What happens when every individual decision is syntactically valid and semantically true, but the aggregate behavior of the agent begins to erode business margins over time? Imagine a retention agent that learns to successfully keep customers from churning by consistently offering the maximum allowed 15% discount. The agent is technically obeying all rules, but over a thousand interactions, it silently destroys the company’s profitability.

By leveraging decision telemetry, specifically attaching a unique Decision Flow ID (DFID) to every interaction, we transform opaque AI conversations into structured, relational database rows. Because every decision, context snapshot, and outcome is permanently linked by a DFID, we can run asynchronous, postexecution monitors over rolling windows of data.

A practical “day three” monitor in customer retention and autonomous billing can be as simple as SQL:

-- Trigger a circuit breaker if an agent keeps maxing discounts
SELECT agent_id
     , AVG(CAST(params->>'discount_pct' AS DECIMAL)) AS rolling_avg_discount
     , COUNT(dfid) AS total_decisions
  FROM execution_log
 WHERE executed_at >= CURRENT_TIMESTAMP - INTERVAL '7 days'
   AND status="SUCCESS"
 GROUP BY agent_id
HAVING AVG(CAST(params->>'discount_pct' AS DECIMAL)) > 14.5;
-- assuming a hard limit at 15.0

If an aggregate monitor detects that an agent’s average discount rate is creeping dangerously high, it trips a circuit breaker. The system immediately suspends the agent’s authority in the registry, cutting off its compute budget and execution rights until a human operator intervenes.

This is temporal governance. When you combine syntactic, semantic, and temporal defenses, the paradigm shifts entirely. You are no longer praying that the model is perfect. Its imperfections are structurally contained before they can become systemic losses.

Accuracy as a financial slider

Once a deterministic airlock enforces context, authority, evidence, and time, the risk of catastrophic failure drops drastically. You no longer need the underlying large language model to be perfect; you simply need to know how much its imperfection costs. At this point, model intelligence (intent) ceases to be a question of operational safety and becomes a pure economic variable.

Governance by exception

When a proposal fails the syntactic or semantic gates, we don’t blindly loop the model. Once deterministic gates exist, failed decisions no longer require blind retries. They become bounded exceptions.

Escalations aren’t a failure mode of the architecture; they’re a predictable cost component. By intentionally accepting false-positive escalations from the semantic airlock, we trade unbounded business risk for a bounded operational expense.

Different organizations may handle those exceptions differently. Some may escalate directly to human operators. Others may route failures through progressively more capable models before escalation. Research such as “FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance” demonstrates that model cascades can significantly reduce inference cost while maintaining quality, making them one possible implementation of this broader principle.

The architectural insight, however, is independent of any specific routing strategy. Deterministic governance transforms retries into explicit exceptions, allowing organizations to decide whether additional compute, additional context, or human intervention is the most economical next step. The system operates by governance by exception: Human operators and expensive premium models don’t review routine transactions. They only review the genuine anomalies where the baseline machine could not mathematically or semantically prove its own rationale.

Bounding the cost variance

With the execution infrastructure stabilized, the focus shifts to a critical operational challenge: cost variance.

In traditional software, execution costs are predictable. In probability-based systems, the exact same task might consume 500 tokens on Monday and 15,000 tokens on Tuesday if an agent enters a prolonged reasoning loop to resolve an edge case. For enterprise deployments, this unpredictable variance is often a more severe blocker than the base cost of inference.

By enforcing a strict computation budget per decision flow and utilizing the intent retry governor, the architecture places a hard ceiling on this variance. If an agent reaches its retry limit without producing a compliant policy, the runtime aborts the process and safely escalates it. While this doesn’t make AI operational costs perfectly static, it structurally bounds the financial exposure, ensuring that the compute cost of handling any single transaction never exceeds a defined limit.

The financial slider equation

With safety guaranteed by the runtime and cost variance capped by the infrastructure, the economics of agentic AI can be distilled into a single, formal equation:

Total Decision Cost = Compute Cost + (Escalation Rate × Human Cost)

This equation fundamentally changes the optimization problem. Traditional agent architectures treat model capability as a prerequisite for safety. Once governance is externalized, capability primarily influences escalation frequency. The question is no longer “Which model is intelligent enough to be safe?” but “Which combination of model cost and escalation rate minimizes total decision cost?”

In both scenarios, the system is deterministically compliant. The choice is purely unit economics.

While a smarter model may reduce escalations by making better use of available evidence, no model can eliminate escalations caused by genuine business ambiguity. A $100 billion reasoning model can’t invent context it doesn’t possess.

By decoupling safety from intelligence, you’re no longer hostage to the pursuit of perfect accuracy. Intelligence becomes a tunable economic variable, finally making agentic AI viable for the enterprise.

Engineering for imperfection

As we scale these systems from isolated pilots to enterprise-grade operations, a stark reality comes into focus: The greatest risk in agentic AI is no longer hallucination. It’s unlimited spending performed by a system that believes it’s still making progress.

We don’t need smarter, infinitely expanding models to safely deploy autonomous systems into high-stakes production environments. We need smarter systems that fundamentally assume the underlying model will eventually fail, drift, or lie.

Consider how civil engineers build a suspension bridge. They don’t spend decades searching for “perfect steel” that will never bend, rust, or fatigue. They accept that the material is inherently flawed and subject to the laws of entropy. To compensate, they build redundancies. They calculate margins of error. They construct hard, load-bearing physical frameworks that dictate exactly how much stress the material is allowed to absorb before the structure safely redistributes the weight.

The software industry has spent the last three years searching for perfect steel. We’ve poured billions of dollars into massive evaluation suites, prompt engineering alchemy, and ever-expanding context windows, hoping to forge a probabilistic model that never hallucinates. It’s a mirage.

Engineering maturity in the AI era doesn’t mean removing all imperfection from machine reasoning. It means designing an architecture so rigid, deterministic, and resilient that the model’s imperfections cease to be an operational liability.

The future of agentic AI is unlikely to be won by the organization with the smartest model. It will be won by the organization that most effectively separates intelligence from authority. Once reasoning and execution are decoupled, intelligence becomes a tunable economic parameter. Safety becomes infrastructure. And the endless pursuit of perfect model accuracy finally stops being a business requirement.

The end of that pursuit isn’t the end of AI. It’s the moment AI finally becomes engineering.

Note: The runtime described here is a reference architecture, not a specific implementation technology. The same principles can be realized through workflow engines, policy platforms, orchestration frameworks, or custom infrastructure. A sample implementation of these concepts is available in the GitHub repository.