This edition continues last week's discussion of agent authority, moving from what an agent is allowed to do to whether it completes the intended business outcome.
In this edition
- Why passing evaluations does not guarantee production reliability.
- Six capabilities in the Agent Reliability Loop.
- Five metrics that connect agent execution to business value.
Imagine an AI agent built to automate supplier onboarding.
During testing, everything looks promising.
The agent extracts supplier information, validates documents, checks internal records, and prepares an approval request.
The demonstration is successful.
The evaluation dashboard shows encouraging results.
Leadership approves the production rollout.
Three weeks later, operations reports a different picture.
Some suppliers are being processed twice. Others remain stuck in approval queues. The agent occasionally retrieves outdated information. A downstream API times out, but the workflow still reports completion.
Employees are spending time correcting transactions that were supposed to be automated.
The AI model is functioning.
The application is running.
The dashboards are green.
But the business process is failing.
This is an illustrative scenario, but the underlying problem is increasingly relevant as enterprises move AI agents from experimentation into operational workflows.
It exposes a distinction that deserves more attention:
An agent that produces a correct response is not necessarily an agent that completes a business process correctly.
And that distinction may determine which enterprise AI investments deliver measurable value.
The problem isn't always the model
When an AI agent underperforms, the first instinct is often to improve the model.
Try a more capable model.
Refine the prompt.
Add another agent.
Increase the context window.
Introduce additional reasoning steps.
Sometimes these changes help.
But many production failures originate outside the model.
The enterprise application may return incomplete data.
A document may have changed since it was indexed.
An API may fail midway through execution.
A workflow may contain exceptions that were never included in testing.
A retry may unintentionally create duplicate transactions.
Or the agent may produce a convincing final response without verifying that the requested business action actually succeeded.
These are not simply model-quality problems.
They are end-to-end workflow reliability problems.
The distinction matters because improving model accuracy alone cannot guarantee that the surrounding business process becomes more reliable.
What production evidence tells us
A study presented at ICML 2026, Characterizing Agents in Production, examined 20 production-agent case studies and surveyed 306 practitioners across 26 domains.
The researchers identified reliability—consistent correct behavior over time—as the leading development challenge.
The study also found that 68% of agents executed no more than ten steps before human intervention, while 74% relied primarily on human evaluation.
This suggests that successful production deployments frequently depend on bounded workflows, human judgment, and systems-level engineering rather than unrestricted autonomy.
Another revealing signal comes from LangChain's March 2026 analysis of agent engineering practices.
According to its survey:
- 89% of teams had implemented agent observability.
- 52% performed offline evaluations.
- 37% performed online evaluations.
The gap is important.
Organizations are increasingly able to observe agent execution.
But observing an execution is not the same as systematically measuring whether it achieved the intended outcome.
In an article updated October 13, Kore.ai introduced Autoloop, a production agent-optimization capability emphasizing evaluation, root-cause diagnosis, targeted improvements, and regression validation.
This is a vendor description of its own product, rather than independent evidence of effectiveness. The significance here is the operating model it describes.
It is the direction of the market.
Enterprise AI is moving from building agents toward continuously operating and improving them.
And that requires a different engineering discipline.
The hidden gap: Task Completion vs. Business Completion
Consider a customer-service agent asked to update a delivery address.
The agent understands the request.
It identifies the correct customer.
It calls the order-management API.
It responds:
Your delivery address has been updated.
From the conversational perspective, the interaction appears successful.
But what if the API returned an error?
What if the update applied to an old order?
What if the customer had two active orders?
What if the update succeeded but was subsequently overwritten?
The response may be accurate in tone and format while the business outcome remains incorrect.
For production systems, success should be measured across the complete execution chain:
User Intent → Agent Decision → Tool Execution → System State → Verified Business Outcome
The last two stages are particularly important.
A successful tool call does not always establish that the intended business state was achieved.
And a successful business-state change does not necessarily establish that the action was appropriate.
For consequential workflows, both correctness and authorization must be verified.
This leads to a more useful definition:
Agent reliability is the ability to achieve the intended business outcome consistently, within defined operational, cost, and risk boundaries.
Five mistakes enterprises make when scaling AI agents
1. Evaluating responses instead of outcomes
Many evaluations focus on whether an answer is relevant, coherent, grounded, or factually correct.
These are important measures.
But an invoice-processing agent should also be evaluated on whether it selected the correct invoice, avoided duplicate processing, applied the correct business rules, and completed the intended transaction.
A high-quality response does not guarantee a high-quality workflow.
Organizations need evaluations that cover the final business state, not just the model's output.
2. Testing only the happy path
A demonstration typically uses clean data, available APIs, and predictable scenarios.
Production environments behave differently.
Documents are incomplete.
Customer records conflict.
External systems become unavailable.
Business rules change.
Users provide ambiguous instructions.
And workflows encounter exceptions.
An agent that succeeds on 100 carefully selected test cases may still perform poorly against thousands of real-world variations.
The solution is not simply to increase the number of tests.
It is to test the failure conditions that matter.
3. Treating observability as evaluation
Tracing tells us what happened during an execution.
Evaluation determines whether that behavior met an expected standard.
These are complementary capabilities.
A trace may show that an agent called three tools and completed in six seconds.
But that alone does not tell us whether the customer received the correct refund or whether the ERP transaction was valid.
Observability explains execution. Evaluation judges its correctness.
Production teams need both.
4. Improving prompts without regression protection
Suppose an engineering team modifies a prompt to resolve an invoice-classification problem.
The change improves classification accuracy.
But it also increases tool calls, causes more timeouts, or changes how exceptions are handled.
One metric improves while another deteriorates.
Without regression testing, the team may deploy a change that makes the overall business workflow worse.
Agent updates should be treated as production software changes, with measurable release criteria and rollback options.
5. Measuring AI usage instead of business value
An organization may report:
10,000 agent interactions.
A 95% task-completion rate.
A 30% reduction in average response time.
These metrics are useful, but incomplete.
What percentage of transactions required human correction?
How many were completed correctly without rework?
How much did each verified completion cost?
Did the automation reduce operational workload?
Did customer outcomes improve?
These questions connect AI engineering to business performance.
A practical framework: The Agent Reliability Loop
I recommend approaching production-agent reliability through six connected capabilities.
1. DEFINE — Establish the business outcome
Before selecting evaluation tools, define what success means.
For a supplier-onboarding agent, success might require:
- Correct supplier identification.
- Required document validation.
- Appropriate approval routing.
- No duplicate supplier creation.
- Correct final state in the ERP system.
The evaluation contract should also specify unacceptable outcomes, escalation conditions, and when human approval is required.
This creates an explicit boundary between a completed AI interaction and a completed business transaction.
2. INSTRUMENT — Trace the complete workflow
Capture meaningful execution events across the agent and its integrations.
A useful trace connects:
Request → Agent → Tool → API → Business System → Result
Include correlation identifiers, tool outcomes, relevant versions, execution times, and error conditions.
Where permitted, record the evidence required to verify the resulting business state.
Avoid logging sensitive information unnecessarily.
For an Azure-based implementation, Application Insights, Azure Monitor, and distributed tracing can provide much of the operational foundation.
Microsoft Foundry also supports agent monitoring and evaluation capabilities, although preview features should be assessed carefully before depending on them for critical production controls.
The objective is straightforward:
When a workflow fails, engineering should be able to identify where and why.
3. EVALUATE — Test the complete execution path
Build evaluations at multiple levels.
Model evaluation: Did the agent understand the request and produce an appropriate response?
Tool evaluation: Did it select the correct tool and provide valid parameters?
Workflow evaluation: Did the required sequence execute correctly, including exceptions and retries?
Business evaluation: Did the final system state match the intended outcome?
Safety evaluation: Did the workflow remain within its permitted operational boundaries?
For workflows with real-world side effects, use isolated test environments, mocks, or controlled replay where appropriate.
The key principle is that evaluation should reflect the actual work the agent is expected to perform.
4. LEARN — Convert production failures into regression tests
This is where many implementations stop too early.
A production failure is investigated.
The prompt is adjusted.
The issue is closed.
But the failure never becomes part of the permanent evaluation suite.
A better process is:
Production Failure → Root-Cause Analysis → Reproducible Test → Targeted Fix → Regression Validation
Over time, this creates a growing library of real enterprise scenarios.
Examples include ambiguous supplier identities, conflicting document versions, expired credentials, API timeouts, and duplicate transaction attempts.
Each resolved failure should make the next release more reliable.
The evaluation dataset becomes an operational asset.
5. CONTROL — Introduce risk-based release gates
Not every agent requires the same level of control.
A read-only research assistant and an agent capable of initiating payments have very different risk profiles.
For low-risk workflows, automated evaluation and monitoring may be sufficient.
For workflows that modify business records, additional approval and rollback controls may be appropriate.
For financial transactions, production infrastructure changes, or regulated operations, stricter release criteria and human oversight may be necessary.
Release decisions should consider outcome correctness, exception rates, safety, latency, cost, and regression results.
The greater the consequence of failure, the stronger the required evidence before granting autonomy.
6. OPTIMIZE — Improve reliability and unit economics together
A more capable model is not automatically the better production choice.
An agent may achieve higher accuracy but require significantly more tokens, tool calls, or processing time.
Another configuration may be cheaper but generate more human rework.
The correct comparison is not simply cost per model request.
It is:
Total Operating Cost ÷ Verified Successful Business Outcomes
Total operating cost should include inference, infrastructure, integrations, monitoring, evaluation, and human exception handling.
For production AI, the cheapest execution is not necessarily the cheapest completed transaction.
The business case: Why small failure rates matter
Consider an illustrative enterprise workflow processing 10,000 cases each month.
Assume 4% of cases require human correction.
That represents 400 cases.
If each correction costs an average of $35 in staff time and operational overhead, the monthly rework cost is:
400 × $35 = $14,000
Now suppose better evaluation, workflow design, and exception handling reduce the rework rate from 4% to 1%.
The remaining correction cost becomes:
100 × $35 = $3,500
That represents a potential reduction of:
$10,500 per month, or $126,000 annually.
These figures are illustrative, not reported customer results. They exclude the cost of implementing and operating the improvement system.
But they demonstrate why seemingly small failure rates deserve attention.
In high-volume workflows, reliability is an economic variable.
For a CTO, it affects operational stability and engineering workload.
For a CIO, it affects the return on technology investment.
For a Head of AI, it determines whether successful pilots can be scaled confidently.
For business leaders, it affects service quality, productivity, and customer trust.
The commercial value of improving an agent depends on the frequency, cost, and consequences of its failures—not simply the sophistication of the underlying AI model.
What this means across industries
Manufacturing
Agents supporting procurement, supplier management, quality documentation, and production planning must operate against complex business rules and changing operational data.
A seemingly minor processing error can create downstream delays, duplicate orders, or incorrect records.
Outcome verification and exception handling are essential.
Financial services
Agents supporting reconciliation, document processing, customer servicing, and transaction workflows require accurate state changes, traceability, and controlled execution.
Reliability must be measured alongside authorization, auditability, and regulatory obligations.
SaaS companies
AI agents are increasingly embedded in customer-facing products and internal operations.
Failures can affect customer experience, support costs, service commitments, and retention.
Production evaluation should become part of the product-release lifecycle.
Consulting organizations
Reusable agent frameworks can accelerate client delivery.
But client-specific workflows, integrations, policies, and data create different failure conditions.
A repeatable evaluation framework allows consulting teams to standardize quality assurance while adapting business tests to each engagement.
Mid-market organizations
Smaller technology teams may not have dedicated AI reliability engineers.
They need a practical operating model built on existing CI/CD, monitoring, testing, and incident-management capabilities.
The priority should be a small number of high-value workflow metrics rather than an unnecessarily complex AI operations platform.
The five metrics I would put on an enterprise AI dashboard
For every consequential production agent, I would start with these measurements.
| Metric | What it reveals |
|---|---|
| Verified outcome success rate | Percentage of requests that achieve the intended business result |
| Human rework rate | Percentage requiring correction after agent execution |
| Exception recovery rate | Percentage of recoverable failures resolved safely |
| Cost per verified outcome | Actual economics of successfully completed work |
| Time to detect and resolve failures | Operational responsiveness and reliability |
These should be segmented by workflow type, business impact, and risk.
A single aggregate success rate can conceal serious problems.
For example, a 98% overall success rate may appear acceptable until the remaining 2% includes high-value financial transactions or production changes.
The severity of failure matters as much as its frequency.
What technology leaders should do in the next 30 days
Start with one production agent that already performs meaningful business work.
Do not begin by changing its model.
First, establish whether you can measure the intended outcome.
Review a representative sample of successful and unsuccessful executions.
Identify where the final business state differs from the agent's reported completion.
Classify the most common failures.
Turn those failures into repeatable tests.
Add a small set of operational and business metrics.
Then establish release criteria for the next change.
This exercise often reveals whether the real bottleneck is model behavior, data quality, integration reliability, workflow design, or exception handling.
Only then should the team decide which component needs improvement.
The larger shift: From Agent Development to Agent Operations
Building an agent is becoming easier.
Operating one reliably inside an enterprise remains difficult.
The next phase of enterprise AI maturity will require organizations to combine traditional software-engineering disciplines with AI-specific evaluation.
That means treating agents as production systems with owners, operational objectives, regression tests, monitoring, incident response, and measurable business outcomes.
The operating model should evolve from:
Build → Test → Deploy
toward:
Define Outcomes → Evaluate → Deploy → Observe → Diagnose → Improve → Revalidate
This does not require a large new technology platform from day one.
It requires a disciplined feedback loop.
Because the question that ultimately matters is not:
Did the AI agent produce a good answer?
It is:
Did the AI agent complete the business process correctly, consistently, and economically?
And can we demonstrate that it continues to do so as the enterprise environment changes?
A question for technology leaders
When your production AI agent reports a task as completed, how do you independently verify that the intended business outcome actually happened?
I'd be interested in hearing how organizations are approaching outcome-based evaluation, production monitoring, and continuous improvement.
— Atul Yadav
Technology • AI • Strategy • Impact
Continue reading
Agent authority and access reviews · Cost per business outcome
Keep a clearer view of what’s next
Subscribe to XTechStacks Weekly for practical perspectives on enterprise AI, architecture and engineering. If this edition helped you frame a decision, share its published link with a colleague.
Working through agent reliability in your organization? Talk to XTechStacks about your architecture questions.
About the author — Atul Yadav
Atul Yadav writes XTechStacks Weekly, exploring enterprise AI, technology architecture and practical engineering decisions. This newsletter connects emerging technology with the controls and tradeoffs involved in real systems.
Follow XTechStacks on LinkedIn · Explore newsletter editions