Agent Correctness in Production: Moving Beyond Text Hallucination

I have a job interview question that tells me everything I need to know about a team’s understanding of agent reliability.

Tell me how you monitor your agents in production.

The typical answer sounds reasonable at first. They describe an evaluation suite that runs before deployment. They measure token-level accuracy against reference outputs. They check that tool calls match the expected schemas. They say their guardrails catch hallucinations before they reach the user.

Then I ask a follow-up: when was the last time an agent in production made a structurally valid decision that was nevertheless wrong for the business?

Silence.

That silence is where the real problem lives. Not in the text you can see but in the intent you cannot.

The Evaluation Trap

There is a deep disconnect between how agents are evaluated and how agents behave in production.

The evaluation process runs agents through a controlled dataset. The dataset contains carefully constructed test cases. The evaluation framework checks whether the agent produces the expected output. The evaluation metrics measure accuracy, precision, recall. An agent that scores 94 percent on the eval dataset is considered production-ready.

The production environment is not the eval dataset.

In production, the agent encounters inputs that are messier, more ambiguous, and more adversarial than anything in the test set. Users ask questions that overlap with the test data but are not identical. Systems that the agent integrates with have changed. Network conditions introduce timing variations that affect token generation order. The eval metrics never measured these things because they are not test cases.

But they are production problems. And they create failures that look correct from the outside while producing wrong outcomes on the inside.

This is not an argument against evaluation. It is an argument that evaluation alone is insufficient. You need a different layer of protection specifically for production.

Structural vs Semantic vs Intent Errors

Let me classify the failure modes that actually matter in production.

Structural errors are the ones eval frameworks catch. The agent produces a malformed tool call. The JSON does not parse. A required field is missing. These are easy to detect because the tool itself rejects the input. The validation layer catches them before execution. The question is whether catching structural errors is enough.

It is not.

Semantic errors happen when the agent calls the right tool with the right parameters but applies it in the wrong context. The agent looks up a customer record by ID and finds the correct record. It then generates a response to that customer. But the customer ID came from a request that was intended for a different customer. The tool call was structurally and semantically correct. The outcome was applied to a wrong entity. The eval dataset probably never tested this edge case.

Intent errors are the most dangerous. The agent calls the right tool with the right parameters in the right context. Everything checks out structurally and semantically. But the agent’s underlying reasoning was wrong. It decided to call that tool because it misunderstood the user’s true intent. A user asked the agent to cancel their subscription. The agent correctly located the subscription, correctly formatted a cancellation request, correctly sent it to the API endpoint. But the user was trying to modify the plan, not cancel it. The agent interpreted the word cancel in the simplest way available and never escalated the ambiguity. The eval dataset likely included cancellation patterns that the agent handled perfectly. It did not include the pattern where a user uses a word with multiple meanings.

The hierarchy is important. Structural errors are caught by schema validation. Semantic errors require runtime context verification. Intent errors require a fundamentally different approach: monitoring the agent’s reasoning, not just its output.

The Monitoring Gap

Most organizations that deploy agents have monitoring. They track latency, error rates, token counts, and API costs. They have dashboards. They set alerts. When the error rate spikes, someone investigates. When the cost exceeds a threshold, someone adjusts the budget.

But monitoring those metrics tells you nothing about agent correctness.

A team might have 0.01 percent error rate and still be making hundreds of semantic errors per day. The errors are not failures that trigger alerts. They are successful executions of wrong intentions. They register in your dashboards as happy paths.

The monitoring gap is not a technical limitation. It is a design decision. Organizations optimized their monitoring for infrastructure health because that is what observability tools were built for. They were not built to monitor agent reasoning. And they could not have been because the standard tooling assumes that a successful HTTP response means a successful operation.

Bridging this gap requires monitoring the agent’s decision points, not just its outputs. Every time an agent selects a tool, it makes a decision based on its interpretation of the input and its current state. Tracking those decisions, storing the reasoning chain, and periodically evaluating whether the reasoning was sound is what production monitoring should look like.

This is not a small addition to existing observability infrastructure. It is a separate monitoring layer that captures structured representations of the agent’s thoughts, not just its actions.

Production Guardrails That Actually Work

Schema validation at the parser level catches structural errors. It is necessary but insufficient.

Runtime context verification catches semantic errors. When the agent selects an entity to operate on, you verify that the entity is the correct one for the user’s stated intent. This requires a verification step that exists between the agent’s reasoning and the tool execution. The verification does not need to be perfect. It needs to catch the cases where the entity selection deviates from the user’s intent.

Intent monitoring catches intent errors. This is the hardest layer to build because it requires the system to understand what the user actually wants, not just what they asked for. The most pragmatic approach is to create a simple confidence score on each agent decision. When the confidence drops below a threshold, the decision is escalated to a human or routed through a more expensive frontier model that has better interpretive capability.

These three layers together form a production-grade guardrail system. Schema validation is cheap and near-instant. Runtime context verification is slightly more expensive because it requires additional computation but remains fast enough for most workflows. Intent monitoring is the most expensive layer because it involves evaluating the agent’s reasoning chain and may trigger escalation.

The cost structure matters. You want the cheap layer to catch as many errors as possible so that the expensive layers only trigger when needed. This is similar to how a well-designed API uses rate limiting and input validation to protect expensive downstream services. The intent monitoring layer is the downstream service. It should only be called when the cheaper layers cannot resolve the ambiguity.

Eval Benchmarks vs Production Reality

The disconnect between eval benchmarks and production behavior is not a problem that can be solved by better testing. It is structural.

Eval benchmarks test a closed set of known failure cases. Production presents an open set of unknown failure cases. You can prepare for known failures with targeted test cases. You cannot prepare for open-set failures without a fundamentally different approach.

The approach is gradual deployment with continuous monitoring. An agent that moves to production in stages — first for a small percentage of users, first for low-risk operations, first for operations that have built-in rollback — generates production data that no eval dataset can replicate. That production data is used to continuously update the eval dataset, creating an expanding safety net rather than a static one.

This process is not elegant. It is not the kind of approach you describe in a conference talk or a board presentation. It is the kind of operational discipline that distinguishes organizations that ship reliable agents in production from organizations that ship agents that work in eval but fail in the real world.

The teams I have seen succeed at long-term agent reliability share a common trait. They have accepted that eval benchmarks are a necessary but insufficient milestone. They measure them because they have to. They are not fooled by the numbers. They know that the real measurement of correctness happens over weeks and months of production operation, not during a two-week eval sprint before deployment.

The Architecture of Trust

Building agents that are trustworthy in production is not a feature. It is an architectural constraint that shapes every layer of the system from the ground up.

The agent architecture itself needs to be designed for transparency and verifiability. Every decision the agent makes should be recorded in a structured log that can be reviewed, audited, and evaluated. The reasoning chain should be preserved in a format that allows both automated analysis and human review. The tools used by the agent should expose their preconditions and postconditions so that the system can verify that expected state transitions occurred.

The deployment architecture should support gradual rollout, feature flags, and instant rollback. A production issue should be isolated and reversible within minutes, not hours or days.

The monitoring architecture should track not just infrastructure metrics but agent decision quality over time. A decline in decision quality across a subset of operations is an early warning that something is drifting. The system should detect that drift before it becomes a production incident.

This is not easy work. It requires more discipline than most teams bring to production deployments. It requires a commitment to incremental improvement rather than a one-and-done launch-and-forget approach. But the cost of being wrong in production is exponentially higher than the cost of doing it carefully in the first place.

When an agent calls the wrong API endpoint once, you get a support ticket. When it calls the wrong endpoint to the wrong entity with the wrong parameters on repeat, you get a regulatory investigation. The gap between one error and systemic failure is measured in seconds of monitoring coverage.