Agent Correctness: Evaluating Tool Use Errors and Hallucinations

Most enterprises evaluating agent systems measure the wrong thing.

They check whether the agent produced the right final answer. The agent retrieved data from the database. The agent generated a correct email. The agent deployed the code to staging. Final answer: correct. Pass the test.

But the agent called the wrong database field. It included sensitive information in the email that should have been redacted. It deployed to the staging environment but with a configuration that only works in development. The final output looks right. The intermediate steps were fundamentally wrong. And the damage is already done.

This is the central problem of agent evaluation in production. Text output correctness is easy to measure. Tool use correctness is hard. And tool use is where agents create real value and real risk - a problem made more tractable by protocol-level standardization, which reduces the surface area of tool hallucinations by making tool schemas explicit and machine-readable.

Why Tool Use Correction is Different

A text generation model has one job: produce the next plausible token given the context. If the output reads correctly and is semantically aligned with the prompt, the model succeeded. Evaluation is straightforward. Compare the generated text against a reference. Compute an accuracy score. Done.

An agent has a different job. It perceives the environment through available tools. It reasons about what state the environment is in. It selects the appropriate tools sequentially. It interprets tool results. It adapts its strategy based on feedback. It produces actions that change the environment.

Each step in this sequence is a potential point of failure. The agent picks the wrong tool. It provides the wrong parameters. It misinterprets a tool’s output. It makes an invalid state transition. Each of these failures may or may not affect the final answer. A correctly formatted error email and a correctly formatted email sent to the wrong recipient are functionally different outcomes, even though both pass a naive text comparison.

The failure modes fall into three categories.

Tool hallucination is the most common. The agent calls an action that does not exist. It invents parameters that the schema does not define. It assumes capabilities that the tool does not provide. An agent that calls a delete_all_users() function because it inferred the intent from the prompt is hallucinating a tool. An agent that calls a valid update_user_preferences() function but with a parameter value that matches no existing user is using a tool correctly by structure but incorrectly by semantics.

Structural errors are less common but more visible. The agent produces malformed tool calls that fail to parse. JSON syntax errors. Missing required fields. Type mismatches. These errors are easy to detect because the tool itself rejects the call. They are also relatively easy to prevent with schema validation at the parser level.

Semantic errors are the hardest to detect and the most dangerous. The agent calls the right tool with the right parameters. The tool executes successfully. The action is valid. The outcome is correct. But the action was applied in the wrong context. The agent updated user preferences for the wrong user. It deployed the pipeline to the wrong cluster. It sent the notification with correct content to the wrong recipient list. Every structural check passed. The agent failed. The question of whether to replace a single monolithic agent with an agentic SDLC framework where multiple specialized agents handle verification independently becomes critical at this point.

Measuring Agent Correctness

Standard benchmarks failed for agent evaluation because they were designed for language models, not autonomous systems. MMLU tests knowledge across fifty-seven subjects. TruthfulQA tests factual accuracy. HELM tests broad model capabilities. None of these measure whether an agent can use tools correctly.

The shift to tool-correctness metrics started with two observations. First, tool calling is a structured output problem that can be evaluated independently from text generation. Second, tool success rate is a better predictor of agent reliability than text output accuracy.

The evaluation framework needs to measure at least three dimensions.

Structural correctness evaluates whether tool calls parse correctly against the available tool schema. This is a binary measurement: the tool call matches the schema or it does not. Schema validation catches this. You can measure structural correctness at the token level for each tool call the agent produces.

Semantic correctness evaluates whether the tool call achieved the intended outcome. This requires ground truth data. You need a label for what the correct tool call sequence is for each input. Then you compare the agent’s actual sequence against the correct sequence. The comparison can be step-by-step, checking each tool call in order, or outcome-based, checking whether the final environment state matches the expected state.

Efficiency correctness evaluates whether the agent took the optimal number of steps to reach the goal. An agent that calls fifty tools to accomplish a task that requires five is technically correct but economically wasteful. Efficiency correctness captures the trade-off between thoroughness and cost.

Evaluating Hallucination in Tool Use

Agent hallucination in tool use follows patterns. Understanding these patterns helps you design evaluation datasets that target them specifically.

The most common hallucination pattern is parameter fabrication. The agent correctly identifies which tool to call but invents parameter values that do not exist. An agent needs to set a user’s role to admin. It calls set_user_role() with the correct tool name, the correct user ID field, and a role value of superadmin that does not exist in the valid role enumeration. The tool call parses correctly. The tool executes. The result is a silent failure that goes undetected by structural validation.

The second pattern is tool conflation. The agent uses the wrong tool because two tools have similar names or similar purposes. An agent needs to retrieve an existing report. It calls the report generation tool instead of the report retrieval tool. Both tools return data. Both produce output that looks correct. The user receives fresh data instead of the cached report they expected. The action is valid. The intent is wrong.

The third pattern is state misattribution. The agent assumes the environment is in a state different from its actual state. This is the semantic error category. The agent sends a deployment notification assuming the deployment completed successfully. The deployment actually failed. The notification is sent to the wrong state. The downstream consequences are real.

Designing evaluation datasets to catch these patterns requires adversarial thinking. You need inputs where the correct action is obvious to a human but non-obvious to an agent operating through tool schemas alone. You need cases where the tool names are similar but the functions differ. You need edge cases that push the boundaries of the tool definitions.

The Trajectory Evaluation Problem

The central insight of trajectory-level evaluation is this: the path an agent takes to a correct answer matters as much as the answer itself.

An agent that reaches the correct final outcome after exploring thirty dead-end tool calls is consuming thirty times more compute than an agent that reaches the same outcome in two calls. Both produce the correct result. One is efficient. The other is wasteful. In production, the wasteful agent becomes expensive at scale.

An agent that reaches an incorrect outcome through the correct tool call sequence is fundamentally different from an agent that reaches an incorrect outcome through a fundamentally wrong sequence. The first agent has a reasoning problem. The second agent does not understand the tools.

Measuring trajectories requires recording the complete sequence of tool calls, the inputs to each tool, the outputs from each tool, and the environment state after each action. This trace allows you to reconstruct the agent’s decision-making process and compare it against a reference trajectory.

The comparison can check for equivalence at different levels of granularity. Tool-level equivalence checks whether the agent used the same tool at each step. Sequence equivalence checks whether the ordering of tools matches. State equivalence checks whether the environment state after each step matches the expected trajectory.

A trajectory that passes all three tests is functionally equivalent to the reference, even if the token-level explanations differ. A trajectory that passes tool-level equivalence but fails state equivalence indicates that the agent called the same tools in the same order but in a different context. A trajectory that fails tool-level equivalence has a fundamentally different approach.

Guardrails and Prevention

Measurement is only half the equation. You need automated safeguards that prevent agent errors in production.

Schema validation is the first line of defense. All tool calls must be validated against their schema before execution. This catches structural errors: missing required fields, type mismatches, invalid parameter values. Modern LLM inference frameworks support grammar-constrained generation that forces tool call output to adhere to a schema. This eliminates parsing errors before they occur.

Permission boundaries are the second line. Each tool call is checked against the agent’s permission set before execution. An agent that retrieves public data can be granted read access to a public API. The same agent deploying to production needs different permissions. The permission check is a simple look-up. The tool name is checked against the agent’s permission list. The parameter values are checked against allowed ranges.

Anomaly detection is the third line. The agent’s tool call pattern is monitored in real time. Unusual actions trigger alerts. Unusual sequences trigger pre-execution review. A sudden spike in the agent’s tool call rate or a new tool type that the agent has not used before are both signals that warrant monitoring attention. The same principles apply whether you are running a single agent or a full agentic SDLC — every autonomous actor needs observation.

A Practical Evaluation Framework

Build your evaluation framework starting with a small set of high-value user stories for each agent. Each story maps a user intent to a tool use task. A user wants to reset their password. A user wants to generate a sales report for Q4. A user wants to deploy a new version of the application.

For each story, define the expected tool call sequence. Define what parameters each tool call needs. Define the expected environment state before and after. This sequence is your reference trajectory.

Run your agent against each story. Record the complete trajectory. Compare against the reference on three dimensions: tool correctness, sequence correctness, and state correctness.

Repeat with adversarial variations of each story. Introduce ambiguous user inputs. Include misleading context. Provide incomplete information. Check whether the agent’s trajectory changes in predictable ways or collapses into error.

Track four metrics weekly: structural tool call error rate. Semantic tool call error rate. Average trajectory length relative to reference. Rate of novel, unclassified tool call patterns. These four metrics give you a complete view of agent correctness over time. If error rates climb, investigate which tools are failing and why. If trajectory length grows, the agent may be exploring inefficient paths that need to be constrained. If novel patterns appear, evaluate whether they represent improvements or new failure modes.

The Cost of Getting This Wrong

Text hallucinations cost credibility. Users notice when an agent produces incorrect text. They question its reliability. They lose trust in the system. But they can usually correct the text manually.

Tool use errors cost systems. An agent that calls the wrong production database with the wrong parameters does not just produce bad output. It changes the state of your infrastructure. It corrupts data. It deploys the wrong configuration. It sends the wrong notification and damages customer relationships.

Text errors are user-facing. Tool errors are system-facing. Both need evaluation. But the consequences of tool errors are orders of magnitude more expensive.

Invest in trajectory-level evaluation. Build adversarial test suites. Implement schema validation and permission boundaries. Monitor tool call patterns continuously. The agents that pass these tests reliably are the ones that survive in production. The ones that do not will fail in ways that are expensive and sometimes irreversible.

FAQ

What is the difference between structural and semantic tool use errors?

Structural errors occur when the tool call does not match the schema: missing fields, wrong types, invalid syntax. Semantic errors occur when the tool call is structurally valid but targets the wrong resource, uses incorrect values, or is applied in the wrong context.

Why do standard benchmarks like MMLU fail for agent evaluation?

Standard benchmarks measure text output quality, not tool execution correctness. Agents are evaluated by what they do, not what they say. MMLU cannot measure whether an agent called the correct database endpoint with the correct parameters.

How many test cases do I need for meaningful agent evaluation?

Start with twenty to thirty high-value user stories. Each story should have at least three adversarial variations. The total should be at least one hundred test cases. Expand based on which tools and scenarios generate the most errors.

What is trajectory length and why does it matter?

Trajectory length is the number of tool calls an agent makes to complete a task. It matters for efficiency. A trajectory that takes fifty calls to solve a problem solvable in five is consuming fifty times more compute and latency for the same result.

How often should I run agent evaluations in production?

Run the full evaluation suite weekly. Run targeted evaluations on any tool or agent that shows degradation in weekly metrics. Run continuous monitoring on tool call patterns to catch novel failures in real time.