Tool Error detects errors or failures during the execution of Tools.
This metric is particularly valuable for monitoring agentic AI systems where the model uses various tools to complete tasks. Tool execution failures can lead to incomplete or incorrect responses, affecting the overall user experience. Here’s a scale that shows the relationship between Tool Error detection and the potential impact on your AI system:
01
High Error Rate
Many tools failed during execution, causing incomplete or incorrect responses.
Low Error Rate
All tools executed correctly without errors.

​
Calculation method

Tool Error detection is computed through a multi-step process:
1

Model Request

Additional evaluation requests are sent to an LLM evaluator (e.g., OpenAI’s GPT4o-mini) to analyze tool execution outcomes.
2

Prompt Engineering

A carefully engineered chain-of-thought prompt guides the model to evaluate whether each tool executed successfully without errors.
3

Log Analysis

The system performs a detailed analysis of execution logs and outputs from each tool call to identify potential issues.
4

Error Detection

The evaluation process identifies specific errors, exceptions, and unexpected behaviors that occurred during tool execution.
5

Impact Assessment

A detailed explanation is generated describing the detected errors and their potential impact on the system’s functionality.
We also surface a generated explanation that helps understand the nature of the error and its potential causes.
This metric is computed by prompting an LLM, which requires additional LLM calls to compute, potentially impacting usage and billing.

​
Understanding tool error

​
Common Types of Tool Errors

Tool Error detection identifies various failure modes:
API Failures: External services or APIs that tools depend on may be unavailable or return errors.
Parameter Errors: Tools may receive invalid parameters that cause execution failures.
Timeout Issues: Tools may take too long to execute and exceed allocated time limits.
Permission Errors: Tools may lack necessary permissions to access required resources.

​
Optimizing your AI system

​
Addressing Tool Errors

When your system experiences tool execution errors, consider these improvements:
Implement robust error handling: Ensure tools can gracefully handle exceptions and provide meaningful error messages.
Add parameter validation: Validate input parameters before tool execution to prevent runtime errors.
Monitor external dependencies: Set up monitoring for external services that your tools depend on.
Implement fallback mechanisms: Design tools with fallback options when primary execution paths fail.

​
Best practices

Comprehensive Logging

Implement detailed logging for all tool executions to facilitate debugging and error analysis.

Graceful Degradation

Design tools to provide partial results or alternative responses when they encounter errors.

Error Categorization

Categorize different types of errors to identify patterns and prioritize fixes based on frequency and impact.

User-Friendly Error Messages

Translate technical errors into user-friendly messages that help users understand what went wrong.
This metric helps you detect whether your tools executed correctly. It’s most useful in Agentic Workflows where many Tools get called. It helps you detect and understand patterns in your Tool failures, allowing you to improve reliability over time.