Overview
Action Advancement measures whether an assistant successfully accomplishes or makes progress toward at least one user goal in a conversation.
- Provides a complete or partial answer to the user’s question
- Requests clarification or additional information to better understand the user’s needs
- Confirms that a requested action has been successfully completed
- Factually accurate
- Directly addressing the user’s request
- Consistent with any tool outputs used
Action Advancement at a glance
| Property | Description |
|---|---|
| Name of Metric | Action Advancement |
| Metric Category | Agentic Metrics |
| Use this metric for | Evaluating whether AI agents make progress toward user goals in conversations |
| Can be applied to | session, trace, all span types (agent, workflow, retriever, LLM and tool) |
| LLM/Luna Support | Supported with both LLM + Luna models |
| Protect Runtime Protection | No - Not applicable for this metric |
| Constants | None - Uses dynamic evaluation |
| Usage Context | Agentic workflows, multi-step tasks, tool-using assistants |
| Value Type | Confidence score (0.0 to 1.0) - Confidence that any one action has advanced |
| Input/Output Requirements | Requires conversation context, user goals, and assistant responses |
When to Use This Metric
When to Use This Metric
Agentic Workflows: When an AI agent must decide on actions and select appropriate tools.
Multi-step Tasks: When completing a user’s request requires multiple steps or decisions.
Tool-using Assistants: When evaluating if the assistant used available tools effectively.
Customer Service Agents: Resolving user issues through multi-step problem-solving.
Task-Oriented Assistants: Completing specific actions like booking flights or processing orders.
Research Assistants: Gathering and synthesizing information across multiple sources.
Creative Assistants: Understanding and building upon user requests iteratively.
Calculation method
If the Action Advancement score is less than 100%, it means at least one evaluator determined the assistant failed to make progress on any user goal. Action Advancement is calculated by:1
Model Request
Multiple evaluation requests are sent to an LLM evaluator to analyze the assistant’s progress toward user goals.
2
Prompt Engineering
A specialized chain-of-thought prompt guides the model to evaluate whether the assistant made progress on user goals based on the metric’s definition.
3
Evaluation Process
Each evaluation analyzes the interaction and produces both a detailed explanation and a binary judgment (yes/no) on goal advancement.
4
Score Calculation
The final Action Advancement score is computed as the confidence score or probability that any one user ask is advanced.
This metric requires multiple LLM calls to compute, which may impact usage and billing.
Score Interpretation
Expected Score: 1.0 (Excellent) - The assistant made clear progress toward the booking goal by gathering necessary information and providing options.0.00.51.0
Poor
Assistant failed to make any progress toward user goalsFair
Assistant made some progress but didn't fully address the user's needsExcellent
Assistant successfully advanced user goals with clear progressWhat different scores mean
- 0.0 - 0.3 (Poor): The assistant completely failed to address the user’s request or made no meaningful progress. Common causes include ignoring the user’s question, providing irrelevant information, or failing to use available tools when needed.
- 0.4 - 0.7 (Fair): The assistant made some progress but didn’t fully accomplish the user’s goal. This might include partial answers, requesting clarification when not needed, or missing key aspects of the request.
- 0.8 - 1.0 (Excellent): The assistant successfully advanced the user’s goal by providing complete answers, making appropriate requests for clarification, or confirming successful task completion.
How to improve Action Advancement scores
To improve Action Advancement scores, focus on ensuring your AI agents make meaningful progress toward user goals in every interaction.Common issues and solutions
| Issue | Cause | Solution |
|---|---|---|
| Assistant ignores user requests | Poor prompt engineering or context understanding | Improve system prompts to emphasize goal-oriented responses and ensure the assistant understands user intent |
| Incomplete responses | Insufficient context or tool usage | Provide better context and ensure the assistant uses available tools effectively |
| Irrelevant information | Lack of focus on user goals | Train the assistant to stay focused on the specific user request and avoid tangential information |
| No progress on multi-step tasks | Poor task breakdown | Implement better task decomposition and ensure the assistant can handle complex, multi-step processes |
Best practices for optimization
- Clear goal identification: Ensure your assistant can identify and prioritize user goals
- Progressive disclosure: Break complex tasks into manageable steps
- Tool integration: Make sure the assistant effectively uses available tools and APIs
- Context awareness: Maintain conversation context to build on previous interactions
Comparison to other metrics
| Property | Action Advancement | Instruction Adherence | Completeness |
|---|---|---|---|
| Metric Category | Agentic Metrics | Response Quality | Response Quality |
| Use this metric for | Evaluating goal progress in conversations | Measuring how well responses follow instructions | Assessing response completeness |
| Best for | Multi-step tasks and agentic workflows | Single-turn instruction following | Ensuring comprehensive responses |
| LLM/Luna Support | Yes | Yes | Yes |
| Protect Runtime Protection | No | No | No |
| Value Type | Percentage (0.0-1.0) | Percentage (0.0-1.0) | Percentage (0.0-1.0) |
| Limitations | Requires conversation context | May not capture goal progress | Doesn’t measure goal advancement |
Best practices
To effectively implement and optimize Action Advancement in your AI systems, consider these key practices:Track progress over time
Monitor Action Advancement scores across different versions of your agent to ensure improvements in task completion capabilities. This helps you identify whether your optimizations are actually improving goal advancement.Analyze failure patterns
When Action Advancement scores are low, examine the specific steps where agents fail to make progress to identify systematic issues. Look for patterns in where agents get stuck or fail to advance user goals.Combine with other metrics
Use Action Advancement alongside other agentic metrics to get a comprehensive view of your assistant’s effectiveness. This provides a more complete picture of your agent’s performance beyond just goal advancement.Test edge cases
Create evaluation datasets that include complex, multi-step tasks to thoroughly assess your agent’s ability to advance user goals. This ensures your agent can handle challenging scenarios that require multiple steps.When optimizing for Action Advancement, ensure you’re not sacrificing other important aspects like safety, factual accuracy, or user experience in pursuit of task completion.