Class: Scorers
Defined in: src/entities/scorers.ts Service class for scorer operations.Constructors
Constructor
Returns
Scorers
Methods
create()
Parameters
options
The scorer creation options.defaults?
{cot_enabled?: null | boolean;
filters?: | null
| (
| {
case_sensitive?: boolean;
filter_type?: "string";
name: "node_name";
operator: "eq" | "ne" | "contains";
value: string;
}
| {
filter_type?: "map";
key: string;
name: "metadata";
operator: "eq" | "ne" | "not_in" | "one_of";
value: string | string[];
})[];
input_type?: | null
| "basic"
| "llm_spans"
| "retriever_spans"
| "sessions_normalized"
| "sessions_trace_io_only"
| "tool_spans"
| "trace_input_only"
| "trace_io_only"
| "trace_normalized"
| "trace_output_only"
| "agent_spans"
| "workflow_spans";
model_name?: null | string;
num_judges?: null | number;
output_type?: | null
| "boolean"
| "categorical"
| "count"
| "discrete"
| "freeform"
| "percentage"
| "multilabel";
scoreable_node_types?: null | string[];
}
(Optional) Default settings for the scorer.
defaults.cot_enabled?
null | boolean
Cot Enabled
Description
Whether to enable chain of thought for this scorer. Defaults to False for llm scorers.
defaults.filters?
|null
| (
| {
case_sensitive?: boolean;
filter_type?: "string";
name: "node_name";
operator: "eq" | "ne" | "contains";
value: string;
}
| {
filter_type?: "map";
key: string;
name: "metadata";
operator: "eq" | "ne" | "not_in" | "one_of";
value: string | string[];
})[]
Filters
Description
List of filters to apply to the scorer.
defaults.input_type?
|null
| "basic"
| "llm_spans"
| "retriever_spans"
| "sessions_normalized"
| "sessions_trace_io_only"
| "tool_spans"
| "trace_input_only"
| "trace_io_only"
| "trace_normalized"
| "trace_output_only"
| "agent_spans"
| "workflow_spans"
Description
What type of input to use for model-based scorers (sessions_normalized, trace_io_only, etc..).
defaults.model_name?
null | string
Model Name
defaults.num_judges?
null | number
Num Judges
defaults.output_type?
|null
| "boolean"
| "categorical"
| "count"
| "discrete"
| "freeform"
| "percentage"
| "multilabel"
Description
What type of output to use for model-based scorers (boolean, categorical, etc.).
defaults.scoreable_node_types?
null | string[]
Scoreable Node Types
Description
List of node types that can be scored by this scorer. Defaults to llm/chat.
defaultVersionId?
string
(Optional) The default version ID for the scorer.
description?
string
(Optional) A description for the scorer.
inputType?
InputType
(Optional) The input type for the scorer.
modelType?
"llm" | "code" | "slm"
(Optional) The model type for the scorer.
name
string
The name of the scorer.
outputType?
OutputType
(Optional) The output type for the scorer.
scoreableNodeTypes?
("agent" | "llm" | "retriever" | "tool" | "workflow" | "trace" | "session")[]
(Optional) The node types that can be scored.
scorerType
ScorerTypes
The type of the scorer.
tags?
string[]
(Optional) Tags to associate with the scorer.
Returns
Promise<ScorerResponse>
A promise that resolves to the created scorer.
createCodeScorerVersion()
Parameters
scorerId
string
The unique identifier of the scorer.
codeContent
string
The Python code content for the scorer.
validationResult?
string
(Optional) The validation result JSON string.
Returns
Promise<BaseScorerVersionResponse>
A promise that resolves to the created scorer version.
createLlmScorerVersion()
Parameters
options
The LLM scorer version creation options.chainPollTemplate?
{explanation_field_name?: string;
metric_description?: null | string;
metric_few_shot_examples?: object[];
metric_system_prompt?: null | string;
response_schema?: | null
| {
[key: string]: unknown;
};
template: string;
value_field_name?: string;
}
(Optional) Chain poll template configuration.
chainPollTemplate.explanation_field_name?
string
Explanation Field Name
Description
Field name to look for in the chainpoll response, for the explanation.
Default
chainPollTemplate.metric_description?
null | string
Metric Description
Description
Description of what the metric should do.
chainPollTemplate.metric_few_shot_examples?
object[]
Metric Few Shot Examples
Description
Few-shot examples for the metric.
chainPollTemplate.metric_system_prompt?
null | string
Metric System Prompt
Description
System prompt for the metric.
chainPollTemplate.response_schema?
|null
| {
[key: string]: unknown;
}
Response Schema
Description
Response schema for the output
chainPollTemplate.template
string
Template
Description
Chainpoll prompt template.
chainPollTemplate.value_field_name?
string
Value Field Name
Description
Field name to look for in the chainpoll response, for the rating.
Default
cotEnabled?
boolean
(Optional) Whether chain-of-thought is enabled.
instructions?
string
(Optional) Instructions for the LLM scorer.
modelName?
string
(Optional) The model name to use.
numJudges?
number
(Optional) The number of judges for consensus.
scorerId
string
The unique identifier of the scorer.
userPrompt?
string
(Optional) User prompt for the LLM scorer.
Returns
Promise<BaseScorerVersionResponse>
A promise that resolves to the created scorer version.
delete()
Parameters
scorerId
string
The unique identifier of the scorer to delete.
Returns
Promise<DeleteScorerResponse>
A promise that resolves to a response containing a success message.
getScorerVersion()
Parameters
scorerId
string
The unique identifier of the scorer.
version
number
The version number to retrieve.
Returns
Promise<BaseScorerVersionResponse>
A promise that resolves to the scorer version.
list()
Parameters
options?
(Optional) The filtering options.name?
string
(Optional) Filter by a single scorer name.
names?
string[]
(Optional) Filter by multiple scorer names.
types?
ScorerTypes[]
(Optional) Filter by scorer types.
Returns
Promise<ScorerResponse[]>
A promise that resolves to an array of scorers.