ℹ️ These docs are for the v2.0 version of Galileo. Documentation for v1.0 version can be found here.
Python SDK Reference
TypeScript SDK Reference
API Reference
- POSTCreate
- GETGet Scorer
- DELDelete Scorer
- PATCHUpdate
- POSTCreate Llm Scorer Version
- POSTValidate Code Scorer
- GETGet Validate Code Scorer Task Result
- GETGet Scorer Version Code
- POSTCreate Code Scorer Version
- POSTCreate Preset Scorer Version
- POSTCreate Luna Scorer Version
- POSTList Scorers With Filters
- GETList Tags
- GETGet Scorer Version Or Latest
- GETList All Versions For Scorer
- GETList Projects For Scorer Route
- GETList Projects For Scorer Version Route
- POSTRestore Scorer Version
- POSTAutogen Llm Scorer
- POSTManual Llm Validate
- POSTValidate Llm Scorer Log Record
- POST
List Scorers With Filters
Copy
Ask AI
curl --request POST \
--url https://api.galileo.ai/v2/scorers/list \
--header 'Content-Type: application/json' \
--header 'Galileo-API-Key: <api-key>' \
--data '
{
"filters": [
{
"operator": "eq",
"value": "<string>",
"name": "name",
"case_sensitive": false
}
],
"sort": {
"name": "name",
"ascending": true,
"sort_type": "column"
}
}
'Copy
Ask AI
{
"starting_token": 0,
"limit": 100,
"paginated": false,
"next_starting_token": 123,
"scorers": [
{
"id": "<string>",
"name": "<string>",
"scorer_type": "llm",
"tags": [
"<string>"
],
"defaults": {
"model_name": "<string>",
"num_judges": 123,
"filters": [
{
"value": "<string>",
"operator": "eq",
"name": "node_name",
"filter_type": "string",
"case_sensitive": true
}
],
"scoreable_node_types": [
"<string>"
],
"cot_enabled": true,
"output_type": "boolean",
"input_type": "basic"
},
"latest_version": {
"id": "<string>",
"version": 123,
"scorer_id": "<string>",
"generated_scorer": {
"id": "<string>",
"name": "<string>",
"chain_poll_template": {
"template": "<string>",
"metric_system_prompt": "<string>",
"metric_description": "<string>",
"value_field_name": "rating",
"explanation_field_name": "explanation",
"metric_few_shot_examples": [
{
"generation_prompt_and_response": "<string>",
"evaluating_response": "<string>"
}
],
"response_schema": {}
},
"instructions": "<string>",
"user_prompt": "<string>"
},
"registered_scorer": {
"id": "<string>",
"name": "<string>",
"score_type": "<string>"
},
"finetuned_scorer": {
"id": "<string>",
"name": "<string>",
"lora_task_id": 123,
"prompt": "<string>",
"luna_input_type": "span",
"luna_output_type": "float",
"class_name_to_vocab_ix": {},
"executor": "action_completion_luna"
},
"model_name": "<string>",
"num_judges": 123,
"scoreable_node_types": [
"<string>"
],
"cot_enabled": true,
"output_type": "boolean",
"input_type": "basic"
},
"model_type": "slm",
"ground_truth": true,
"default_version_id": "<string>",
"default_version": {
"id": "<string>",
"version": 123,
"scorer_id": "<string>",
"generated_scorer": {
"id": "<string>",
"name": "<string>",
"chain_poll_template": {
"template": "<string>",
"metric_system_prompt": "<string>",
"metric_description": "<string>",
"value_field_name": "rating",
"explanation_field_name": "explanation",
"metric_few_shot_examples": [
{
"generation_prompt_and_response": "<string>",
"evaluating_response": "<string>"
}
],
"response_schema": {}
},
"instructions": "<string>",
"user_prompt": "<string>"
},
"registered_scorer": {
"id": "<string>",
"name": "<string>",
"score_type": "<string>"
},
"finetuned_scorer": {
"id": "<string>",
"name": "<string>",
"lora_task_id": 123,
"prompt": "<string>",
"luna_input_type": "span",
"luna_output_type": "float",
"class_name_to_vocab_ix": {},
"executor": "action_completion_luna"
},
"model_name": "<string>",
"num_judges": 123,
"scoreable_node_types": [
"<string>"
],
"cot_enabled": true,
"output_type": "boolean",
"input_type": "basic"
},
"user_prompt": "<string>",
"scoreable_node_types": [
"<string>"
],
"output_type": "boolean",
"input_type": "basic",
"required_scorers": [
"<string>"
],
"label": "",
"included_fields": [
"<string>"
],
"description": "<string>",
"created_by": "<string>",
"created_at": "2023-11-07T05:31:56Z",
"updated_at": "2023-11-07T05:31:56Z"
}
]
}data
List Scorers With Filters
POST
/
v2
/
scorers
/
list
List Scorers With Filters
Copy
Ask AI
curl --request POST \
--url https://api.galileo.ai/v2/scorers/list \
--header 'Content-Type: application/json' \
--header 'Galileo-API-Key: <api-key>' \
--data '
{
"filters": [
{
"operator": "eq",
"value": "<string>",
"name": "name",
"case_sensitive": false
}
],
"sort": {
"name": "name",
"ascending": true,
"sort_type": "column"
}
}
'Copy
Ask AI
{
"starting_token": 0,
"limit": 100,
"paginated": false,
"next_starting_token": 123,
"scorers": [
{
"id": "<string>",
"name": "<string>",
"scorer_type": "llm",
"tags": [
"<string>"
],
"defaults": {
"model_name": "<string>",
"num_judges": 123,
"filters": [
{
"value": "<string>",
"operator": "eq",
"name": "node_name",
"filter_type": "string",
"case_sensitive": true
}
],
"scoreable_node_types": [
"<string>"
],
"cot_enabled": true,
"output_type": "boolean",
"input_type": "basic"
},
"latest_version": {
"id": "<string>",
"version": 123,
"scorer_id": "<string>",
"generated_scorer": {
"id": "<string>",
"name": "<string>",
"chain_poll_template": {
"template": "<string>",
"metric_system_prompt": "<string>",
"metric_description": "<string>",
"value_field_name": "rating",
"explanation_field_name": "explanation",
"metric_few_shot_examples": [
{
"generation_prompt_and_response": "<string>",
"evaluating_response": "<string>"
}
],
"response_schema": {}
},
"instructions": "<string>",
"user_prompt": "<string>"
},
"registered_scorer": {
"id": "<string>",
"name": "<string>",
"score_type": "<string>"
},
"finetuned_scorer": {
"id": "<string>",
"name": "<string>",
"lora_task_id": 123,
"prompt": "<string>",
"luna_input_type": "span",
"luna_output_type": "float",
"class_name_to_vocab_ix": {},
"executor": "action_completion_luna"
},
"model_name": "<string>",
"num_judges": 123,
"scoreable_node_types": [
"<string>"
],
"cot_enabled": true,
"output_type": "boolean",
"input_type": "basic"
},
"model_type": "slm",
"ground_truth": true,
"default_version_id": "<string>",
"default_version": {
"id": "<string>",
"version": 123,
"scorer_id": "<string>",
"generated_scorer": {
"id": "<string>",
"name": "<string>",
"chain_poll_template": {
"template": "<string>",
"metric_system_prompt": "<string>",
"metric_description": "<string>",
"value_field_name": "rating",
"explanation_field_name": "explanation",
"metric_few_shot_examples": [
{
"generation_prompt_and_response": "<string>",
"evaluating_response": "<string>"
}
],
"response_schema": {}
},
"instructions": "<string>",
"user_prompt": "<string>"
},
"registered_scorer": {
"id": "<string>",
"name": "<string>",
"score_type": "<string>"
},
"finetuned_scorer": {
"id": "<string>",
"name": "<string>",
"lora_task_id": 123,
"prompt": "<string>",
"luna_input_type": "span",
"luna_output_type": "float",
"class_name_to_vocab_ix": {},
"executor": "action_completion_luna"
},
"model_name": "<string>",
"num_judges": 123,
"scoreable_node_types": [
"<string>"
],
"cot_enabled": true,
"output_type": "boolean",
"input_type": "basic"
},
"user_prompt": "<string>",
"scoreable_node_types": [
"<string>"
],
"output_type": "boolean",
"input_type": "basic",
"required_scorers": [
"<string>"
],
"label": "",
"included_fields": [
"<string>"
],
"description": "<string>",
"created_by": "<string>",
"created_at": "2023-11-07T05:31:56Z",
"updated_at": "2023-11-07T05:31:56Z"
}
]
}Authorizations
Body
application/json
filters
(ScorerNameFilter · object | ScorerTypeFilter · object | ScorerModelTypeFilter · object | ScorerTagsFilter · object | ScorerCreatorFilter · object | ScorerCreatedAtFilter · object | ScorerUpdatedAtFilter · object | ScorerLabelFilter · object)[]
- ScorerNameFilter
- ScorerTypeFilter
- ScorerModelTypeFilter
- ScorerTagsFilter
- ScorerCreatorFilter
- ScorerCreatedAtFilter
- ScorerUpdatedAtFilter
- ScorerLabelFilter
Response
Successful Response
Show child attributes
Show child attributes
Available options:
llm, code, luna, preset Show child attributes
Show child attributes
List of filters to apply to the scorer.
Filters on node names in scorer jobs.
- NodeNameFilter
- MetadataFilter
Show child attributes
Show child attributes
Available options:
eq, ne, contains Allowed value:
"node_name"Allowed value:
"string"List of node types that can be scored by this scorer. Defaults to llm/chat.
Whether to enable chain of thought for this scorer. Defaults to False for llm scorers.
What type of output to use for model-based scorers (boolean, categorical, etc.).
Available options:
boolean, categorical, count, discrete, freeform, percentage, multilabel What type of input to use for model-based scorers (sessions_normalized, trace_io_only, etc..).
Available options:
basic, llm_spans, retriever_spans, sessions_normalized, sessions_trace_io_only, tool_spans, trace_input_only, trace_io_only, trace_normalized, trace_output_only, agent_spans, workflow_spans Scorer version from the scorer_versions table.
Show child attributes
Show child attributes
Show child attributes
Show child attributes
Template for a chainpoll metric prompt, containing all the info necessary to send a chainpoll prompt.
Show child attributes
Show child attributes
Chainpoll prompt template.
System prompt for the metric.
Description of what the metric should do.
Field name to look for in the chainpoll response, for the rating.
scorers.latest_version.generated_scorer.chain_poll_template.explanation_field_name
string
default:explanation
Field name to look for in the chainpoll response, for the explanation.
scorers.latest_version.generated_scorer.chain_poll_template.metric_few_shot_examples
FewShotExample · object[]
Few-shot examples for the metric.
scorers.latest_version.generated_scorer.chain_poll_template.response_schema
Response Schema · object
Response schema for the output
Show child attributes
Show child attributes
Available options:
span, trace_object, trace_input_output_only Available options:
float, string, string_list Executor pipeline. Defaults to finetuned scorer pipeline but can run custom galileo score pipelines.
Available options:
action_completion_luna, action_advancement_luna, agentic_session_success, agentic_session_success, agentic_workflow_success, agentic_workflow_success, agent_efficiency, agent_flow, bleu, chunk_attribution_utilization_luna, chunk_attribution_utilization, completeness_luna, completeness, context_adherence, context_adherence_luna, context_relevance, context_relevance_luna, conversation_quality, correctness, ground_truth_adherence, input_pii, input_pii_gpt, input_sexist, input_sexist, input_sexist_luna, input_sexist_luna, input_tone, input_tone_gpt, input_toxicity, input_toxicity_luna, instruction_adherence, output_pii, output_pii_gpt, output_sexist, output_sexist, output_sexist_luna, output_sexist_luna, output_tone, output_tone_gpt, output_toxicity, output_toxicity_luna, prompt_injection, prompt_injection_luna, prompt_perplexity, rouge, tool_error_rate, tool_error_rate_luna, tool_selection_quality, tool_selection_quality_luna, uncertainty, user_intent_change List of node types that can be scored by this scorer. Defaults to llm/chat.
Whether to enable chain of thought for this scorer. Defaults to False for llm scorers.
What type of output to use for model-based scorers (sessions_normalized, trace_io_only, etc.).
Available options:
boolean, categorical, count, discrete, freeform, percentage, multilabel What type of input to use for model-based scorers (sessions_normalized, trace_io_only, etc.).
Available options:
basic, llm_spans, retriever_spans, sessions_normalized, sessions_trace_io_only, tool_spans, trace_input_only, trace_io_only, trace_normalized, trace_output_only, agent_spans, workflow_spans Available options:
slm, llm, code Scorer version from the scorer_versions table.
Show child attributes
Show child attributes
Show child attributes
Show child attributes
Template for a chainpoll metric prompt, containing all the info necessary to send a chainpoll prompt.
Show child attributes
Show child attributes
Chainpoll prompt template.
System prompt for the metric.
Description of what the metric should do.
Field name to look for in the chainpoll response, for the rating.
scorers.default_version.generated_scorer.chain_poll_template.explanation_field_name
string
default:explanation
Field name to look for in the chainpoll response, for the explanation.
scorers.default_version.generated_scorer.chain_poll_template.metric_few_shot_examples
FewShotExample · object[]
Few-shot examples for the metric.
scorers.default_version.generated_scorer.chain_poll_template.response_schema
Response Schema · object
Response schema for the output
Show child attributes
Show child attributes
Available options:
span, trace_object, trace_input_output_only Available options:
float, string, string_list Executor pipeline. Defaults to finetuned scorer pipeline but can run custom galileo score pipelines.
Available options:
action_completion_luna, action_advancement_luna, agentic_session_success, agentic_session_success, agentic_workflow_success, agentic_workflow_success, agent_efficiency, agent_flow, bleu, chunk_attribution_utilization_luna, chunk_attribution_utilization, completeness_luna, completeness, context_adherence, context_adherence_luna, context_relevance, context_relevance_luna, conversation_quality, correctness, ground_truth_adherence, input_pii, input_pii_gpt, input_sexist, input_sexist, input_sexist_luna, input_sexist_luna, input_tone, input_tone_gpt, input_toxicity, input_toxicity_luna, instruction_adherence, output_pii, output_pii_gpt, output_sexist, output_sexist, output_sexist_luna, output_sexist_luna, output_tone, output_tone_gpt, output_toxicity, output_toxicity_luna, prompt_injection, prompt_injection_luna, prompt_perplexity, rouge, tool_error_rate, tool_error_rate_luna, tool_selection_quality, tool_selection_quality_luna, uncertainty, user_intent_change List of node types that can be scored by this scorer. Defaults to llm/chat.
Whether to enable chain of thought for this scorer. Defaults to False for llm scorers.
What type of output to use for model-based scorers (sessions_normalized, trace_io_only, etc.).
Available options:
boolean, categorical, count, discrete, freeform, percentage, multilabel What type of input to use for model-based scorers (sessions_normalized, trace_io_only, etc.).
Available options:
basic, llm_spans, retriever_spans, sessions_normalized, sessions_trace_io_only, tool_spans, trace_input_only, trace_io_only, trace_normalized, trace_output_only, agent_spans, workflow_spans Enumeration of output types.
Available options:
boolean, categorical, count, discrete, freeform, percentage, multilabel Enumeration of input types.
Available options:
basic, llm_spans, retriever_spans, sessions_normalized, sessions_trace_io_only, tool_spans, trace_input_only, trace_io_only, trace_normalized, trace_output_only, agent_spans, workflow_spans Fields that can be used in the scorer to configure it. i.e. model, num_judges, etc. This enables the ui to know which fields a user can configure when they're setting a scorer
Was this page helpful?
⌘I