Create Code Scorer Version

curl --request POST \
  --url https://api.galileo.ai/v2/scorers/{scorer_id}/version/code \
  --header 'Content-Type: multipart/form-data' \
  --header 'Galileo-API-Key: <api-key>' \
  --form file='@example-file' \
  --form 'validation_result=<string>'

{
  "id": "<string>",
  "version": 123,
  "scorer_id": "<string>",
  "created_at": "2023-11-07T05:31:56Z",
  "updated_at": "2023-11-07T05:31:56Z",
  "generated_scorer": {
    "id": "<string>",
    "name": "<string>",
    "chain_poll_template": {
      "template": "<string>",
      "metric_system_prompt": "<string>",
      "metric_description": "<string>",
      "value_field_name": "rating",
      "explanation_field_name": "explanation",
      "metric_few_shot_examples": [
        {
          "generation_prompt_and_response": "<string>",
          "evaluating_response": "<string>"
        }
      ],
      "response_schema": {}
    },
    "created_by": "<string>",
    "created_at": "2023-11-07T05:31:56Z",
    "updated_at": "2023-11-07T05:31:56Z",
    "scoreable_node_types": [
      "chain"
    ],
    "scorer_configuration": {
      "model_alias": "gpt-4.1-mini",
      "num_judges": 3,
      "output_type": "boolean",
      "scoreable_node_types": [
        "<string>"
      ],
      "cot_enabled": false,
      "ground_truth": false
    },
    "instructions": "<string>",
    "user_prompt": "<string>"
  },
  "registered_scorer": {
    "id": "<string>",
    "name": "<string>",
    "score_type": "<string>",
    "created_at": "2023-11-07T05:31:56Z",
    "updated_at": "2023-11-07T05:31:56Z",
    "created_by": "<string>",
    "data_type": "unknown",
    "scoreable_node_types": [
      "<string>"
    ]
  },
  "finetuned_scorer": {
    "id": "<string>",
    "name": "<string>",
    "lora_task_id": 123,
    "prompt": "<string>",
    "created_at": "2023-11-07T05:31:56Z",
    "updated_at": "2023-11-07T05:31:56Z",
    "created_by": "<string>",
    "luna_input_type": "span",
    "luna_output_type": "float",
    "class_name_to_vocab_ix": {},
    "executor": "action_completion_luna"
  },
  "model_name": "<string>",
  "num_judges": 123,
  "scoreable_node_types": [
    "<string>"
  ],
  "cot_enabled": true,
  "output_type": "boolean",
  "input_type": "basic",
  "chain_poll_template": {
    "template": "<string>",
    "metric_system_prompt": "<string>",
    "metric_description": "<string>",
    "value_field_name": "rating",
    "explanation_field_name": "explanation",
    "metric_few_shot_examples": [
      {
        "generation_prompt_and_response": "<string>",
        "evaluating_response": "<string>"
      }
    ],
    "response_schema": {}
  },
  "allowed_model": true
}

POST

scorers

{scorer_id}

version

code

Create Code Scorer Version

curl --request POST \
  --url https://api.galileo.ai/v2/scorers/{scorer_id}/version/code \
  --header 'Content-Type: multipart/form-data' \
  --header 'Galileo-API-Key: <api-key>' \
  --form file='@example-file' \
  --form 'validation_result=<string>'

{
  "id": "<string>",
  "version": 123,
  "scorer_id": "<string>",
  "created_at": "2023-11-07T05:31:56Z",
  "updated_at": "2023-11-07T05:31:56Z",
  "generated_scorer": {
    "id": "<string>",
    "name": "<string>",
    "chain_poll_template": {
      "template": "<string>",
      "metric_system_prompt": "<string>",
      "metric_description": "<string>",
      "value_field_name": "rating",
      "explanation_field_name": "explanation",
      "metric_few_shot_examples": [
        {
          "generation_prompt_and_response": "<string>",
          "evaluating_response": "<string>"
        }
      ],
      "response_schema": {}
    },
    "created_by": "<string>",
    "created_at": "2023-11-07T05:31:56Z",
    "updated_at": "2023-11-07T05:31:56Z",
    "scoreable_node_types": [
      "chain"
    ],
    "scorer_configuration": {
      "model_alias": "gpt-4.1-mini",
      "num_judges": 3,
      "output_type": "boolean",
      "scoreable_node_types": [
        "<string>"
      ],
      "cot_enabled": false,
      "ground_truth": false
    },
    "instructions": "<string>",
    "user_prompt": "<string>"
  },
  "registered_scorer": {
    "id": "<string>",
    "name": "<string>",
    "score_type": "<string>",
    "created_at": "2023-11-07T05:31:56Z",
    "updated_at": "2023-11-07T05:31:56Z",
    "created_by": "<string>",
    "data_type": "unknown",
    "scoreable_node_types": [
      "<string>"
    ]
  },
  "finetuned_scorer": {
    "id": "<string>",
    "name": "<string>",
    "lora_task_id": 123,
    "prompt": "<string>",
    "created_at": "2023-11-07T05:31:56Z",
    "updated_at": "2023-11-07T05:31:56Z",
    "created_by": "<string>",
    "luna_input_type": "span",
    "luna_output_type": "float",
    "class_name_to_vocab_ix": {},
    "executor": "action_completion_luna"
  },
  "model_name": "<string>",
  "num_judges": 123,
  "scoreable_node_types": [
    "<string>"
  ],
  "cot_enabled": true,
  "output_type": "boolean",
  "input_type": "basic",
  "chain_poll_template": {
    "template": "<string>",
    "metric_system_prompt": "<string>",
    "metric_description": "<string>",
    "value_field_name": "rating",
    "explanation_field_name": "explanation",
    "metric_few_shot_examples": [
      {
        "generation_prompt_and_response": "<string>",
        "evaluating_response": "<string>"
      }
    ],
    "response_schema": {}
  },
  "allowed_model": true
}

Authorizations

Galileo-API-Key

string

header

required

Path Parameters

scorer_id

string<uuid4>

required

Body

multipart/form-data

file

required

validation_result

string | null

Pre-validated result as JSON string to skip validation

Response

Successful Response

string<uuid4>

required

version

integer

required

scorer_id

string<uuid4>

required

created_at

string<date-time>

required

updated_at

string<date-time>

required

generated_scorer

GeneratedScorerResponse · object

Show child attributes

generated_scorer.id

string<uuid4>

required

generated_scorer.name

string

required

generated_scorer.chain_poll_template

ChainPollTemplate · object

required

Template for a chainpoll metric prompt, containing all the info necessary to send a chainpoll prompt.

Show child attributes

generated_scorer.chain_poll_template.template

string

required

Chainpoll prompt template.

generated_scorer.chain_poll_template.metric_system_prompt

string | null

System prompt for the metric.

generated_scorer.chain_poll_template.metric_description

string | null

Description of what the metric should do.

generated_scorer.chain_poll_template.value_field_name

string

default:rating

Field name to look for in the chainpoll response, for the rating.

generated_scorer.chain_poll_template.explanation_field_name

string

default:explanation

Field name to look for in the chainpoll response, for the explanation.

generated_scorer.chain_poll_template.metric_few_shot_examples

FewShotExample · object[]

Few-shot examples for the metric.

Show child attributes

generated_scorer.chain_poll_template.metric_few_shot_examples.generation_prompt_and_response

string

required

generated_scorer.chain_poll_template.metric_few_shot_examples.evaluating_response

string

required

generated_scorer.chain_poll_template.response_schema

Response Schema · object

Response schema for the output

generated_scorer.created_by

string<uuid4>

required

generated_scorer.created_at

string<date-time>

required

generated_scorer.updated_at

string<date-time>

required

generated_scorer.scoreable_node_types

enum<string>[] | null

required

Available options:

chain,

chat,

llm,

retriever,

tool,

agent,

workflow,

trace,

session

generated_scorer.scorer_configuration

GeneratedScorerConfiguration · object

required

Show child attributes

generated_scorer.scorer_configuration.model_alias

string

default:gpt-4.1-mini

generated_scorer.scorer_configuration.num_judges

integer

default:3

Required range: 1 <= x <= 10

generated_scorer.scorer_configuration.output_type

enum<string>

default:boolean

Output type of the generated scorer.

Available options:

boolean,

categorical,

count,

discrete,

freeform,

percentage,

multilabel

generated_scorer.scorer_configuration.scoreable_node_types

string[]

Types of nodes that can be scored by this scorer.

generated_scorer.scorer_configuration.cot_enabled

boolean

default:false

Whether chain of thought is enabled for this scorer.

generated_scorer.scorer_configuration.ground_truth

boolean

default:false

Whether ground truth is enabled for this scorer.

generated_scorer.instructions

string | null

generated_scorer.user_prompt

string | null

registered_scorer

CreateUpdateRegisteredScorerResponse · object

Show child attributes

registered_scorer.id

string<uuid4>

required

registered_scorer.name

string

required

registered_scorer.score_type

string | null

required

registered_scorer.created_at

string<date-time>

required

registered_scorer.updated_at

string<date-time>

required

registered_scorer.created_by

string<uuid4>

required

registered_scorer.data_type

enum<string> | null

required

Available options:

unknown,

text,

label,

floating_point,

integer,

timestamp,

milli_seconds,

boolean,

uuid,

percentage,

dollars,

array,

template_label,

thumb_rating_percentage,

user_id,

text_offsets,

segments,

hallucination_segments,

thumb_rating,

score_rating,

star_rating,

tags_rating,

thumb_rating_aggregate,

score_rating_aggregate,

star_rating_aggregate,

tags_rating_aggregate

registered_scorer.scoreable_node_types

string[] | null

required

finetuned_scorer

FineTunedScorerResponse · object

Show child attributes

finetuned_scorer.id

string<uuid4>

required

finetuned_scorer.name

string

required

finetuned_scorer.lora_task_id

integer

required

finetuned_scorer.prompt

string

required

finetuned_scorer.created_at

string<date-time>

required

finetuned_scorer.updated_at

string<date-time>

required

finetuned_scorer.created_by

string<uuid4>

required

finetuned_scorer.luna_input_type

enum<string> | null

Available options:

span,

trace_object,

trace_input_output_only

finetuned_scorer.luna_output_type

enum<string> | null

Available options:

float,

string,

string_list

finetuned_scorer.class_name_to_vocab_ix

Class Name To Vocab Ix · object

Class Name To Vocab Ix
Class Name To Vocab Ix

Show child attributes

finetuned_scorer.class_name_to_vocab_ix.{key}

integer[]

finetuned_scorer.executor

enum<string> | null

Executor pipeline. Defaults to finetuned scorer pipeline but can run custom galileo score pipelines.

Available options:

action_completion_luna,

action_advancement_luna,

agentic_session_success,

agentic_workflow_success,

agent_efficiency,

agent_flow,

bleu,

chunk_attribution_utilization_luna,

chunk_attribution_utilization,

completeness_luna,

completeness,

context_adherence,

context_adherence_luna,

context_relevance,

context_relevance_luna,

conversation_quality,

correctness,

ground_truth_adherence,

input_pii,

input_pii_gpt,

input_sexist,

input_sexist_luna,

input_tone,

input_tone_gpt,

input_toxicity,

input_toxicity_luna,

instruction_adherence,

output_pii,

output_pii_gpt,

output_sexist,

output_sexist_luna,

output_tone,

output_tone_gpt,

output_toxicity,

output_toxicity_luna,

prompt_injection,

prompt_injection_luna,

prompt_perplexity,

rouge,

tool_error_rate,

tool_error_rate_luna,

tool_selection_quality,

tool_selection_quality_luna,

uncertainty,

user_intent_change

model_name

string | null

num_judges

integer | null

scoreable_node_types

string[] | null

cot_enabled

boolean | null

output_type

enum<string> | null

Enumeration of output types.

Available options:

boolean,

categorical,

count,

discrete,

freeform,

percentage,

multilabel

input_type

enum<string> | null

What type of input to use for model-based scorers (sessions_normalized, trace_io_only, etc.).

Available options:

basic,

llm_spans,

retriever_spans,

sessions_normalized,

sessions_trace_io_only,

tool_spans,

trace_input_only,

trace_io_only,

trace_normalized,

trace_output_only,

agent_spans,

workflow_spans

chain_poll_template

ChainPollTemplate · object

Template for a chainpoll metric prompt, containing all the info necessary to send a chainpoll prompt.

Show child attributes

chain_poll_template.template

string

required

Chainpoll prompt template.

chain_poll_template.metric_system_prompt

string | null

System prompt for the metric.

chain_poll_template.metric_description

string | null

Description of what the metric should do.

chain_poll_template.value_field_name

string

default:rating

Field name to look for in the chainpoll response, for the rating.

chain_poll_template.explanation_field_name

string

default:explanation

Field name to look for in the chainpoll response, for the explanation.

chain_poll_template.metric_few_shot_examples

FewShotExample · object[]

Few-shot examples for the metric.

Show child attributes

chain_poll_template.metric_few_shot_examples.generation_prompt_and_response

string

required

chain_poll_template.metric_few_shot_examples.evaluating_response

string

required

chain_poll_template.response_schema

Response Schema · object

Response schema for the output

allowed_model

boolean | null

Get Scorer Version Code

Create Preset Scorer Version

⌘I

Python SDK Reference

TypeScript SDK Reference

API Reference

Create Code Scorer Version

Authorizations

Path Parameters

Body

Response