As you progress from initial testing to systematic evaluation, you’ll want to run experiments to validate your application’s performance and behavior. Here are several ways to structure your experiments, starting from the simplest approaches and moving to more sophisticated implementations.

Experiments fit both into the initial prompt engineering and model selection phases of your app, as well as during application development time, such as during testing or in a CI/CD pipeline. This allows you to fit experiments into your SDLC for evaluation-driven development.

AI Engineers and data scientists can use experiments in notebooks or in simple applications to test out prompts or different models. AI Engineers can then add experiments into their production apps allowing these experiments to be run against complex applications or scenarios, including RAG and agentic flows.

​

Configure an LLM Integration

To calculate metrics, you will either need to configure an integration with an LLM, or set up the Luna 2 SLM.

To configure an LLM, visit the relevant API platform to obtain an API key, then add it using the integrations page in the Galileo console.

​

Experiment Flow

The entry point for running experiments is a call to the run experiments function (see the run_experiment Python SDK docs, or the runExperiment TypeScript SDK docs for more details).

Experiments take a dataset, and can either pass it to a prompt template, or to a custom function. This custom function can go from a simple call to an LLM right up to a full agentic workflow.

For each row in a dataset, a new trace is created, and either the prompt template is logged as an LLM span, or every span created in the custom function is logged to that trace.

If you are building experiments into your production application, you will need to enable a way to call the experiment runner. For example, you can do this inside a unit test.

​

Work with Prompts

The simplest way to get started with experimentation is by evaluating prompts directly against datasets. This is especially valuable during the initial prompt development and refinement phase, where you want to test different prompt variations. Assuming you’ve previously created a dataset, you can use the following code to run an experiment:

from galileo import Message, MessageRole
from galileo.prompts import create_prompt_template 
from galileo.experiments import run_experiment
from galileo.datasets import get_dataset
from galileo.schema.metrics import GalileoScorers

project = "my-project"

# 1a. If the prompt template does not exist, create it:
prompt_template = create_prompt_template(
	name="geography-prompt",
	project=project,
	messages=[
		Message(role=MessageRole.system, 
				content="""
				You are a geography expert.
				Respond with only the continent name.
				"""),
		Message(role=MessageRole.user, content="{{input}}")
	]
)

# 1b. (OPTIONAL) If the prompt template already exists, fetch it:
# prompt_template = get_prompt_template(name="geography-prompt",
#                                       project=project)  

# 2. Run the experiment and get results
results = run_experiment(
	"geography-experiment",
	# Name of a dataset you created
	dataset=get_dataset(name="countries"), 
	prompt_template=prompt_template,
	# Optional
	prompt_settings={
		"max_tokens": 256,
		# Make sure you have an integration set up
		# for the model alias you're using
		"model_alias": "GPT-4o",
		"temperature": 0.8
	},
	metrics=[GalileoScorers.correctness],
	project=project
)

​

Run Experiments with Custom Functions

Once you’re comfortable with basic prompt testing, you might want to evaluate more complex parts of your app using your datasets. This approach is particularly useful when you have a generation function in your app that takes a set of inputs, which you can model with a dataset.

If your experiment runs code that uses the log decorator, or a third-party SDK integration, then all the spans created by these will be logged to the experiment.

This example uses the log decorator. The workflow span created by the log decorator will be logged to the experiment.

from galileo import log
from galileo.experiments import run_experiment
from galileo.datasets import get_dataset
from galileo.schema.metrics import GalileoScorers

dataset = get_dataset(name="countries")

@log(span_type="llm", name= "My Span")
def llm_call(input):
  # Custom function code
	return result

results = run_experiment(
	"geography-experiment",
	dataset=dataset,
	function=llm_call,
	metrics=[GalileoScorers.correctness],
	project="my-project",
)

This example uses the OpenAI SDK wrapper. The LLM span created by the wrapper will be logged to the experiment.

import os
from galileo.experiments import run_experiment
from galileo.datasets import get_dataset
from galileo.openai import openai
from galileo.schema.metrics import GalileoScorers

client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
dataset = get_dataset(name="countries")

def llm_call(input):
	return client.chat.completions.create(
        model="gpt-4o",
        messages=[
          {
            "role": "system",
            "content": "You are a geography expert."
          },
          {
            "role": "user",
            "content": f"""
            Which continent does the following country belong to: {input}
            """
          }
        ],
    ).choices[0].message.content

results = run_experiment(
	"geography-experiment",
	dataset=dataset,
	function=llm_call,
	metrics=[GalileoScorers.correctness],
	project="my-project",
)

​

Run Experiments Against Complex Code with Custom Functions

Custom functions can be as complex as required, including multiple steps, agents, RAG, and more. This means you can build experiments around an existing application, allowing you to run experiments against the full application you have built, using datasets to mimic user inputs.

For example, if you have a multi-agent LangGraph chatbot application, you can run an experiment against it using a dataset to define different user inputs, and log every stage in the agentic flow as part of that experiment.

To enable this, you will need to make some small changes to your application logic to handle the logging context from the experiment.

When functions in your application are run by the run_experiment call, a logger is created by the experiment runner, and a trace is started. This logger can be passed through the application, accessed using the @log decorator or by calling galileo_context.get_logger_instance() in Python, or getLogger in TypeScript.

You will need to change your code to use this instead of creating a new logger and starting a new trace.

​

Get an existing logger and check for an existing trace

The Galileo SDK maintains a context that tracks the current logger. You can get this logger with the following code:

from galileo import galileo_context

# Get the current logger
galileo_logger=galileo_context.get_logger_instance()

If there isn’t a current logger, one will be created by this call, so this will always return a logger.

Once you have the logger, you can check for an existing trace by accessing the current parent trace from the logger. If this is not set, then there is no active trace.

has_existing_trace = galileo_logger.current_parent() is not None

You can use this to decide if you need to create a new trace in your application. If there is no parent trace, you can safely create a new one.

def process_message(input):
    # Get the Galileo logger instance
    galileo_logger = galileo_context.get_logger_instance()

    # If there is a current parent trace, we are in an experiment
    # Otherwise, we start a new trace for the chat workflow
    is_in_experiment = False
    if not galileo_logger.current_parent():
        galileo_logger.start_trace(
            input=input,
            name="Chat Workflow"
        )
    else:
        is_in_experiment = True

    # Your code goes here to process the input and create log spans as needed
    # You can also pass this log to other functions, or access it in those using
    # galileo_context.get_logger_instance()

    # If we are not in an experiment, we conclude and flush the trace
    if not is_in_experiment:
        galileo_logger.conclude("Some output")
        galileo_logger.flush()

You can then safely call your code from the experiment runner as well as in your normal application logic. When called from the experiment runner, your traces will be logged to that experiment. When called from your application code, the traces will be logged as normal.

​

Using LangChain or LangGraph in an experiment

When using LangChain or LangGraph, Galileo provides a callback class that handles creating a logger, starting a trace, logging spans, then concluding and flushing the trace. This behavior is inconsistent with that required for experiments, where the logger is created and trace started at the start of the experiment, and the logger is concluded and flushed at the end.

To work around this, you can tell the callback to not start or flush the trace by detecting if there is already an active trace. If there is, then don’t start a new trace or flush it on completion.

The easiest way to do this is to get the current logger from the Galileo context, and check to see if it contains a parent trace. If there is no parent trace, then it is a new logger instance and you can start and flush the trace. If there is a parent trace, then it is an existing logger created from the experiment, and you can create the callback setting parameters to not start or flush the trace.

# Get the logger from the current Galileo context
logger = galileo_context.get_logger_instance()

is_in_experiment = logger.current_parent() is not None

# Create the callback checking to see if we are in an experiment
# If we are, set start_new_trace and flush_on_chain_end to False
# so that the existing trace is used, and not flushed
galileo_callback = GalileoAsyncCallback(
    logger,
    start_new_trace=not is_in_experiment,
    flush_on_chain_end=not is_in_experiment,
)
callbacks: Callbacks = [galileo_callback]

​

Custom Function Logging Principles

There are a few important principles to understand when logging experiments in code.

• When running an experiment, a new logger is created for you and set in the Galileo context. If you create a new logger manually in the application code used in your experiment, this logger will not be used in the experiment.
• To access the logger to manually add traces inside the experiment code, you can call galileo_context.get_logger_instance() (Python) or getLogger() (TypeScript) to get the current logger.
• To detect if there is an active trace, use the current_parent() (Python) or currentParent (TypeScript) method on the logger. This will return None/undefined if there isn’t an active trace.
• Be sure to handle cases in your application code where a logger is created or a trace is started, and make sure this doesn’t happen in an experiment, and the experiment logger and trace is used instead.
• Every row in a dataset is a new trace. If you create new traces manually, they will not be used.
• Do not conclude or flush the logger in your experiment, the experiment will do this for you.

​

Ground truth

For BLUE, ROUGE, and Ground Truth Adherence, you also need to set the ground truth in your dataset. This is set in the output column.

dataset = [
  {
    "input": "Spain"
    "output": "Spain is in Europe"
  }
]

​

Custom Dataset Evaluation

As your testing needs become more specific, you might need to work with custom or local datasets. This approach is perfect for focused testing of edge cases or when building up your test suite with specific scenarios:

import os
from galileo.experiments import run_experiment
from galileo.openai import openai
from galileo.schema.metrics import GalileoScorers

dataset = [
  {
    "input": "Spain"
  }
]

def llm_call(input):
  client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
  return client.chat.completions.create(
        model="gpt-4",
        messages=[
          {
            "role": "system",
            "content": "You are a geography expert"
          },
          {
            "role": "user",
            "content": f"""
            Which continent does the following country belong to: {input}
            """
          }
        ],
    ).choices[0].message.content

results = run_experiment(
	"geography-experiment",
	dataset=dataset,
	function=llm_call,
	metrics=[GalileoScorers.correctness],
	project="my-project"
)

​

Custom Metrics for Deep Analysis

For the most sophisticated level of testing, you might need to track specific aspects of your application’s behavior. Custom metrics provide the flexibility to define precisely what you want to measure, enabling deep analysis and targeted improvement:

import os
from galileo import Trace, Span
from galileo.experiments import run_experiment
from galileo.openai import openai
from galileo.schema.metrics import LocalMetricConfig

# 1. Scorer Function
def brevity_rank(step: Span | Trace) -> str:
    """Rank response brevity based on word count."""
    word_count = len(step.output.content.split(" "))
    if word_count <= 3:
        return "Terse"
    if word_count <= 5:
        return "Temperate"
    return "Talkative"

# 2. Aggregator Function
def brevity_aggregator(ranks: list[str]) -> str:
    """Extract the single rank and adjust if it's 'Talkative'."""
    return "Terrible" if ranks[0] == "Talkative" else ranks[0]

# 3. Configure the Local Metric
terseness = LocalMetricConfig[str](
    name="Terseness",
    scorer_fn=brevity_rank,
    aggregator_fn=brevity_aggregator,
)

# 4. Dataset
countries_dataset = [
    {"input": "Indonesia"},
    {"input": "New Zealand"},
    {"input": "Greenland"},
    {"input": "China"},
]

# 5. LLM-Call Function
def llm_call(input):
    client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
    return (
        client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {
                    "role": "system", 
                    "content": """
                    You are a geography expert. Always answer as succinctly as possible.
                    """
                },
                {
                    "role": "user", 
                    "content": f"""
                    Which continent does the following country belong to: {input}
                    """
                },
            ],
        )
        .choices[0]
        .message.content
    )

# 6. Run the Experiment!
results = run_experiment(
    "terseness-custom-metric",
    dataset=countries_dataset,
    function=llm_call,
    metrics=[terseness],  # You can add multiple custom metrics here
    project="My first project",
)

Each of these experimentation approaches fits into different stages of your development and testing workflow. As you progress from simple prompt testing to sophisticated custom metrics, Galileo’s experimentation framework provides the tools you need to gather insights and improve your application’s performance at every level of complexity.

​

Experimenting with Agentic and RAG Applications

The experimentation framework extends naturally to more complex applications like agentic AI systems and RAG (Retrieval-Augmented Generation) applications. When working with agents, you can evaluate various aspects of their behavior, from decision-making capabilities to tool usage patterns. This is particularly valuable when testing how agents handle complex workflows, multi-step reasoning, or tool selection.

For RAG applications, experimentation helps validate both the retrieval and generation components of your system. You can assess the quality of retrieved context, measure response relevance, and ensure that your RAG pipeline maintains high accuracy across different types of queries. This is especially important when fine-tuning retrieval parameters or testing different reranking strategies.

The same experimentation patterns shown above apply to these more complex systems. You can use predefined datasets to benchmark performance, create custom datasets for specific edge cases, and define specialized metrics that capture the unique aspects of agent behavior or RAG performance. This systematic approach to testing helps ensure that your advanced AI applications maintain high quality and reliability in production environments.

​

Best Practices

1. Use consistent datasets: Use the same dataset when comparing different prompts or models to ensure fair comparisons.
2. Test multiple variations: Run experiments with different prompt variations to find the best approach.
3. Use appropriate metrics: Choose metrics that are relevant to your specific use case.
4. Start small: Begin with a small dataset to quickly iterate and refine your approach before scaling up.
5. Document your experiments: Keep track of what you’re testing and why to make it easier to interpret results.

​

Next Steps

​

Experiments SDK

​

Metrics