Getting Started with Galileo

Create a new project folder and navigate to it in your terminal:

mkdir first-AI-evaluation
cd first-AI-evaluation

Install the Galileo SDK with OpenAI support and the dotenv package using the following command in your terminal:

pip install "galileo[openai]" python-dotenv

In your project folder, create a new blank application file and .env file:

first-AI-evaluation/
│── app.py
│── .env

In your .env file, add the following credentials:

GALILEO_API_KEY=your-Galileo-api-key-here
OPENAI_API_KEY=your-OpenAI-api-key-here
GALILEO_PROJECT=My first project
GALILEO_LOG_STREAM=physics-questions

To access your Galileo API keys, open the Galileo Console and log in or create an account.

The Galileo Console is the hub for viewing all your Galileo projects, configuring your experiments, and viewing your logged data.

In the .env file, fill in the API key values:

• OPENAI_API_KEY : Obtain your OpenAI API key by logging in to your OpenAI account on their website.
• GALILEO_API_KEY : Obtain your Galileo API key from the Galileo Console. Open “Settings & Users” from the drop down menu in the top-right corner. Then, click “API Keys” in the left column, and generate your API key by clicking ”+ Create new key”.

Copy the code below into your application file:

import os
from galileo import galileo_context # The Galileo context manager
from galileo.openai import openai # The Galileo OpenAI client wrapper
from dotenv import load_dotenv
load_dotenv()

# Initialize the OpenAI client using the Galileo wrapper and your
# OpenAI API key
client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

# The Galileo context manager will automatically create a trace and flush
# the logs when exiting
with galileo_context():
    prompt = f"Explain the following topic succinctly: Newton's First Law"
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
    )
    print(response.choices[0].message.content.strip())

If you are using TypeScript, you will also need to configure your code to use ESM. Add the following to your package.json file:

{
  "type": "module",
  ... // Existing contents
}

Run your application using the following command in your terminal:

python app.py

Congratulations! You have run your first AI inference using the Galileo SDK.

You can see the model’s output in your terminal. Next, you’ll use the Galileo Console to see even more details.

To see the results of your inference in the Galileo Console, you will need to open your project, and then select your log stream.

Use the drop down in the top-left to select your project, “My first project”.

Click “Log Streams” in the top menu to open your project’s Log Streams.

Log Streams are where you can see the activity logged from running this tutorial application.

The first row in your Log Stream is the run Trace from running your application. Click it to view more details.

In the Trace, you can see the prompt and the model’s response. However, there is no data about the model’s performance. In the next section we will add a metric to measure how well the model followed our instructions.

Click “Configure Metrics” on the right to open the menu of built-in Metrics.

Metrics are used to evaluate and track the performance of AI models in your experiments.

Metrics are evaluated using an LLM evaluator, so you will need to configure an LLM to evaluate the instruction adherence.

Click the “OpenAI” link, add your OpenAI API key, and save.

You can also connect to any supported LLM using the integrations page in the Galileo console.

Click the toggle next to “Instruction Adherence” to add the Metric.

The instruction adherence Metric measures how closely the model’s output matches the instructions in the prompt.

With the Instruction Adherence Metric set to be logged, run your application again with the same command as before.

python app.py

The new row in your Log Stream is the run Trace from re-running your application. Click it to view more details.

You can see the instruction adherence measurement in the top-right corner of the Trace.