Log Streams

• Separate logs from different environments (development, staging, production)
• Isolate logs from different applications or services
• Group logs by feature, team, or business unit
• Organize logs for specific experiments or testing scenarios

​

Why Log streams are important

1. Organization: Log Streams provide a clean separation between different parts of your application ecosystem, preventing logs from becoming mixed and difficult to analyze.
2. Environment Isolation: They allow you to keep logs from development, staging, and production environments separate, making it easier to focus on relevant data.
3. Lifecycle Management: Different Log Streams can have different retention policies, allowing you to keep critical production logs longer while purging development logs more frequently.

​

How Log streams support workflows

• Debugging: When issues arise, you can focus on logs from the specific environment or application where the problem occurred, reducing noise and speeding up troubleshooting.
• Monitoring: Log Streams enable targeted monitoring of specific applications or environments, making it easier to set up relevant alerts and dashboards.
• Analysis: By organizing logs into logical groups, Log Streams facilitate more focused analysis and pattern recognition within specific contexts.

​

Key components of a Log stream

1. Name: A unique identifier for the Log Stream within a project
2. Description: Optional information about the purpose or content of the Log Stream
3. Tags: Labels for additional categorization and filtering
4. Metadata: Additional contextual information about the Log Stream
5. Access Controls: Permissions determining who can view or modify the Log Stream
6. Metrics: Configured metrics that will be evaluated for each trace and session

​

How logs are grouped in Log streams

• Explicit Assignment: When creating logs, you can explicitly specify which Log Stream they should be sent to.
• Default Assignment: If no Log Stream is specified, logs may be sent to a default Log Stream.
• Pattern-Based Routing: In some configurations, logs can be routed to different Log Streams based on patterns or rules.

• Timestamp
• Log type
• Severity or importance
• Source application or component
• User or session information
• Custom tags or metadata

​

Role of metadata in Log streams

• Stream-Level Metadata: Provides context about the Log Stream itself, such as its purpose, owner, or associated application.
• Log-Level Metadata: Enriches individual logs with additional context that can be used for filtering and analysis.

• Environment information (dev, staging, prod)
• Application or service name
• Version information
• Deployment identifiers
• Team or owner information
• Geographic or regional data

​

Creating and managing Log streams

​

How to create Log streams

​

Using the Galileo Console

1. Navigate to your project in the Galileo Console
2. Go to the Log Streams section
3. Click “Create Log Stream”
4. Provide a name and optional description
5. Configure any additional settings
6. Save the new Log Stream

​

Using the Galileo SDK

from galileo import GalileoLogger

# Initialize with a new Log stream (will be created if it doesn't exist)
logger = GalileoLogger(project="my-project", log_stream="new-log-stream")

​

Using environment variables

GALILEO_PROJECT=my-project
GALILEO_LOG_STREAM=my-log-stream

​

Required configurations

• Name: A unique identifier within the project
• Project: The Galileo project the Log Stream belongs to

• Description: Information about the purpose or content of the Log Stream
• Tags: Labels for categorization
• Metadata: Additional contextual information
• Retention Policy: How long logs should be kept in the Log Stream

​

Customization options

• Naming Conventions: Establish consistent naming patterns for different types of Log Streams
• Tagging: Add tags to categorize Log Streams and make them easier to filter
• Metadata: Include additional context such as environment, application, or team information
• Access Controls: Configure who can view or modify the Log Stream
• Retention Policies: Set how long logs should be kept in the Log Stream

​

Using Log streams in practice

​

Organizing logs for different environments

• Development Log Stream: Captures logs from local development environments, typically with verbose logging enabled
• Staging Log Stream: Contains logs from the staging or QA environment, used for pre-production testing
• Production Log Stream: Holds logs from the live production environment, often with more selective logging

• Apply different retention policies to each environment
• Set up environment-specific monitoring and alerts
• Focus on relevant logs when debugging issues
• Prevent development logs from cluttering production analysis

​

Best practices

​

Naming conventions

• Use consistent, descriptive names
• Include the environment in the name (e.g., app-name-prod, app-name-dev)
• Consider including version information for major releases

​

Organization strategies

• Create separate Log Streams for different applications or services
• Use distinct Log Streams for different environments
• Consider creating special-purpose Log Streams for experiments or debugging

​

Maintenance

• Regularly review and clean up unused Log Streams
• Document the purpose and ownership of each Log Stream
• Establish clear retention policies based on importance and compliance requirements

​

Isolating workflows and failure modes

• Feature-Specific Log Streams: Create dedicated Log Streams for new or experimental features
• Error Log Streams: Route error logs to a dedicated stream for easier monitoring
• Critical Path Log Streams: Isolate logs from critical business workflows for heightened monitoring

• Focus on specific areas when debugging
• Set up targeted alerts for critical components
• Analyze patterns in specific workflows without noise from unrelated logs

​

Integration with the Galileo ecosystem

​

Interaction with other components

1. Logs: Log Streams are the organizational containers for logs, providing structure and context.
2. Traces: Traces are typically kept within a single Log Stream, maintaining the context of related operations.
3. Projects: Log Streams exist within projects, which provide the top-level organization in Galileo.
4. Metrics: Metrics can be calculated and displayed per Log Stream, allowing for comparative analysis.
5. Alerts: Alerts can be configured to monitor specific Log Streams for anomalies or issues.

​

Supporting workflows

• Evaluation: Compare performance across different Log Streams to evaluate changes or experiments
• Debugging: Isolate issues by focusing on relevant Log Streams
• Experimentation: Use separate Log Streams for control and experimental groups
• Monitoring: Set up dashboards and alerts for specific Log Streams

​

Dependencies and connections

• Projects: All Log Streams belong to a specific project
• API Keys: Authentication is required to create and access Log Streams
• SDK: The Galileo SDK is used to interact with Log Streams
• Logs: Log Streams contain and organize logs

​

Examples and practical applications

​

Basic Log stream creation

import os
from galileo import GalileoLogger
from galileo.openai import openai

# Initialize with a specific Log stream

logger = GalileoLogger(project="my-project", log_stream="production-app")

# Any logs created will go to the specified Log stream

client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Tell me about AI"}]
)

​

Multi-Environment Log stream management

import os
from galileo import GalileoLogger
from galileo.openai import openai

# Determine environment from environment variable
env = os.getenv("APP_ENVIRONMENT", "development")

# Create a logger with an environment-specific Log stream
logger = GalileoLogger(
    project="my-ai-app",
    log_stream=f"my-ai-app-{env}"
)

# All logs will now go to the environment-specific stream

​

Custom tagging for Log streams

from galileo import GalileoLogger, galileo_context

# Initialize the base logger

logger = GalileoLogger(project="my-project", log_stream="production")

# Use context to add tags for a specific feature

with galileo_context(tags=["recommendation-engine", "v2.3"]): # All logs in this context will have the specified tags # making them easier to filter within the Log stream
result = recommendation_engine.get_recommendations(user_id)

​

Further resources

• Projects - Learn about the top-level organization of Galileo resources
• Traces - Learn more about how logs are organized into traces
• Spans - Understand the different types of spans in Galileo
• Metrics - Discover how to measure key aspects of your AI application
• Experiments - Learn how to compare different approaches