Labeling

​

Overview

• Build evaluation datasets from real production data
• Train automated graders by providing labeled examples
• Track quality trends over time
• Identify patterns in failures

​

Viewing Traces

​

Trace List

​

Filtering

• Status: Success or error
• Date Range: Filter by time period
• Grader Results: Pass or fail on specific graders
• Labels: Filter by human-applied labels
• Tags: Filter by assigned tags

​

Searching

• Input content
• Output content
• Error messages

​

Trace Details

​

Input

{
  "order_id": "order-123",
  "items": ["item1", "item2"]
}

​

Output

{
  "order_id": "order-123",
  "total": 20
}

​

Span Tree

Trace: agent-workflow
├── Span: validate-input (guardrail)
├── Span: fetch-data (function)
└── Span: generate-response (llm)

​

Applying Labels

1. Open a trace
2. Review the input, output, and span tree
3. Apply a pass/fail label or add tags

​

Tagging

​

Adding Tags

1. Open a trace
2. Click Add Tag
3. Select an existing tag or create a new one

​

Managing Tags

• Create new tags
• Edit tag names and colors
• Archive unused tags

​

Building Evaluation Datasets

1. Filter to traces you want to include
2. Label traces with pass/fail outcomes
3. Apply tags to organize by feature, environment, or use case
4. Use these labeled traces to train and validate graders

​

Best Practices

• Label regularly: Consistent labeling improves grader accuracy
• Tag strategically: Use tags to organize by feature, environment, or failure mode
• Label both passes and failures: Graders need examples of both to learn effectively
• Start with the most impactful function: Focus labeling effort on the trace function with the highest volume or most critical failures