Skip to main content

​
Overview

Reviewing lets you read traces, record a pass/fail verdict on them, and tag them. Every verdict carries its reason, and you can change both. Use it to:
  • Build evaluation datasets from real production data
  • Train automated graders by providing examples with verdicts
  • Track quality trends over time
  • Identify patterns in failures

​
Viewing Traces

Navigate to a trace function and turn on the Verdicts toggle to see all captured traces with their verdicts. A trace with no verdict yet shows a No verdict pill. Click Start reviewing to work through them.

​
Trace List

The list shows:

​
Filtering

Filter traces by:
  • Status: Success or error
  • Date Range: Filter by time period
  • Grader Results: Pass or fail on specific graders
  • Verdicts: Filter by pass or fail, and by who wrote the verdict
  • Tags: Filter by assigned tags

​
Searching

Use the search bar to find traces by:
  • Input content
  • Output content
  • Error messages

​
Trace Details

Click on a trace to view details:

​
Input

The exact input passed to the function:

​
Output

The output returned:

​
Span Tree

When functions call other wrapped functions, you see a hierarchical view:
Click on individual spans to see their inputs, outputs, context, and timing.

​
Recording verdicts

Record a verdict on a trace to say whether it passed:
  1. Open a trace
  2. Review the input, output, and span tree
  3. Record a pass or fail verdict with its reason, or add tags
Verdicts feed into grader training: the more examples with verdicts you provide, the better automated graders perform. Recording a pass on a trace a grader failed also records a correction on that trace, which is the signal to tighten the grader’s criteria. See the Graders guide.

​
Tagging

Organize traces with tags for filtering and dataset building:

​
Adding Tags

  1. Open a trace
  2. Click Add Tag
  3. Select an existing tag or create a new one

​
Managing Tags

Navigate to Tags in the user menu to:
  • Create new tags
  • Edit tag names and colors
  • Archive unused tags

​
Building Evaluation Datasets

Use traces with verdicts to build evaluation datasets:
  1. Filter to traces you want to include
  2. Record pass/fail verdicts on them
  3. Apply tags to organize by feature, environment, or use case
  4. Use these traces and their verdicts to train and validate graders

​
Best Practices

  • Review regularly: Consistent verdicts improve grader accuracy
  • Tag strategically: Use tags to organize by feature, environment, or failure mode
  • Record both passes and failures: Graders need examples of both to learn effectively
  • Start with the most impactful function: Focus review effort on the trace function with the highest volume or most critical failures