Overview
An experiment is a replay of a dataset against your current code, scored against what the original traces did. It answers the only question that matters after a change: did this fix what I meant to fix, and did it break anything else? Each experiment replays historical inputs through your code now, pairs every replay with the original trace it came from, and classifies the pair.Running an Experiment
Ask your coding agent:replay(), which returns a testRunId and a testRunUrl for the resulting experiment. See the TypeScript, Python, and Ruby references for the call signature, and Replay Mocking for controlling which child spans re-run for real.
A trace replays only when its root span has serializable inputs, or it was captured through a framework handler. Traces whose inputs were stubbed as non-serializable at capture time cannot be replayed.
Status
Verdicts
Each replayed trace is classified by comparing the original trace’s label with the replay’s:
Fixed and regressed are the two numbers to read first. A run with 8 fixed and 3 regressed is not a win.
Totals and Pass Rate
Alongside verdicts, an experiment tracks how each replay ended: succeeded, failed, errored, still pending, awaiting labels (the replay finished but no verdict has been written yet), and skipped. The pass rate counts only resolved traces, so awaiting-labels and skipped traces are excluded from the denominator rather than counted as failures.Graders on an Experiment
When an experiment completes, its grader set is the union of the graders attached directly to the experiment and the runnable graders on its dataset at that moment. That set is then frozen onto the experiment, so its results stay reproducible even if the dataset’s attachments change later. Attaching a grader to an already completed experiment records the assignment but does not regrade the traces it already produced. Use Re-run on the experiment, or launch a fresh replay. See Graders.Experiment Groups
Every experiment launched in a single iteration shares an experiment group, so the runs you kicked off together stay together in the list instead of interleaving with older work. Groups are ordered by their most recent run. Runs launched ad hoc, without a group, collect in a single Ungrouped bucket, which is ordered by recency like any other group. Your agent can also group existing experiments after the fact, as long as they are not already split across different groups.Comparing Runs
An experiment always shows you the before and after: the original trace’s input and output next to the replay’s. Per-trace results also carry token usage for both sides, so you can see the cost and cache-read delta of a change rather than just its pass rate. That makes an experiment as useful for a cost optimization pass as it is for a correctness one.Best Practices
- Change one thing per experiment. Two changes in one run give you a pass rate you cannot attribute.
- Read regressions before fixes. The traces that used to pass are the ones your users already rely on.
- Mock what you are not changing. Keep the code under test real and mock the expensive setup around it. See Replay Mocking.
- Attach graders to the dataset, not the run. Dataset graders flow into every experiment automatically, so each run is scored the same way.