> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bitfab.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Experiments

> Replay a dataset against your changed code and score every trace as fixed, regressed, or unchanged

## Overview

An **experiment** is a replay of a dataset against your current code, scored against what the original traces did. It answers the only question that matters after a change: did this fix what I meant to fix, and did it break anything else?

Each experiment replays historical inputs through your code now, pairs every replay with the original trace it came from, and classifies the pair.

## Running an Experiment

Ask your coding agent:

```
Run an experiment on checkout-agent to improve pass rate
```

You can also drive replays directly from the SDK with `replay()`, which returns a `testRunId` and a `testRunUrl` for the resulting experiment. See the [TypeScript](/reference/typescript), [Python](/reference/python), and [Ruby](/reference/ruby) references for the call signature, and [Replay Mocking](/replay-mocking) for controlling which child spans re-run for real.

<Note>
  A trace replays only when its root span has serializable inputs, or it was captured through a framework handler. Traces whose inputs were stubbed as non-serializable at capture time cannot be replayed.
</Note>

## Status

| Status        | Meaning                                         |
| ------------- | ----------------------------------------------- |
| **Pending**   | Replays are still running                       |
| **Completed** | Every replay settled. Graders run at completion |
| **Failed**    | The run itself failed                           |

## Verdicts

Each replayed trace is classified by comparing the original trace's label with the replay's:

| Verdict                | Original                         | Replay | Read it as                                                    |
| ---------------------- | -------------------------------- | ------ | ------------------------------------------------------------- |
| **Fixed**              | Fail                             | Pass   | The change did what you wanted                                |
| **Regressed**          | Pass                             | Fail   | The change broke something that worked                        |
| **Still passing**      | Pass                             | Pass   | Protected, no change                                          |
| **Still failing**      | Fail                             | Fail   | Not fixed yet                                                 |
| **Original unlabeled** | Not scored                       | Any    | Nothing to compare against, but you can still A/B the outputs |
| **Unpaired**           | No partner, or replay unresolved |        | No comparison available                                       |
| **Skipped**            |                                  |        | The labeler deliberately declined to score it                 |

Fixed and regressed are the two numbers to read first. A run with 8 fixed and 3 regressed is not a win.

### Totals and Pass Rate

Alongside verdicts, an experiment tracks how each replay ended: succeeded, failed, errored, still pending, awaiting labels (the replay finished but no verdict has been written yet), and skipped. The pass rate counts only resolved traces, so awaiting-labels and skipped traces are excluded from the denominator rather than counted as failures.

## Graders on an Experiment

When an experiment completes, its grader set is the union of the graders attached directly to the experiment and the runnable graders on its dataset at that moment. That set is then frozen onto the experiment, so its results stay reproducible even if the dataset's attachments change later.

Attaching a grader to an already completed experiment records the assignment but does not regrade the traces it already produced. Use **Re-run** on the experiment, or launch a fresh replay. See [Graders](/primitives/graders).

## Experiment Groups

Every experiment launched in a single iteration shares an **experiment group**, so the runs you kicked off together stay together in the list instead of interleaving with older work. Groups are ordered by their most recent run.

Runs launched ad hoc, without a group, collect in a single **Ungrouped** bucket, which is ordered by recency like any other group.

Your agent can also group existing experiments after the fact, as long as they are not already split across different groups.

## Comparing Runs

An experiment always shows you the before and after: the original trace's input and output next to the replay's. Per-trace results also carry token usage for both sides, so you can see the cost and cache-read delta of a change rather than just its pass rate. That makes an experiment as useful for a cost optimization pass as it is for a correctness one.

## Best Practices

* **Change one thing per experiment.** Two changes in one run give you a pass rate you cannot attribute.
* **Read regressions before fixes.** The traces that used to pass are the ones your users already rely on.
* **Mock what you are not changing.** Keep the code under test real and mock the expensive setup around it. See [Replay Mocking](/replay-mocking).
* **Attach graders to the dataset, not the run.** Dataset graders flow into every experiment automatically, so each run is scored the same way.