Experiments

Overview

An experiment is a replay of a dataset against your current code, scored against what the original traces did. It answers the only question that matters after a change: did this fix what I meant to fix, and did it break anything else? Each experiment replays historical inputs through your code now, pairs every replay with the original trace it came from, and classifies the pair.

Running an Experiment

Ask your coding agent:

Run an experiment on checkout-agent to improve pass rate

You can also drive replays directly from the SDK with replay(), which returns the experiment ID and a link to the resulting experiment (experimentId and experimentUrl in TypeScript). See the TypeScript, Python, Ruby, and Go references for the call signature, and Replay Mocking for controlling which child spans re-run for real.

A trace replays only when its root span has serializable inputs, or it was captured through a framework handler. Traces whose inputs were stubbed as non-serializable at capture time cannot be replayed.

A dataset does not have to be built from production traffic. Seeded traces, written from cases you already hold, replay in an experiment exactly as captured ones do and are graded the same way. They carry no database snapshot pin, so an experiment that needs database branching has to run against captured traces.

Status

Status	Meaning
Pending	Replays are still running
Completed	Every replay settled. Graders run at completion
Failed	The run itself failed

Verdicts

Each replayed trace is classified by comparing the original trace’s verdict with the replay’s:

Verdict	Original	Replay	Read it as
Fixed	Fail	Pass	The change did what you wanted
Regressed	Pass	Fail	The change broke something that worked
Still passing	Pass	Pass	Protected, no change
Still failing	Fail	Fail	Not fixed yet
Original has no verdict	Not scored	Any	Nothing to compare against, but you can still A/B the outputs
Replay has no verdict	Pass or Fail	Not scored yet	The replay is still waiting for a verdict
Capture incomplete		Did not finish capturing	The replay has no result to compare
Unpaired	No original linked	Any	No comparison available
Skipped			The judge deliberately declined to score it

Fixed and regressed are the two numbers to read first. A run with 8 fixed and 3 regressed is not a win. An assertion compares only with its own verdict on the original trace. Starting an experiment judges assertions on the replays, never on the original traces, so an assertion nobody judged or labeled on the original reads pass or fail with nothing to compare against. Graders compare with their own verdict on the original, and a trace as a whole compares with the original trace’s overall verdict.

Totals and Pass Rate

Alongside verdicts, an experiment tracks how each replay ended: succeeded, failed, errored, still pending, awaiting verdicts (the replay finished but no verdict has been written yet), and skipped. The pass rate counts only resolved traces, so traces awaiting verdicts and skipped traces are excluded from the denominator rather than counted as failures.

Graders on an Experiment

When an experiment completes, its grader set is the union of the graders attached directly to the experiment and the runnable graders on its dataset at that moment. That set is then frozen onto the experiment, so its results stay reproducible even if the dataset’s attachments change later. Attaching a grader to an already completed experiment records the assignment but does not regrade the traces it already produced. Use Re-run on the experiment, or launch a fresh replay. See Graders.

Experiment Groups

Every experiment launched in a single iteration shares an experiment group, so the runs you kicked off together stay together in the list instead of interleaving with older work. Groups are ordered by their most recent run. Runs launched ad hoc, without a group, collect in a single Ungrouped bucket, which is ordered by recency like any other group. Your agent can also group existing experiments after the fact, as long as they are not already split across different groups.

Who Ran It, and From Where

Every experiment records who launched it and exactly which code ran, so a pass rate three weeks old still says whose change it measured. Six fields are stored on the run:

Field	What it is
Bitfab user id	The owner of the API key the replay authenticated with, resolved on the server rather than reported by the SDK, so it cannot be forged. A team sharing one key in CI attributes every run to that key’s owner.
GitHub email	`user.email` from the launching machine’s git config.
Branch	The checked-out branch, or empty on a detached head.
Commit sha	`HEAD` on the launching machine.
Base sha	The tree at `HEAD`: the committed state.
Experiment sha	The tree that actually ran, uncommitted and untracked files included.

The last two are the useful pair. HEAD alone lies whenever the working tree is dirty, so the experiment sha is written from a throwaway git index and covers what was really on disk. That gives you two properties:

git diff <base sha> <experiment sha> is the change the experiment tested, computed by git with no tooling from us.
Two runs with the same experiment sha ran byte-identical code. Two runs where base and experiment match ran a clean tree.

Writing that tree never touches your index or your stash: it uses a temporary GIT_INDEX_FILE, so a replay is safe to run mid-edit. Everything is read locally from git, with no network call. BITFAB_DISABLE_GIT_STATE opts a process out, and older SDKs simply report nothing. Both appear on the experiment row and in its code change view.

Reading Results From the SDK

Every SDK has an experiments namespace for reading runs back after the fact, so a dashboard or a report outside Bitfab can show pass rates without scraping the web app. It is a read surface: experiments are still created by running a replay.

list pages the organization’s experiments newest first. Every filter is optional and they combine: dataset, traced function, git branch, experiment group, status, metadata, and a creation window. With no filter and a limit of one it returns the latest run.
get reads one experiment with everything above plus who ran it, its git state, completedAt, and its metadata.
getRollup reads one run’s pass/fail rollup, and getRollupAll reads up to 100 runs in one request. The rollup tallies traces, scored verdicts, raw checks, and one row per assertion category, each split into passing, failing, fixed, regressed, still passing, still failing, no verdict, and unpaired, plus how many of the failing ones the judge abstained on in every attempt because it could not find the evidence. A trace or an assertion passes when at least 75 percent of its attempts passed, the same threshold the dashboard applies.

A run tagged with metadata at replay start carries those tags on every read, and list filters on them by containment, so { schedule: "eod" } finds every run that pair was set on. Use metadata to tell a scheduled run apart from an ad hoc one on the same branch. It never stands in for the recorded git state, which the SDK writes on its own.

const { experiments } = await bitfab.experiments.list({
  datasetId: PROD_RELEASE_DATASET_ID,
  gitBranch: "dev",
  metadata: { schedule: "eod" },
  status: "completed",
  createdAfter: new Date(Date.now() - 30 * 24 * 60 * 60 * 1000),
})
const rollups = await bitfab.experiments.getRollupAll(
  experiments.map((experiment) => experiment.id),
)
for (const { experimentId, totals, rollup } of rollups) {
  const passRate = rollup.traces.passing / (rollup.traces.passing + rollup.traces.failing)
  console.log(experimentId, totals.total, passRate)
  for (const { category, verdicts } of rollup.categories) {
    console.log(`  ${category.title}: ${verdicts.passing}/${verdicts.passing + verdicts.failing}`)
  }
}

The batch runs a fixed set of SQL queries however many ids it carries, so a month of daily runs is one call. Categories with no assertions in a run are absent from its rollup; compare against categories.list() to show them as gaps.

Running a Dataset on a Schedule

A nightly run of a dataset against a branch is a scheduled GitHub Actions workflow that checks out the branch by name and runs the replay command with the dataset and a metadata tag. Checking out by branch name matters: a checkout by sha leaves a detached head, and the run records no branch.

name: Nightly eval
on:
  schedule:
    - cron: "0 0 * * 2-6" # 5pm Pacific during daylight time; adjust when clocks change
  workflow_dispatch:
jobs:
  replay:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: dev
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: npx bitfab-replay --registry scripts/replayRegistry.ts <pipeline> --dataset-id ${{ vars.PROD_RELEASE_DATASET_ID }} --metadata schedule=eod --db-branch
        env:
          BITFAB_API_KEY: ${{ secrets.BITFAB_API_KEY }}

A rollup only counts verdicts, so the scheduled run must produce them: assertion judging, on by default, scores every approved assertion on each replay as it finishes (so leave off --skip-assertion-judging), and graders attached to the dataset score their checks at completion. A run with neither reports empty tallies. bitfab:setup cloud installs a dispatch-only workflow for replaying from your editor; a scheduled run is a separate workflow so the two never compete for the same secrets or the same run.

Comparing Runs

An experiment always shows you the before and after: the original trace’s input and output next to the replay’s. Per-trace results also carry token usage for both sides, so you can see the cost and cache-read delta of a change rather than just its pass rate. That makes an experiment as useful for a cost optimization pass as it is for a correctness one.

Best Practices

Change one thing per experiment. Two changes in one run give you a pass rate you cannot attribute.
Read regressions before fixes. The traces that used to pass are the ones your users already rely on.
Mock what you are not changing. Keep the code under test real and mock the expensive setup around it. See Replay Mocking.
Attach graders to the dataset, not the run. Dataset graders flow into every experiment automatically, so each run is scored the same way.

Getting Started

Primitives

Account

SDKs

Framework Integrations

Integrations

Web Portal

Reference

Changelog

Overview

Running an Experiment

Status

Verdicts

Totals and Pass Rate

Graders on an Experiment

Experiment Groups

Who Ran It, and From Where

Reading Results From the SDK

Running a Dataset on a Schedule

Comparing Runs

Best Practices

​Overview

​Running an Experiment

​Status

​Verdicts

​Totals and Pass Rate

​Graders on an Experiment

​Experiment Groups

​Who Ran It, and From Where

​Reading Results From the SDK

​Running a Dataset on a Schedule

​Comparing Runs

​Best Practices

Overview

Running an Experiment

Status

Verdicts

Totals and Pass Rate

Graders on an Experiment

Experiment Groups

Who Ran It, and From Where

Reading Results From the SDK

Running a Dataset on a Schedule

Comparing Runs

Best Practices