Datasets

Overview

A dataset is a named bucket of traces for one trace function: the traces your team reviews, that graders score, and that experiments replay against. It is the unit of evaluation in Bitfab. Without a dataset, a grader has nothing to run on and an experiment has nothing to replay. Datasets usually start as a failure mode you want to fix, for example “Hallucinated order numbers” or “Long multi-turn threads”, and grow as you find more traces that belong in them.

Creating a Dataset

Ask your coding agent:

Build a dataset of failing checkout-agent traces from last week

The agent calls save_dataset with the trace function key, a short name, and an optional description, then populates it with add_traces_to_dataset. The portal’s New dataset button shows this prompt rather than opening a form.

Membership

Adding traces is idempotent. Re-adding a trace already in the dataset does nothing.
Traces are validated against the dataset’s trace function key. Ids that belong to another organization or another trace function are skipped, and the response reports how many were added versus skipped.
Up to 100 traces per call.
Removing a trace from a dataset does not delete the trace.

Your agent can also add the trace you are currently working on, which is the normal way a dataset grows during a debugging session: fix the trace, confirm the fix by replaying it, then add it to the dataset so a future change cannot silently break it again.

Filling a dataset before production has traces

A dataset holds traces, so a function that has not run in production yet has nothing to collect. If you already hold the cases somewhere else (a spreadsheet, a fixtures file, an export from another tool), seed them into Bitfab as traces and add those trace IDs to the dataset like any other:

const traceId = await bitfab.seedTrace("checkout-agent", runCheckout, {
  args: [caseRow.input],
  metadata: { caseUid: caseRow.id },
  name: caseRow.id,
})
await bitfab.datasets.addTraces(dataset.id, [traceId])

trace_id = bitfab.seed_trace(
    "checkout-agent",
    run_checkout,
    args=[case_row["input"]],
    metadata={"case_uid": case_row["id"]},
    name=case_row["id"],
)
bitfab.datasets.add_traces(dataset["id"], [trace_id])

Seeded traces behave like captured ones from here on. Experiments replay them, graders score them, and the dataset’s pass rate counts them. Seeding is available in the TypeScript and Python SDKs. See Seeding traces for TypeScript and Seeding traces for Python.

Reading a Dataset

The datasets list for a trace function shows each dataset’s name, trace count, and one verdict signal:

Signal	When it shows
Grader pass-rate pills	The dataset has graders attached. One pill per grader, filled green to red at its pass rate
Human pass/fail	No graders attached, but reviewers have labeled traces by hand
Nothing	The dataset is unlabeled and ungraded

A dataset shows exactly one of these. Grader results and human labels are never mixed into a single verdict.

A grader that is attached but has not run yet reads as 0/0 rather than a wall of failures, so an empty pill means “not scored”, not “everything failed”.

Open a dataset to see its traces, the graders attached to it, and the experiments that have replayed it.

Attaching Graders

The dataset detail view has a Graders section. Use Manage to attach or detach graders, and Re-run to grade the dataset’s traces with the graders you select. Only one re-run runs at a time per dataset. Graders must belong to the same organization and trace function as the dataset. Detaching a grader from a dataset never deletes the grader. See Graders for what the graders themselves do.

Running Experiments Against a Dataset

An experiment replays a dataset’s traces against your current code and scores the results. The experiment inherits the dataset’s runnable graders at completion, so attaching a grader to the dataset is usually all the setup an experiment needs.

From the SDK

Every operation above is also available programmatically through client.datasets in the TypeScript, Python, Ruby, and Go SDKs, for scripts and CI jobs that build or maintain datasets without an agent in the loop.

const { dataset } = await bitfab.datasets.save({
  traceFunctionKey: "checkout-agent",
  name: "Refund failures",
})
await bitfab.datasets.addTraces(dataset.id, traceIds)
await bitfab.datasets.addGraders(dataset.id, [graderId])
const { run } = await bitfab.datasets.rerunGraders(dataset.id)

dataset = bitfab.datasets.save("checkout-agent", "Refund failures")["dataset"]
bitfab.datasets.add_traces(dataset["id"], trace_ids)
bitfab.datasets.add_graders(dataset["id"], [grader_id])
run = bitfab.datasets.rerun_graders(dataset["id"])["run"]

dataset = client.datasets.save(trace_function_key: "checkout-agent", name: "Refund failures")["dataset"]
client.datasets.add_traces(dataset["id"], trace_ids)
client.datasets.add_graders(dataset["id"], [grader_id])
run = client.datasets.rerun_graders(dataset["id"])["run"]

saved, _ := client.Datasets.Save(ctx, bitfab.SaveDatasetParams{TraceFunctionKey: "checkout-agent", Name: "Refund failures"})
client.Datasets.AddTraces(ctx, saved.Dataset.ID, traceIDs)
client.Datasets.AddGraders(ctx, saved.Dataset.ID, []string{graderID})
rerun, _ := client.Datasets.RerunGraders(ctx, saved.Dataset.ID, bitfab.RerunGradersOptions{})

Dataset listing and trace-ID listing automatically fetch every page in all four SDKs and return the complete result. You can keep using list / List and listTraces / list_traces / ListTraces without managing cursors. The methods are save, list, get, list traces, add and remove traces, add and remove graders, re-run graders, and read a re-run’s status. They follow the membership rules on this page: adds are idempotent, foreign or wrong-function ids are reported as skipped rather than rejected, and removing a trace never deletes it. See the TypeScript reference, the Python reference, the Ruby reference, the Go reference, and the HTTP API.

Best Practices

Include both directions. A dataset of only failures cannot tell you when a change makes things worse. Add the passing traces you refuse to regress.
Keep a dataset about one thing. “Hallucinated order numbers” tells you what broke. “Bad outputs” does not.
Grow it from real failures. Every production failure worth fixing is worth adding, so the fix is locked in.
Size it for the loop you want. Small enough to replay quickly while iterating, large enough that the pass rate means something.

Getting Started

Primitives

Account

SDKs

Framework Integrations

Integrations

Web Portal

Reference

Changelog

Overview

Creating a Dataset

Membership

Filling a dataset before production has traces

Reading a Dataset

Attaching Graders

Running Experiments Against a Dataset

From the SDK

Best Practices

​Overview

​Creating a Dataset

​Membership

​Filling a dataset before production has traces

​Reading a Dataset

​Attaching Graders

​Running Experiments Against a Dataset

​From the SDK

​Best Practices

Overview

Creating a Dataset

Membership

Filling a dataset before production has traces

Reading a Dataset

Attaching Graders

Running Experiments Against a Dataset

From the SDK

Best Practices