Skip to main content

​
Overview

A dataset is a named bucket of traces for one trace function: the traces your team reviews, that graders score, and that experiments replay against. It is the unit of evaluation in Bitfab. Without a dataset, a grader has nothing to run on and an experiment has nothing to replay. Datasets usually start as a failure mode you want to fix, for example “Hallucinated order numbers” or “Long multi-turn threads”, and grow as you find more traces that belong in them.

​
Creating a Dataset

Ask your coding agent:
The agent calls create_dataset with the trace function key, a short name, and an optional description, then populates it with add_traces_to_dataset. The portal’s New dataset button shows this prompt rather than opening a form.

​
Membership

  • Adding traces is idempotent. Re-adding a trace already in the dataset does nothing.
  • Traces are validated against the dataset’s trace function key. Ids that belong to another organization or another trace function are skipped, and the response reports how many were added versus skipped.
  • Up to 100 traces per call.
  • Removing a trace from a dataset does not delete the trace.
Your agent can also add the trace you are currently working on, which is the normal way a dataset grows during a debugging session: fix the trace, confirm the fix by replaying it, then add it to the dataset so a future change cannot silently break it again.

​
Reading a Dataset

The datasets list for a trace function shows each dataset’s name, trace count, and one verdict signal: A dataset shows exactly one of these. Grader results and human labels are never mixed into a single verdict.
A grader that is attached but has not run yet reads as 0/0 rather than a wall of failures, so an empty pill means “not scored”, not “everything failed”.
Open a dataset to see its traces, the graders attached to it, and the experiments that have replayed it.

​
Attaching Graders

The dataset detail view has a Graders section. Use Manage to attach or detach graders, and Re-run to grade the dataset’s traces with the graders you select. Only one re-run runs at a time per dataset. Graders must belong to the same organization and trace function as the dataset. Detaching a grader from a dataset never deletes the grader. See Graders for what the graders themselves do.

​
Running Experiments Against a Dataset

An experiment replays a dataset’s traces against your current code and scores the results. The experiment inherits the dataset’s runnable graders at completion, so attaching a grader to the dataset is usually all the setup an experiment needs.

​
Best Practices

  • Include both directions. A dataset of only failures cannot tell you when a change makes things worse. Add the passing traces you refuse to regress.
  • Keep a dataset about one thing. “Hallucinated order numbers” tells you what broke. “Bad outputs” does not.
  • Grow it from real failures. Every production failure worth fixing is worth adding, so the fix is locked in.
  • Size it for the loop you want. Small enough to replay quickly while iterating, large enough that the pass rate means something.