Skip to main content

​
Overview

A grader is an automated judge that scores a trace as pass or fail against one specific property of a traced function, for example “the reply never invents an order number”. Graders turn a quality bar you would otherwise check by hand into a check that runs across every trace in a dataset and every replay in an experiment. Each grader belongs to one organization and one trace function. A trace function usually has several graders, one per property worth protecting.
Graders only run where they are attached. A grader with no dataset or experiment attached is a definition that never executes.
Graders are LLM-as-judge: a judge model reads the trace and returns pass or fail with a reason. You define what it should look for in plain language, not in code.

​
Viewing Graders

Open a trace function and select Graders. The list shows one row per grader with: Sort by status, most evaluated, recently updated, or name, and switch between the Active and Archived views. Click a row to open the grader.

​
Grader Detail

The detail page shows the grader’s criteria, its judge model, and every dataset and experiment it is attached to. From here you can change the judge model, archive the grader, or restore it.

​
Status

​
Creating and Editing Graders

Graders are authored from your coding agent through the Bitfab plugin, not from a form in the web portal. Ask your agent in plain language:
The agent calls the save_grader tool, which upserts. Pass a grader id to edit a specific grader, or reuse the name of an existing grader on the same trace function to update that one instead of creating a duplicate. The same tool renames graders, clears pass and fail criteria, selects the judge model, and archives or restores. list_graders enumerates what is already defined for a function. See the Claude Code, Cursor, or Codex plugin reference for the full tool signatures.

​
Writing Good Criteria

A grader is defined by a small set of fields:
One property per grader. A grader that checks tone, factuality, and formatting at once produces a fail you cannot act on. Three graders tell you which one broke.
Keep the focus checkable from the trace alone, and spell out acceptable variations so the judge does not fail a correct answer for phrasing it differently.

​
Judge Model

Every grader runs on a selectable judge model: Change it from the dropdown on the grader detail page, or have your agent pass a model to save_grader. Start on the default and move up only for criteria the default judge gets wrong.

​
Running Graders

​
On a Dataset

Open a dataset and use Manage in its Graders section to attach or detach graders. Your agent can do the same with add_graders_to_dataset and remove_graders_from_dataset. Only graders belonging to the same organization and trace function as the dataset can be attached, and detaching an assignment never deletes the grader itself. Once graders are attached, Re-run grades the dataset’s traces with the graders you select. Only one re-run can be in flight per dataset at a time.

​
On an Experiment

When an experiment finishes replaying a dataset, its grader set is the union of the graders attached directly to that experiment and the dataset’s runnable graders at that moment. That set is then frozen onto the experiment, so its results stay reproducible even if the dataset’s attachments change afterwards. Attaching a grader to an experiment that has already completed records the assignment but does not regrade the traces it already produced. Use Re-run on the experiment, or run a fresh replay.

​
On New Traces

Graders do not currently run automatically as new production traces arrive. Add traces to a dataset, attach the graders you want, and use Re-run to grade them.

​
Reading Results

  • Trace lists show grader outcomes in the Grader Results column, and you can filter traces by pass or fail on a specific grader.
  • Dataset rows show a pass-rate pill once graders are attached, splitting green to red at the trace’s pass rate across those graders.
  • Trace detail shows each grader’s verdict with the reason it gave. A failing grader also produces a failure diagnostic explaining what went wrong.
  • Corrections appear when a reviewer labels a trace as a pass that a grader failed. The trace shows who corrected it, which is the clearest signal that the grader’s criteria need tightening.
Human labeling and automated grading are complementary: see the Labeling guide for how reviewers mark traces by hand.

​
Archiving

Archive a grader from its row in the list or from its detail page. Archived graders are hidden from the default view, stop running everywhere, and keep their history. Switch the list to Archived to restore one.

​
Best Practices

  • One property per grader: a focused grader produces an actionable failure.
  • Attach before expecting results: a grader runs only on the datasets and experiments it is attached to.
  • Re-run after editing criteria: existing evaluations reflect the criteria in place when they ran.
  • Grade both directions: a dataset of only failures cannot tell you whether a grader is too strict.
  • Escalate the judge model deliberately: reach for a stronger model when the default judge is measurably wrong, not by default.