> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bitfab.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Graders

> Define automated pass/fail judges for a traced function and run them against datasets and experiments

## Overview

A **grader** is an automated judge that scores a trace as pass or fail against one specific property of a traced function, for example "the reply never invents an order number". Graders turn a quality bar you would otherwise check by hand into a check that runs across every trace in a dataset and every replay in an experiment.

Each grader belongs to one organization and one trace function. A trace function usually has several graders, one per property worth protecting.

<Note>
  Graders only run where they are attached. A grader with no dataset or experiment attached is a definition that never executes.
</Note>

Graders are LLM-as-judge: a judge model reads the trace and returns pass or fail with a reason. You define what it should look for in plain language, not in code.

## Viewing Graders

Open a trace function and select **Graders**. The list shows one row per grader with:

| Column          | Description                                           |
| --------------- | ----------------------------------------------------- |
| **Grader**      | Name, status badge, and the grader's evaluation focus |
| **Datasets**    | How many datasets the grader is attached to           |
| **Experiments** | How many experiments have run it                      |

Sort by status, most evaluated, recently updated, or name, and switch between the **Active** and **Archived** views. Click a row to open the grader.

### Grader Detail

The detail page shows the grader's criteria, its judge model, and every dataset and experiment it is attached to. From here you can change the judge model, archive the grader, or restore it.

### Status

| Status       | Meaning                                                          |
| ------------ | ---------------------------------------------------------------- |
| **Active**   | Usable. Runs when you grade an attached dataset or experiment    |
| **Archived** | Retired. Hidden from lists and runs nowhere. Restore at any time |

## Creating and Editing Graders

Graders are authored from your coding agent through the Bitfab plugin, not from a form in the web portal. Ask your agent in plain language:

```
Add a grader to checkout-agent: the reply must never invent an order number
```

The agent calls the `save_grader` tool, which upserts. Pass a grader id to edit a specific grader, or reuse the name of an existing grader on the same trace function to update that one instead of creating a duplicate. The same tool renames graders, clears pass and fail criteria, selects the judge model, and archives or restores. `list_graders` enumerates what is already defined for a function.

See the [Claude Code](/claude-plugin), [Cursor](/cursor-plugin), or [Codex](/codex-plugin) plugin reference for the full tool signatures.

## Writing Good Criteria

A grader is defined by a small set of fields:

| Field                     | Purpose                                                                                 |
| ------------------------- | --------------------------------------------------------------------------------------- |
| **Evaluation focus**      | The property a trace must satisfy, written as a concrete, checkable statement. Required |
| **Pass criteria**         | What specifically counts as a pass                                                      |
| **Fail criteria**         | What specifically counts as a fail                                                      |
| **Evaluation steps**      | The order in which the judge should check things                                        |
| **Acceptable variations** | Differences that must not be penalized, such as wording or formatting                   |
| **Span selection**        | Which part of the trace the judge should look at                                        |

<Tip>
  One property per grader. A grader that checks tone, factuality, and formatting at once produces a fail you cannot act on. Three graders tell you which one broke.
</Tip>

Keep the focus checkable from the trace alone, and spell out acceptable variations so the judge does not fail a correct answer for phrasing it differently.

## Judge Model

Every grader runs on a selectable judge model:

| Model                | Notes                                                 |
| -------------------- | ----------------------------------------------------- |
| **Gemini 2.5 Flash** | Default. Fast and cost-effective                      |
| **GPT-5.6 Sol**      | OpenAI flagship. Strongest reasoning for hard grading |
| **GPT-5.6 Terra**    | OpenAI balanced. Faster and cheaper than Sol          |
| **Claude Opus 4.8**  | Anthropic flagship reasoning model                    |
| **Gemini 3.1 Pro**   | Google advanced reasoning model                       |

Change it from the dropdown on the grader detail page, or have your agent pass a model to `save_grader`. Start on the default and move up only for criteria the default judge gets wrong.

## Running Graders

### On a Dataset

Open a [dataset](/primitives/datasets) and use **Manage** in its Graders section to attach or detach graders. Your agent can do the same with `add_graders_to_dataset` and `remove_graders_from_dataset`. Only graders belonging to the same organization and trace function as the dataset can be attached, and detaching an assignment never deletes the grader itself.

Once graders are attached, **Re-run** grades the dataset's traces with the graders you select. Only one re-run can be in flight per dataset at a time.

### On an Experiment

When an [experiment](/primitives/experiments) finishes replaying a dataset, its grader set is the union of the graders attached directly to that experiment and the dataset's runnable graders at that moment. That set is then frozen onto the experiment, so its results stay reproducible even if the dataset's attachments change afterwards.

Attaching a grader to an experiment that has already completed records the assignment but does not regrade the traces it already produced. Use **Re-run** on the experiment, or run a fresh replay.

### On New Traces

Graders do not currently run automatically as new production traces arrive. Add traces to a dataset, attach the graders you want, and use **Re-run** to grade them.

## Reading Results

* **Trace lists** show grader outcomes in the Grader Results column, and you can filter traces by pass or fail on a specific grader.
* **Dataset rows** show a pass-rate pill once graders are attached, splitting green to red at the trace's pass rate across those graders.
* **Trace detail** shows each grader's verdict with the reason it gave. A failing grader also produces a failure diagnostic explaining what went wrong.
* **Corrections** appear when a reviewer labels a trace as a pass that a grader failed. The trace shows who corrected it, which is the clearest signal that the grader's criteria need tightening.

Human labeling and automated grading are complementary: see the [Labeling guide](/web-portal/labeling) for how reviewers mark traces by hand.

## Archiving

Archive a grader from its row in the list or from its detail page. Archived graders are hidden from the default view, stop running everywhere, and keep their history. Switch the list to **Archived** to restore one.

## Best Practices

* **One property per grader**: a focused grader produces an actionable failure.
* **Attach before expecting results**: a grader runs only on the datasets and experiments it is attached to.
* **Re-run after editing criteria**: existing evaluations reflect the criteria in place when they ran.
* **Grade both directions**: a dataset of only failures cannot tell you whether a grader is too strict.
* **Escalate the judge model deliberately**: reach for a stronger model when the default judge is measurably wrong, not by default.