> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bitfab.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Assertions

> Say what should happen when one specific trace is replayed, approve it, and have every replay of that trace checked against it

## Overview

An **assertion** says what should happen when one specific trace is replayed. It is written against the original trace and checked against every replay of it, so it states the right answer for that one input.

```
The reply offers the 6:10am flight to Denver, not the 9:45am one
The agent asks the user to confirm before charging the card
No email is sent to ana@example.com
```

An assertion turns a trace you fixed into a test. Once a person approves it, every later replay of that trace gets a verdict on it: pass, fail, or abstained. An experiment counts those verdicts alongside its graders, so a fix you made last month cannot quietly break again.

> [!NOTE]
>
> Assertions are in alpha. The **Assertions** page in the sidebar carries an alpha badge, and the details on this page may change as the feature settles.

### Assertions and graders

Assertions and [graders](https://docs.bitfab.ai/primitives/graders.md) are both pass/fail checks. They differ in what they cover.

| | Assertion | Grader |
| - | - | - |
| **Covers** | One trace and every replay of it | Every trace of a traced function |
| **States** | The right answer for that one input, with its real names, times, and amounts | A property every run should have, such as "never invents an order number" |
| **Goes into force** | When a person approves it | When it is attached to a dataset or an experiment |
| **Decided by** | Bitfab's assertion judge, or code | An LLM judge |
| **Best for** | Locking in a fix for one case | A quality bar across all cases |

Use both. A grader catches a failure mode wherever it shows up. An assertion pins down exactly what one case should do.

## How an Assertion Works

**1. Write it**

Your coding agent writes an assertion on an original trace, Bitfab generates drafts from the trace, or a person labels a [scenario](https://docs.bitfab.ai/primitives/scenarios.md) and Bitfab drafts assertions from the label.

**2. Bitfab checks the wording**

Each new assertion is checked against the writing rules after it is saved. One that breaks a rule, such as two claims in one sentence, is rewritten in place. This is the [guided rewrite](#guided-rewrite).

**3. A person approves it**

Every new assertion starts as **Not reviewed**. Nothing checks it until someone approves it on the **Assertions** page or through the coding agent.

**4. Every replay is checked**

When the trace is replayed, Bitfab's assertion judge reads the replay and gives each approved assertion a verdict. An [experiment](https://docs.bitfab.ai/primitives/experiments.md) compares that verdict with the assertion's verdict on the original trace.

## Writing Assertions

### From your coding agent

The common path is to ask your coding agent, in plain language, while you are fixing a trace:

```
This booking should have picked the 6am flight, not the 9am. Add an assertion.
Generate assertions on the failing checkout-agent traces from last week
```

The agent writes with the `save_trace_assertions` tool, or asks Bitfab to draft them with `generate_trace_assertions`. See the [Claude Code plugin](https://docs.bitfab.ai/claude-plugin.md#save_trace_assertions) reference for both tools.

### Generating assertions

Bitfab can read one original trace and draft assertions about what it actually did. Pass **guidance** to steer the drafts toward one failure. Guidance is free text, most often a root cause analysis of why the trace went wrong. It narrows the evidence Bitfab reads and produces smaller assertions aimed at that failure.

```typescript TypeScript theme={null}
const drafts = await client.traces.generateAssertions(traceId, {
  guidance: "The agent booked a flight after the user asked for a train.",
})
```

```python Python theme={null}
drafts = client.traces.generate_assertions(
    trace_id,
    guidance="The agent booked a flight after the user asked for a train.",
)
```

Generated drafts are saved as **Not reviewed**, like any other new assertion.

* A trace keeps one set of generated drafts. Asking again returns the same drafts. Pass `regenerate: true` to start over, which archives the previous drafts once the new ones are saved.
* Assertions a person wrote are never replaced by a regeneration.
* `categoryIds` limits the drafts to those [categories](https://docs.bitfab.ai/primitives/categories.md). Leave it out and Bitfab picks from every category in your organization.
* A replay trace is refused. Generate on the original.

### Writing them yourself

Scripts and replay processes can write assertions directly through the SDK, with no coding agent in the loop.

```typescript TypeScript theme={null}
await client.traces.saveAssertions({
  traceId,
  assertions: [
    {
      assertion: "The reply offers the 6:10am flight to Denver",
      targetOnEvaluatedTrace: { kind: "output" },
    },
  ],
})
```

```python Python theme={null}
client.traces.save_assertions(
    trace_id,
    [
        {
            "assertion": "The reply offers the 6:10am flight to Denver",
            "targetOnEvaluatedTrace": {"kind": "output"},
        }
    ],
)
```

Saving with an assertion's `id` edits it in place. A field left out of the save keeps its previous value. `saveAssertionsAll` / `save_assertions_all` writes up to 500 traces in one request, and a rejected batch writes nothing. The SDK can write and edit assertions, but it cannot approve them. See the [TypeScript](https://docs.bitfab.ai/reference/typescript.md#traces), [Python](https://docs.bitfab.ai/reference/python.md#traces), [Ruby](https://docs.bitfab.ai/reference/ruby.md#assertions), and [Go](https://docs.bitfab.ai/reference/go.md#assertions) references for every field.

### Writing a good assertion

An assertion is one statement of the answer the trace settles. Write it so a reader holding only the replay can say pass or fail.

| Do | Instead of |
| - | - |
| State the value: "offers the 6:10am flight" | Point at it: "offers the time the user picked" |
| One claim per assertion | "Books the flight and emails the receipt" |
| A plain claim | A condition: "if the meeting is booked, it sends an invite" |
| The outcome a user would notice | A span, function, field, or tool name |
| Names, dates, and amounts that hold on any replay | An id that exists only in the recorded run |

### Guided rewrite

Bitfab checks each new or edited assertion against the rules above after it is saved. When one breaks a rule, Bitfab rewrites it in place as one well-formed assertion, usually within a minute. The rewrite cleans up how the assertion is phrased and keeps what you meant. The outcome you asked for stays, even where the trace shows something else.

When the text you sent held more than one claim, the rewrite keeps one and hands the others back as ready-to-send text under `separateAssertions`. Those are not saved for you. Save each as its own assertion.

Each assertion reports the check as `rewrite.status`.

| Status | Meaning |
| - | - |
| `pending` | The check is still running |
| `rewritten` | The text broke a rule and was replaced. `sentAssertion` holds what you sent and `changes` says what moved |
| `unchanged` | The text followed the rules and was kept as sent |
| `check_failed` | The check could not run, or had not finished 15 minutes after the save. The text stays as sent |
| `not_checked` | No check ran, for example because the rewrite was skipped |

The rewrite is on by default. Pass `skipGuidedRewrite` (`skip_guided_rewrite` in Python and Ruby, `SkipGuidedRewrite` in Go) to save text exactly as sent. Over MCP, `save_trace_assertions` takes `guidedRewrite: false`. Drafts made from a scenario label are always checked.

## What an Assertion Checks

By default an assertion is checked against the whole replay. Set `targetOnEvaluatedTrace` to narrow it.

| Target | Checks |
| - | - |
| None | The whole trace |
| `{ kind: "output" }` | The trace's final output |
| `{ kind: "span", name: "search_flights", occurrence: "last" }` | One span, found by name. `occurrence` is `"first"`, `"last"` (the default), or a 0-based index |

Targets name spans, never span ids. A span id from the original trace matches nothing on a replay. When the target cannot be found on the trace being checked, the assertion gets no pass.

### Judge, code, and hybrid assertions

Every assertion has a `kind` that says who decides it on a replay.

| Kind | Decided by | Cost |
| - | - | - |
| **Judge** (the default) | Bitfab's assertion judge reads the replay and decides | Judge credits per verdict |
| **Code** | A short TypeScript function over the replay's spans returns pass or fail, the same way every time | No credits |
| **Hybrid** | The code settles what a program can read, such as the recipient or that a tool ran, and returns `"defer"` to hand the rest to the judge | Credits only when the judge is asked |

```ts theme={null}
const sends = await named("send_email")
return sends.some((span) => span.input?.to === "ana@example.com") ? "pass" : "fail"
```

Bitfab drafts the code for you after an assertion is saved. The assertion then shows a **Code** or **Hybrid** badge, with its code in review, where you can edit it or turn it back into a judge assertion. You can also write `kind` and `code` yourself through the SDKs and `save_trace_assertions`.

Code that cannot run on a replay is reported as broken, not as a fail. That covers code that throws, runs past one second of compute, or reads a span no correct run produces. The judge decides that replay instead, and `codeBrokenReason` says why, so a renamed span never reads as a regression.

> [!NOTE]
>
> Code and hybrid assertions are turned on per organization. Contact us to enable them.

See the [HTTP reference](https://docs.bitfab.ai/reference/http.md#assertions) for the full code contract: the `spans`, `input`, `output`, and `named()` it is called with, and its limits.

## Reviewing Assertions

Open **Assertions** in the sidebar to work through the review queue, for one traced function or across all of them.

The **Approval** tabs split the queue into **Not reviewed**, **Approved**, **Rejected**, and **All**, each with its count. Narrow the list by category or by assignee, or group it by trace to review one trace's assertions together.

Selecting an assertion opens it beside the trace it was written on. The detail shows:

* The assertion, who wrote it, and what it checks.
* **Assertion justification**: why it was written, with links to the spans it cites. Open a cited span to read what the original run did there.
* Its category, and **Why this category** when Bitfab filed it.
* Its assignee, the organization member on the hook for reviewing it.
* **Approve** and **Reject**, and **Edit** and **Archive** buttons that give you the prompt to hand your coding agent.

| Key | Action |
| - | - |
| `j` or `↓` | Next assertion |
| `k` or `↑` | Previous assertion |
| `a` | Approve, or undo an approval |
| `r` | Reject, or undo a rejection |

Rewording an assertion, or changing its criteria, target, or category, sends it back to **Not reviewed**. A reworded assertion has to be approved again before replays check it.

Your coding agent can approve or reject an existing assertion too, through `save_trace_assertions` with the assertion's `id` and an `approvalState`. A new assertion cannot be approved in the same call that writes it. The approver recorded is the owner of the API key the agent used, so an approval means a second author looked. The SDKs cannot approve.

## Verdicts on Assertions

A replay gives each approved assertion one verdict.

| Verdict | Meaning | Counts as |
| - | - | - |
| **Pass** | The evidence shows the run did what the assertion says | Pass |
| **Fail** | The evidence shows it did not | Fail |
| **Abstained** | The evidence the judge read does not show the part of the run the assertion checks | Fail |
| **Skipped** | No verdict, for a check that never ran, such as an attempt that crashed | Nothing |

An abstained verdict counts as a fail in every pass rate, because anything short of an evidenced pass is a fail. It is shown apart from a fail, so you can tell a gap in what the judge could see from a mistake the agent made.

Every verdict carries its reasoning and the evidence behind it: quotes of what named spans did on the run. Each quote links back to its span.

### Who wrote a verdict

| Source | Written by |
| - | - |
| **Bitfab's judge** (`bitfab_judge`) | Bitfab's assertion judge, or an assertion's code |
| **Human** (`human`) | A person, in the web app or through `save_human_verdicts` |
| **Agent** (`agent`) | A coding agent, through `save_agent_verdicts` or an SDK's `verdicts.save` |

When more than one exists, the judge's verdict counts first, then a person's, then a coding agent's. A verdict Bitfab's judge decided stands. A person or an agent cannot replace it. When the judge gets one wrong, rewrite the assertion so the right answer is clearer. An assertion the judge skipped can still be judged by a person or an agent.

## Assertions in Experiments

An [experiment](https://docs.bitfab.ai/primitives/experiments.md) checks the approved assertions on every trace it replays.

* **Approved only.** An assertion left in review is not judged, carries no verdict, and does not count toward pass rates.
* **Judged as each replay finishes.** Assertion judging is on by default. Pass `--skip-assertion-judging` to the replay command, or `skipAssertionJudging` to `replay()`, to turn it off.
* **Inherited by replays.** A replay with no assertions of its own reads its original trace's, so you write each assertion once.
* **Compared with the original.** Each assertion's verdict on the replay is compared with its verdict on the original trace, giving fixed, regressed, still passing, or still failing. An assertion nobody judged on the original reads pass or fail with nothing to compare against.
* **Only the asserted traces.** Pass `--only-with-assertions` (`onlyWithAssertions`) to replay just the traces that carry an approved assertion.
* **Repeated attempts.** When an experiment replays each trace several times, an assertion passes when at least 75 percent of its attempts passed.

The experiment's rollup has one row per category, so you can see which kind of check moved.

Replay mocking never hides an assertion's evidence. When a replay selects which calls to run for real, Bitfab keeps running the calls every non-archived assertion on the trace needs, whether the assertion is approved or not. See [Replay Mocking](https://docs.bitfab.ai/replay-mocking.md).

## Organizing Assertions

* **[Categories](https://docs.bitfab.ai/primitives/categories.md)** group assertions by what they check, such as "Mandate compliance" or "Rebooking". An assertion belongs to at most one. Bitfab picks one when it generates assertions, and the experiment rollup reports each category separately.
* **Assignee** names the one organization member who should review an assertion. Assigning routes it to a person without approving anything. Pass `assigneeEmail` on a save, or pick a member on the **Assertions** page.
* **Human note** holds context meant for people only. Agents can read and edit it, but it is never used as evidence when judging a replay.

## Archiving

Archive an assertion to retire it. Later replays stop checking it, and the verdicts it already received are kept. Ask your coding agent ("Archive the assertion about the 9am flight"), which calls `archive_trace_assertions`, or call `archiveAssertions` / `archive_assertions` from an SDK.

## Tools and Methods

| Operation | MCP tool | TypeScript SDK |
| - | - | - |
| Write or edit assertions | `save_trace_assertions` | `traces.saveAssertions`, `traces.saveAssertionsAll` |
| Approve or reject | `save_trace_assertions` (`approvalState`) | Not available |
| Generate drafts | `generate_trace_assertions` | `traces.generateAssertions` |
| Read a trace's assertions | `get_trace_assertions` | `traces.getAssertions` |
| Find assertions by review state | `list_trace_assertions` | Not available |
| Archive | `archive_trace_assertions` | `traces.archiveAssertions` |
| Suggest evidence for a verdict | `generate_verdict_evidence` | `verdicts.generateEvidence` |
| Write verdicts | `save_agent_verdicts`, `save_human_verdicts` | `verdicts.save`, `verdicts.saveHuman` |
| Read verdicts | `get_trace_verdicts` | `verdicts.get`, `verdicts.getAll` |

`get_trace_assertions` returns only approved assertions unless you pass `approvalStates`, because only approved ones are checked on a replay. The SDK read returns every state, each with its `approvalState`. The Python, Ruby, and Go SDKs use the same methods in their own naming. See the [SDK reference overview](https://docs.bitfab.ai/reference/overview.md#assertions-and-verdicts).

## Best Practices

* **Write the assertion while you fix the trace.** You know the right answer best right after you found the bug. Add the trace to a [dataset](https://docs.bitfab.ai/primitives/datasets.md) too, so experiments replay it.
* **Read the cited spans before approving.** Approval decides whether replays check the assertion at all.
* **One claim per assertion.** A failure on a one-claim assertion tells you exactly what broke.
* **Pass guidance when generating.** A root cause analysis produces a few sharp assertions instead of many loose ones.
* **Prefer code for what a program can read.** A recipient, an amount, or whether a tool ran is cheaper and steadier as a code or hybrid assertion.
* **Treat an abstained verdict as a coverage gap.** It usually means the run's evidence is not captured where the judge can read it, or the assertion targets the wrong part of the run.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.