Assertions

Overview

An assertion says what should happen when one specific trace is replayed. It is written against the original trace and checked against every replay of it, so it states the right answer for that one input.

The reply offers the 6:10am flight to Denver, not the 9:45am one
The agent asks the user to confirm before charging the card
No email is sent to ana@example.com

An assertion turns a trace you fixed into a test. Once a person approves it, every later replay of that trace gets a verdict on it: pass, fail, or abstained. An experiment counts those verdicts alongside its graders, so a fix you made last month cannot quietly break again.

Assertions are in alpha. The Assertions page in the sidebar carries an alpha badge, and the details on this page may change as the feature settles.

Assertions and graders

Assertions and graders are both pass/fail checks. They differ in what they cover.

	Assertion	Grader
Covers	One trace and every replay of it	Every trace of a traced function
States	The right answer for that one input, with its real names, times, and amounts	A property every run should have, such as “never invents an order number”
Goes into force	When a person approves it	When it is attached to a dataset or an experiment
Decided by	Bitfab’s assertion judge, or code	An LLM judge
Best for	Locking in a fix for one case	A quality bar across all cases

Use both. A grader catches a failure mode wherever it shows up. An assertion pins down exactly what one case should do.

How an Assertion Works

Write it

Your coding agent writes an assertion on an original trace, Bitfab generates drafts from the trace, or a person labels a scenario and Bitfab drafts assertions from the label.

Bitfab checks the wording

Each new assertion is checked against the writing rules after it is saved. One that breaks a rule, such as two claims in one sentence, is rewritten in place. This is the guided rewrite.

A person approves it

Every new assertion starts as Not reviewed. Nothing checks it until someone approves it on the Assertions page or through the coding agent.

Every replay is checked

When the trace is replayed, Bitfab’s assertion judge reads the replay and gives each approved assertion a verdict. An experiment compares that verdict with the assertion’s verdict on the original trace.

Writing Assertions

From your coding agent

The common path is to ask your coding agent, in plain language, while you are fixing a trace:

This booking should have picked the 6am flight, not the 9am. Add an assertion.
Generate assertions on the failing checkout-agent traces from last week

The agent writes with the save_trace_assertions tool, or asks Bitfab to draft them with generate_trace_assertions. See the Claude Code plugin reference for both tools.

Generating assertions

Bitfab can read one original trace and draft assertions about what it actually did. Pass guidance to steer the drafts toward one failure. Guidance is free text, most often a root cause analysis of why the trace went wrong. It narrows the evidence Bitfab reads and produces smaller assertions aimed at that failure.

const drafts = await client.traces.generateAssertions(traceId, {
  guidance: "The agent booked a flight after the user asked for a train.",
})

drafts = client.traces.generate_assertions(
    trace_id,
    guidance="The agent booked a flight after the user asked for a train.",
)

Generated drafts are saved as Not reviewed, like any other new assertion.

A trace keeps one set of generated drafts. Asking again returns the same drafts. Pass regenerate: true to start over, which archives the previous drafts once the new ones are saved.
Assertions a person wrote are never replaced by a regeneration.
categoryIds limits the drafts to those categories. Leave it out and Bitfab picks from every category in your organization.
A replay trace is refused. Generate on the original.

Writing them yourself

Scripts and replay processes can write assertions directly through the SDK, with no coding agent in the loop.

await client.traces.saveAssertions({
  traceId,
  assertions: [
    {
      assertion: "The reply offers the 6:10am flight to Denver",
      targetOnEvaluatedTrace: { kind: "output" },
    },
  ],
})

client.traces.save_assertions(
    trace_id,
    [
        {
            "assertion": "The reply offers the 6:10am flight to Denver",
            "targetOnEvaluatedTrace": {"kind": "output"},
        }
    ],
)

Saving with an assertion’s id edits it in place. A field left out of the save keeps its previous value. saveAssertionsAll / save_assertions_all writes up to 500 traces in one request, and a rejected batch writes nothing. The SDK can write and edit assertions, but it cannot approve them. See the TypeScript, Python, Ruby, and Go references for every field.

Writing a good assertion

An assertion is one statement of the answer the trace settles. Write it so a reader holding only the replay can say pass or fail.

Do	Instead of
State the value: “offers the 6:10am flight”	Point at it: “offers the time the user picked”
One claim per assertion	”Books the flight and emails the receipt”
A plain claim	A condition: “if the meeting is booked, it sends an invite”
The outcome a user would notice	A span, function, field, or tool name
Names, dates, and amounts that hold on any replay	An id that exists only in the recorded run

Guided rewrite

Bitfab checks each new or edited assertion against the rules above after it is saved. When one breaks a rule, Bitfab rewrites it in place as one well-formed assertion, usually within a minute. The rewrite cleans up how the assertion is phrased and keeps what you meant. The outcome you asked for stays, even where the trace shows something else. When the text you sent held more than one claim, the rewrite keeps one and hands the others back as ready-to-send text under separateAssertions. Those are not saved for you. Save each as its own assertion. Each assertion reports the check as rewrite.status.

Status	Meaning
`pending`	The check is still running
`rewritten`	The text broke a rule and was replaced. `sentAssertion` holds what you sent and `changes` says what moved
`unchanged`	The text followed the rules and was kept as sent
`check_failed`	The check could not run, or had not finished 15 minutes after the save. The text stays as sent
`not_checked`	No check ran, for example because the rewrite was skipped

The rewrite is on by default. Pass skipGuidedRewrite (skip_guided_rewrite in Python and Ruby, SkipGuidedRewrite in Go) to save text exactly as sent. Over MCP, save_trace_assertions takes guidedRewrite: false. Drafts made from a scenario label are always checked.

What an Assertion Checks

By default an assertion is checked against the whole replay. Set targetOnEvaluatedTrace to narrow it.

Target	Checks
None	The whole trace
`{ kind: "output" }`	The trace’s final output
`{ kind: "span", name: "search_flights", occurrence: "last" }`	One span, found by name. `occurrence` is `"first"`, `"last"` (the default), or a 0-based index

Targets name spans, never span ids. A span id from the original trace matches nothing on a replay. When the target cannot be found on the trace being checked, the assertion gets no pass.

Judge, code, and hybrid assertions

Every assertion has a kind that says who decides it on a replay.

Kind	Decided by	Cost
Judge (the default)	Bitfab’s assertion judge reads the replay and decides	Judge credits per verdict
Code	A short TypeScript function over the replay’s spans returns pass or fail, the same way every time	No credits
Hybrid	The code settles what a program can read, such as the recipient or that a tool ran, and returns `"defer"` to hand the rest to the judge	Credits only when the judge is asked

const sends = await named("send_email")
return sends.some((span) => span.input?.to === "ana@example.com") ? "pass" : "fail"

Bitfab drafts the code for you after an assertion is saved. The assertion then shows a Code or Hybrid badge, with its code in review, where you can edit it or turn it back into a judge assertion. You can also write kind and code yourself through the SDKs and save_trace_assertions. Code that cannot run on a replay is reported as broken, not as a fail. That covers code that throws, runs past one second of compute, or reads a span no correct run produces. The judge decides that replay instead, and codeBrokenReason says why, so a renamed span never reads as a regression.

Code and hybrid assertions are turned on per organization. Contact us to enable them.

See the HTTP reference for the full code contract: the spans, input, output, and named() it is called with, and its limits.

Reviewing Assertions

Open Assertions in the sidebar to work through the review queue, for one traced function or across all of them. The Approval tabs split the queue into Not reviewed, Approved, Rejected, and All, each with its count. Narrow the list by category or by assignee, or group it by trace to review one trace’s assertions together. Selecting an assertion opens it beside the trace it was written on. The detail shows:

The assertion, who wrote it, and what it checks.
Assertion justification: why it was written, with links to the spans it cites. Open a cited span to read what the original run did there.
Its category, and Why this category when Bitfab filed it.
Its assignee, the organization member on the hook for reviewing it.
Approve and Reject, and Edit and Archive buttons that give you the prompt to hand your coding agent.

Key	Action
`j` or `↓`	Next assertion
`k` or `↑`	Previous assertion
`a`	Approve, or undo an approval
`r`	Reject, or undo a rejection

Rewording an assertion, or changing its criteria, target, or category, sends it back to Not reviewed. A reworded assertion has to be approved again before replays check it. Your coding agent can approve or reject an existing assertion too, through save_trace_assertions with the assertion’s id and an approvalState. A new assertion cannot be approved in the same call that writes it. The approver recorded is the owner of the API key the agent used, so an approval means a second author looked. The SDKs cannot approve.

Verdicts on Assertions

A replay gives each approved assertion one verdict.

Verdict	Meaning	Counts as
Pass	The evidence shows the run did what the assertion says	Pass
Fail	The evidence shows it did not	Fail
Abstained	The evidence the judge read does not show the part of the run the assertion checks	Fail
Skipped	No verdict, for a check that never ran, such as an attempt that crashed	Nothing

An abstained verdict counts as a fail in every pass rate, because anything short of an evidenced pass is a fail. It is shown apart from a fail, so you can tell a gap in what the judge could see from a mistake the agent made. Every verdict carries its reasoning and the evidence behind it: quotes of what named spans did on the run. Each quote links back to its span.

Who wrote a verdict

Source	Written by
Bitfab’s judge (`bitfab_judge`)	Bitfab’s assertion judge, or an assertion’s code
Human (`human`)	A person, in the web app or through `save_human_verdicts`
Agent (`agent`)	A coding agent, through `save_agent_verdicts` or an SDK’s `verdicts.save`

When more than one exists, the judge’s verdict counts first, then a person’s, then a coding agent’s. A verdict Bitfab’s judge decided stands. A person or an agent cannot replace it. When the judge gets one wrong, rewrite the assertion so the right answer is clearer. An assertion the judge skipped can still be judged by a person or an agent.

Assertions in Experiments

An experiment checks the approved assertions on every trace it replays.

Approved only. An assertion left in review is not judged, carries no verdict, and does not count toward pass rates.
Judged as each replay finishes. Assertion judging is on by default. Pass --skip-assertion-judging to the replay command, or skipAssertionJudging to replay(), to turn it off.
Inherited by replays. A replay with no assertions of its own reads its original trace’s, so you write each assertion once.
Compared with the original. Each assertion’s verdict on the replay is compared with its verdict on the original trace, giving fixed, regressed, still passing, or still failing. An assertion nobody judged on the original reads pass or fail with nothing to compare against.
Only the asserted traces. Pass --only-with-assertions (onlyWithAssertions) to replay just the traces that carry an approved assertion.
Repeated attempts. When an experiment replays each trace several times, an assertion passes when at least 75 percent of its attempts passed.

The experiment’s rollup has one row per category, so you can see which kind of check moved. Replay mocking never hides an assertion’s evidence. When a replay selects which calls to run for real, Bitfab keeps running the calls every non-archived assertion on the trace needs, whether the assertion is approved or not. See Replay Mocking.

Organizing Assertions

Categories group assertions by what they check, such as “Mandate compliance” or “Rebooking”. An assertion belongs to at most one. Bitfab picks one when it generates assertions, and the experiment rollup reports each category separately.
Assignee names the one organization member who should review an assertion. Assigning routes it to a person without approving anything. Pass assigneeEmail on a save, or pick a member on the Assertions page.
Human note holds context meant for people only. Agents can read and edit it, but it is never used as evidence when judging a replay.

Archiving

Archive an assertion to retire it. Later replays stop checking it, and the verdicts it already received are kept. Ask your coding agent (“Archive the assertion about the 9am flight”), which calls archive_trace_assertions, or call archiveAssertions / archive_assertions from an SDK.

Tools and Methods

Operation	MCP tool	TypeScript SDK
Write or edit assertions	`save_trace_assertions`	`traces.saveAssertions`, `traces.saveAssertionsAll`
Approve or reject	`save_trace_assertions` (`approvalState`)	Not available
Generate drafts	`generate_trace_assertions`	`traces.generateAssertions`
Read a trace’s assertions	`get_trace_assertions`	`traces.getAssertions`
Find assertions by review state	`list_trace_assertions`	Not available
Archive	`archive_trace_assertions`	`traces.archiveAssertions`
Suggest evidence for a verdict	`generate_verdict_evidence`	`verdicts.generateEvidence`
Write verdicts	`save_agent_verdicts`, `save_human_verdicts`	`verdicts.save`, `verdicts.saveHuman`
Read verdicts	`get_trace_verdicts`	`verdicts.get`, `verdicts.getAll`

get_trace_assertions returns only approved assertions unless you pass approvalStates, because only approved ones are checked on a replay. The SDK read returns every state, each with its approvalState. The Python, Ruby, and Go SDKs use the same methods in their own naming. See the SDK reference overview.

Best Practices

Write the assertion while you fix the trace. You know the right answer best right after you found the bug. Add the trace to a dataset too, so experiments replay it.
Read the cited spans before approving. Approval decides whether replays check the assertion at all.
One claim per assertion. A failure on a one-claim assertion tells you exactly what broke.
Pass guidance when generating. A root cause analysis produces a few sharp assertions instead of many loose ones.
Prefer code for what a program can read. A recipient, an amount, or whether a tool ran is cheaper and steadier as a code or hybrid assertion.
Treat an abstained verdict as a coverage gap. It usually means the run’s evidence is not captured where the judge can read it, or the assertion targets the wrong part of the run.

Getting Started

Primitives

Account

SDKs

Framework Integrations

Integrations

Web Portal

Reference

Changelog

Overview

Assertions and graders

How an Assertion Works

Writing Assertions

From your coding agent

Generating assertions

Writing them yourself

Writing a good assertion

Guided rewrite

What an Assertion Checks

Judge, code, and hybrid assertions

Reviewing Assertions

Verdicts on Assertions

Who wrote a verdict

Assertions in Experiments

Organizing Assertions

Archiving

Tools and Methods

Best Practices

​Overview

​Assertions and graders

​How an Assertion Works

​Writing Assertions

​From your coding agent

​Generating assertions

​Writing them yourself

​Writing a good assertion

​Guided rewrite

​What an Assertion Checks

​Judge, code, and hybrid assertions

​Reviewing Assertions

​Verdicts on Assertions

​Who wrote a verdict

​Assertions in Experiments

​Organizing Assertions

​Archiving

​Tools and Methods

​Best Practices

Overview

Assertions and graders

How an Assertion Works

Writing Assertions

From your coding agent

Generating assertions

Writing them yourself

Writing a good assertion

Guided rewrite

What an Assertion Checks

Judge, code, and hybrid assertions

Reviewing Assertions

Verdicts on Assertions

Who wrote a verdict

Assertions in Experiments

Organizing Assertions

Archiving

Tools and Methods

Best Practices