Skip to main content

Overview

An assertion says what should happen when one specific trace is replayed. It is written against the original trace and checked against every replay of it, so it states the right answer for that one input.
An assertion turns a trace you fixed into a test. Once a person approves it, every later replay of that trace gets a verdict on it: pass, fail, or abstained. An experiment counts those verdicts alongside its graders, so a fix you made last month cannot quietly break again.
Assertions are in alpha. The Assertions page in the sidebar carries an alpha badge, and the details on this page may change as the feature settles.

Assertions and graders

Assertions and graders are both pass/fail checks. They differ in what they cover. Graders are deprecated and new ones can no longer be created, so write assertions for new checks. Graders you already have keep running.

How an Assertion Works

1

Write it

Your coding agent writes an assertion on an original trace, Bitfab generates drafts from the trace, or a person labels a scenario and Bitfab drafts assertions from the label.
2

Bitfab checks the wording

Each new assertion is checked against the writing rules after it is saved. One that breaks a rule, such as two claims in one sentence, is rewritten in place. This is the guided rewrite.
3

A person approves it

Every new assertion starts as Not reviewed. Nothing checks it until someone approves it on the Assertions page or through the coding agent.
4

Every replay is checked

When the trace is replayed, Bitfab’s assertion judge reads the replay and gives each approved assertion a verdict. An experiment compares that verdict with the assertion’s verdict on the original trace.

Writing Assertions

From your coding agent

The common path is to ask your coding agent, in plain language, while you are fixing a trace:
The agent writes with the save_trace_assertions tool, or asks Bitfab to draft them with generate_trace_assertions. See the Claude Code plugin reference for both tools.

Generating assertions

Bitfab can read one original trace and draft assertions about what it actually did. Pass guidance to steer the drafts toward one failure. Guidance is free text, most often a root cause analysis of why the trace went wrong. It narrows the evidence Bitfab reads and produces smaller assertions aimed at that failure.
Generated drafts are saved as Not reviewed, like any other new assertion.
  • A trace keeps one set of generated drafts. Asking again returns the same drafts. Pass regenerate: true to start over, which archives the previous drafts once the new ones are saved.
  • Assertions a person wrote are never replaced by a regeneration.
  • categoryIds limits the drafts to those categories. Leave it out and Bitfab picks from every category in your organization.
  • A replay trace is refused. Generate on the original.

Writing them yourself

Scripts and replay processes can write assertions directly through the SDK, with no coding agent in the loop.
Saving with an assertion’s id edits it in place. A field left out of the save keeps its previous value. saveAssertionsAll / save_assertions_all writes up to 500 traces in one request, and a rejected batch writes nothing. The SDK can write and edit assertions, but it cannot approve them. See the TypeScript, Python, Ruby, and Go references for every field.

Writing a good assertion

An assertion is one statement of the answer the trace settles. Write it so a reader holding only the replay can say pass or fail.

Guided rewrite

Bitfab checks each new or edited assertion against the rules above after it is saved. When one breaks a rule, Bitfab rewrites it in place as one well-formed assertion, usually within a minute. The rewrite cleans up how the assertion is phrased and keeps what you meant. The outcome you asked for stays, even where the trace shows something else. When the text you sent held more than one claim, the rewrite keeps one and hands the others back as ready-to-send text under separateAssertions. Those are not saved for you. Save each as its own assertion. Each assertion reports the check as rewrite.status. The rewrite is on by default. Pass skipGuidedRewrite (skip_guided_rewrite in Python and Ruby, SkipGuidedRewrite in Go) to save text exactly as sent. Over MCP, save_trace_assertions takes guidedRewrite: false. Drafts made from a scenario label are always checked.

What an Assertion Checks

By default an assertion is checked against the whole replay. Set targetOnEvaluatedTrace to narrow it. Targets name spans, never span ids. A span id from the original trace matches nothing on a replay. When the target cannot be found on the trace being checked, the assertion gets no pass.

Assertions with code

An assertion can carry code: a short TypeScript function over the replay’s spans. The code decides what a program can read, and hands the rest to Bitfab’s assertion judge. An assertion without code is decided by the judge on every replay. An external assertion is never decided by Bitfab: you post its verdict yourself, so it uses no judge credits.
data is what the code found that the judge needs to decide, such as the email it sent. Keep it to the few values the rest of the assertion is about. Code that throws is treated as "defer" with no data. Bitfab decides whether an assertion should have code, and drafts it for you after the assertion is saved. An assertion with code shows a Code badge, with its code in review, where you can read it or remove it. You can also write code yourself through the SDKs and save_trace_assertions, and send code: null to remove it. Code that cannot run on a replay is reported as broken, not as a fail. That covers code that runs past one second of compute, returns something other than a result, or reads a span no correct run produces. The judge decides that replay instead, and codeBrokenReason says why, so a renamed span never reads as a regression. When an assertion’s code turns out to be wrong, turn it off with Turn code off in review, or send codeOff: true through the SDKs and save_trace_assertions. The code is kept, the LLM judge decides every replay, and Bitfab does not write new code for that assertion. Turn code on, or codeOff: false, uses the code again. Removing the code also turns it back on, so Bitfab can write new code for the assertion.
Assertion code is turned on per organization. Contact us to enable it.
See the HTTP reference for the full code contract: the spans, input, output, and named() it is called with, and its limits.

External assertions

An external assertion is an assertion Bitfab never judges. You evaluate the trace with your own harness, CI job, or review queue, and post each verdict to Bitfab. Use one in place of an external grader when the check belongs to one trace. Create it by saving an assertion with kind: "external" through save_trace_assertions or an SDK’s assertion save (bitfab.AssertionKindExternal in Go). It takes no code. An assertion is external only from the save that creates it, and an existing assertion cannot be changed to or from external. Like any assertion, it must be approved before it takes a verdict. Post a verdict by assertion id alone. Bitfab finds the trace the assertion is on. Pass an experiment id, and optionally a 0-based attempt (default 0), to save the verdict on that experiment’s replay of the trace instead.
  • Over MCP, save_assertion_verdicts posts up to 200 verdicts in one call. The SDKs batch with saveExternalAll / save_external_all.
  • A call that names an assertion that is not external, not approved, or not in your organization fails and writes nothing. Write verdicts on other assertions with save_human_verdicts or save_agent_verdicts.
  • The verdict is stored where Bitfab’s judge would store its own (source bitfab_judge), so it counts in every pass rate as soon as it is posted. A person’s verdict, then a coding agent’s, still counts over it.
  • The assertion shows an External badge in the web app.

Reviewing Assertions

Open Assertions in the sidebar to work through the review queue, for one traced function or across all of them. The Approval tabs split the queue into Not reviewed, Approved, Rejected, and All, each with its count. Narrow the list by category or by assignee, or group it by trace to review one trace’s assertions together. Selecting an assertion opens it beside the trace it was written on. The detail shows:
  • The assertion, who wrote it, and what it checks.
  • Assertion justification: why it was written, with links to the spans it cites. Open a cited span to read what the original run did there.
  • Its category, and Why this category when Bitfab filed it.
  • Its assignee, the organization member on the hook for reviewing it.
  • Approve and Reject, and Edit and Archive buttons that give you the prompt to hand your coding agent.
Rewording an assertion, or changing its criteria, target, or category, sends it back to Not reviewed. A reworded assertion has to be approved again before replays check it. Your coding agent can approve or reject an existing assertion too, through save_trace_assertions with the assertion’s id and an approvalState. A new assertion cannot be approved in the same call that writes it. The approver recorded is the owner of the API key the agent used, so an approval means a second author looked. The SDKs cannot approve.

Verdicts on Assertions

A replay gives each approved assertion one verdict. An abstained verdict counts as a fail in every pass rate, because anything short of an evidenced pass is a fail. It is shown apart from a fail, so you can tell a gap in what the judge could see from a mistake the agent made. Every verdict carries its reasoning and the evidence behind it: quotes of what named spans did on the run. Each quote links back to its span.

Who wrote a verdict

When more than one exists, a person’s verdict counts first, then a coding agent’s, then the judge’s. When the judge gets one wrong, a person or an agent can save their own verdict on it through save_human_verdicts, save_agent_verdicts, or an SDK’s verdicts methods: it is kept beside the judge’s and becomes the one counted, and reads mark which verdict counts. Archiving an agent’s verdict makes the judge’s count again. The web app does not yet offer this on a verdict the judge decided.

Assertions in Experiments

An experiment checks the approved assertions on every trace it replays.
  • Approved only. An assertion left in review is not judged, carries no verdict, and does not count toward pass rates.
  • Judged as each replay finishes. Assertion judging is on by default. Pass --skip-assertion-judging to the replay command, or skipAssertionJudging to replay(), to turn it off.
  • Inherited by replays. A replay with no assertions of its own reads its original trace’s, so you write each assertion once.
  • Compared with the original. Each assertion’s verdict on the replay is compared with its verdict on the original trace, giving fixed, regressed, still passing, or still failing. An assertion nobody judged on the original reads pass or fail with nothing to compare against.
  • Only the asserted traces. Pass --only-with-assertions (onlyWithAssertions) to replay just the traces that carry an approved assertion.
  • Only the traces asserted in chosen categories. Pass --category-ids (categoryIds) with --trace-ids or --dataset-ids to replay just the traces among them that carry an approved assertion in one of those categories. Which assertions are judged on each replay stays the same.
  • Repeated attempts. When an experiment replays each trace several times, an assertion passes when at least 75 percent of its attempts passed.
The experiment’s rollup has one row per category, so you can see which kind of check moved. Replay mocking never hides an assertion’s evidence. When a replay selects which calls to run for real, Bitfab keeps running the calls every non-archived assertion on the trace needs, whether the assertion is approved or not. See Replay Mocking.

Organizing Assertions

  • Categories group assertions by what they check, such as “Mandate compliance” or “Rebooking”. An assertion belongs to at most one. Bitfab picks one when it generates assertions, and the experiment rollup reports each category separately.
  • Assignee names the one organization member who should review an assertion. Assigning routes it to a person without approving anything. Pass assigneeEmail on a save, or pick a member on the Assertions page.
  • Human note holds context meant for people only. Agents can read and edit it, but it is never used as evidence when judging a replay.

Archiving

Archive an assertion to retire it. Later replays stop checking it, and the verdicts it already received are kept. Ask your coding agent (“Archive the assertion about the 9am flight”), which calls archive_trace_assertions, or call archiveAssertions / archive_assertions from an SDK.

Tools and Methods

get_trace_assertions returns only approved assertions unless you pass approvalStates, because only approved ones are checked on a replay. The SDK read returns every state, each with its approvalState. The Python, Ruby, and Go SDKs use the same methods in their own naming. See the SDK reference overview.

Best Practices

  • Write the assertion while you fix the trace. You know the right answer best right after you found the bug. Add the trace to a dataset too, so experiments replay it.
  • Read the cited spans before approving. Approval decides whether replays check the assertion at all.
  • One claim per assertion. A failure on a one-claim assertion tells you exactly what broke.
  • Pass guidance when generating. A root cause analysis produces a few sharp assertions instead of many loose ones.
  • Prefer code for what a program can read. A recipient, an amount, or whether a tool ran is cheaper and steadier decided by code.
  • Treat an abstained verdict as a coverage gap. It usually means the run’s evidence is not captured where the judge can read it, or the assertion targets the wrong part of the run.