> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bitfab.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Writing a Good Assertion

> The rules Bitfab checks every assertion against, with a before and after for each, plus when to give an assertion code and how to read an abstained verdict

An [assertion](https://docs.bitfab.ai/primitives/assertions.md) is one statement of what a correct run does with one trace's input. A replay runs that input again, and a reader holding only the replay decides whether the statement held. Everything on this page serves that reader: when they can say pass or fail without guessing, the verdict means what you meant.

The examples use an overnight flight-booking agent. While the traveller sleeps, it loads their trip, checks their calendar, searches fares, decides whether to book, then charges the card, books the ticket, emails a confirmation, and adds the flight to their calendar. In the trace they are written against, the traveller is Ana Ruiz ([ana@example.com](mailto:ana@example.com)), the trip is 9512 from SFO to Denver, and the fare found is United UA 1432, leaving at 8:30 AM on March 12 for $248, against a $300 budget.

## The shape of a good assertion

Write the agent, a verb, and the values this trace settles: names, dates, times, amounts, and recipients. Name the person ("Ana Ruiz", not "the traveller") and the thing ("UA 1432", not "the flight") whenever the trace says which one.

```
Agent books the 8:30 AM March 12 flight from SFO to Denver for $248.
Agent charges Ana Ruiz's card $248.
Agent does not book the trip, because the $412 fare is over the $300 budget.
```

Each line states one definite thing the run must do, or must not do, and every word names something the reader checks.

## The rules

Bitfab checks each new or edited assertion against these rules after it is saved. An assertion that breaks one is rewritten in place by the [guided rewrite](https://docs.bitfab.ai/primitives/assertions.md#guided-rewrite), so the rules below are the same ones the product applies.

### State the value

Write the value the trace settles, not a description of where to find it. A reader who has to work out "the fare the budget allows" can work it out wrongly.

| Before | After |
| - | - |
| Agent books the fare the traveller's budget allows. | Agent books the \$248 fare to Denver. |
| The confirmation gives the time the agent picked. | The confirmation gives the 8:30 AM departure. |

A name, a date, a time, an amount, an email address, or a quoted message is a stated value. So is a bounded range or a named set of allowed values: "a fare from $200 to $300", "United or Alaska".

### One claim per assertion

Put each requirement that can pass or fail on its own in its own assertion. When two share one, a fail does not tell you which broke.

| Before | After |
| - | - |
| Agent books the Denver flight and emails the traveller a confirmation. | Agent books UA 1432 from SFO to Denver on March 12. |
| | Agent emails [ana@example.com](mailto:ana@example.com) a confirmation for UA 1432 on March 12. |

Parts that only count together are one claim: "the calendar entry is on March 12 at 8:30 AM" fails if either the date or the time is wrong, and neither alone is correct. Equally correct answers joined by "or" are one claim too. When you send text with more than one claim, the guided rewrite keeps one and hands the others back under `separateAssertions` for you to save.

### A plain claim, not a condition

A trace's input never changes. Whether the trip is still active, the fare is in budget, or the calendar has a conflict is already settled, so read the trace and say what the run should do.

| Before | After |
| - | - |
| If the agent books the trip, it charges no more than \$300. | Agent charges Ana Ruiz's card \$248. |
| If the agent sends a confirmation, it goes to [ana@example.com](mailto:ana@example.com). | Agent emails the confirmation to [ana@example.com](mailto:ana@example.com). |

A condition that never arises on the replay leaves the reader nothing to decide. An option the claim allows is not a condition: "the calendar entry may include the return flight" is fine.

Opening on a condition about the input ("if the trip was cancelled, the agent does not charge the card") is not rewritten, but it reads as hedging. It is always true or always false for this trace, so drop it: "Agent does not charge Ana Ruiz's card for trip 9512."

### The outcome a user would notice

Describe what the run did in plain words, not the code's names for it. A span, function, tool, or field name checks the plumbing, and a rename fails a correct run.

| Before | After |
| - | - |
| ChargeCard is called with amount\_cents 24800. | Agent charges Ana Ruiz's card \$248. |
| The Decide output has should\_book set to false. | Agent does not book UA 1432 for Ana Ruiz. |

Pass and fail criteria may name the call to look at, such as "No BookTicket call runs." That points the reader at where to check, which is what criteria are for. The assertion above them still says the outcome in plain words.

### Values that hold on any replay

Leave out ids that belong to the recorded run. A replay makes its own charge ids, ticket numbers, and span ids, so a check that names the recorded one can never match.

| Before | After |
| - | - |
| The confirmation email names ticket TKT-8F21A. | Agent emails [ana@example.com](mailto:ana@example.com) a confirmation for UA 1432 on March 12. |
| The booking uses charge ch\_3PqL9e. | Agent books UA 1432 with a \$248 charge on Ana Ruiz's card. |

Values from the trace's input stay the same on every replay: the traveller, the route, the dates, the fare, and the budget.

### A clear pass line

Use words two careful readers would score the same way. "Short", "reasonable", "about", and "appropriate" point in a direction without saying what passes.

| Before | After |
| - | - |
| The confirmation is short and gives a reasonable summary of the trip. | The confirmation gives the March 12, 8:30 AM departure and the \$248 fare. |
| Agent books a flight leaving roughly in the morning. | Agent books UA 1432, which leaves at 8:30 AM on March 12. |

A word that names one thing the run returns or does is exact, whatever that thing is called.

### Any wording that reaches the outcome

Check what the run achieves, not the words it uses. A required phrase, word order, or layout fails a correct run that says the same thing differently.

| Before | After |
| - | - |
| The confirmation starts with "You're all set" and gives the time before the airport. | The confirmation tells Ana Ruiz that UA 1432 on March 12 at 8:30 AM is booked. |

Wording the input asks for is the outcome, so state it exactly. When Ana Ruiz asked for the subject line "Denver trip booked", "The confirmation's subject is 'Denver trip booked'" is right.

### About this run

Write about what this run does with this input, not the state of the thread, an earlier turn, or a different run.

| Before | After |
| - | - |
| Agent waits for the traveller to confirm before booking. | Agent books UA 1432 without asking Ana Ruiz first. |
| Having already charged the card, the agent books the ticket. | Agent books UA 1432 for Ana Ruiz on March 12. |

### Criteria never say to skip

Pass and fail criteria that tell the reader to skip the check ("if no fare search ran, skip this") leave the verdict undecided. Say what a run like that counts as instead, usually a fail.

### One assertion per claim on a trace

When another assertion on the same trace already makes a claim, a single failure is counted twice. The guided rewrite cannot fix this by rewording: archive one of them.

### Smaller things to fix

The check also notes these, but does not rewrite an assertion for them alone:

* **Filler words.** "Proactively", "clearly", "successfully", and "correctly" add nothing a reader can check. "Agent correctly books the Denver flight" says the same as "Agent books the Denver flight".
* **Too much prose.** When what must hold is hard to find, cut until it is easy.
* **A judgement call with no criteria.** When a claim needs a reader's judgement, such as whether the confirmation explains why the trip was not booked, add pass and fail criteria that mark the edge.

## The outcome is yours to set

The recorded run is one sample, and it may be wrong. An assertion says what a correct run does, so it can and often should disagree with what the trace shows: the trace you are fixing booked the \$412 fare, and the assertion says it does not. The guided rewrite changes how an assertion is worded and keeps the outcome you asked for, even where the trace shows something else.

## When to give an assertion code

An assertion can carry [code](https://docs.bitfab.ai/primitives/assertions.md#assertions-with-code): a short JavaScript function over the replay's spans that returns the verdict when what it reads settles the claim, and hands the replay to the LLM judge when it does not. You write the assertion the same way either way. Bitfab decides whether an assertion should have code and drafts it after the save.

| What settles the claim | Example | Best decided by |
| - | - | - |
| Which calls ran, and with what arguments | Agent charges Ana Ruiz's card \$248. | Code |
| The exact text the run sent, such as a time or a recipient | The confirmation goes to [ana@example.com](mailto:ana@example.com). | Code, or code that finds the message and hands it to the judge |
| What the run meant, its tone, or its intent | The confirmation tells Ana Ruiz that UA 1432 was not booked because of the March 12 board meeting. | The LLM judge |

Code is cheaper (1 credit per 10 verdicts, against 10 credits for a judge verdict) and gives the same verdict on the same replay every time. When a claim is half of each, such as the board meeting example, code can check that an email went to [ana@example.com](mailto:ana@example.com) and hand that email to the judge to read.

Following the rules above is what makes code possible. "ChargeCard is called with amount\_cents 24800" and "Agent charges Ana Ruiz's card \$248" can both be checked by code, but only the second still passes when the span is renamed, and only the second is an outcome the reader can check by hand.

## Reading an abstained verdict

An abstained verdict means the evidence that was read does not show the part of the run the assertion checks, so nothing in it confirms or denies the claim. It counts as a fail in every pass rate, and it is shown apart from a fail so you can tell a gap in what could be seen from a mistake the agent made.

Read the verdict's reason first. It says what the evidence would have had to show. Then look for which of these caused it:

| Cause | Example | Fix |
| - | - | - |
| The step it checks is not traced, or its input and output are not captured | The claim is about the confirmation email, but the step that sends it is not recorded as a span | Trace that step, so replays record what it was called with |
| The assertion targets a part of the run that does not hold the evidence | It targets the final output, but the charge amount is only in the call that charges the card | Target the whole trace, or the span that holds it |
| The claim is about something other than this run | "Agent waits for the traveller to confirm" asks about a later turn | Rewrite it as what this run does, following [About this run](#about-this-run) |
| The claim depends on a condition that does not arise | "If the agent books, it charges no more than \$300" on a replay that does not book | State the outcome the input settles, following [A plain claim, not a condition](#a-plain-claim-not-a-condition) |

An abstained verdict is a gap in coverage, not a verdict on the agent. Fix the cause, and the next replay gets a pass or a fail you can act on.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.