The shape of a good assertion
Write the agent, a verb, and the values this trace settles: names, dates, times, amounts, and recipients. Name the person (“Ana Ruiz”, not “the traveller”) and the thing (“UA 1432”, not “the flight”) whenever the trace says which one.The rules
Bitfab checks each new or edited assertion against these rules after it is saved. An assertion that breaks one is rewritten in place by the guided rewrite, so the rules below are the same ones the product applies.State the value
Write the value the trace settles, not a description of where to find it. A reader who has to work out “the fare the budget allows” can work it out wrongly.
A name, a date, a time, an amount, an email address, or a quoted message is a stated value. So is a bounded range or a named set of allowed values: “a fare from $200 to $300”, “United or Alaska”.
One claim per assertion
Put each requirement that can pass or fail on its own in its own assertion. When two share one, a fail does not tell you which broke.
Parts that only count together are one claim: “the calendar entry is on March 12 at 8:30 AM” fails if either the date or the time is wrong, and neither alone is correct. Equally correct answers joined by “or” are one claim too. When you send text with more than one claim, the guided rewrite keeps one and hands the others back under
separateAssertions for you to save.
A plain claim, not a condition
A trace’s input never changes. Whether the trip is still active, the fare is in budget, or the calendar has a conflict is already settled, so read the trace and say what the run should do.
A condition that never arises on the replay leaves the reader nothing to decide. An option the claim allows is not a condition: “the calendar entry may include the return flight” is fine.
Opening on a condition about the input (“if the trip was cancelled, the agent does not charge the card”) is not rewritten, but it reads as hedging. It is always true or always false for this trace, so drop it: “Agent does not charge Ana Ruiz’s card for trip 9512.”
The outcome a user would notice
Describe what the run did in plain words, not the code’s names for it. A span, function, tool, or field name checks the plumbing, and a rename fails a correct run.
Pass and fail criteria may name the call to look at, such as “No BookTicket call runs.” That points the reader at where to check, which is what criteria are for. The assertion above them still says the outcome in plain words.
Values that hold on any replay
Leave out ids that belong to the recorded run. A replay makes its own charge ids, ticket numbers, and span ids, so a check that names the recorded one can never match.
Values from the trace’s input stay the same on every replay: the traveller, the route, the dates, the fare, and the budget.
A clear pass line
Use words two careful readers would score the same way. “Short”, “reasonable”, “about”, and “appropriate” point in a direction without saying what passes.
A word that names one thing the run returns or does is exact, whatever that thing is called.
Any wording that reaches the outcome
Check what the run achieves, not the words it uses. A required phrase, word order, or layout fails a correct run that says the same thing differently.
Wording the input asks for is the outcome, so state it exactly. When Ana Ruiz asked for the subject line “Denver trip booked”, “The confirmation’s subject is ‘Denver trip booked’” is right.
About this run
Write about what this run does with this input, not the state of the thread, an earlier turn, or a different run.Criteria never say to skip
Pass and fail criteria that tell the reader to skip the check (“if no fare search ran, skip this”) leave the verdict undecided. Say what a run like that counts as instead, usually a fail.One assertion per claim on a trace
When another assertion on the same trace already makes a claim, a single failure is counted twice. The guided rewrite cannot fix this by rewording: archive one of them.Smaller things to fix
The check also notes these, but does not rewrite an assertion for them alone:- Filler words. “Proactively”, “clearly”, “successfully”, and “correctly” add nothing a reader can check. “Agent correctly books the Denver flight” says the same as “Agent books the Denver flight”.
- Too much prose. When what must hold is hard to find, cut until it is easy.
- A judgement call with no criteria. When a claim needs a reader’s judgement, such as whether the confirmation explains why the trip was not booked, add pass and fail criteria that mark the edge.
The outcome is yours to set
The recorded run is one sample, and it may be wrong. An assertion says what a correct run does, so it can and often should disagree with what the trace shows: the trace you are fixing booked the $412 fare, and the assertion says it does not. The guided rewrite changes how an assertion is worded and keeps the outcome you asked for, even where the trace shows something else.When to give an assertion code
An assertion can carry code: a short JavaScript function over the replay’s spans that returns the verdict when what it reads settles the claim, and hands the replay to the LLM judge when it does not. You write the assertion the same way either way. Bitfab decides whether an assertion should have code and drafts it after the save.
Code is cheaper (1 credit per 10 verdicts, against 10 credits for a judge verdict) and gives the same verdict on the same replay every time. When a claim is half of each, such as the board meeting example, code can check that an email went to ana@example.com and hand that email to the judge to read.
Following the rules above is what makes code possible. “ChargeCard is called with amount_cents 24800” and “Agent charges Ana Ruiz’s card $248” can both be checked by code, but only the second still passes when the span is renamed, and only the second is an outcome the reader can check by hand.
Reading an abstained verdict
An abstained verdict means the evidence that was read does not show the part of the run the assertion checks, so nothing in it confirms or denies the claim. It counts as a fail in every pass rate, and it is shown apart from a fail so you can tell a gap in what could be seen from a mistake the agent made. Read the verdict’s reason first. It says what the evidence would have had to show. Then look for which of these caused it:
An abstained verdict is a gap in coverage, not a verdict on the agent. Fix the cause, and the next replay gets a pass or a fail you can act on.