Skip to main content
This page is for whoever writes a scenario’s contents: a coding agent calling save_scenario, your code calling traces.saveScenario, or a scenario callback answering Bitfab. Read it before you save one. Scenarios covers how scenarios are created and reviewed in the product. Every example on this page is one you can copy. Each part of the scenario has a Good example and a Not this example, with the reason the second one is wrong.

What a scenario is for

A scenario is one test case for a traced function. It holds two things:
  1. The situation a run starts in: the trigger that starts it, and the data it has when the trigger arrives.
  2. What is supposed to happen in that situation.
Assertions are written from what is supposed to happen, and every replay of the trace is judged against them. That is the whole job of a scenario: to give assertions something exact to check, and a situation that stays the same from one replay to the next. A scenario is not a record of what a run did. The trace already records that. The scenario records what should happen, so a replay of changed code can be checked against it. A good scenario passes two tests:
  • Every assertion can be written from it alone. The names, times, and values an assertion needs are in the trigger, history, or records.
  • A replay given its trigger and data faces the same situation the original run did. Nothing in it comes from after the trigger arrived.

The parts of a scenario

The example used on this page

A scheduling assistant gets a text from the executive it works for, Maya Chen, on Tuesday, October 6, 2026 at 11:42am Pacific: “Can you push my 3pm with Dana to tomorrow, same time?” Maya’s calendar has a 3pm meeting with Dana Ortiz that day, and a Board prep meeting on Wednesday from 2:30pm to 4pm. An earlier text from Dana said she is out Wednesday mornings. What should happen: Wednesday at 3pm is taken, so the assistant leaves the meeting where it is, tells Maya about the clash, and offers her a time on Wednesday when she is free. It does not contact Dana until Maya picks a time.

Name and description

The name says what the situation is, so someone scanning a list can tell scenarios apart. The description says the situation in a few sentences and then what should happen in it. Write what should happen as outcomes a replay can be checked for.
The second one describes what one run did, which the trace already shows, and the one thing it says should happen (“handle reschedules better”) is nothing a replay could pass or fail.

Trigger

The trigger is what starts the run, exactly as it arrives. Copy the content word for word. Do not summarize, correct, or tidy it, because a replay receives it and assertions quote it.
The second one paraphrases the message. “Tomorrow” is already resolved to “Wednesday”, so a replay never has to work that out, and the assertions end up checking a request nobody sent. It also claims to be the whole message when it is not the message at all.
  • eventTime is when the trigger arrived, with its time zone. Everything else in the scenario is read relative to it: “tomorrow” means the day after this.
  • type is a few plain words on what was asked, such as reschedule request or refund request.
  • contentPassages splits the content into the parts a person would read separately, such as the paragraphs of a long email. For a one-line message, it holds the content once.
  • sources lists where the content appears in the run, as { spanId, sourceField, text }. Leave it empty when you do not have span ids.

History

History is what came before the trigger, in time order: earlier messages in the thread, and facts the run has in front of it. Each entry is one fact, quoted from where it appeared.
The bad entry arrived after the trigger. A replay never sees it, so an assertion that leans on it checks something the run could not have known.
  • Quote the text as it appeared. quote is the exact text. summary restates it in a sentence when the quote alone is hard to follow, and is null otherwise.
  • Set source when the text is in the trace. Name the span with { spanId, sourceField, text }. An entry with a source can back an assertion. An entry without one is still data the run had, but an assertion cannot cite it.
  • Use role: "thread start" for the message that opened the thread, and fact for everything else.
  • Do not repeat the trigger. The trigger is not history.

Records

Records are the rows your own systems hold when the trigger arrives: messages, contacts, calendar events, memories, orders, tickets. Include every record that what should happen depends on. Here the Board prep event is the reason the meeting cannot move, so it has to be in the scenario, or no assertion can name the clash.
The first bad record is the meeting as it is today, after the original run moved it, so a replay would start from that run’s change instead of the real situation. The second one is what should happen, dressed up as a record. A replay reads records as data it has, so it would be handed the answer. What should happen goes in the description and the assertions.
  • Records are as of the trigger. Set asOf: "run" when the record is as it was when the run started. When your system keeps no history for a kind of record and you can only read it as it is today, send it with asOf: "now". Never send today’s copy of a record the run itself changed.
  • Include what bears on the trigger, not everything you have. Here that is the meeting being moved, what is on Maya’s calendar at the new time, and Dana’s contact, not every event on Maya’s calendar this month.
  • quote is a short exact string to look for in the run’s spans, such as a title. Leave it null to look for the whole text. A record found in a span cites that span, and assertions can rest on it. One found in no span is kept as context the run never saw.
  • ref is your own pointer to the row, such as Event:88213, so the record can be traced back to your system.
  • Leave out secrets. No passwords, API keys, tokens, or full payment card numbers, even when the record holds them.
Send records with recordsPath rather than records when you can, so they never pass through the conversation. Records go in a save of their own, after the scenario has a trigger.

Missing items

Missing items are what the run needs and cannot see. They keep assertions from expecting the run to use data it never has.
The bad one is a claim about data, not something missing, and nobody read Dana’s calendar to know it. An assertion built on it would expect the assistant to know Dana is free. Write each item as the thing that was not available, in a few words. Never write that something did not exist because it was not in the records.

Sources

sources says how your records were read: when the run started, when the data was captured, and for each kind of data whether it was read in full as of the run, cut to a time window, read as of capture, or not captured at all. Send it beside the records when your capture knows these things. It stops an assertion from concluding “Maya had nothing else on Wednesday” from a calendar that was only read three days out. Leave it out rather than guess. A kind with no sources is never treated as read in full.

What should happen: the assertions

Each outcome in the description becomes an assertion: one claim a replay passes or fails. Draft them with generate_scenario_assertions, passing what should happen as the note, or write them yourself with save_trace_assertions. Either way they wait as pending until someone approves them.
The first bad one restates what the original run did, which is not what a replay should do. The second has no line between pass and fail. The third checks a step on the way instead of the outcome: a replay that reads Board prep some other way and texts Maya the right thing fails it, and one that reads the calendar and still moves the meeting passes it. The fourth packs three outcomes into one, so a replay that gets two of them right still fails the whole thing.
  • One outcome per assertion, stated as what a correct run does.
  • Take every value from the scenario: the names, dates, times, and record ids in the trigger, history, and records. An assertion can only be checked against what the scenario holds. Never use an id the original run created, such as the id of a message it sent, because a replay creates its own.
  • Require an order only when the outcome depends on it. “The assistant looks up Ajay Shah before sending the brief” is a good assertion, because the brief has to be written from what the lookup found.
  • Say what must not happen when that is part of the case, such as not contacting Dana.
  • Check what the run leaves saved when that is the outcome, not only what it says. If Maya had asked the assistant to stop booking her on Wednesday mornings, the assertion is about the rule it stores: “After the turn, Maya has an active rule saying she takes no meetings on Wednesday mornings.”
  • Name a tool only when which tool is used is the outcome. If the email has to come from the user’s own mailbox and not the assistant’s, the tool is what a person would check:
Good
When the claim has an edge case a reviewer would argue about, add passCriteria or failCriteria with the assertion, such as “A calendar read that returns an access error still counts as checking the calendar.” When a program can settle the claim by reading the trace (which tools ran, the values passed to them, their order, or an exact phrase in a message), pass kind: "code" to save_trace_assertions and Bitfab writes code that decides it on every replay without an LLM judge. Leave it as a judged assertion when deciding it means reading what a message means.

What never goes in a scenario

Saving it

For a trace of any function Bitfab does not build triggers for, create the scenario with its name, description, trigger, and data in one call. Send the records in a second call once the trigger is saved, then draft the assertions. Read the scenario back with get_scenario before any later edit, so it starts from what is saved.
First call: save_scenario
Second call: save_scenario
The file at recordsPath holds the three records from Records, as an array or as { "records": [...] }.
Third call: generate_scenario_assertions
topic is one line on what the situation is about. hasHistory is true when history holds entries. To add history or missing items later without resending the data, use appendData. See the save_scenario reference for every argument.

Fixing a scenario someone flagged

When someone marks a scenario Needs updating, their note says what is wrong with the scenario itself: a missing message, a wrong trigger, a record that should be there. Fix only what the note names, using the same rules as above, and send needsUpdatingResolution in the same call saying what you changed.
A resolution describes the change to the scenario. Changing what the assistant does is a code change, and the scenario is what checks it.

Before you save

  • The description says what should happen as outcomes a replay can pass or fail.
  • Every assertion can be written from the trigger, history, and records alone.
  • Nothing restates what the original run did. The trace has that.
  • The trigger content is copied word for word, and eventTime has its time zone.
  • Every history entry and record existed before the trigger arrived, as it was then.
  • Every record the expected outcome depends on is included, whether or not the run looked at it.
  • What should happen appears only in the description and the assertions, never as a record or history entry.
  • No secrets, and no rows unrelated to the trigger.