Skip to main content

​
Overview

A scenario is the situation one run of a traced function was in: the trigger that started the run and the data available when it arrived, with a name and a description. It is built from one source trace. A scenario describes the situation only. It never says what the run did or should have done. That separation is what makes it reviewable. A person reads the situation, says what the right outcome was, and Bitfab turns that label into assertions on the source trace.
Scenarios are in beta. The Scenarios page in the sidebar carries a beta badge, and the details on this page may change as the feature settles.
Scenarios matter most for agents whose runs depend on more than their input, such as an assistant that reads a user’s messages, calendar, and contacts before it acts. The trace shows what the run looked at. The scenario records everything it had, including what it missed, so a reviewer can judge whether the run did the right thing with what was there.

​
What a Scenario Holds

Each record that Bitfab finds in one of the trace’s spans cites that span, so an assertion drafted from it can point at the evidence.

​
Creating Scenarios

A scenario is created in one of these ways:
  • Adding a trace to a dataset. The trace is linked to an empty scenario. When your organization has a scenario callback, Bitfab also starts building it.
  • Saving an assertion on a trace, or filing the trace under a category. The trace gets an empty scenario to hold its categories.
  • Asking for one. Your coding agent calls generate_scenario, or your code calls traces.generateScenario. Either starts the build in the background and returns at once.
  • Saving one directly. Your coding agent calls save_scenario, or your code calls traces.saveScenario, with the trigger and data.
A build needs something that knows your data. The usual way is a scenario callback: a URL in your own system that Bitfab calls with the trace, and that answers with the trigger, data, and records your system held when the run happened. Without one, your coding agent can read the trace and save the trigger and data with save_scenario. An empty scenario stands in until it is built. Its trace’s input and output take the place of its trigger and data, so it can still be labeled.

​
Creation status

Every scenario shows how far building it has come. Asking for a build while one is running starts nothing more. Asking again for one that failed starts it over. To rebuild a scenario that was already created, for example because your callback came back empty the first time, pass regenerate: true. That clears and rebuilds its name, description, trigger, data, and actual outcome, and keeps its categories, review status, labels, and assertions.
In Ruby pass regenerate: true, and in Go bitfab.WithRegenerate().

​
Reviewing Scenarios

Open Scenarios in the sidebar. The page has three tabs, each with a count. The Filter menu narrows the list by status, by whether the scenario was generated, by dataset, and by category. It shows generated scenarios by default, so switch it to Not generated or All scenarios to see empty ones. Move between scenarios with j and k, or the Next and Back buttons. ⌘ Enter runs the main button. The page moves on to the next scenario as soon as you act, without waiting for the save.

​
Labeling a run

1

Read the situation

Read the trigger, the conversation history, what the run didn’t have, and where the data came from. Open trace shows the source trace beside it.
2

Write the label

In the Label box, answer “What should the run have done here, and what did it do instead?” in plain words.
3

Generate assertions

Press Generate assertions. Bitfab drafts up to four assertions from your label, the trigger, the history, and the records the run saw. Records the run never saw are context only. Each draft is checked against the writing rules before it is shown.
4

Review the drafts

Set each draft to Approve, Pending, Archive, or Edit, and use Show why to read the spans it cites. Press Save & next, or Update with changes when you marked a draft for editing.
A label that gives nothing a replay could be checked against yields No assertions drafted. Accept the scenario without assertions, or rewrite the label and draft again.

​
When the scenario itself is wrong

Sometimes the problem is the scenario, not the run: a missing message, a wrong trigger, or a record that should be there. Press Needs updating and write what is wrong instead of labeling the run. The scenario leaves the inbox until it is fixed. Your coding agent finds these with list_scenarios and fixes them with save_scenario, sending a note on what it changed. The scenario then comes back to the inbox as Updated, showing what you asked for and what changed. Press Looks right to confirm the fix, or flag it again.

​
Review status

Each scenario has one review status, which the Status filter uses.

​
Opening a Trace’s Scenario

The trace panel on the Traces page and in dataset review has a Scenario button that opens the scenario built from that trace. While the scenario is being built, the button shows a moving border and becomes clickable once it is ready. It is disabled when no scenario has been built from the trace.

​
Tools and Methods

The SDKs find a scenario by its trace id. Python and Ruby use snake_case names, and Go uses GetScenario, GenerateScenario, and SaveScenario. See the Claude Code plugin reference for every tool argument and the TypeScript reference for the SDK shapes.

​
Best Practices

  • Set up a scenario callback. It is how Bitfab gets the data your system held at the time of the run, which a trace alone does not show. See Scenario callback.
  • Label the outcome, not the steps. “It should have offered the refund the customer asked for, not a store credit” drafts better assertions than a list of tool calls.
  • Flag the scenario, not the run, when data is missing. An assertion drafted from an incomplete scenario checks the wrong thing.
  • Add labeled traces to a dataset. The assertions you approve only protect the trace when an experiment replays it. See Datasets.