save_agent_verdicts | Set, skip, abstain on, or archive agent pass/fail verdicts on a trace’s assertions. An abstention, for an assertion whose evidence the agent could not find, counts as a fail but reads back apart from one. Supports confidence, annotations, and { spanId, text } evidence. Every verdict needs the assertionId it scores; verdicts are never written directly on a trace | |
save_human_verdicts | Write validated human pass/fail verdicts on a trace’s approved assertions, with annotations and optional { spanId, text } evidence. Every verdict needs the assertionId it scores; verdicts are never written directly on a trace. Used by assistant fix before adding a trace to a dataset | |
save_assertion_verdicts | Post pass/fail verdicts on external assertions (kind: "external"), which Bitfab never judges because your own harness, CI, or review queue decides them. Each verdict names only the assertionId, and Bitfab finds its trace; pass experimentId (and an optional 0-based attempt) to save on that experiment’s replays. Up to 200 per call, all or nothing; an assertion that is not external or not approved fails the call. Counts as soon as it is written, and a person’s or an agent’s verdict still counts over it | |
generate_trace_assertions | Generate review-ready assertion drafts from one original trace and existing categories. Optionally focus with guidance, such as a root cause analysis, and limit to categoryIds. Waits up to four minutes, then reports it still running, and calling again checks on it without starting a second one. Reuses the prior result unless regenerate is true, and never replaces human-authored assertions | |
regenerate_assertion | Have Bitfab write one named assertion again by its assertionId, optionally steered by guidance, and save the new wording and criteria in place. Keeps the id, trace, category, assignee, human note, and justification, saves everything else as if it were a new assertion (so it loses any code, the span it was limited to and every verdict given on the old wording, and Bitfab decides afterward whether code can check it), and always returns the assertion to awaiting review | |
save_trace_assertions | Record what a trace SHOULD do when replayed: one assertion per statement, stating the answer the trace settles with its real names, dates, times, and amounts rather than a condition, with optional pass/fail criteria, target, and people-only human note, and an assigneeEmail naming the member who should review it. Saved awaiting review, and checked on replays once a reviewer approves it. Agents can edit the note but must never use it as assessment evidence. approvalState records a review decision on an assertion that already exists: { id, approvalState } alone, since an omitted assertion keeps the stored wording. Refused on a new assertion, so putting a claim into force is never the call that wrote it | |
list_organization_members | List the people in the organization with their names and emails, so assigneeEmail on save_trace_assertions carries an address someone actually holds. Takes no arguments | |
get_trace_assertions | Read the assertions and people-only human notes on up to 100 traces by ID, no span content. Approved assertions come back by default, since only those are checked on a replay; pass approvalStates (pending, approved, rejected) to read drafts awaiting review or rejected ones. Waits up to 45 seconds for an assertion still being checked against the writing rules after a save, then shows what changed. Notes are never assessment evidence; a replay with no assertions of its own reads its original’s | |
list_trace_assertions | Find assertions across traces by review state, optionally on one traced function or category, without knowing the trace IDs: pass approvalStates: ["pending"] for the review queue. Newest first, one page at a time, with a cursor that holds its place while you approve or reject what you have read | |
archive_trace_assertions | Retire assertions on a trace so later replays stop checking them. Non-destructive and all-or-nothing: one unknown, already-archived, or wrong-trace id fails the call and archives nothing | |
list_scenarios | List scenarios, newest first, with each one’s id, name, trace function, and review status (needs updating, updated, needs label, drafting, drafting failed, no assertions, needs review, done, or waiting on data), optionally for one traceFunctionKey and one queue: inbox, needs_label, needs_review, needs_updating, or done. A scenario that needs updating is printed with its note. Pass queue needs_updating to find the scenarios people flagged for fixing, read each with get_scenario, and fix it with save_scenario. Returns a cursor when more remain | |
get_scenario | Read one scenario, found by scenarioId or by a traceId it was built from (send one, not both): the situation the trace function’s run was in, as the trigger that started the run and the data available when it arrived, with a name and a description. It describes the situation only, never what the run did or should do. Returns the creation status (empty until something starts building it, generating while Bitfab builds it or waits for your scenario callback, created once built, or error with the reason), the review status and, when a person flagged the scenario, the note saying what needs updating (and the resolution once one was sent), then the trigger and data summaries, then the actual outcome when one was saved, then the name, description, trigger, data, and actualOutcome as JSON in the shape save_scenario accepts. A scenario not generated yet is returned with its review status, its categories, and the assertions on the trace it was made from, whose input and output stand in for its trigger and data. | |
save_scenario | Create or update one scenario. Send a traceId with records, or with recordsPath naming a local JSON file the plugin reads so the records stay out of the conversation, to save the records your system holds for that run: Bitfab builds only the trigger from the trace when it has no scenario yet (so far only for traces of Noah’s noah-agent-turn function) and starts the data empty, so the data holds only what you send, looks for each record’s quote (else its text) in the trace’s span inputs and outputs, and keeps a record found in no span as context the run never saw. Send sources beside records (or let recordsPath forward the file’s frameAt, capturedAt, windows, and coverage) to record how much of each kind of data was read; the scenario page and get_scenario then show which kinds were read in full as of the run, and drafted assertions never treat a record missing from the rest as proof it did not exist. Records replace the saved ones whole, need traceId rather than scenarioId, cannot be sent with other fields, and a replay is refused. To edit, find the scenario by scenarioId or by a traceId it was built from (send one, not both). name, description, trigger, and data each replace the saved value whole, and fields left out are kept. appendData adds history entries and missing items after the saved ones without resending the data, and appends made at the same time all land. Send data or appendData, not both. A history entry’s source span is optional, and one naming a span that is not on a trace linked to the scenario is refused. The trigger and data summaries are rebuilt from what is saved. An edit to a trace with no scenario, or to an unknown scenarioId, is refused. Returns the saved scenario To record what the run actually did, send actualOutcome as { summary, actions, source }: a summary in a few sentences and the actions it took, in order, each { name, details, result, failed, at }, read from the trace. It is shown below the trigger as one example, never as what the run should have done. Bitfab never fills it, and null clears it. To say the scenario itself needs fixing, send needsUpdatingNote with what is wrong. Once a flagged scenario is fixed, send needsUpdatingResolution saying what you changed, in the same call as the fix, and the scenario goes back to the person’s inbox to check. | For a trace of any other function, create the scenario by sending traceId with trigger, data, or both, in the shape get_scenario returns. The part not sent starts empty and can be sent later. |
generate_scenario | Start building the scenario for one trace and return at once; its creation status becomes generating. Read it later with get_scenario, which shows the creation status until the scenario is created or building it failed. Asking again while it is generating starts nothing more, and asking for one that failed starts it over. Bitfab builds the trigger for a noah-agent-turn trace, and your scenario callback, when set up, fills in the rest for any trace. A trace that already has a scenario is left as it is. A replay, or a trace with no way to build one (not noah-agent-turn and no callback), is refused. Adding traces to a dataset links each to an empty scenario, and starts the same build only when you have a scenario callback. | |
generate_scenario_assertions | Turn a person’s verdict and note on one scenario, found by its id or by a trace it was built from, into up to four pending agent assertions for review on the trace the scenario was built from. Each cites spans from the scenario’s trigger, history, and records the run saw; records the run never saw are context only. A note with nothing checkable yields none. A trace with no scenario, an unknown scenario id, or a scenario built from a replay is refused. A scenario not generated yet drafts from its trace’s top-level input and output and cites the trace’s top-level span. Each draft is then checked and rewritten like save_trace_assertions after the call returns, and get_trace_assertions shows what changed and any claim to save separately. | |
save_trace_categories | File one trace under categories, replacing its whole set (an empty list clears it). Categories are held by the trace’s scenario, so every trace linked to it shows the same categories. A trace with no scenario yet gets a dummy one, which save_scenario later fills in without losing its categories. | |
save_dataset | Create a dataset for a traced function (named buckets of traces for review and replay), or rename one and change its description by id | |
list_datasets | List all datasets for a traced function with trace counts and assigned graders | |
add_traces_to_dataset | Add traces to a dataset (idempotent, 1-100 per call) | |
remove_traces_from_dataset | Remove traces from a dataset without deleting the traces themselves | |
add_graders_to_dataset | Assign graders to a dataset (idempotent, organization- and function-scoped, 1-100 per call) | |
remove_graders_from_dataset | Remove grader assignments from a dataset without deleting the graders | |
save_grader | Edit an existing grader (a pass/fail check) by id: rename, clear pass/fail criteria, archive/restore, select the judge model (not for external graders). Graders are deprecated and new ones can no longer be created | |
list_graders | List the graders for a traced function, LLM-as-judge and external, with optional name search and cursor pagination (20 by default, 50 maximum; archived hidden by default) | |