save_agent_verdicts | Set, skip, abstain on, or archive agent pass/fail verdicts on a trace’s assertions. An abstention, for an assertion whose evidence the agent could not find, counts as a fail but reads back apart from one. Supports confidence, annotations, and { spanId, text } evidence. Every verdict needs the assertionId it scores; verdicts are never written directly on a trace | |
save_human_verdicts | Write validated human pass/fail verdicts on a trace’s approved assertions, with annotations and optional { spanId, text } evidence. Every verdict needs the assertionId it scores; verdicts are never written directly on a trace. Used by assistant fix before adding a trace to a dataset | |
generate_trace_assertions | Generate review-ready assertion drafts from one original trace and existing categories. Optionally focus with guidance, such as a root cause analysis, and limit to categoryIds. Waits up to four minutes, then reports it still running, and calling again checks on it without starting a second one. Reuses the prior result unless regenerate is true, and never replaces human-authored assertions | |
save_trace_assertions | Record what a trace SHOULD do when replayed: one assertion per statement, stating the answer the trace settles with its real names, dates, times, and amounts rather than a condition, with optional pass/fail criteria, target, and people-only human note, and an assigneeEmail naming the member who should review it. Saved awaiting review, and checked on replays once a reviewer approves it. Agents can edit the note but must never use it as assessment evidence. approvalState records a review decision on an assertion that already exists: { id, approvalState } alone, since an omitted assertion keeps the stored wording. Refused on a new assertion, so putting a claim into force is never the call that wrote it | |
list_organization_members | List the people in the organization with their names and emails, so assigneeEmail on save_trace_assertions carries an address someone actually holds. Takes no arguments | |
get_trace_assertions | Read the assertions and people-only human notes on up to 100 traces by ID, no span content. Approved assertions come back by default, since only those are checked on a replay; pass approvalStates (pending, approved, rejected) to read drafts awaiting review or rejected ones. Notes are never assessment evidence; a replay with no assertions of its own reads its original’s | |
list_trace_assertions | Find assertions across traces by review state, optionally on one traced function or category, without knowing the trace IDs: pass approvalStates: ["pending"] for the review queue. Newest first, one page at a time, with a cursor that holds its place while you approve or reject what you have read | |
archive_trace_assertions | Retire assertions on a trace so later replays stop checking them. Non-destructive and all-or-nothing: one unknown, already-archived, or wrong-trace id fails the call and archives nothing | |
list_scenarios | List scenarios, newest first, with each one’s id, name, trace function, and review status (needs updating, updated, needs label, drafting, drafting failed, no assertions, needs review, or done), optionally for one traceFunctionKey and one queue: inbox, needs_label, needs_review, needs_updating, or done. A scenario that needs updating is printed with its note. Pass queue needs_updating to find the scenarios people flagged for fixing, read each with get_scenario, and fix it with save_scenario. Returns a cursor when more remain | |
get_scenario | Read one scenario, found by scenarioId or by a traceId it was built from (send one, not both): the situation the trace function’s run was in, as the trigger that started the run and the data available when it arrived, with a name and a description. It describes the situation only, never what the run did or should do. Returns the review status and, when a person flagged the scenario, the note saying what needs updating (and the resolution once one was sent), then the trigger and data summaries, then the actual outcome when one was saved, then the name, description, trigger, data, and actualOutcome as JSON in the shape save_scenario accepts. A scenario not generated yet is returned with its review status, its categories, and the assertions on the trace it was made from, whose input and output stand in for its trigger and data. | |
save_scenario | Create or update one scenario. Send a traceId with records, or with recordsPath naming a local JSON file the plugin reads so the records stay out of the conversation, to save the records your system holds for that run: Bitfab builds only the trigger from the trace when it has no scenario yet (so far only for traces of Noah’s noah-agent-turn function) and starts the data empty, so the data holds only what you send, looks for each record’s quote (else its text) in the trace’s span inputs and outputs, and keeps a record found in no span as context the run never saw. Send sources beside records (or let recordsPath forward the file’s frameAt, capturedAt, windows, and coverage) to record how much of each kind of data was read; the scenario page and get_scenario then show which kinds were read in full as of the run, and drafted assertions never treat a record missing from the rest as proof it did not exist. Records replace the saved ones whole, need traceId rather than scenarioId, cannot be sent with other fields, and a replay is refused. To edit, find the scenario by scenarioId or by a traceId it was built from (send one, not both). name, description, trigger, and data each replace the saved value whole, and fields left out are kept. appendData adds history entries and missing items after the saved ones without resending the data, and appends made at the same time all land. Send data or appendData, not both. A history entry’s source span is optional, and one naming a span that is not on a trace linked to the scenario is refused. The trigger and data summaries are rebuilt from what is saved. An edit to a trace with no scenario, or to an unknown scenarioId, is refused. Returns the saved scenario To record what the run actually did, send actualOutcome as { summary, actions, source }: a summary in a few sentences and the actions it took, in order, each { name, details, result, failed, at }, read from the trace. It is shown below the trigger as one example, never as what the run should have done. Bitfab never fills it, and null clears it. To say the scenario itself needs fixing, send needsUpdatingNote with what is wrong. Once a flagged scenario is fixed, send needsUpdatingResolution saying what you changed, in the same call as the fix, and the scenario goes back to the person’s inbox to check. | For a trace of any other function, create the scenario by sending traceId with trigger, data, or both, in the shape get_scenario returns. The part not sent starts empty and can be sent later. |
generate_scenario_assertions | Turn a person’s verdict and note on one scenario, found by its id or by a trace it was built from, into up to four pending agent assertions for review on the trace the scenario was built from. Each cites spans from the scenario’s trigger, history, and records the run saw; records the run never saw are context only. A note with nothing checkable yields none. A trace with no scenario, an unknown scenario id, or a scenario built from a replay is refused. A scenario not generated yet drafts from its trace’s top-level input and output and cites the trace’s top-level span. Each draft is then checked and rewritten like save_trace_assertions, and the response says what changed and any claim to save separately. | |
save_trace_categories | File one trace under categories, replacing its whole set (an empty list clears it). Categories are held by the trace’s scenario, so every trace linked to it shows the same categories. A trace with no scenario yet gets a dummy one, which save_scenario later fills in without losing its categories. | |
save_dataset | Create a dataset for a traced function (named buckets of traces for review and replay), or rename one and change its description by id | |
list_datasets | List all datasets for a traced function with trace counts and assigned graders | |
add_traces_to_dataset | Add traces to a dataset (idempotent, 1-100 per call) | |
remove_traces_from_dataset | Remove traces from a dataset without deleting the traces themselves | |
add_graders_to_dataset | Assign graders to a dataset (idempotent, organization- and function-scoped, 1-100 per call) | |
remove_graders_from_dataset | Remove grader assignments from a dataset without deleting the graders | |
save_grader | Create or edit a grader (a pass/fail check) for a traced function: an LLM-as-judge Bitfab runs by default, or an external grader with external: true whose verdicts you post with save_grader_verdicts. Upsert by id or name, rename, clear pass/fail criteria, archive/restore, select the judge model (not for external graders) | |
list_graders | List the graders for a traced function, LLM-as-judge and external, with optional name search and cursor pagination (20 by default, 50 maximum; archived hidden by default) | |
