{serviceUrl}/api/sdk/. This page documents the wire protocol so you can integrate directly without using an SDK, or debug SDK behavior.
Base URL defaults to https://bitfab.ai.
Authentication
All endpoints require the API key in theAuthorization header:
401 Unauthorized.
Request Compression
Request bodies may be sent uncompressed or compressed withgzip (also accepted
as x-gzip) or deflate, declared with the Content-Encoding header. This is
what lets an OpenTelemetry Collector export to Bitfab with its default gzip
compression without extra configuration.
Any other Content-Encoding, including zstd, snappy, and br, returns
415 Unsupported Media Type.
A compressed body that expands beyond 8,000,000 bytes returns
413 Payload Too Large. Bitfab’s SDKs target at most 3,000,000 bytes on the
wire and never send more than 8,000,000 decompressed bytes.
Beneath that, every SDK lets a single span’s whole payload (its input, output,
contexts, prompt, and metadata together) use up to 7,800,000 carrier bytes when
its single-span request gzips below the 3,000,000-byte wire target. The carrier
is the payload re-escaped into its OTLP attribute. If the request does not
compress enough, compression is disabled, or preparing it fails, the SDK
replaces the largest fields with <unserializable: too_large_N_bytes>
placeholders until the carrier fits the 2,800,000-byte fallback budget. See
the OTel architecture for the full behavior.
The SDKs gzip their own requests once a body reaches 8,192 bytes and gzip makes
it smaller. Smaller or incompressible requests are sent uncompressed.
Set BITFAB_DISABLE_COMPRESSION to any value to send everything uncompressed.
Common Request Fields
The SDKs append these fields to most request bodies:Endpoints
POST /api/sdk/functions/lookup
Look up a function by name. Blocking.
Request:
{ "id": null } - SDKs raise a typed error pointing users to /functions.
POST /api/sdk/functions/{functionId}/traces
Record a trace produced by a server-side function execution (e.g. client.call(...)). Fire-and-forget.
Request:
200 OK.
POST /api/sdk/externalSpans
Record a single span produced by withSpan / @span / Span / bitfab_span. Fire-and-forget.
Request:
experimentId attributes the span to the experiment a replay is running under. The server
still accepts the older name testRunId in its place, and uses experimentId when both
are present.
Response: 200 OK.
POST /api/sdk/otel/v1/traces
Record a batch from the TypeScript, Python, Ruby, or Go SDK, or from an OpenTelemetry
Collector, using the standard OTLP/JSON ExportTraceServiceRequest shape.
Those SDKs enrich each OTel span with two string attributes:
bitfab.operation:external_span,external_trace, orinternal_tracebitfab.payload: the JSON-encoded Bitfab replay/rendering payload
externalSpans, externalTraces, and functions/{functionId}/traces. Because
OTLP carriers have no URL to read path parameters from, an internal_trace
carrier holds that route’s body plus a functionId field. Ordinary OTel spans
without those Bitfab attributes are ignored. A fully successful OTLP/JSON
response is {}. If one or more enriched spans are malformed or unsupported,
valid carriers in the same request are still ingested and the response uses the
standard partialSuccess object with rejectedSpans and errorMessage.
Those SDKs keep each request at most around 3 MB even when OTel’s
count-based processor produces a larger candidate batch. Direct requests also
contain at most 128 carriers in TypeScript or eight in Python, Ruby, and Go. Set
BITFAB_OTEL_MAX_REQUEST_BYTES to a positive integer no greater than
3000000 to use a smaller request target. An oversized carrier gets its own
request. The SDK tries compression and, if needed, trims the largest payload
fields to placeholders before reporting an export failure if the carrier still
cannot fit. HTTP 413 responses from ingress are reported as export failures.
See Batching and payload size
for the wire limits and trimming behavior.
When configuring a Collector, set its otlphttp exporter endpoint to
https://bitfab.ai/api/sdk/otel, use JSON encoding, and supply the Bitfab API
key as an Authorization: Bearer ... exporter header. The exporter appends
/v1/traces.
See OpenTelemetry Transport Architecture
for the carrier model, direct and Collector flows, size partitioning, and
replay persistence barrier.
POST /api/sdk/externalTraces
Record the completion of a root span’s trace. Sent once per trace when the outermost span ends. Fire-and-forget.
The request’s top-level id is the canonical Bitfab trace UUID. The nested raw trace ID remains an ingestion correlation value.
Request:
name is the trace’s title: what Bitfab shows for the trace, and a field search_traces and the trace list filter match on. Unset, the trace is titled by its trace function key. workflow_name is accepted as a legacy alias of name (older SDKs sent the trace function key there, and the OpenAI Agents SDK emits it natively); when both are present, name wins.
ingestion_type records how the trace came to exist and is stored on the trace’s ingestion_type column. seeded marks a trace written by seedTrace / seed_trace from a case rather than produced by production traffic. It is read back on replay item selection, and a seeded trace is refused a database branch because it carries no snapshot pin. An absent, unknown, or malformed value resolves to captured, so a trace from an older SDK is never mistaken for a seeded one. rawTrace is accepted as an alias of externalTrace.
GET /api/sdk/sim-plan
Read the organization’s sim plan decisions about content capture, as the SDKs do in the background. Blocking, and safe to call once a minute. The SDKs give the read a five second timeout and send it over the same kept-alive connections as their span uploads.
Response:
save_sim_plan MCP tool) are listed, so every entry carries captureContent false; a node absent from the list has content capture on. A node whose content capture is locked (recorded by a framework integration, the root of its traces, imported from another platform, or named by Bitfab) is never listed, even when a stale decision for it is stored, so the SDKs never strip it. A node is matched by the trace function key of the trace’s root span plus the span’s span_data.name. When a node has captureContent false, the SDK does not send its spans, except a trace’s root span, a span recorded by a framework integration, and a span that recorded an error. Children of a span that is not sent keep their original parent id, and also carry nearest_captured_ancestor_id, the id of the closest span above them in the recorded call chain whose capture is on. The SDK resolves it from its own span stack while the trace runs and sends it only when it differs from parent_id, so a span whose own parent was sent carries no redirect. Bitfab nests the span under that ancestor. A span without the field, from an SDK older than 0.59.0 or from an import, is placed by timing instead. A capture-off span that is sent goes without span_data.input, input_meta, output, output_meta (and Python’s input_serialized and output_serialized), with span_data.content_off_by_simulation_plan set to true. A node declared in code and a nested trace root ignore the plan and are always sent with their content and no marker. The server stores that mark on the span’s content_off_by_simulation_plan column, so a span with no payload because of the sim plan is never counted as a span that recorded nothing. The span tree omits such spans unless they recorded an error.
Every span from the TypeScript and Python SDKs also carries a span_origin record beside span_data, holding the SDK name (bitfab.sdk.typescript or bitfab.sdk.python), its version, and instrumentation.name, which says what produced the span: span (withSpan / @span), trace (withTrace / @trace and the subtree beneath it), openai-agents, langgraph, claude-agent-sdk, or vercel-ai (the framework integrations). The server stores instrumentation.name on the span’s instrumentation column and the SDK name and version on its span_origin_name and span_origin_version columns. Spans recorded before this change have all three empty, so they resolve their instrumentation from the trace source at read time and show no SDK name or version. Spans recorded by a framework integration always keep their content, and so does a node that is the root of its traces. The sim plan refuses to turn content off for such a node, and the SDKs never strip them.
GET /api/sdk/traces/{traceId}/span
Fetch one persisted span without loading the full trace. traceId and the optional exact id selector are canonical Bitfab UUIDs. Select one of:
id={canonicalSpanId}for an exact matchname={spanName}and/ortype=llm|agent|function|guardrail|handoff|custom, with optionaloccurrence=first|last|{zeroBasedIndex}; the default islast. With both, a span must match both.type=llm&occurrence=firstis the trace’s first model call.
id together with name or type, or none of the three, returns a 400. The response includes the span’s canonical id, traceId, and parentSpanId, plus its name, type, input, output, contexts, prompt, metadata, metrics, errors, and timestamps. It never exposes ingestion source IDs.
POST /api/sdk/spans/list
Read every span of one trace in the order it ran, one page at a time. The body takes traceId and optionally spanType, spanName (exact), fields, cursor, and limit (default 50, maximum 200, or 50 when fields is set). No filter is required.
Each span carries id, traceId, parentSpanId, name, type, startedAt, and endedAt, plus one key per field named in fields (input, output, reasoning, content, errors, contexts). Content is fetched only for the fields named. The response contains spans, nextCursor, and hasMore; pass nextCursor back as cursor while hasMore is true. A trace outside the API key’s organization returns a 404.
PATCH /api/sdk/traces/{traceId}
Update a detached current trace by its canonical Bitfab ID. The body accepts appendContexts, mergeMetadata, setSessionId, and setName. Blocking: SDKs call this synchronously and surface a rejection to the caller.
POST /api/sdk/traces/search
Search the API key’s organization by caller metadata, by trace name, or by experiment. The body requires at least one of callerMetadata, name, nameContains, or experimentId, and optionally accepts traceFunctionKey, cursor, and limit (default 50, maximum 500). Every filter supplied must match.
callerMetadata is a non-empty object with at most 10 string key/value pairs, matched exactly. name matches the trace name’s words in order, so "Order ABC" also matches "Order ABC 123", and it matches on word boundaries rather than inside a word. nameContains matches any part of a trace name, including inside a word, and must be at least 3 characters, the shortest string the trigram index can serve. All three match case-insensitively, and the two name filters are capped at 256 characters. experimentId returns only the replay traces that experiment produced. A request with no filter, or a nameContains shorter than 3 characters, returns a 400.
The response contains traces, nextCursor, and hasMore. Each trace summary contains id, traceFunctionKey, name, status, createdAt, callerMetadata, experimentId, durationMs, and tokens. experimentId is the experiment that produced the trace, or null for a captured trace. durationMs is the trace’s wall-clock time, or null when it has no recorded end time. tokens is { "input", "output", "cached", "total" } (each a number or null) summed across the trace’s spans, or null when none were recorded.
POST /api/sdk/replay/start
Begin a replay session. Blocking. Timeout: 30 s on the client side.
Request:
name is an optional display name stored on the resulting experiment. limit (1-5,000, default 5) caps how many recent traces are fetched. When traceIds is present (max 100 entries) the ID list determines the count and limit is ignored; the field stays accepted because older SDKs always send a defaulted value. When datasetId or datasetIds (max 50) is present, the datasets’ full trace lists determine the count and limit is ignored. datasetIds selects several datasets in one run: the run replays the union of their traces, deduped where they overlap, is graded by the union of their graders, and is attributed to every one of them, so it appears under each dataset’s experiments. datasetId is the single-dataset spelling and is folded in when both arrive. Every dataset is validated against the organization and the replay’s trace function; a foreign or mismatched ID returns a 400. graderIds (max 100) attaches graders directly to this run, independent of the dataset’s own graders; the resulting experiment is graded by the union of these and the dataset’s runnable graders at completion. Each id must be an active grader in the same organization and trace function, or the replay returns a 400. A replay with no dataset can still carry graders this way. judgeAssertions asks the server to judge every approved assertion on each replay as that replay finishes and save it as that assertion’s agent verdict, with the evidence it cited; an assertion that already carries a verdict is left alone. The SDKs send it by default and leave it out only when the caller skips assertion judging (skipAssertionJudging, skip_assertion_judging, or SkipAssertionJudging, or --skip-assertion-judging on bitfab-replay). SDK versions released before judging became the default never send it, so their replays stay unjudged. invocation records how the replay was started and is saved on the experiment: flags holds the replay options the caller set, keyed by the bitfab-replay flag name without dashes (concurrency, dataset-ids) whether the replay came from the command or a direct SDK call, or by the option name in kebab-case when there is no flag (on-item-start); metadata and param carry only their keys, never values (at most 100 flags, each a string, number, boolean, null, or a list of those), sdk holds the SDK’s language and version, and environment is local, ci, or cloud_replay. The SDKs send it on every replay, leave it out only if building it fails in the caller’s process (the replay still starts), and leave out any option whose name contains key, token, secret, or password. It is optional, and a malformed invocation is dropped without failing the replay.
Response 200:
experimentId and experimentUrl identify the experiment this replay records into. The
response also carries them under the older names testRunId and testRunUrl, with the
same values, so SDKs released before the rename keep working.
Each items[] entry carries reference metrics from the original trace being replayed, under originalDurationMs (end-to-end wall time in integer milliseconds), originalTokens (an object with input/output/cached/total counts), and originalModel (the model id). The unprefixed durationMs, tokens, and model are deprecated aliases carrying the same original-trace values, for SDKs that predate the rename. Any field may be null when the underlying trace didn’t capture it. The replayed run’s own token usage is returned later by /api/sdk/replay/complete (see below), once its spans are persisted.
POST /api/sdk/replay/status
Read a replay’s experiment without finalizing it. The Python and Ruby SDKs use this as a
server-authoritative persistence barrier after flushing its OTel pipeline.
expectedSpanCounts is optional. When supplied, traceIds includes an entry
only after that trace has reached a final status and at least the expected
number of spans is persisted. When omitted, traceIds contains every replay
trace currently associated with the experiment, preserving the polling contract
used by older clients. The older name testRunId is still accepted in place of
experimentId.
POST /api/sdk/replay/complete
Signal that a replay session finished.
Request:
testRunId is still accepted in place of experimentId.
Response:
traceIds maps each SDK-generated replay trace id to the persisted server trace id. tokens is each replay trace’s token usage, keyed by server trace id and aggregated from the freshly-uploaded replay spans (the same source the experiments view reads), so it’s the replayed run’s cost rather than the original’s; a trace with no token data maps to null. traceCount is how many traces the server persisted for the run. The SDK maps tokens onto each item to populate its replay tokens.
GET /api/sdk/externalSpans/{id}
Fetch a specific external span by ID. Used by replay internals. Returns the rawSpan object.
Datasets
Datasets are named buckets of traces scoped to one trace function. Every endpoint below is organization-scoped through the API key. A dataset id from another organization returns a 404. Add and remove calls report partial acceptance in their response instead of failing the whole request: ids the organization does not own, or that belong to a different trace function, come back under askipped list.
The dataset object returned by every endpoint:
GET /api/sdk/datasets
List datasets. Pass ?traceFunctionKey=my-function-key to scope to one function, or omit it
for every dataset in the organization.
For bounded responses, pass limit (1–100), then send the returned nextCursor
unchanged as cursor on the next request. Keep the same traceFunctionKey filter
across pages. Results are ordered by most recently updated, with the dataset ID
breaking ties. Stop when nextCursor is null.
Paged response 200: { "datasets": [ ...dataset ], "nextCursor": "...", "hasMore": true }
Requests without limit or cursor retain the complete-list response for older
clients. Current TypeScript, Python, Ruby, and Go SDKs request bounded pages and
collect them automatically.
Response 200: { "datasets": [ ...dataset ] }
POST /api/sdk/datasets
Create a dataset, or update the one that already carries this name under the same trace
function. The upsert key is (traceFunctionKey, name). On an update, an omitted
description keeps the existing one.
Request:
{ "dataset": { ... }, "created": true }
GET /api/sdk/datasets/{id}
Response 200: { "dataset": { ... } }
GET /api/sdk/datasets/{id}/traces
The ids of every trace in the dataset, which is the same membership a replay with
datasetId selects.
Response 200: { "datasetId": "uuid", "traceIds": ["uuid", "..."] }
Pass limit (1–100) for a bounded page, then pass nextCursor unchanged as
cursor to continue. Trace IDs are ordered ascending. A cursor-only request
defaults to 100 items. Stop when nextCursor is null.
Paged response 200: { "datasetId": "uuid", "traceIds": ["uuid"], "nextCursor": "uuid", "hasMore": true }
Requests without limit or cursor retain the complete membership response.
All four SDKs fetch the pages automatically and preserve their existing return
shape. Pagination reads current membership; changes made during traversal can
affect the result.
POST /api/sdk/datasets/{id}/traces
Add traces to the dataset. Between 1 and 100 ids per call. Traces must belong to the
organization and to the dataset’s trace function; the rest are reported under
skippedTraceIds. Adding a trace that is already a member is a no-op reported under
alreadyPresentTraceIds.
Request: { "traceIds": ["uuid", "..."] }
Response 200:
POST /api/sdk/datasets/{id}/removeTraces
Remove traces from the dataset. The traces themselves are never deleted; only their
membership in this dataset is. Ids that were not members come back under
notPresentTraceIds.
Request: { "traceIds": ["uuid", "..."] }
Response 200: { "dataset": { ... }, "removedTraceIds": ["uuid"], "notPresentTraceIds": [] }
POST /api/sdk/datasets/{id}/graders
Assign graders to the dataset. Graders must belong to the organization and to the
dataset’s trace function; the rest come back under skippedGraderIds. Already-assigned
graders are reported under alreadyAssignedGraderIds.
Request: { "graderIds": ["uuid", "..."] }
Response 200:
POST /api/sdk/datasets/{id}/removeGraders
Request: { "graderIds": ["uuid", "..."] }
Response 200: { "dataset": { ... }, "removedGraderIds": ["uuid"], "notAssignedGraderIds": [] }
POST /api/sdk/datasets/{id}/rerunGraders
Re-run graders over every trace in the dataset. graderIds is optional and defaults to
every grader assigned to the dataset; a grader that is not assigned returns a 400. A dataset
with no graders assigned returns a 400. Only one re-run can be active per dataset: a request
that matches the active run’s graders joins it (joinedExisting: true), while a different
selection returns a 400 until the active run finishes. The call returns as soon as the run
is queued; poll the GET below for progress.
Request: { "graderIds": ["uuid", "..."] } (optional body)
Response 200:
GET /api/sdk/datasets/{id}/rerunGraders
The dataset’s active re-run, or the run named by ?runId=. status is one of pending,
running, completed, or errored. While running, progress carries
{ completedTraces, totalTraces, graderCount }. When completed, result carries
{ tracesGraded, gradersRun }. When errored, error carries the message.
Response 200: { "run": { ... } } or { "run": null } when nothing is active.
Assertions
An assertion says what a trace SHOULD do the next time it is replayed, as opposed to a verdict on a run that already happened. Every endpoint is organization-scoped through the API key, and a trace from another organization returns a 404. A new assertion is saved awaiting review, and only one a person has approved is checked on a replay: an assertion left in review is not judged, carries no verdict, and does not qualify its trace foronlyWithAssertions selection or count toward an experiment’s pass
rates. approvalState on the object below is "pending" (awaiting review), "approved",
or "rejected", with approvedBy and approvedAt set once someone decides. These
endpoints read and write assertions in every state, so a tool that lists them for review
sees the ones still awaiting it.
Write assertions against the ORIGINAL trace. A replay that carries none of its own reads
its original’s through the replay lineage, and the response names the original in
inheritedFrom.
The assertion object returned by every endpoint:
targetOnEvaluatedTrace names what on the trace under evaluation the assertion checks. It
is null for the whole trace, { "kind": "output" } for the final output, or
{ "kind": "span", "name": "search_flights", "occurrence": "last" } for one span.
occurrence accepts "first", "last" (the default), or a 0-based index. Targets are span
names, never span ids, because an id captured on the original resolves to nothing on the
replay. An assertion whose target cannot be found on the trace being evaluated is errored,
never passed.
GET /api/sdk/traces/{traceId}/assertions
Read a trace’s assertions. Returns { "assertions": [...], "inheritedFrom": null }, or
the original trace’s id in inheritedFrom when this trace is a replay reading its
original’s.
Each assertion also carries rewrite, the result of the latest guided rewrite check on
it, in the shape described under POST /api/sdk/traces/assertions, or null when no
check ever ran on it. Read it here after a save to see whether the check rewrote what you
sent and which other claims to save as their own assertions.
GET /api/sdk/traces/{traceId}/assertions/{assertionId}/evidence
Suggest evidence for an assertion from the selected original or replay trace. The
assertion must belong to that trace or be inherited through its replay lineage. The
response contains { "traceId", "assertionId", "evidence" }; every evidence item comes
from the selected trace and includes spanId, text, spanName, signature-only
parameters, spanType, and isMocked. text is a word-for-word quote from the field named by sourceField.
POST /api/sdk/traces/assertions
Create or edit assertions across one or many traces. A call carries 1 to 500 traces, 1 to
50 assertions per trace, and at most 1000 assertions in total.
{ "assertions": [...] }, one flat list covering every trace in the batch, each
row carrying its own traceId and a rewrite object.
guidedRewrite: true checks each entry that carries assertion text against the
writing rules after the save and, when it breaks one (two claims in one, a condition, a
value pointed at instead of stated, an id from the recorded run, wording with no clear
pass line), rewrites it into one well-formed assertion that replaces the saved text. The
rewrite keeps the outcome the caller asked for and only changes how it is written. A request without
guidedRewrite, or with false, saves the text exactly as sent. The SDKs send true
unless the caller skips the rewrite. Every entry in the batch is checked, each as its own
background job, so the response comes back as soon as the batch is saved:
status is pending for each entry being checked and not_checked for the rest (the
rewrite was off, or the entry sent no assertion text). The check usually finishes within
a minute, and GET /api/sdk/traces/{traceId}/assertions then shows rewritten,
unchanged, or check_failed (a check that could not run, or had not finished 15 minutes
after the save, which keeps the text as sent and says why in error). changes has one
sentence per change. separateAssertions holds other claims the sent text bundled, which
were not saved, so save each as its own assertion. unresolved lists problems rewording
could not fix. Saving an assertion again before its check finishes starts a new check, and
the earlier one is discarded.
Pass an entry’s id to edit an existing assertion and omit it to add a new one, so two
callers adding different assertions to one trace never overwrite each other. On an edit,
an omitted field is preserved and an explicit null clears it. An edit keeps the original
author and source. source defaults to "agent". The author is recorded from the API
key’s user and is required, so every assertion has one.
The whole batch is written in one transaction. A trace the organization does not own, an
id that names an assertion on a different trace, the same id sent twice, or a total
over the 1000-assertion cap rejects the request with a 400 and writes nothing.
POST /api/sdk/traces/{traceId}/assertions/archive
Archive assertions by id. Archiving hides them from every read and keeps the rows for
audit; it never deletes a trace or an assertion outright.
{ "archived": ["uuid"] }.
GET /api/sdk/traces/{traceId}/scenario
Read the scenario built from a trace. A scenario is one situation a trace function’s run
was in: the trigger that started the run and the data available when it arrived, with a
name and a description, built from one source trace. It describes the situation only,
never what the run did or should do. Returns { "scenario": {...} } with name,
description, trigger, data, actualOutcome (or null), triggerSummary, and
dataSummary. A trace with no scenario returns a 404.
POST /api/sdk/traces/{traceId}/scenario
Update the scenario built from a trace. Send at least one field.
name, description, trigger, and data each replace the saved value whole, and fields
left out are kept. trigger and data keep the shape the GET returns. appendData takes
{ "history"?: [...], "missing"?: [...] } and adds those entries after the saved ones
without resending the data, so appends made at the same time all land. Send data or
appendData, not both. A history entry’s source names the span its text came from, on
this trace or another trace linked to the same scenario, and may be left out or null; a
source naming a span on any other trace is refused. The server rebuilds triggerSummary and dataSummary from what is saved.
actualOutcome records what the run actually did, read from the trace by the caller:
{ "summary": "...", "actions": [{ "name", "details", "result", "failed", "at" }], "source"?: {...} }.
Each action has a name, its arguments as details, what came back as result (or null),
whether it failed, and when it happened as at (ISO 8601, or null). It is one example of
how the run handled the situation, never what it should have done, and Bitfab never fills it.
It replaces the saved value whole, null clears it, and leaving it out keeps it.
Returns { "scenario": {...} }. A trace with no scenario returns a 404.
POST /api/sdk/traces/verdicts
Record pass/fail verdicts, 1 to 200 per call. These are the same writes the
save_agent_verdicts MCP tool performs, reachable from a replay process rather than a
coding-agent session.
assertionId. Verdicts are only
written on assertions, never on a whole trace, so an entry without assertionId is a 400
unless it is { "archive": true }, which may omit it to clear an old whole-trace agent
verdict. A verdict counts as soon as it is written: verdicts have no approval step.
verdict is "pass", "fail", or "abstained". A "pass" or "fail" needs an
annotation, and may carry confidence and evidence.
Key a direct verdict by traceId. Key a replay verdict by originalTraceId plus the
top-level experimentId (the older name testRunId is still accepted), adding attempt when the experiment ran each trace more than once;
the server resolves it to the replay trace through the lineage, so a caller never needs a
server-generated replay trace id. { "skip": true } withholds a verdict and
{ "archive": true } clears a previous one. Use skip for an attempt that crashed, was
punctured, or fell back to a schema default, and for an assertion whose target could not
be resolved, where a FAIL would read as a behavior regression rather than a check that never
ran. { "verdict": "abstained" } records an abstention. An abstention is the judge’s third answer: the evidence it read does not show the part of the run the assertion checks, so nothing can confirm or deny it. It counts as a fail in every pass rate, since anything short of an evidenced pass is a fail, but it is stored and read back apart from a fail, so a reader sees judge coverage rather than the agent’s behavior. It is not a skip: a skip is for a check that never ran, an abstention is for a check that ran and found nothing to judge. annotation beside it
says what the evidence would have had to show.
Returns { "verdicts": [{ "key": "uuid", "traceId": "uuid", "action": "set" }] }, where key
echoes the id you addressed the verdict by and action is one of set, skipped,
abstained, archived, or no-active-verdict.
Deprecated: the old path POST /api/sdk/traces/labels serves the same handler, and the old
body spelling (a top-level labels array, label: true or false for a pass or a fail, and
abstain: true for an abstention) is still accepted on both paths. An entry carrying both
label and verdict is a 400. Responses also carry the old labels array beside
verdicts, with no-active-label in place of no-active-verdict.
GET /api/sdk/traces/verdicts
Read verdicts back. Pass traceIds as a comma-separated list, 1 to 100 per call.
verdict, annotation, and approved describe the trace’s effective verdict.
verdictStatus is judged, skipped, or not-judged. assertions carries one entry per scored
assertion, keyed by the same assertionId the write used, so a verdict written per
assertion is readable per assertion rather than only as the passed / failed tally. An
assertion’s verdict is pass, fail, or abstained, or null when there is none, and
source is agent or human. The trace’s abstained count says how many of its failed
verdicts are abstentions. An older trace scored only as a whole, from before verdicts were
limited to assertions, returns an empty assertions.
Deprecated: GET /api/sdk/traces/labels serves the same handler. The response also carries
the same rows under labels, and each row and assertion keeps its old label (a boolean,
false for both fail and abstained), labelStatus (labeled, skipped, or
unlabeled), and labelSource fields.
POST /api/sdk/traces/verdicts/human
Record verdicts a person gave, 1 to 200 per call. These are the same writes the
save_human_verdicts MCP tool performs. Every entry names the assertion it scores with
assertionId, and an entry without it is a 400. The verdict is recorded as given by a
person, so it reads back with approved: true.
verdict is "pass" or "fail": a person never abstains. Use this only when a person
decided the verdict. An agent’s own verdicts belong on POST /api/sdk/traces/verdicts.
The call is all-or-nothing, matching POST /api/sdk/traces/verdicts. A trace id outside the
organization, the same target twice in one batch, or an assertionId that is not active on
its trace fails the whole request with a 400 and writes nothing.
Returns { "verdicts": [{ "traceId": "uuid", "assertionId": "uuid", "verdict": "pass", "action": "set" }] }.
Deprecated: POST /api/sdk/traces/labels/human serves the same handler and the old
labels / label body is still accepted. The response also carries
labels: [{ "traceId", "assertionId", "label": true, "action": "set" }].
GET /api/sdk/graderVerdicts
Read the individual verdicts each grader recorded, one row per grader per trace.
The same breakdown the get_grader_verdicts MCP tool returns.
traceIds (1 to 100, comma separated) to see every grader’s verdict on those traces,
graderId to see one grader’s most recent verdicts across traces (newest first), or both to
narrow. Passing neither is a 400. limit applies only when reading by grader alone, and
defaults to 50 with a maximum of 200.
Returns { "verdicts": [{ "traceId": "uuid", "graderId": "uuid", "graderName": "...", "graderStatus": "active", "graderType": "external", "verdict": "fail", "reason": "...", "failureDiagnostic": "...", "confidence": "High", "source": "live_grader", "evaluatedAt": "..." }] }. verdict is pass, fail, or null. source is human for a verdict a person recorded on the grader and live_grader for one a grader run produced; it is a different enum from the source on trace verdicts.
POST /api/sdk/graderVerdicts
Record verdicts for external graders, the graders Bitfab never runs, 1 to 200 per call.
These are the same writes the save_grader_verdicts MCP tool performs.
verdict is "pass" or "fail". The call is all-or-nothing, and a grader Bitfab runs
fails the whole request with a 400. Returns
{ "verdicts": [{ "graderId": "uuid", "traceId": "uuid", "verdict": "fail", "action": "set" }] }.
Deprecated: /api/sdk/graderLabels serves the same GET and POST handlers, and the old
labels / label body is still accepted. Responses also carry the old labels array beside
verdicts: on GET in today’s shape (label, labelReason, labelConfidence), on POST as
{ "graderId", "traceId", "label": false, "action": "set" }.
Experiment rollups
GET /api/sdk/experiments/{experimentId}/rollup and
GET /api/sdk/experiments/rollups?experimentIds=uuid,uuid carry the verdict tallies under
rollup.verdicts, rollup.graderVerdicts, rollup.assertionVerdicts,
rollup.categories[].verdicts, and rollup.graders[].verdicts, and the count of finished
traces still waiting for a verdict under totals.awaitingVerdicts. Each tally counts
originalWithoutVerdict, the traces whose original has no verdict. Deprecated: the same
tallies are also sent under the old names rollup.labels, rollup.graderLabels,
rollup.assertionLabels, categories[].labels, graders[].labels,
totals.awaitingLabels, and originalUnlabeled.
Serialization
- Inputs and outputs in
span_data.input/span_data.outputare serialized using superjson by the TypeScript SDK. Theinput_meta/output_metafields are the superjson meta descriptors. - Python, Ruby, and Go SDKs use JSON-compatible fallbacks. Objects that don’t round-trip through JSON are coerced to string via the SDK’s
serializehelper.
Error Responses
SDKs swallow all non-2xx responses on fire-and-forget endpoints and log to stderr. Blocking endpoints (
lookup, replay/start) raise typed errors.