Ruby SDK Reference

The gem is called bitfab. It requires Ruby 3.4 or newer. It uses a global singleton client pattern (Bitfab.configure / Bitfab.client).

Framework Integrations

No framework-native adapters are shipped for Ruby yet. Instrument Ruby code manually via Bitfab::Traceable, Bitfab::Traceable.wrap, or a bound Bitfab::BitfabFunction. See the Ruby SDK guide. Framework coverage status lives in the Frameworks overview.

Module Layout

Bitfab                   # top-level module & singleton facade
Bitfab::Client           # the underlying client class
Bitfab::BitfabFunction   # fluent wrapper bound to one trace_function_key
Bitfab::Traceable        # mixin for declarative span tracing
Bitfab::Datasets         # dataset namespace reached as client.datasets
Bitfab::Traces           # assertions and scenario namespace reached as client.traces
Bitfab::Verdicts         # verdict namespace reached as client.verdicts
Bitfab::Graders          # grader namespace reached as client.graders
Bitfab::Subtree          # TracePoint subtree capture (internal)
Bitfab::CurrentSpan      # span handle
Bitfab::CurrentTrace     # trace handle
Bitfab::NoOpCurrentSpan  # returned outside a span context
Bitfab::NoOpCurrentTrace # returned outside a span context
Bitfab::Replay           # replay orchestration (internal)
Bitfab::ReplayContext    # thread-local replay context shared by same-thread fibers (internal)
Bitfab::ReplayBranch     # per-trace DB branch exposed during replay
Bitfab::DbBranchReplayError # a requested database branch could not be resolved
Bitfab::ReplayError      # whole-run replay failure carrying partial items and the experiment ID and URL
Bitfab::ReplayRegistry   # project-owned replay root registry
Bitfab::ReplayCommand    # installed bitfab-replay command implementation
Bitfab::ReplayCli        # argument parsing and execution for bitfab-replay (internal)
Bitfab::NO_MOCK_OVERRIDE # resolver sentinel that continues mock resolution
Bitfab::SpanContext      # thread-local span stack (internal)
Bitfab::TraceState       # trace state storage (internal)
Bitfab::Transport        # span/trace delivery seam (internal)
Bitfab::Otel             # OpenTelemetry batch transport (internal)
Bitfab::MOCK_STRATEGIES  # %w[none all marked]
Bitfab::BITFAB_PROGRESS_PREFIX # wire prefix used by report_replay_progress

Module-Level API

`Bitfab.configure(api_key: nil, service_url: nil, capture_enabled: nil, enabled: nil, **options)`

Creates the global client. Subsequent calls replace it.

Param	Type	Default
`api_key:`	`String, Proc, nil`	`nil`
`service_url:`	`String, nil`	`"https://bitfab.ai"`
`capture_enabled:`	`Boolean, nil`	`nil`
`enabled:`	`Boolean, nil`	Deprecated alias, `nil`
`simulation_plan:`	`Boolean`	`true`
`timeout:`	`Numeric`	HTTP seconds, `120`
`trace_across_threads:`	`Boolean, nil`	Environment setting
`db_snapshot:`	`Hash, nil`	Optional `{provider: "neon"}`
`env_vars:`	`Hash, nil`	Execution credentials
`baml_client:`	BAML client	Optional generated client
`strict:`	`Boolean`	`false`

The key is resolved lazily, at the first traced call. If you passed a Proc, it is called at that point. A missing or empty configured value falls back to ENV["BITFAB_API_KEY"]. In normal mode, a missing key warns and disables tracing. With strict: true, a missing key raises instead. capture_enabled: false leaves ordinary calls unrecorded without resolving credentials while replay and seeding still record. The deprecated enabled: option has the same behavior and warns once. When neither option is passed, ENV["BITFAB_CAPTURE_ENABLED"] controls capture: 1, true, or yes enables it; 0, false, or no disables it (case-insensitive and trimmed). The environment is read at the first traced call.

`Bitfab.client`

Returns the configured Bitfab::Client. Raises RuntimeError("Bitfab not configured...") if configure has not been called.

`Bitfab.client_or_nil`

Returns the configured Bitfab::Client, or nil if configure has not been called. The traced-call path for bitfab_span resolves the client this way, not through Bitfab.client. As a result, a method invoked before Bitfab.configure runs untraced instead of raising.

`Bitfab.reset!`

Resets the global client to nil. Intended for tests.

`Bitfab.current_span`

Returns the active CurrentSpan, or Bitfab::NO_OP_SPAN outside a span context.

`Bitfab.current_trace`

Returns the active CurrentTrace, or Bitfab::NO_OP_TRACE outside a span context.

`Bitfab.current_replay_branch`

Returns the Bitfab::ReplayBranch resolved for the replay item currently running. Returns nil outside a replay item. It also returns nil when the item’s source trace carried no DB snapshot reference. Call it inside the replayed method, then fall back to your normal connection string:

branch = Bitfab.current_replay_branch
url = branch ? branch.database_url : ENV["DATABASE_URL"]

The branch is a frozen, per-item value object. It exposes database_url, neon_branch_id, env_key, expires_at, snapshot_timestamp, provider_console_url, read_only, region, and trace_id. It is built from the thread-local replay context, so parallel items each see their own branch. Reading database_url marks the trace as having used the branch, reported as accessed. The other readers inspect the branch without exposing the connection string, and deliberately leave that mark untouched. inspect redacts the URL. Every field the service puts on the lease gets a reader under its snake_case name. A field added server-side is readable before you upgrade the SDK. database_url is the one exception. It carries the credential. It is also the only member whose read marks the branch as accessed. as_json and to_json serialize only those exposed fields. Without this, ActiveSupport would serialize the object’s instance variables by default. That would put the connection string into any Rails log line or API response that touched a branch. Because of this override, a serialized branch is safe to hand around. database_url still stays available to code that explicitly asks for it.

`Bitfab.report_replay_progress(progress)`

A ready-made replay lifecycle callback. Pass it into replay as both on_item_start: and on_item_finish:. It writes lifecycle events to stderr, prefixed with Bitfab::BITFAB_PROGRESS_PREFIX. The Bitfab plugin reads those events to identify in-flight traces and finished results. stdout stays free for direct-run ReplayResult JSON. It never raises.

`Bitfab.serialize_replay_result(result)`

Returns indented JSON. It preserves structured fields from :trace_error and :replay_error, including Bitfab::DbBranchReplayError#code, original_trace_id, and nested cause. Use it for direct-run stdout instead of JSON.pretty_generate(result). JSON.pretty_generate(result) raises when a result contains an exception object.

`Bitfab.flush_traces(timeout: 30)`

Waits for queued spans and trace completions from every live client to reach the server within one total deadline. Returns true when everything landed and false on delivery failure or timeout. Records still held awaiting a simulation plan also make it return false. Ready records are still flushed when other records remain held. Client flush and close methods report held records the same way. Repeating close preserves the first shutdown result. It does not close the clients.

`class Bitfab::Client`

`SPAN_TYPES`

Bitfab::Client::SPAN_TYPES  # => %w[llm agent function guardrail handoff custom]

Readable client state

client.api_key    # configured String/nil, or the result of the configured Proc
client.service_url
client.enabled    # effective state; lazily resolves the key
client.datasets
client.traces
client.verdicts
client.graders

api_key does not apply the ENV["BITFAB_API_KEY"] fallback. It also never warns. enabled reports the effective state. Reading it resolves the key, the same way the first traced call would. service_url, datasets, traces, verdicts, and graders are attribute readers.

`get_trace`

trace = client.get_trace(trace_id)
trace.trace_id
trace.add_context({"review" => "complete"})
trace.set_metadata({"region" => "west"})
trace.set_session_id("session-2")
trace.set_name("Reviewed case")

Returns a handle to an existing trace by its canonical Bitfab UUID. No active span is required. The ID is validated locally. Each update waits for the server response and raises when the trace is missing or inaccessible. add_context appends an entry. set_metadata merges keys into existing metadata. set_session_id and set_name replace their respective values. Non-hash context or metadata values and empty session IDs or names are ignored. A disabled client sends no updates.

`get_trace_span`

get_trace_span(trace_id, id: nil, name: nil, type: nil, occurrence: "last")

Fetches one persisted span without loading its trace. trace_id is the canonical Bitfab trace ID. Pass the span’s Bitfab id alone, or name: and/or type: ("llm", "agent", "function", "guardrail", "handoff", or "custom"); with both, a span must match both. Any other combination raises ArgumentError, "Provide id, or name and/or type", and an unknown type raises ArgumentError before the request is sent. occurrence picks among the matching spans in start order and accepts "first", "last", or a zero-based integer. It defaults to "last". Returns nil when no trace or span matches.

first_model_call = client.get_trace_span(trace_id, type: "llm", occurrence: "first")

`get_span`

get_span(span_id)

Reads one span from its span ID alone, either Bitfab’s span ID or the one the SDK sent when recording it, so no trace ID is needed. Returns the same span hash as get_trace_span, or nil when the span is not in the API key’s organization.

`traces`

Manage the assertions that describe what a trace should do, and the scenario built from it.

client.traces.get_assertions(trace_id)
client.traces.save_assertions(trace_id, assertions, source: "agent", skip_guided_rewrite: false)
client.traces.save_assertions_all(updates, source: "agent", skip_guided_rewrite: false)
client.traces.archive_assertions(trace_id, assertion_ids)
client.traces.get_scenario(trace_id)
client.traces.generate_scenario(trace_id)
client.traces.save_scenario(trace_id, name: nil, description: nil, trigger: nil, data: nil, append_data: nil)
client.traces.save_scenario(trace_id, actual_outcome: {"summary" => "...", "actions" => [...]})
client.traces.save_scenario(trace_id, category_ids: ["category-id", ...])
client.traces.save_scenario(trace_id, records: [{"kind" => "...", "ref" => "...", ...}], sources: nil)

get_assertions returns {"assertions" => [...], "inheritedFrom" => String | nil, "generation" => Hash | nil}. "generation" reports the trace’s assertion generation ("status", "error", "assertionIds", "startedAt", "finishedAt") and is nil when none ran. Each assertion carries "rewrite", the latest guided rewrite check of its text (see below), or nil when none ran. Assertions include separate "justification" and "categoryJustification" evidence plus one read-only decision ("approvalState", "approvedBy", "approvedAt") covering the assertion and its current category assignment.
"assigneeEmail" on an assertion hash names the organization member who should review it. The server resolves the address against that organization’s members and refuses one belonging to nobody in it, so a typo fails the call instead of quietly leaving the assertion unrouted. Omit the key on an edit to preserve the current assignee, and pass nil to leave it to nobody. Reads return a nullable "assignee" hash with "id", "fullName", "email", and "imageUrl". Assigning routes an assertion rather than deciding it, so it never returns the assertion to "pending" the way an edit to the wording, criteria, target, or category does.
save_assertions returns the saved assertions. Each input hash requires "assertion". Include "id" to update an existing assertion. Optional "humanNote", "passCriteria", and "failCriteria" accept strings or nil. "justification" and "categoryJustification" each accept an array of {"spanId" => ..., "text" => ...} evidence entries; omission preserves the existing value and nil clears it.
Each hash that carries "assertion" text is checked against the writing rules after it is saved, and text that breaks one is rewritten in place as one assertion. The save returns as soon as the text is saved, and each returned assertion carries a "rewrite" hash: "status" ("pending", "rewritten", "unchanged", "not_checked", or "check_failed"), "sentAssertion", "changes", "separateAssertions", "unresolved", and "error". A checked assertion comes back "pending", and Bitfab rewrites it in the background, usually within a minute. Read the result later with get_assertions, whose assertions carry the same "rewrite" hash, or nil when no check ever ran on them. A check still unfinished 15 minutes after the save reads as "check_failed", with the text as sent. "separateAssertions" holds other claims the sent text bundled. They were not saved, so save each as its own assertion. Pass skip_guided_rewrite: true to save text exactly as sent, with no check.
"targetOnEvaluatedTrace" accepts nil, {"kind" => "output"}, or {"kind" => "span", "name" => "lookup", "occurrence" => "last"}. Occurrence accepts "first", "last", or a zero-based integer.
save_assertions_all accepts [{"traceId" => trace_id, "assertions" => assertions}]. Empty updates return [] without an HTTP request. Source defaults to "agent". Pass source: "human" for assertions supplied by a person.
archive_assertions returns the archived assertion IDs.
A scenario is one situation a trace function’s run was in: the trigger that started the run and the data available when it arrived, with a name and a description, built from one source trace. Those fields describe the situation only, never what the run did or should do. get_scenario returns the scenario built from trace_id as a hash with camelCase keys ("name", "description", "trigger", "data", "actualOutcome", "triggerSummary", "dataSummary", and the rest). A trace with no scenario raises Net::HTTPError with a 404, from either method.
generate_scenario starts building the scenario for trace_id and returns at once with {"status", "traceId", "scenarioId"}, without waiting for it: "status" is "started" while Bitfab builds it in the background, or "already_generated" when the trace already has one, named by "scenarioId". Read it later with get_scenario. Bitfab builds the trigger itself for a noah-agent-turn trace, and your scenario callback, when one is set up, fills in the rest for any trace. A trace that already has a scenario is left as it is. A replay, or a trace Bitfab has no way to build a scenario for (not noah-agent-turn and no scenario callback), is refused. Adding a trace to a dataset links it to an empty scenario, and starts the same build only when your organization has a scenario callback.
save_scenario updates that scenario and returns the saved hash. name:, description:, trigger:, and data: each replace the saved value whole, and keywords left out are kept. append_data: takes {"history" => [...], "missing" => [...]} and adds those entries after the saved ones without resending the data; appends made at the same time by different callers all land. Pass data: or append_data:, not both. Passing both, or none of the eight, raises ArgumentError before any request.
records: replaces the scenario’s records whole, each {"kind", "ref", "at", "asOf" => "run" | "now", "title", "text", "quote"}, and Bitfab matches each to the span its "quote", or failing that its "text", shows up in. Records are sent on their own with only sources: ({"frameAt", "capturedAt", "windows", "coverage"}) beside them, a non-empty array is refused while the trigger is empty, and leaving sources: out clears the saved sources. Passing sources: without records:, or records: beside any other field, raises ArgumentError before any request. data: never changes the records: its "records" and "sources" keys are left out of the request and the saved ones are kept, so a data hash read back from get_scenario can be saved as it is.
actual_outcome: is what the run actually did in this one example, never what it should have done. Bitfab never fills it in, so it stays nil until you send it. It takes {"summary" => String, "actions" => [...], "source" => Hash | nil}, where each action is {"name" => String, "details" => Hash, "result" => String | nil, "failed" => true | false, "at" => String | nil} and "source" takes the same {"spanId", "sourceField", "text"} shape as a history entry’s and can be left out. It replaces the saved value whole. Passing actual_outcome: nil clears it, and leaving the keyword out keeps it. A save that passes only actual_outcome:, nil included, is accepted.
category_ids: files the scenario under those categories and replaces its whole set of categories. Leave it out to keep the current set, and pass [] to clear it. An ID outside the organization fails the call with a 400 that names the missing IDs. Both methods return the scenario’s categories under "categories", each {"id", "title", "description"}, sorted by title.
A history entry’s "source" names the span its text came from, on this trace or another trace linked to the same scenario. Leave it out or set it to nil when the text is not in such a trace. A "source" naming a span on any other trace is refused. The server rebuilds "triggerSummary" and "dataSummary" from what is saved.
client.categories.save(title, id: nil, description: nil) creates a category or updates one by ID. Omitting description: preserves it; pass "" to clear it. Categories group assertions and scenarios. They are organizational metadata and do not carry approval fields.
client.categories.get(id), .list, and .delete(id) read and manage the organization’s categories. Deleting one detaches its assertions while preserving them and their verdicts.
client.organization_members.list returns the people in the organization, each a hash with "id", "fullName", "email", and "imageUrl". It takes no arguments, because an API key belongs to one organization. Read it to find the address to pass as an assertion’s "assigneeEmail", which is refused when it belongs to nobody here. Membership reaches Bitfab through the identity provider’s mirror, so an empty list means nothing has been mirrored rather than that the organization is empty.
Assertion hashes accept "category_id": omit it to preserve the assignment or pass nil to clear it. Reads include both that ID and the nullable "category" summary.

`verdicts`

Save and read verdicts on an individual assertion. Verdicts are only written on assertions, never on a whole trace, and have no approval step.

client.verdicts.save(verdict:, assertion_id:, annotation: nil, trace_id: nil, original_trace_id: nil,
  attempt: nil, confidence: nil, experiment_id: nil, evidence: ...)
client.verdicts.save_all(verdicts, experiment_id: nil)
client.verdicts.skip(assertion_id:, trace_id: nil, original_trace_id: nil, attempt: nil,
  experiment_id: nil, annotation: nil)
client.verdicts.abstain(assertion_id:, trace_id: nil, original_trace_id: nil, attempt: nil,
  experiment_id: nil, annotation: nil)
client.verdicts.archive(trace_id: nil, original_trace_id: nil, attempt: nil,
  experiment_id: nil, assertion_id: nil)
client.verdicts.save_human(verdict:, annotation:, trace_id:, assertion_id:, confidence: nil, evidence: ...)
client.verdicts.save_human_all(verdicts)
client.verdicts.get(trace_id)
client.verdicts.get_all(trace_ids)
client.verdicts.generate_evidence(trace_id, assertion_id)

save, skip, and archive require exactly one of trace_id: or original_trace_id:. For a replay verdict, use the original trace ID with experiment_id:. test_run_id: is a deprecated alias for experiment_id:, and passing both with different values raises ArgumentError. attempt: selects a replay attempt. assertion_id: names the approved assertion the verdict scores. It is a required keyword on save and skip, and a nil or empty value raises ArgumentError before any request, since verdicts are only written on assertions. archive may omit it to clear an old whole-trace agent verdict.
save requires verdict: ("pass", "fail", or "abstained"). annotation: is the reason, required for "pass" and "fail": omitting it, or passing nil or "", raises ArgumentError before any request. Only "abstained" may omit it. Any other verdict raises ArgumentError before any request. Optional confidence is "VeryLow", "Low", "Medium", "High", or "VeryHigh", and applies to a pass or a fail.
evidence: accepts one { "spanId" => ..., "text" => ... } entry per cited span on the evaluated trace. Omit it to preserve saved evidence. Pass nil to clear it. get and get_all return the evidence with each whole-trace and per-assertion verdict.
generate_evidence generates assertion verdict evidence. Pass the original or replay trace being judged; the assertion must belong to that trace or be inherited through its replay lineage. Each candidate comes from the selected trace and contains spanId, text, sourceField (the span field the quote was copied from), spanName, signature-only parameters (names and runtime types, never values), spanType, and isMocked. Save only the candidate’s { spanId, text } fields as verdict evidence; saved evidence remains on verdicts.get and verdicts.get_all.
skip withholds a verdict when the check did not run. annotation: records why, stored on the withheld verdict and read back by get and get_all. A skip sent without it clears the note the previous verdict carried. archive clears a previous verdict. Single writes return {"key" => ..., "traceId" => ..., "action" => ...}. Action is "set", "archived", "no-active-verdict", "skipped", or "abstained".
abstain records an abstention, the same write as save(verdict: "abstained") or "verdict" => "abstained" in a save_all entry. An abstention is the judge’s third answer: the evidence it read does not show the part of the run the assertion checks, so nothing can confirm or deny it. It counts as a fail in every pass rate, since anything short of an evidenced pass is a fail, but it is stored and read back apart from a fail, so a reader sees judge coverage rather than the agent’s behavior. It is not a skip: a skip is for a check that never ran, an abstention is for a check that ran and found nothing to judge. annotation: says what the evidence would have had to show. It reads back as "verdict" => "abstained", and the trace’s "abstained" count says how many of its failed verdicts are abstentions.
save_all accepts API-shaped hashes with "traceId" or "originalTraceId", optional "attempt", "assertionId", and a verdict. "assertionId" is required on every entry except "archive" => true, and a batch missing one raises ArgumentError before any request. A verdict uses "verdict" plus "annotation" and optional "confidence", or "skip" => true with an optional "annotation", or "archive" => true. It returns the outcomes array.
save_human records a person’s verdict on the assertion assertion_id: names on trace_id:. assertion_id: is a required keyword, verdict: is "pass" or "fail", since a person never abstains, and annotation: is required, so omitting it or passing nil or "" raises ArgumentError before any request. save_human_all accepts hashes with "traceId", "assertionId", "verdict", "annotation", and optional "confidence", and a batch with a missing or empty "assertionId" raises ArgumentError before any request. Human outcomes carry "traceId", "assertionId", "verdict", and "action" => "set".
Reads use actual trace IDs, including the replay trace ID returned by a replay. get returns nil when the trace has no verdict. get_all([]) returns [] without an HTTP request.
Read results include "traceId", "verdictStatus" ("judged", "skipped", or "not-judged"), "verdict" ("pass", "fail", "abstained", or nil), "annotation", "evidence", "approved", "passed", "failed", "abstained", and "assertions". Each assertion verdict includes "assertionId", "assertion", "verdictStatus", "verdict", "annotation", "evidence", "confidence", "source", and "approved".
client.labels, Bitfab::Labels, generate_label_evidence, and the boolean label: keyword (and "label" / "abstain" entry keys) are deprecated and still work for one minor, with a one-time warning. They return the new outcome values, so archiving through client.labels when there is nothing to clear returns the action "no-active-verdict". Bitfab::Labels::LABELS_PATH, Bitfab::Labels::HUMAN_LABELS_PATH, and Bitfab::Graders::GRADER_LABELS_PATH are deprecated aliases of the new verdict paths, and require "bitfab/labels" still loads the client. Passing both label: and verdict: raises ArgumentError, and an entry with "abstain" => false and no "verdict" raises ArgumentError: send "verdict" instead. An "abstained" verdict that carries confidence or evidence, and an entry that carries both the deprecated "label" and "abstain", raise ArgumentError before any request.

`graders`

Read the verdicts a grader produced.

client.graders.get_verdicts(trace_ids: nil, grader_id: nil, limit: nil)

Pass a nonempty trace_ids: list, a grader_id:, or both. An absent selector raises ArgumentError. limit: caps results. Each returned hash carries "traceId", "graderId", "graderName", "graderStatus", "graderType", "verdict" ("pass", "fail", or nil), "reason", "failureDiagnostic", "confidence", "source", and "evaluatedAt".

client.graders.save_verdict(grader_id:, trace_id:, verdict:, reason: nil, failure_diagnostic: nil, confidence: nil)
client.graders.save_verdict_all(verdicts)

Posts verdicts for external graders: graders created with external: true, which Bitfab never runs because you evaluate traces yourself, in your own eval harness, CI, or review queue. Each verdict is stored as that grader’s own, so it counts in dataset and experiment pass rates exactly like a grader Bitfab ran, and get_verdicts reads it back. The same write as the save_grader_verdicts MCP tool. verdict: is "pass" or "fail". save_verdict_all takes up to 200 verdicts and is all-or-nothing: a grader Bitfab runs, an archived grader, a grader or trace outside the organization, a trace from another trace function, or two verdicts for the same grader and trace rejects the call and writes nothing. Saving again for the same grader and trace replaces the earlier verdict. Each entry passed to save_verdict_all is a hash with "graderId", "traceId", "verdict", and optional "reason", "failureDiagnostic", and "confidence". Each returned hash carries "graderId", "traceId", "verdict", and "action". The old get_labels, save_label, and save_label_all are deprecated and still work for one minor, returning their old shapes.

`experiments`

Read experiments back after the fact, for a dashboard or a report outside the replay process.

client.experiments.list(dataset_id: nil, trace_function_key: nil, git_branch: nil, experiment_group_id: nil, status: nil, metadata: nil, created_after: nil, created_before: nil, cursor: nil, limit: nil)
client.experiments.get(id)
client.experiments.get_rollup(id)
client.experiments.get_rollup_all(ids)

list pages newest first and returns the page hash with "experiments", "nextCursor", and "hasMore". Every filter is optional and they combine; metadata: is a hash matched by containment, and created_after: / created_before: take a Time or an ISO 8601 string. get returns the experiment hash with its provenance ("gitBranch", "commitSha", "baseSha", "experimentSha", "runBy", "completedAt", "metadata"). get_rollup returns one {"experimentId", "totals", "rollup"} hash and get_rollup_all returns an array of them for up to 100 ids in one request, or [] with no request for an empty list. "totals" counts the run’s replays by state: "total" counts one replay per trace per attempt, "traces" counts distinct traces, and "judging" counts traces, replays or originals, with an assertion verdict still being judged. "rollup" carries "traces", "verdicts", "checks", and "categories" tallies, each with "passing", "failing", "abstained" (failing verdicts the judge abstained on in every attempt), "fixed", "regressed", "stillPassing", "stillFailing", "originalWithoutVerdict", "unpaired", and "classified"; a trace or an assertion passes when at least 75 percent of its attempts passed, and "passing" / "failing" count every verdict, compared to the original or not, so their ratio is the pass rate. An id outside the organization raises on a 404, and a batch with one unknown id fails whole. These namespaces return hashes with the API’s string keys. HTTP errors propagate to the caller. The Ruby example’s scripts/test_evaluation.rb exercises assertions, verdicts, and grader reads. Set BITFAB_API_KEY, BITFAB_SERVICE_URL, and BITFAB_TRACE_ID to run it against an existing trace. Optional BITFAB_EXPERIMENT_ID and BITFAB_ATTEMPT demonstrate writing a replay verdict.

`datasets`

client.datasets.save(trace_function_key:, name:, description: nil, golden: nil)
client.datasets.list(trace_function_key: nil)
client.datasets.get(dataset_id)
client.datasets.update(dataset_id, name: nil, description: nil, golden: nil)
client.datasets.list_traces(dataset_id)
client.datasets.add_traces(dataset_id, trace_ids)
client.datasets.remove_traces(dataset_id, trace_ids)
client.datasets.add_graders(dataset_id, grader_ids)
client.datasets.remove_graders(dataset_id, grader_ids)
client.datasets.rerun_graders(dataset_id, grader_ids: nil, timeout: 90)
client.datasets.get_grader_rerun(dataset_id, run_id: nil)

Dataset operations for the authenticated organization. These are the same operations the Bitfab MCP tools expose to a coding agent. Every method returns the parsed JSON response as a Hash with string keys matching the HTTP API ("traceCount", "addedTraceIds", and so on).

save is an upsert keyed on (trace_function_key, name). "created" is true for a new dataset. It is false when an existing one was updated instead. A nil description leaves an existing description untouched.
list takes an optional trace function key. Without it, every dataset in the organization is returned.
update renames a dataset or changes its description in place and returns the updated dataset. It keeps its id, traces, and graders. A nil argument leaves that field untouched, and an empty description clears it. Passing none of name, description, and golden raises ArgumentError before any request. A blank name, or a name another dataset under the same trace function already uses, raises Net::HTTPError with a 400.
A dataset carries "id", "traceFunctionKey", "name", "description", "traceCount", "graders" ([{"id", "name"}]), "golden", "createdAt", and "updatedAt".
golden: true on save or update marks the dataset as golden for the whole organization, a trusted reference set listed first in the app and in list. golden: false unmarks it, and nil leaves it as it is. "golden" reads false from an older server.
list_traces returns {"datasetId", "traceIds"}. This is the same membership a replay with dataset_id: selects.
add_traces and add_graders accept 1 to 100 ids. They report partial acceptance instead of rejecting the whole call. Ids outside the organization, or under another trace function, come back in "skippedTraceIds" / "skippedGraderIds". Ids already present come back in "alreadyPresentTraceIds" / "alreadyAssignedGraderIds".
remove_traces never deletes a trace, only its membership in the dataset. Ids that were not members come back in "notPresentTraceIds". remove_graders reports "notAssignedGraderIds" the same way.
rerun_graders re-scores every trace in the dataset. grader_ids: defaults to every assigned grader. An unassigned id rejects the call. It blocks until the run finishes and returns {"run", "joinedExisting"} with the completed run. A run that errors raises Bitfab::GraderRerunError, whose run holds the errored run and its "error". A run still going after timeout: seconds (default 90) raises Bitfab::GraderRerunTimeoutError, a subclass of GraderRerunError that carries the last run seen. The timeout only stops the wait. The run keeps going on Bitfab, and get_grader_rerun(dataset_id, run_id:) reads it later. A request matching an in-flight run joins it instead, reporting "joinedExisting" as true.
A run has "status" (pending | running | completed | errored), "graderIds", "progress" ({completedTraces, totalTraces, graderCount} while running), "result" ({tracesGraded, gradersRun} when completed), and "error".
A dataset id from another organization raises Net::HTTPError with a 404.

`traces`

Bitfab.client.traces.search(
  caller_metadata: nil,
  name: nil,
  name_contains: nil,
  experiment_id: nil,
  trace_function_key: nil,
  cursor: nil,
  limit: nil
)

search finds traces by the caller metadata your code stamped with set_metadata, by the name it set with set_name, or by the experiment that produced them. It returns a page of stable trace summaries matching every filter supplied. caller_metadata: matches exactly and case-insensitively, with at most 10 pairs. name: matches the name’s words in order, so it also finds a name that continues past the query, and it matches on word boundaries rather than inside a word. name_contains: matches any part of the name, including inside a word, and must be at least 3 characters. Both are case-insensitive and capped at 256 characters. experiment_id: returns only the replay traces that experiment produced. At least one of caller_metadata:, name:, name_contains:, or experiment_id: is required, so a search never returns unfiltered traces. The server defaults limit to 50 and caps it at 500. The result is a Hash with "traces", "nextCursor", and "hasMore". Each entry carries "id", "traceFunctionKey", "name", "status", "callerMetadata", "createdAt", "experimentId", "durationMs", and "tokens". "experimentId" is the experiment that produced the trace, nil for a captured trace. "durationMs" is the trace’s wall-clock time, nil when it has no recorded end time. "tokens" is a Hash of "input", "output", "cached", and "total" (each an Integer or nil) summed across the trace’s spans, or nil when none were recorded. "nextCursor" is nil on the last page; pass it back as cursor: to fetch the next one.

`spans`

Bitfab.client.spans.list(
  trace_id,
  span_type: nil,
  span_name: nil,
  fields: nil,
  cursor: nil,
  limit: nil
)

list returns the spans of one trace in start order, as the same page hash with "spans", "nextCursor", and "hasMore". No filter is required, so a call with only trace_id lists every span. span_type: ("llm", "agent", "function", "guardrail", "handoff", or "custom") and span_name: (exact, case-sensitive) narrow the list. fields: picks which of "input", "output", "reasoning", "content", "errors", and "contexts" each span carries (default none, nil when that field is empty), and limit: is 1 to 200, or at most 50 when fields: is set (default 50). Pass "nextCursor" back as cursor: for the next page. An unknown trace raises on a 404. Unset arguments are left out of the request. Each span hash has "id", "traceId", "parentSpanId", "name", "type", "startedAt", and "endedAt", plus only the fields you asked for.

page = client.spans.list(trace_id, fields: ["input"])
page["spans"].each { |span| puts "#{span["name"]}: #{span["input"]}" }

`initialize(api_key: nil, service_url: nil, enabled: nil, strict: false)`

Same semantics as Bitfab.configure.

`flush(timeout: 30)`

Waits for the spans and traces this client queued to be delivered, without closing it. Returns true when everything landed inside the deadline.

`close(timeout: 30)`

Flushes, then permanently shuts down this client’s OpenTelemetry batch worker. Returns true when everything queued was delivered inside the deadline. Later submissions on the closed client are dropped with a one-time warning. A shared client does not need this. at_exit already shuts remaining transports down.

Replay registry and command

Experimental: Add --cloud to any replay command to run the same replay on your own GitHub Actions. Every replay option works as it does locally, your registry is loaded only on the runner, and the command prints the replay’s progress and a short summary of its result and exits with its exit code. Every trace’s input and output are in the experiment, where the Bitfab MCP tools read them; add --cloud-full-output to print the whole result instead. It runs the Python helper that ships with the SDK, using the one workflow bitfab-replay --cloud-init writes. See Cloud replay for setup, snapshot review, and recovery commands.

REGISTRY = Bitfab::ReplayRegistry.new
REGISTRY.register("checkout", CheckoutService.new, :run)
REGISTRY.register(
  "agent",
  AgentService.new,
  :run,
  trace_function_key: "support-agent",
  mock: "marked",
  adapt_inputs:,
  options_factory: ->(ctx) {
    {mock_override: SupportAgentMocks.build(ctx.params["scenario"])}
  }
)

Bitfab::ReplayRegistry#register stores the exact production receiver method. It also stores static per-function replay behavior and an optional options_factory:. A bitfab_span method supplies its trace function key automatically. A plain handler method must pass trace_function_key: explicitly. Pass client: to route this entry’s replay through a specific client. It defaults to Bitfab.client. Unlike Traceable.wrap’s client:, which resolves on every call, this one resolves once, when register is called. The factory receives JSON values from --params and --param. It can use them to construct executable options such as mock_override:. A direct --param value overrides one loaded from a --params file. Registered and factory-produced options are limited to these names:

limit, trace_ids, name, notes, concurrency, max_concurrency, attempts, only_with_assertions, skip_assertion_judging, dry_run
code_change_description, code_change_files
experiment_group_id, experiment_group_name, dataset_id, dataset_ids, grader_ids
mock, mock_override, adapt_inputs, db_branch, on_item_finish

An unknown option name is rejected, either when the registry loads or when the factory runs. on_item_finish is chained after the command’s progress reporter. Other lifecycle callbacks remain owned by the command. The gem installs the bitfab-replay executable. Run it as bundle exec bitfab-replay --registry <path> <pipeline> [options]. The registry module must define REGISTRY. When the registry cannot load one of its requires, the command raises ArgumentError instead of LoadError. Its message keeps the original error and adds a note that a registry belongs in the app that builds the traced method’s call. The command prints that message and exits 1. When assertion-only selection and a limit narrow an explicit ID list or dataset, the command reads approved assertions in selection order until the bound is filled. An assertion still awaiting review does not make a trace eligible. Inherited assertions do not make a replay trace eligible. A failed assertion read aborts selection. The server revalidates the pinned IDs, so assertions changed concurrently can still reduce the final count.

Flag	Value	Effect
`--limit`	`N`	Most traces to select (default 10). It bounds whatever selection is in effect, whether that selector came from the command line or the registry, keeping explicit IDs in caller order and dataset members in ID order. With assertion-only selection, it counts eligible traces before applying the bound
`--trace-ids`	`id1,id2`	Replay exactly these traces. Mutually exclusive with `--dataset-ids`
`--dataset-ids`	`uuid1,uuid2`	Replay the membership of one or more datasets, as their deduped union. Mutually exclusive with `--trace-ids`. `--dataset-id` is the same flag
`--name`	`NAME`	What this run is testing, such as `baseline`. Bitfab already records the commit, branch, and datasets
`--notes`	`TEXT`	Run conditions Bitfab cannot see, such as a forced feature flag
`--concurrency`, `--max-concurrency`	`N`	Items in flight at once
`--attempts`	`N`	Attempts per source, from 1 to 100
`--primitive`	`async` or `process`	Execution mode
`--memory-throttle`, `--no-memory-throttle`		Configure process memory admission
`--only-with-assertions`		Select sources with approved own assertions
`--skip-assertion-judging`		Don’t judge approved assertions on each replay. Judging is on by default and costs model calls. Omitting it leaves a registry-set `skip_assertion_judging` in place
`--dry-run`		Resolve inputs without executing customer code
`--fail-on-error`		Exit 1 after printing the result when any replayed item errored, printing `[replay] N of M replayed items errored` to stderr. Without it, errored items are reported in the summary and the command still exits 0. Under `--dry-run`, items whose inputs failed to resolve count
`--experiment-group-id`	`UUID`	Add this run to an experiment group. A new UUID creates the group
`--experiment-group-name`	`NAME`	Name the experiment group when this run creates it or it has no name yet. A group that already has a name keeps it. Requires `--experiment-group-id`
`--grader-ids`	`id1,id2`	Attach graders to this run only
`--code-change`	`PATH`	Load a code-change description from a file
`--no-code-change`		Record no code change, overriding a registry default
`--mock`	`none\|all\|marked`	Which recorded child spans return their historical output
`--db-branch`, `--no-db-branch`		Turn per-item database branching on or off
`--params`	`PATH`	JSON file of values passed to `options_factory:`
`--param`	`name=value`	One value passed to `options_factory:`. Repeatable, and overrides `--params`
`-h`, `--help`		Print usage

As soon as the server creates the experiment, the command prints [replay] Experiment <id>: <url> to stderr. If the run later fails with Bitfab::ReplayError, it prints Experiment <id>: <url> again before the error message, so the experiment ID is never lost.

`register_mock_override`

register_mock_override(override)
register_mock_override(match, value)
register_mock_override(trace_function_key, resolver_or_override)
register_mock_override(match: matcher, value: value)

Registers an instance-scoped override for every subsequent replay. A keyed resolver runs only for child spans with that trace function key. A keyed { match:, value: } hash also applies its own matcher. A global resolver can route on ctx[:node][:trace_function_key]. Return Bitfab::NO_MOCK_OVERRIDE to continue to lower-priority overrides and the base mock strategy. nil remains a valid mocked output. Per-call overrides take precedence over registered overrides.

`clear_mock_overrides`

Removes every override registered on this client.

replay(receiver, method_name, attempts: 1, only_with_assertions: false, skip_assertion_judging: false, dry_run: false, trace_function_key:, limit: nil, trace_ids: nil, name: nil, metadata: nil, max_concurrency: 10, code_change_description: Replay::CODE_CHANGE_UNSET, code_change_files: Replay::CODE_CHANGE_UNSET, mock: "marked", mock_override: nil, adapt_inputs: nil, db_branch: nil, experiment_group_id: nil, experiment_group_name: nil, dataset_id: nil, dataset_ids: nil, grader_ids: nil, on_item_start: nil, on_item_finish: nil, on_progress: nil, on_experiment_start: nil)

Param	Type	Description
`receiver`	`Object` or `Class`	Instance for instance methods. Class for class methods
`method_name`	`Symbol`	Method to replay
`trace_function_key:`	`String`	Trace function key for this method. It must match the key the method is traced under, via `bitfab_span` or `Bitfab::Traceable.wrap`. A contradicting key raises `ArgumentError`
`limit:`	`Integer, nil`	Max recent traces to replay (default `5`, maximum 5,000). Ignored when `trace_ids:` or a dataset selector is passed
`trace_ids:`	`Array<String>, nil`	Optional explicit trace IDs. Max 100. Passed alongside `dataset_id:` or `dataset_ids:` it pins which members of that selection replay, and the server rejects any ID none of those datasets contains. The ID count determines how many traces replay
`name:`	`String, nil`	Optional display name for the resulting experiment
`notes:`	`String, nil`	Optional run conditions Bitfab cannot see on its own, such as an environment override or a forced feature flag. Kept on the experiment next to its name (max 10,000 characters)
`metadata:`	`Hash, nil`	Your own string tags on the experiment, such as which schedule launched it (`--metadata KEY=VALUE`, repeatable, on the replay command). Keys match `^[A-Za-z0-9_.:-]{1,40}$`, values are at most 500 characters, at most 50 keys. Read back on every `experiments` read and filtered on by `experiments.list`
`max_concurrency:`	`Integer, nil`	Positive maximum parallel threads (default `10`, `nil` means unlimited). Other values raise before an experiment starts
`code_change_description:`	`String, nil`	Optional rationale for the code change being tested. It is stored on the experiment. When supplied by itself, it is preserved. Files are still captured automatically in that case
`code_change_files:`	`Array<Hash>, nil`	Optional list of edited files, each `{ path:, before:, after: }`. Omit it for a rename-aware diff of uncommitted working-tree changes against HEAD, or of the branch against the branch it was cut from when the working tree is clean. Pass `nil` instead to suppress capture entirely
`mock:`	`String`	Mock strategy for child spans. One of `"marked"` (default), `"none"`, or `"all"`. Raises `ArgumentError` on any other value
`experimental_selective_replay:`	`Hash, nil`	Experimental `{must_run: [{trace_function_key:, span_name:}]}` manifest. Requires marked mocks and rejects overrides. See selective replay for reuse declarations and coverage reports
`mock_override:`	`Hash, Proc, Array<Hash, Proc>, nil`	Per-call `{ match:, value: }` overrides or global resolvers. The first matching result other than `Bitfab::NO_MOCK_OVERRIDE` wins. These run before registered overrides and the base `mock:` strategy
`adapt_inputs:`	`#call, nil`	Reshapes recorded positional and keyword arguments for the method’s current signature
`experiment_group_id:`	`String, nil`	Optional UUID that groups multiple replay runs into a single experiment batch. An ID no group has yet creates the group
`experiment_group_name:`	`String, nil`	Optional display name for the experiment group, applied when this replay creates the group or the group has no name yet. A group that already has a name keeps it. Requires `experiment_group_id:`; passing it alone raises `ArgumentError`
`dataset_id:`	`String, nil`	Optional UUID of the dataset this replay runs against. Mutually exclusive with `dataset_ids:`. Alone it replays the dataset’s full membership, and with `trace_ids:` only those members. Durably attributes the experiment to that dataset
`dataset_ids:`	`Array<String>, nil`	Optional UUIDs of the datasets this replay runs against. Mutually exclusive with `dataset_id:`. Replays the union of their traces, or only the selected members when combined with `trace_ids:`. Uses the union of their graders and attributes the experiment to every selected dataset
`grader_ids:`	`Array<String>, nil`	Optional UUIDs of graders attached directly to this run, up to 100. They are unioned with the dataset’s runnable graders when the experiment is graded. Each grader must be active, in the same organization and trace function. Otherwise the replay is rejected with a 400
`db_branch:`	`Boolean, Hash, nil`	Optional. `db_branch: true` requests a DB branch for each replay item, using the mirror’s own sizing. `false` or `nil` leaves branching off. Each bounded worker resolves its own branch. As a result, `max_concurrency` also bounds how many branches are live at once. `Bitfab.current_replay_branch` exposes the branch inside the replayed method. Its keys tune the branch, for example `db_branch: {min_cu: 2, max_cu: 2, warmup_sql: "SELECT 1;"}`. All of these keys are optional. Symbol or string keys both work. This matters because a hash built from JSON or YAML arrives string-keyed. `min_cu` and `max_cu` set the compute’s autoscaling floor and ceiling, from 0.25 to 56. Equal `min_cu`/`max_cu` values pin a fixed size, up to 56. This keeps items comparable to each other. An autoscaling range may not span more than 8 CU. It may also not exceed 16 CU overall. `warmup_sql` is appended to the branch’s readiness check, warming the cache before the replayed method sees the branch. That way, warm-up time is not charged to the call itself. Invalid warm-up SQL fails the branch, rather than quietly handing back a cold one. Omitted keys leave the mirror’s own defaults in place
`on_item_start:`	`#call, nil`	Optional callback fired when a worker begins each item, before replay setup or customer code runs. It carries `type: "started"`, running lifecycle totals, and the historical trace/span identity. Pair it with `on_item_finish:` to distinguish queued work from in-flight work. A raising callback never crashes the run
`on_item_finish:`	`#call, nil`	Optional callback fired exactly once per item, as it finishes. It receives a running-totals hash: `{ experiment_id:, completed:, total:, succeeded:, errored:, item: }` (plus the deprecated alias `test_run_id:`). Every invocation carries `:item`. The callback itself never represents whole-run completion. `item[:trace_id]` is the server replay `traces.id`. It is read back off the ingest response. It is surfaced once the item finishes. Its trace is flushed on finish. `item[:trace_id]` is `nil` only if that flush could not confirm delivery in time. `item[:original_trace_id]` is the original historical trace being replayed. `:source_trace_id` is a deprecated alias for it. The item also carries `input`, `result`, `original_output`, `error`, `trace_error`, `replay_error`, `duration_ms`, `tokens`, `model`, and `db_snapshot_ref`. Use this callback to render live progress while the replay runs. Its running totals only split ran-ok from errored. Pass or fail isn’t known yet at this point. A raising callback never crashes the run. Pass `Bitfab.method(:report_replay_progress)` as both lifecycle callbacks. The Bitfab plugin uses its stderr events to identify in-flight traces. It also uses them to write finished per-item result files
`on_experiment_start:`	`#call, nil`	Optional callback fired once, right after the server creates the experiment and before any item runs, with a hash `{ experiment_id:, experiment_url: }`. It does not fire on a dry run, which creates no experiment. Use it to hold the experiment ID even if the run later fails. A raising callback never crashes the run
`on_progress:`	`#call, nil`	Deprecated compatibility callback. It receives the same per-item events, plus its own legacy item-less terminal `"complete"` event. It is ignored when both `on_progress:` and `on_item_finish:` are provided

attempts: accepts integers from 1 through 100. Work is queued attempt-major. Each attempt gets its own replay trace and historical database branch. Start and finish callbacks carry the zero-based :attempt on their item. By default every approved assertion is judged on each replay as it finishes, and the verdict is saved as that assertion’s agent verdict with the evidence it cited. An assertion that already carries a verdict is left alone. Pass skip_assertion_judging: true to turn judging off, since each judgement costs model calls. Replays from SDK versions released before judging became the default never ask for it, so they stay unjudged. only_with_assertions: selects traces carrying an approved assertion. An assertion still awaiting review does not qualify a trace, because only an approved one is checked on a replay. An input adapter receives the original trace’s stored metadata as ctx[:metadata]. The SDK requests metadata only when an adapter is configured. dry_run: resolves and adapts inputs without invoking the method or provisioning database branches. It creates no experiment, so the result’s experiment_id and experiment_url are nil, and it does not require any persisted replay traces. The registry CLI exposes --attempts N, --only-with-assertions, --skip-assertion-judging, and --dry-run. The item’s :input contains the resolved positional arguments followed by a keyword-argument hash when keywords are present. A keyword-only call therefore reports [{key: value}]. Item-finish callbacks use the same shape. Returns: a Hash with the keys :items, :experiment_id, :experiment_url, and :attempts. :test_run_id and :test_run_url are deprecated aliases carrying the same values. A root returning an Enumerator is consumed inside its replay context before the item finishes. The item’s :result is the collected array. Iteration failures are reported as :trace_error. The duration includes iteration, and a database branch stays leased until iteration finishes. Each item in :items carries these fields:

:attempt, the zero-based attempt index
:ingestion_type, "captured" or "seeded", or nil against older servers
:input, :result, :original_output, :error, :trace_error, :replay_error
:duration_ms (Integer, nil), this replay’s own duration
:original_duration_ms, :original_tokens, :original_model
:db_branch_timings (Hash, nil)
:tokens ({ input:, output:, cached:, total: } or nil)
:model (String, nil)
:trace_id, :original_trace_id, :original_span_id (each String, nil)
the deprecated aliases :source_trace_id and :source_span_id
:db_snapshot_ref (Hash, nil), the source trace’s snapshot pin, if any

:trace_error retains the actual exception raised while executing the replayed trace. :replay_error retains the actual exception raised before the method could even run, such as a database warmup or input-loading failure. :error holds the compatible message for either one. A database branch resolver failure surfaces as a Bitfab::DbBranchReplayError, carrying code, the server message, and original_trace_id. This is true both in progress JSON and in Bitfab::ReplayError#items. HTTP, timeout, and network failures use the code lease_request_failed, with the original client exception as cause. Malformed refs use invalid_snapshot_ref. Unexpected resolver failures use internal_error. :duration_ms is how long this replay took. :tokens is the replayed run’s own token usage. The trace being replayed is described separately, by :original_duration_ms, :original_tokens, and :original_model. :model is a deprecated alias for :original_model. Compare :tokens against :original_tokens to see the cost delta. :trace_id is the new replay trace’s server id. It becomes available to on_item_finish once that item’s trace has been flushed. It stays nil if delivery could not be confirmed, or against an older server. :original_trace_id and :original_span_id identify the original trace and root span being replayed. A replay whose delivery Bitfab cannot confirm in time is a warning on stderr,

[bitfab] replay <original trace id>#<attempt> could not confirm all spans reached Bitfab within <N>s. The run continues; Bitfab will show this replay as capture incomplete.

, never a failure, in thread, sequential, and process concurrency alike: the run completes and the item is not marked errored. A process child forwards it to the command’s stderr as a [replay] span delivery problem for ... line. If no completed item’s trace persisted at all, or finalization fails after items settle, replay raises Bitfab::ReplayError. Its items, experiment_id, and experiment_url retain the partial result. test_run_id and test_run_url are deprecated aliases for the last two. Its cause retains the original whole-run exception. Mock strategies:

"marked": only child spans declared with mock_on_replay: true return historical output. If a selected occurrence is unavailable, the item errors without executing the real child. Default.
"none": every child span runs real code.
"all": every matched recorded child span returns its historical output. Only the root runs real code. A missing or exhausted occurrence fails the item closed.

Child calls are matched by trace_function_key and span name. Repeated calls with the same key and name are distinguished by call order. Calls that share a trace function key but have different span names are tracked independently.

Replay auto-capture environment variables

Variable	Effect
`BITFAB_CODE_CHANGE_PATH`	Path to a JSON file of `{ "description", "files" }`, written by the Bitfab plugin. `replay` reads it instead of running `git diff` when `code_change_files:` is omitted
`BITFAB_CODE_CHANGE_BASE`	Git ref whose merge-base to diff against instead of `HEAD` (for example `origin/main`), so committed branch changes are included
`BITFAB_DISABLE_CODE_CHANGE_CAPTURE`	Set to opt every replay in this process out of automatic code-change capture
`BITFAB_REPLAY_RESULT_PATH`	When set by the Bitfab plugin, `replay` writes the full final `ReplayResult` JSON to this path automatically, in addition to its return value

`get_function(trace_function_key)`

Returns a Bitfab::BitfabFunction bound to trace_function_key. Mirrors client.get_function in the Python SDK and client.getFunction in TypeScript.

`node(klass, method_name, name: nil, type: "custom", capture: true, experiment_id: nil, mock_on_replay: nil, finalize: nil)`

Client-bound external-class form of bitfab_node. The method is configured only beneath an active bitfab_trace owned by this client and otherwise runs unchanged.

`execute_span(...)` (internal)

Called by Traceable. Not intended for direct use.

Client environment variables

Variable	Default	Description
`BITFAB_API_KEY`	unset	The API key used when `api_key:` is omitted or resolves empty. Read at the first traced call, not at construction
`BITFAB_CAPTURE_ENABLED`	unset	The tracing switch when `enabled:` is left at `nil`. `1`, `true`, or `yes` traces; `0`, `false`, or `no` does not (case-insensitive, trimmed). Unset traces. An unrecognized value warns once and is ignored. Read at the first traced call, so env the app loads after the client is built still counts, and `enabled:` in code always wins

Transport environment variables

Variable	Default	Description
`BITFAB_OTEL_MAX_REQUEST_BYTES`	`3000000`	Request-size target for OTLP/JSON exports. Accepts positive integers up to `3000000`
`BITFAB_OTEL_EXPORT_CONCURRENCY`	`32`	Concurrent direct requests per export window. Accepts `1` through `64`
`BITFAB_DISABLE_COMPRESSION`	unset	Set to send every request body uncompressed instead of gzip-encoding bodies of 8,192 bytes or more that compress smaller

Commit ref environment variables

Variable	Effect
`BITFAB_DISABLE_COMMIT_REF`	Set to any value to send no `commit_ref` at all. Neither the platform variables nor `git` are consulted
`BITFAB_COMMIT_SHA`	The commit sent as `commit_ref.sha` on every root trace. Wins over every deploy platform variable and over `git`
`VERCEL_GIT_COMMIT_SHA`, `GITHUB_SHA`, `RAILWAY_GIT_COMMIT_SHA`, `RENDER_GIT_COMMIT`, `SOURCE_VERSION`, `CF_PAGES_COMMIT_SHA`, `CI_COMMIT_SHA`, `BUILD_SOURCEVERSION`, `CIRCLE_SHA1`	Read in that order when `BITFAB_COMMIT_SHA` is unset, together with the same platform’s branch and repository variables when it has them. The first one set wins, and `git` is never run

`CommitRef`

{
  "sha" => String,
  "branch" => String | nil,
  "dirty" => true | false | nil,
  "remote" => String | nil,
  "root_sha" => String | nil
}

The commit the traced code was running at, sent as commit_ref on every root trace completion. sha is the commit. branch is the checked-out branch, or nil when detached or unknown. dirty is true when the working tree had uncommitted or untracked changes, false when it was clean, and nil when the SDK could not tell, which is always the case when the ref came from environment variables rather than git. remote is the origin URL reduced to host/owner/repo with any credentials removed, so a CI checkout token never reaches the trace. root_sha is the repository’s first commit, so two checkouts of the same repository match even without a remote. Resolution order is BITFAB_COMMIT_SHA, then the deploy platform’s build variables, then git in the process’s working directory (see Commit ref environment variables). Environment resolution is synchronous and free. The git path runs once per process on a background thread, with a two second timeout per command, so it never sits on the code path that ran the traced function. A trace that completes before it lands ships without a commit_ref, and a process with neither variables nor a repository never sends one. The result, including a negative one, is memoized for the life of the process. Set BITFAB_DISABLE_COMMIT_REF to opt the process out entirely. Bitfab::CommitRef.current returns the hash, or nil while unresolved.

`class Bitfab::BitfabFunction`

Fluent wrapper bound to a single trace_function_key, returned by Bitfab::Client#get_function.

`attr_reader :trace_function_key`

`wrap(klass, method_name, name: nil, type: "custom", capture_when: "always", mock_on_replay: false, finalize: nil, experiment_id: nil)`

Wraps method_name on klass with span tracing under the bound trace_function_key. Delegates to Bitfab::Traceable.wrap.

`trace(klass, method_name, name: nil, type: "custom", max_depth: 30, max_captured_subtree_spans: 500, exclude: [], include_wrappers: false, mock_on_replay_default: false)`

Experimental. Wraps method_name as the root and captures its first-party Ruby call subtree via TracePoint. Delegates to Bitfab::Traceable.trace; the bound key is used for the root and every captured call.

`module Bitfab::Traceable`

Mixin for declarative instance-method tracing.

`include Bitfab::Traceable`

Extends the including class with ClassMethods below.

Class Methods (after `include`)

`bitfab_function(key)`

Sets the default trace_function_key for all bitfab_span declarations in this class.

`bitfab_span(method_name, trace_function_key: nil, name: nil, type: "custom", capture_when: "always", mock_on_replay: false, finalize: nil, experiment_id: nil)`

Wraps method_name with span tracing. Spans exist only where you create them. Nesting is automatic, but only between spans that already exist. One wrapper around the outermost function records a single-node trace. See Instrumentation.

Param	Type	Default
`method_name`	`Symbol`	required
`trace_function_key:`	`String, nil`	falls back to the class-level `bitfab_function` key
`name:`	`String, nil`	defaults to the qualified class and method name when the class is named
`type:`	`String`	`"custom"` (one of `SPAN_TYPES`)
`capture_when:`	`String, Symbol`	`"always"` / `:always` (default). `"nested"` / `:nested` records only when there’s an active parent span. Otherwise, it runs the method untraced. Unknown values warn once and default to `"always"`
`mock_on_replay:`	`Boolean`	`false`. When `true`, replay returns this span’s historical output, under the default `"marked"` mock strategy

finalize: receives the completed result and changes only recorded output. For an Enumerator it receives collected values after iteration finishes. A finalizer failure records finalize failed: ... without changing the caller’s return value. experiment_id: attributes the span to an existing experiment. test_run_id: is a deprecated alias for it. Supports three call styles:

Before def: registered via method_added hook, wrapped when method is defined
Inline: bitfab_span def foo ... end, type: "function" (Ruby’s def returns :foo)
After def: wraps immediately if the method already exists

Raises RuntimeError if no trace_function_key is provided and no class-level bitfab_function was set.

`bitfab_trace(method_name, trace_function_key: nil, name: nil, type: "custom", max_depth: 30, max_captured_subtree_spans: 500, exclude: [], include_wrappers: false, mock_on_replay_default: false)`

Experimental. Records the root method and every first-party Ruby method it calls, at any depth, without declaring those methods separately. The root uses name and type; descendants use their method names and type "function". Capture is active only while the root runs, including while a returned Enumerator is consumed. It follows recursive, module, inherited, define_method, singleton-class, cross-file, and same-thread child Fiber calls. An alias is named and matched by exclude using the invoked alias rather than its original definition name. Ruby standard-library, dependency-gem, Bitfab SDK, synthetic dynamic-evaluation, and child-thread methods are excluded. An installed gem containing the traced root is treated as first-party, without admitting neighboring gems or SDK internals. Blocks and lambdas are not method-call events and are not recorded. Methods whose only parameters are *args, **kwargs, and &block are treated as forwarding decorator wrappers and skipped unless include_wrappers: true. Child Fibers created during capture inherit node policies, trace identity, parenting, and depth limits through Ruby Fiber storage, with separate call stacks. Pre-existing Fibers and Fiber.new(storage: nil) do not inherit this context. After an inner capture ends, child Fibers retain any still-active outer capture, with their existing parenting and depth limits. Inherited context expires once all enclosing captures end. While a root remains active, hitting a span limit stops new capture but allows already-open spans in every Fiber to finish. Independently traced Fibers remain isolated during interleaving, including when one root finishes before another. When capture ends, already-open automatic calls in suspended Fibers are emitted once with a nil output and Subtree capture ended before the call returned as the error, unless a real error was already observed. Their inputs and parenting are retained. The SDK does not resume Fibers, and later resumption cannot add spans to the ended capture. If hot reload moves the root to an unresolved source, the last valid first-party boundary and its exclusions remain in use. A returned Enumerator defers capture completion until iteration finishes or raises. Calls already paused in child Fibers retain their inputs, parenting, node context, depth and span-budget state across that handoff. Between the initial return and consumption, the session is inactive and its deferred frames are detached from the thread dispatcher. Calls still unfinished when iteration ends receive the incomplete-capture error described above. An Enumerator that is never consumed remains pending; the SDK does not force iteration. Automatically captured methods that return Enumerators record a <unserializable: Enumerator (lazy_enumerator)> output placeholder without consuming them; the traced root records the values yielded during consumption. Each nested bitfab_trace starts an independent trace during ordinary capture, including recursion and calls using the same key. Every enclosing trace retains its own recording of the shared subtree. Every copy of a span also carries the contexts and prompt the call added, so each trace’s model and token totals include the calls that ran inside it. The enclosing span and nested root carry reciprocal trace and span links. Each trace applies its own limits and exclusions. During replay and seed execution, nested annotations remain in the managed trace. max_depth limits how deep capture goes. There is no cap on the total number of spans one trace root sends. max_spans was removed, and passing it raises ArgumentError. max_captured_subtree_spans (default 500) caps discovered calls sent with their inputs and outputs. It does not count configured nodes, nested trace roots, or spans the loaded sim plan turned capture off for, so those keep recording past it. Spans sent without inputs and outputs only because the sim plan could not be read still count toward it. Past it, further discovered calls with capture on record nothing. Hitting any limit warns once per trace function key, and when the root finishes its trace carries the flat metadata keys bitfab.truncated_by (the limits that fired, in the order max_depth, max_captured_subtree_spans) and bitfab.dropped_spans (how many calls were not recorded). exclude contains unqualified method names as strings or symbols. Automatically captured spans are observations, not replay mock targets. Use bitfab_node to make a descendant replay-mockable. A configured node records once in each enclosing trace and parents its automatically captured descendants. Mixing bitfab_span or Traceable.wrap with subtree tracing raises Bitfab::MixedTracingError. The SDK prepares the project boundary and exclusions when the method is declared, refreshes the boundary if hot reload moves the root definition, caches canonical source paths, and shares one TracePoint dispatcher across active roots on a calling thread. After a span limit is reached, it stops reading inputs and outputs immediately, then stops processing events for that session once its captured calls finish. Each active root still captures and serializes its own copy of every overlapping call. Keep this experimental API for discovery; prefer explicit bitfab_span declarations on production hot paths. Supports the same before-def, inline, and after-def declaration styles as bitfab_span:

class OrderService
  include Bitfab::Traceable
  bitfab_function "order-processing"

  bitfab_trace :process_order, type: "agent", exclude: [:healthcheck]
  def process_order(order_id)
    order = load_order(order_id)
    summarize(order)
  end
end

The wrapper preserves the root method’s public, protected, or private visibility.

`bitfab_node(method_name, name: nil, type: "custom", capture: true, experiment_id: nil, mock_on_replay: nil, finalize: nil)`

Experimental. Configures a method only when it is discovered beneath an active bitfab_trace. It never creates a standalone span or trace and inherits the active root’s trace function key, limits, parent, and lifecycle.

Param	Type	Default
`method_name`	`Symbol`	required
`name:`	`String, nil`	defaults to the qualified class and method name when the class is named
`type:`	`String`	`"custom"` (one of `SPAN_TYPES`)
`capture:`	`Boolean`	`true`; `false` omits the node while captured descendants attach to its nearest captured parent
`experiment_id:`	`String, nil`	optional experiment attribution (`test_run_id:` is a deprecated alias)
`mock_on_replay:`	`Boolean, nil`	Inherits the trace’s `mock_on_replay_default:` when omitted. Explicit `false` overrides the marked-replay default. All-mocking and overrides retain precedence. `true` reuses recorded output under marked replay
`finalize:`	`#call, nil`	transforms the recorded output without changing the method’s return value; failures are recorded on the node and do not escape

Captured nodes count toward the enclosing root’s max_depth, but not max_captured_subtree_spans. A root-level exclude wins over the node configuration for that root. Overlapping roots each record the configured name, type, experiment attribution, and finalized output. Finalization runs once per invocation. An Enumerator is finalized with its collected values after iteration finishes. capture: false applies to every active root because it is an unconditional property of the method. capture: false, mock_on_replay: true raises ArgumentError because an omitted node has no recorded output. Supports the same before-def, inline, and after-def declaration styles as bitfab_span and preserves public, protected, or private visibility. Every pending declaration applies, so declaring bitfab_trace and bitfab_node before def keeps both. A method declared with both bitfab_trace and bitfab_node, in any order, records one root span per call, under the root’s own name and type. Called inside another trace root, the outer trace records only the linked copy of the nested trace, with the method’s subtree beneath it.

Module-Level Wrapping

`Bitfab::Traceable.wrap(klass, method_name, trace_function_key:, name: nil, type: "custom", capture_when: "always", mock_on_replay: false, finalize: nil, experiment_id: nil, client: nil)`

Wraps a method on a class you do not own. Uses Module#prepend.

Param	Type
`klass`	`Class` or `Module`
`method_name`	`Symbol`
`trace_function_key:`	`String`
`name:`	`String, nil`
`type:`	`String`
`capture_when:`	`String, Symbol` (`"always"` / `:always` or `"nested"` / `:nested`). Unknown values warn once and default to `"always"`
`mock_on_replay:`	`Boolean`
`client:`	`Bitfab::Client, nil`. Route through a specific client instead of resolving the global client on every call

`Bitfab::Traceable.trace_function_key_for(receiver, method_name)`

Returns the trace function key carried by a method wrapped with bitfab_span / Traceable.wrap, or nil when the receiver method has no Bitfab wrapper. Replay and replay registries use it to reject key mismatches.

`Bitfab::Traceable.trace(klass, method_name, trace_function_key:, name: nil, type: "custom", max_depth: 30, max_captured_subtree_spans: 500, exclude: [], include_wrappers: false, mock_on_replay_default: false)`

Experimental external-class form of bitfab_trace. Uses Module#prepend; its options have the same semantics as the class macro. The guides lead with client.get_function(key).trace(klass, method_name, ...), which binds the key once, or bitfab_function inside a class you own. Passing trace_function_key: here is equally supported and is what those forms do underneath.

`Bitfab::Traceable.node(klass, method_name, name: nil, type: "custom", capture: true, experiment_id: nil, mock_on_replay: nil, finalize: nil)`

Experimental external-class form of bitfab_node. Uses Module#prepend; pass client: internally or prefer client.node(...) when configuration must belong to a specific client.

`class Bitfab::CurrentSpan`

def id        # => canonical Bitfab span ID
def trace_id  # => String
def add_context(context)  # Hash → appends one entry; ignores non-Hash; never raises
def set_prompt(prompt)    # String → overwrites; ignores non-String; never raises

`class Bitfab::CurrentTrace`

def set_session_id(session_id)  # String
def set_name(name)              # String → the trace's title, searchable; empty strings ignored
def set_metadata(metadata)      # Hash → shallow-merge, later keys win
def add_context(context)        # Hash → appends entry
def drop                        # flags the trace to be dropped at completion

Caller metadata is filterable by exact key/value pairs case-insensitively. It accepts up to 50 keys; keys use 1-40 letters, numbers, _, ., :, or -, and scalar values are at most 500 characters. All methods swallow exceptions internally. They never raise. drop flags the trace to be dropped. Once flagged, spans that complete afterward are not uploaded at all. The flag itself still rides out on the completion payload. At completion, the server scrubs any payloads that already raced out, meaning the trace itself, its external trace, and any sibling spans. It deletes the archived S3 objects. It marks the trace dropped instead of completed, keeping only a skeleton audit row. Calling drop outside a span, on NO_OP_TRACE, is a no-op.

`class Bitfab::NoOpCurrentSpan` / `NoOpCurrentTrace`

Singleton instances exposed as Bitfab::NO_OP_SPAN and Bitfab::NO_OP_TRACE. All methods are no-ops. id and trace_id return "".

Thread & Concurrency Model

Span stack is stored in fiber-local Thread.current[:__bitfab_span_stack]. Enable trace_across_threads: true to propagate it to submitted threads and thread-pool jobs. Traced methods that return an Enumerator bridge their parent span into the same-thread Enumerator.new / enum_for source fiber. This keeps nested spans and replay mocks as descendants while the stream is consumed.
Replay context lives in true thread-variables instead (Thread.current.thread_variable_get / thread_variable_set). This context includes the mock tree, the DB branch lease, and the experiment ID. Unlike the fiber-local span stack, it persists across every fiber on the same thread. The thread propagation option copies it for each submitted job. Process replay creates a fresh context from its assigned historical item.
Trace state is stored in a mutex-protected module-level hash (Bitfab::TraceState), keyed by trace_id.
Spans and trace completions are queued on the client’s private OpenTelemetry BatchSpanProcessor. They are delivered by its batch worker. Every other API call is blocking.
Bitfab.flush_traces(timeout:) waits for every live client’s queued spans. It returns false when delivery fails or the deadline expires. client.close(timeout:) shuts a single client’s worker down permanently.
Subtree capture reuses one TracePoint dispatcher per calling thread and keeps a separate method stack per fiber and active root. Nested roots share the dispatcher, but every root still filters and serializes its own copy of overlapping calls, so capture work grows with the number of overlapping roots. Same-key nested annotations and recursion also start independent roots during ordinary capture. Replay and seed execution keep nested annotations in the managed trace. Automatic TracePoint discovery stays on the root thread. The thread propagation option carries explicit spans and replay context into worker jobs.

Error Behavior Summary

Situation	Behavior
Empty `api_key`	Resolved lazily, including an `ENV["BITFAB_API_KEY"]` fallback. Without `strict`, it warns and disables tracing. With `strict: true`, the first traced call raises `RuntimeError` instead
`bitfab_span`-wrapped method called before `Bitfab.configure`	Runs untraced via `Bitfab.client_or_nil`. No error
`Traceable.wrap` / `BitfabFunction#wrap`-wrapped method called before `Bitfab.configure`, with no `client:` bound	`RuntimeError("Bitfab not configured...")`. This path resolves `Bitfab.client`, not `client_or_nil`
`bitfab_span` method raises	Span records `error` and `error_source: "code"`. The exception is re-raised
`bitfab_trace` descendant raises	Descendant records the exception; root behavior and exception propagation are unchanged
`bitfab_node` finalizer raises	Node records `finalize failed: ...`; the original method result is still returned
`add_context` / `set_prompt` with invalid input	Silently ignored. Never raises
Span transport failure	Swallowed, though it warns once. The host app never crashes. `flush_traces` returns `false`
Span submitted after `client.close`	Warned once and dropped
`Bitfab.client` before `configure`	`RuntimeError("Bitfab not configured...")`
`bitfab_span` without `trace_function_key`	`RuntimeError`
`replay`: `trace_function_key:` differs from the method’s declared key	`ArgumentError` (key mismatch)
`replay`: Bitfab never confirms a replay’s spans in time	One warning per replay on stderr; the run completes and no item is marked errored
`replay`: no completed item’s trace persisted, or finalization fails, after items settle	`Bitfab::ReplayError` with collected `items`, the experiment ID and URL, and original `cause`
`replay`: requested database branch cannot be resolved	Item `:replay_error` is `Bitfab::DbBranchReplayError` with the resolver code, message, and original trace ID

Capture policy and simulation plans

Bitfab::Client.new(capture_enabled: false) turns off ordinary recording while keeping traced methods replayable. Replay and seed execution still record when credentials are available. enabled: remains a deprecated alias; use capture_enabled: for new code. The same options work with Bitfab.configure. The SDK reads the simulation plan in the background and applies capture-off settings using the owning trace function key and span name. Once the plan has loaded, a span whose capture is off is not sent, and its children carry the nearest captured ancestor the SDK resolved while the trace ran, so Bitfab nests them under the closest call above them whose capture is on. The root span of each trace, framework-origin spans, and spans that recorded an error are still sent without content. Configured nodes and nested trace roots ignore the plan and always keep their content, and a discovered call that shares a configured node’s name is still dropped. The client starts reading the sim plan when it is built, or at first use when its API key is resolved lazily (a proc, or read from BITFAB_API_KEY). A bitfab_trace root that starts while that first read is still running waits for it, for at most five seconds after that first read began. Once the first read has finished, whether it succeeded or failed, no later root waits. Nested roots and the calls beneath a root never wait, and nothing waits when the plan is disabled. While the first read is still running, it holds records off the calling thread. When that read succeeds, held spans are sent with the plan applied. When it fails (a network error, a timeout, an HTTP error other than 404, or a response it cannot understand), held spans and every later span are sent without inputs and outputs until a read succeeds, and the SDK warns once. Those spans carry content_off_by_simulation_plan and still count toward max_captured_subtree_spans. The root span of each trace, framework-origin spans, configured nodes, and nested trace roots keep their content. Sending a span without content removes inputs, outputs, their serialized forms, and the prompt while keeping timing, errors, context, and links. A 404 means the server has no plan feature, so records are sent unchanged. A failed refresh after a successful read keeps the last plan. At most 1,000 records are held, and any beyond that are sent without inputs and outputs instead of being dropped. flush and close wait within their timeout for held records and in-progress submission, preserving span-before-completion ordering. close and process exit never drop held records. Records still held because the plan has not loaded are sent without inputs and outputs. Use simulation_plan: false or a nonblank BITFAB_DISABLE_SIM_PLAN to disable plan reads. timeout: configures the default HTTP timeout in seconds; plan reads have a separate five-second bound. Every emitted span carries span_origin with the Ruby SDK version and its instrumentation source (span, trace, or its framework adapter).

Seed and reseed

seed_trace records a supplied case or runs a callable once. Both forms work with capture_enabled: false, require an API key, mark the trace as seeded, and omit database snapshot pins.

double = ->(value) { value * 2 }
id = client.seed_trace("double", input: [3], expected: 6, fn: double,
  metadata: {row: "example"})
run_id = client.seed_trace("double", double, args: [3], name: "fresh case")
adopted = client.reseed_trace("double", double, trace_id: id)

Case inputs are positional argument arrays. Optional fn: validates the callable’s signature without executing it. Run-based seeds also accept kwargs:. Both forms accept metadata:, session_id:, and name:. Reseeding loads the stored inputs and metadata and adopts the successful run under the existing trace ID. Verdicts, assertions, and dataset memberships stay on that trace; :previous_run_trace_id identifies the previous run. A failed execution is recorded and raised without adopting it. The registry accepts plain callables with an explicit key:

REGISTRY = Bitfab::ReplayRegistry.new.register(
  "double", double, client:, trace_function_key: "double"
)

Use bitfab-seed --registry ./registry.rb double --cases cases.jsonl to record cases without execution, add --run to execute each case once, or use --from-trace id1,id2 to reseed existing traces. The replay command also accepts --seed as an alias for --cases. JSON or JSONL case rows require an input array and may include expected, metadata, sessionId, and name. Run-based cases must omit expected. The CLI rejects empty replay selections and emits the final result as JSON. Registry option on_item_finish: receives each item completion after the progress reporter. Use it to save verdicts while items finish. Callback errors are reported without aborting replay; dry runs skip the project callback. Bitfab::Client.new(db_snapshot: {provider: "neon"}) explicitly records Neon as the database snapshot provider on ordinary trace roots. Unsupported providers are rejected at construction. Without this option, roots carry only their timestamp and the server resolves the provider at replay time. This configuration performs no database I/O; seeded traces still carry no snapshot pin. Replay summaries follow TypeScript’s case semantics: seeded rows show “Matched expected” or “Missed expected” against their stored output, while captured rows show “Same” or “Changed.” For a case seed, that stored output is expected; for a run-based seed, it is the previously recorded execution result. Python currently seeds by running code and uses “Same/Changed” summaries for all rows. Errors are counted separately and are never counted as matches.

BAML execution

client.call(name, **inputs) executes a stored BAML prompt locally and records its result and native collector. Configure env_vars: to supply provider credentials. Only OPENAI_API_KEY is forwarded to the native runtime. Install the optional baml and sorbet-runtime gems. client.wrap_baml(client_or_method, method = nil, on_collector: nil) enriches the active span with the rendered prompt, model, provider, token counts, and duration. A method can be a bound Method, symbol, or string. Set baml_client: in the constructor or pass the generated client explicitly. The returned callable exposes collector. The fluent function handle forwards wrap_baml to the owning client. Collector callbacks cannot change the wrapped method’s result. client.wrap_jev(callable = nil, on_reply: nil, &block) records a Jev call (TypeSafe’s System One model) into the span that is current when the call starts: the reply’s model, provider (typesafe), inputTokens, outputTokens, and durationMs. Pass a callable or a block that returns Jev’s reply, as a Hash with string or symbol keys or an object responding to model and usage. It opens no span. on_reply receives each reply after it is recorded, and a failed call records nothing. The function handle forwards wrap_jev to the owning client. See the Jev framework guide.

Worker threads

Set trace_across_threads: true on the client or BITFAB_TRACE_ACROSS_THREADS=1 to carry span, replay, and seed context into newly submitted work. This installs process-wide hooks for Thread.new, Thread.start, and Thread.fork. If your application includes concurrent-ruby, it also hooks thread-pool post calls. SDK transport threads and pool implementation threads are excluded. A worker’s spans nest under the submitting span. Replay mocks and the item’s database lease remain available there. Each job gets its own span stack. Reused workers restore their previous context after the job finishes or raises. This option does not install automatic subtree discovery on worker threads. Every span’s runtime metadata identifies the executing thread. Propagated jobs also record the submitting thread’s ID and name. Join application work before closing the client so its spans can finish.

Process replay

Use Bitfab::ReplayConcurrency on a registry entry when each replay item needs a fresh interpreter.

REGISTRY = Bitfab::ReplayRegistry.new.register(
  "workflow", workflow, client:, trace_function_key: "workflow",
  concurrency: Bitfab::ReplayConcurrency.new(
    primitive: "process", attempts: 2, max_concurrency: 4,
    on_item_finish_in_child_process: ->(event) {
      puts event[:item][:trace_id]
    }
  )
)

Run bitfab-replay --registry ./registry.rb workflow. Process mode reloads that registry for every item and attempt. It requires this installed registry command. Calling client.replay directly with process mode raises before creating an experiment. The parent starts and completes one experiment. Each child resolves its own inputs and database lease. Its result is written after trace delivery is confirmed. The child-process hook runs afterward and receives experiment_id (plus the deprecated alias test_run_id) and the finished item. Hook errors are reported without discarding the result. Registry on_item_finish: still runs in the parent. Dry runs skip customer execution, leases, and grading hooks. primitive: "async" uses the existing thread pool and defaults to ten workers. primitive: "process" defaults to four children. attempts defaults to one and accepts 1–100. Process concurrency must be bounded. Passing the concurrency: object together with scalar attempts: or max_concurrency: is rejected. CLI --primitive, --attempts, and --max-concurrency override registry defaults. Process mode enables memory admission by default. The next launch waits when free memory cannot cover active children’s expected growth, another child, and a reserve. Critical memory pressure also delays launches, meaning the critical kern.memorystatus_vm_pressure_level on macOS or every task stalled on memory for 10% or more of the last ten seconds on Linux. Swap at 85% used delays launches only where pressure cannot be read. At least one child is always permitted to make progress. Linux and macOS use local memory readings. Other platforms continue without a memory estimate. Set memory_throttle: false or CLI --no-memory-throttle to disable it. BITFAB_REPLAY_CHILD_MEMORY_MB and BITFAB_REPLAY_MEMORY_FLOOR_MB both default to 2048. BITFAB_REPLAY_MEMORY_THROTTLE=0 disables admission for the process. child_timeout: defaults to 2400 seconds. Timed-out children are terminated and produce an item error when they have not written a result. A completed result survives a timeout in the later grading hook. Child processes receive the environment captured when the SDK loaded. Mutations made while loading the parent registry do not leak into later children. child_delivery_timeout: is how many seconds a child waits for its spans to reach Bitfab. It bounds two waits. The first runs before the result is written: the child flushes the item’s spans to read back trace_id, and if that flush does not confirm delivery, it asks Bitfab whether the trace is stored for up to the same number of seconds again. The second runs after on_item_finish_in_child_process: the child closes its client, which sends whatever is still queued, including spans the hook itself recorded, before it exits. Raise it when a large trace uploads slowly and items come back with an error or with spans missing. It defaults to 30.0 under primitive: "process", must be a positive number, and is refused under primitive: "async", since no child process exists there. Keep it well below child_timeout:. A child reports span delivery failures, such as a rejected or failed upload or delivery not confirmed within child_delivery_timeout:, as [replay] span delivery problem for <trace>#<attempt>: ... on the command’s own stderr. Repeated messages are counted instead of repeated.

Bitfab::ReplayConcurrency.new(
  primitive: "process",
  on_item_finish_in_child_process: grade,
  child_delivery_timeout: 180
)

​Framework Integrations

​Module Layout

​Module-Level API

​Bitfab.configure(api_key: nil, service_url: nil, capture_enabled: nil, enabled: nil, **options)

​Bitfab.client

​Bitfab.client_or_nil

​Bitfab.reset!

​Bitfab.current_span

​Bitfab.current_trace

​Bitfab.current_replay_branch

​Bitfab.report_replay_progress(progress)

​Bitfab.serialize_replay_result(result)

​Bitfab.flush_traces(timeout: 30)

​class Bitfab::Client

​SPAN_TYPES

​Readable client state

​get_trace

​get_trace_span

​get_span

​traces

​verdicts

​graders

​experiments

​datasets

​traces

​spans

​initialize(api_key: nil, service_url: nil, enabled: nil, strict: false)

​flush(timeout: 30)

​close(timeout: 30)

​Replay registry and command

​register_mock_override

​clear_mock_overrides

​Replay auto-capture environment variables

​get_function(trace_function_key)

​node(klass, method_name, name: nil, type: "custom", capture: true, experiment_id: nil, mock_on_replay: nil, finalize: nil)

​execute_span(...) (internal)

​Client environment variables

​Transport environment variables

​Commit ref environment variables

​CommitRef

​class Bitfab::BitfabFunction

​attr_reader :trace_function_key

​wrap(klass, method_name, name: nil, type: "custom", capture_when: "always", mock_on_replay: false, finalize: nil, experiment_id: nil)

​trace(klass, method_name, name: nil, type: "custom", max_depth: 30, max_captured_subtree_spans: 500, exclude: [], include_wrappers: false, mock_on_replay_default: false)

​module Bitfab::Traceable

​include Bitfab::Traceable

​Class Methods (after include)

​bitfab_function(key)

​bitfab_span(method_name, trace_function_key: nil, name: nil, type: "custom", capture_when: "always", mock_on_replay: false, finalize: nil, experiment_id: nil)

​bitfab_trace(method_name, trace_function_key: nil, name: nil, type: "custom", max_depth: 30, max_captured_subtree_spans: 500, exclude: [], include_wrappers: false, mock_on_replay_default: false)

​bitfab_node(method_name, name: nil, type: "custom", capture: true, experiment_id: nil, mock_on_replay: nil, finalize: nil)

​Module-Level Wrapping

​Bitfab::Traceable.wrap(klass, method_name, trace_function_key:, name: nil, type: "custom", capture_when: "always", mock_on_replay: false, finalize: nil, experiment_id: nil, client: nil)

​Bitfab::Traceable.trace_function_key_for(receiver, method_name)

​Bitfab::Traceable.trace(klass, method_name, trace_function_key:, name: nil, type: "custom", max_depth: 30, max_captured_subtree_spans: 500, exclude: [], include_wrappers: false, mock_on_replay_default: false)

​Bitfab::Traceable.node(klass, method_name, name: nil, type: "custom", capture: true, experiment_id: nil, mock_on_replay: nil, finalize: nil)

​class Bitfab::CurrentSpan

​class Bitfab::CurrentTrace

​class Bitfab::NoOpCurrentSpan / NoOpCurrentTrace

​Thread & Concurrency Model

​Error Behavior Summary

​Capture policy and simulation plans

​Seed and reseed

​BAML execution

​Worker threads

​Process replay

Framework Integrations

Module Layout

Module-Level API

`Bitfab.configure(api_key: nil, service_url: nil, capture_enabled: nil, enabled: nil, **options)`

`Bitfab.client`

`Bitfab.client_or_nil`

`Bitfab.reset!`

`Bitfab.current_span`

`Bitfab.current_trace`

`Bitfab.current_replay_branch`

`Bitfab.report_replay_progress(progress)`

`Bitfab.serialize_replay_result(result)`

`Bitfab.flush_traces(timeout: 30)`

`class Bitfab::Client`

`SPAN_TYPES`

Readable client state

`get_trace`

`get_trace_span`

`get_span`

`traces`

`verdicts`

`graders`

`experiments`

`datasets`

`traces`

`spans`

`initialize(api_key: nil, service_url: nil, enabled: nil, strict: false)`

`flush(timeout: 30)`

`close(timeout: 30)`

Replay registry and command

`register_mock_override`

`clear_mock_overrides`

Replay auto-capture environment variables

`get_function(trace_function_key)`

`node(klass, method_name, name: nil, type: "custom", capture: true, experiment_id: nil, mock_on_replay: nil, finalize: nil)`

`execute_span(...)` (internal)

Client environment variables

Transport environment variables

Commit ref environment variables

`CommitRef`

`class Bitfab::BitfabFunction`

`attr_reader :trace_function_key`

`wrap(klass, method_name, name: nil, type: "custom", capture_when: "always", mock_on_replay: false, finalize: nil, experiment_id: nil)`

`trace(klass, method_name, name: nil, type: "custom", max_depth: 30, max_captured_subtree_spans: 500, exclude: [], include_wrappers: false, mock_on_replay_default: false)`

`module Bitfab::Traceable`

`include Bitfab::Traceable`

Class Methods (after `include`)

`bitfab_function(key)`

`bitfab_span(method_name, trace_function_key: nil, name: nil, type: "custom", capture_when: "always", mock_on_replay: false, finalize: nil, experiment_id: nil)`

`bitfab_trace(method_name, trace_function_key: nil, name: nil, type: "custom", max_depth: 30, max_captured_subtree_spans: 500, exclude: [], include_wrappers: false, mock_on_replay_default: false)`

`bitfab_node(method_name, name: nil, type: "custom", capture: true, experiment_id: nil, mock_on_replay: nil, finalize: nil)`

Module-Level Wrapping

`Bitfab::Traceable.wrap(klass, method_name, trace_function_key:, name: nil, type: "custom", capture_when: "always", mock_on_replay: false, finalize: nil, experiment_id: nil, client: nil)`

`Bitfab::Traceable.trace_function_key_for(receiver, method_name)`

`Bitfab::Traceable.trace(klass, method_name, trace_function_key:, name: nil, type: "custom", max_depth: 30, max_captured_subtree_spans: 500, exclude: [], include_wrappers: false, mock_on_replay_default: false)`

`Bitfab::Traceable.node(klass, method_name, name: nil, type: "custom", capture: true, experiment_id: nil, mock_on_replay: nil, finalize: nil)`

`class Bitfab::CurrentSpan`

`class Bitfab::CurrentTrace`

`class Bitfab::NoOpCurrentSpan` / `NoOpCurrentTrace`

Thread & Concurrency Model

Error Behavior Summary

Capture policy and simulation plans

Seed and reseed

BAML execution

Worker threads

Process replay