# Amp Plugin
Source: https://docs.bitfab.ai/amp-plugin
Bitfab plugin for Amp: trace, diagnose, and iterate on AI workflows without leaving Amp
The Bitfab Amp plugin brings the full evaluation workflow into Amp. It registers the Bitfab tools directly with Amp, ships the setup, assistant, and update skills, and bundles the local commands those skills call, so instrumenting, replaying, and improving your AI workflows all happen in Amp.
## Installation
Run the CLI from your project directory:
```bash theme={null}
npx bitfab-cli init --editor amp
```
This clones the plugin into Amp's plugin directory (`~/.config/amp/plugins/bitfab`), confirms Amp loaded it, opens your browser to log in, and starts Amp with the `bitfab:setup` invocation to type.
Pass an initial setup request with `--prompt` (or `-p`) and the CLI includes it in that invocation:
```bash theme={null}
npx bitfab-cli init --editor amp --prompt "instrument the chat workflow"
```
For the experimental terminal-native flow, run `npx bitfab-cli init --v2` instead. It does not launch Amp: plan review, setup decisions, and edit approvals stay in the CLI while the Claude Agent SDK performs repository analysis and approved edits. Set `ANTHROPIC_API_KEY` or configure a supported Agent SDK cloud provider first. Use `npx bitfab-cli setup --v2 instrument` to run one setup mode, or add `--diagram` to print its declarative state graph without starting the agent.
Clone the plugin into Amp's system plugin directory:
```bash theme={null}
git clone --depth 1 https://github.com/Project-White-Rabbit/bitfab-amp-plugin.git ~/.config/amp/plugins/bitfab
```
Then run `plugins: reload` from the command palette (`Ctrl-O` in the Amp CLI, `Cmd/Alt-Shift-A` in the editor extensions), or restart Amp. Confirm it loaded:
```bash theme={null}
amp plugins list
```
Ask Amp for `bitfab:setup login` to sign in.
Use `.amp/plugins/bitfab` instead of the system directory to install for one project only.
### Requirements
* Node 18 or newer. The plugin's local commands (login, Studio, replay progress) run under node.
* A Bitfab account. The setup skill opens a browser to sign in when it needs to.
## What the Plugin Does
### Automatic Setup
The `bitfab:setup` skill runs a multi-phase workflow:
1. **Login**: opens your browser for OAuth authentication and saves credentials securely
2. **Explain**: walks through the two primitives you instrument with, `withSpan` and `replay`, including the five ways replay can change a method's execution. Everything after this asks you to make per-method decisions, so it comes first
3. **Approach**: asks whether the agent should walk you through instrumenting or hand you the docs so you can do it yourself
4. **Instrument + Replay** (in parallel, per workflow): reads your codebase, finds all AI workflows (LLM calls, agents, AI-driven decisions), and presents them as a numbered list. You choose which to instrument, or name a file, function, or directory yourself and it reads only that instead of scanning. Either way it adds tracing with minimal diffs and creates a registry module for the replay command shipped with the SDK
You can run individual phases by asking Amp for the skill with a mode:
```
bitfab:setup explain # Explain withSpan + replay and list the modes (read-only, no login)
bitfab:setup login # Auth only
bitfab:setup instrument # Trace instrumentation only
bitfab:setup inspect # Diagnose (and offer to fix) your tracing setup
bitfab:setup replay # Replay registry creation only
bitfab:setup analyze-repo # Scan the repo and upload draft trace plans without prompts
```
The setup is interactive: it presents 2-5 concrete options per decision point with a recommended choice, so you stay in control throughout. Trace plans open in Studio for review, and edits are previewed before they are applied.
### Assistant
The `bitfab:assistant` skill turns production traces into code improvements, whether the goal is correctness (improving pass rates) or efficiency (cutting token usage and cost). Amp does the mechanical work and collaborates with you on three steps:
1. **Build a dataset** from production traces: search for failures, label them with expected outcomes
2. **Experiment** against that dataset: make isolated code changes, replay, compare results
3. **Hill climb**: repeat until the best change is found, then present results
Run it with an optional trace function key:
```
bitfab:assistant
bitfab:assistant order-processing
```
#### Building the Dataset
Amp does the data wrangling: it searches production traces for failures, reads full inputs and outputs, and identifies edge cases. It then presents edge cases for your judgment: is this a failure (and what should the output be), correct, or irrelevant? This labeled dataset becomes the benchmark for all experiments.
The plugin opens a rich UI for navigating and labeling the dataset, then brings you back to Amp so you stay in flow. You can label every trace yourself, or label a few and let the agent classify the rest based on the patterns you've established.
#### Running Experiments
The skill reads your code, diagnoses failure patterns, and categorizes proposed changes:
* **Code fixes**: deterministic bugs, bundled into one experiment as a foundation
* **Judgment-based fixes**: prompt changes, search tuning, output formatting, each gets its own experiment
* **Infrastructure proposals**: larger changes noted for future work, not experimented on
Experiments run one at a time in your working tree. Each one edits the code, runs the SDK's replay command with your registry module and labeled dataset, and compares new outputs to expected outcomes.
#### Results
After each round, you see which traces now match expected outcomes, which still diverge, and whether any regressions occurred. The assistant works through the planned experiments in turn without pausing to ask whether to keep going, then wraps up once the plan is complete. The final summary shows pass rate improvement and all files changed, uncommitted in your working tree for review.
### Tools
The plugin registers these tools directly with Amp under the names below, so Amp can call them in any conversation. There is no MCP server in the path: the plugin's tool bridge talks to Bitfab for you. Ask Amp to search your traces and it can, without invoking a skill.
#### Core
| Tool | Description |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `get_bitfab_api_key` | Retrieve your API key for SDK initialization and environment variable configuration |
| `get_api_key_context` | Returns which Bitfab org the plugin reads/writes to. Call before the first plugin write, or when data you wrote isn't visible in Studio |
| `list_organizations` | List the Bitfab organizations available to the signed-in user, marking the current plugin org |
| `get_database_connection_status` | Report whether the org has connected a database for per-trace replay branching (`none`, `checking`, `connected`, or `failed`), identify it as direct Neon or a managed Postgres mirror, and include the pinned project name and ID for direct Neon connections |
#### Trace Inspection
| Tool | Description |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `list_trace_functions` | List all traced functions in your organization |
| `search_traces` | Search and filter traces with keyword search, date ranges, status filters, regex, environment, label filters, dataset, test run, and db-snapshot scoping; replay traces are excluded by default unless `includeReplays` is set or a `testRunId` is provided |
| `get_traces` | Read one or more traces by ID with the trace environment plus summary (truncated) or full span details (input, output, reasoning, context, errors, per-span duration and tokens) |
| `get_trace_labels` | Read just the labels for up to 100 traces by ID in one call, no span content: each trace's verdict, annotation, and approved flag, plus one line per scored assertion carrying that assertion's own verdict, annotation, confidence, and author |
| `get_grader_labels` | Read the individual verdicts each automated grader recorded (reason, failure diagnostic, confidence, human or grader run), by trace IDs, by grader ID, or both |
| `get_span_field` | Fetch the complete, untruncated value of a single span field (input, output, reasoning, content, errors, or contexts) when `get_traces` truncated it |
#### Labeling and Datasets
| Tool | Description |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `save_agent_labels` | Set, skip, or archive agent pass/fail verdicts on traces. Supports confidence levels and annotations for human review. Pass an `assertionId` to score one assertion, omit it for the whole-trace verdict |
| `save_human_labels` | Write validated human pass/fail verdicts (with annotations) on traces. Validated immediately with no UI approval step; used by `assistant fix` before adding a trace to a dataset |
| `save_trace_assertions` | Record what a trace SHOULD do when replayed: one assertion per statement, with optional pass/fail criteria, target, and people-only human note. Agents can edit the note but must never use it as assessment evidence |
| `get_trace_assertions` | Read the assertions and people-only human notes on up to 100 traces by ID, no span content. Notes are never assessment evidence; a replay with no assertions of its own reads its original's |
| `archive_trace_assertions` | Retire assertions on a trace so later replays stop checking them. Non-destructive and all-or-nothing: one unknown, already-archived, or wrong-trace id fails the call and archives nothing |
| `save_dataset` | Create a labeled dataset for a traced function (named buckets of traces for review and replay) |
| `list_datasets` | List all datasets for a traced function with trace counts and assigned graders |
| `add_traces_to_dataset` | Add traces to a dataset (idempotent, 1-100 per call) |
| `remove_traces_from_dataset` | Remove traces from a dataset without deleting the traces themselves |
| `add_graders_to_dataset` | Assign graders to a dataset (idempotent, organization- and function-scoped, 1-100 per call) |
| `remove_graders_from_dataset` | Remove grader assignments from a dataset without deleting the graders |
| `save_grader` | Create or edit an automated grader (LLM-as-judge pass/fail check) for a traced function: upsert by id or name, rename, clear pass/fail criteria, archive/restore, select the judge model |
| `list_graders` | List automated graders for a traced function with optional name search and cursor pagination (20 by default, 50 maximum; archived hidden by default) |
#### Experiments
| Tool | Description |
| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `save_experiment_group` | Create a group from selected experiments with optional name and notes, or update an existing group's metadata without changing membership |
| `list_experiment_groups` | List recent experiment groups with their names, notes, and member experiment ids |
| `get_experiment_group` | Get one experiment group with its metadata and detailed member experiment summaries |
| `save_experiment` | Update an existing experiment's name or notes, move it into an existing experiment group, or remove it from its group |
| `add_graders_to_experiment_group` | Assign one or more graders directly to every current experiment in a group and queue missing evaluations for completed runs; later experiments do not inherit them automatically |
| `remove_graders_from_experiment_group` | Detach one or more graders from every current experiment in a group; unassigned ids are ignored and the graders themselves are not deleted |
| `list_experiments` | List experiments (replay test runs) for a traced function with name, notes, status, totals, and delta (fixed/regressed/still passing/still failing) |
| `get_experiment` | Get a single experiment by id with name, notes, status, totals, delta, experiment group, and grader results, including a per-grader passing/failing breakdown |
| `list_experiment_traces` | Get individual trace results for an experiment with each replay trace's verdict compared to the original, plus token usage (input, output, cached, total) for the replay and paired original |
| `add_graders_to_experiment` / `remove_graders_from_experiment` | Attach or detach active graders on an experiment so they run against its replay traces (idempotent, organization- and function-scoped, 1-100 per call); the effective set at completion is the union with the dataset's runnable graders; detaching a dataset-overlapping grader from an in-progress run is re-added at completion (still runs) but from a completed run permanently drops it from the finalized snapshot, and attaching to a completed experiment grades existing traces only on the next completion/replay |
| `rerun_graders_on_dataset` / `rerun_graders_on_experiment` | Re-score a dataset's traces or an experiment's completed replays with their attached graders, overwriting the previous verdicts (defaults to every attached grader; a subset may be named, and ids that are not attached are rejected); waits up to 90s and reports traces graded, and a repeat call reports the running job instead of starting a second one |
| `get_replay_status` | Read a replay test run's current status and local replay trace ID to server trace ID mapping while replay is still running |
#### Templates
| Tool | Description |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `get_template_reference` | Read the Nunjucks template engine reference, render-context schema, and available filters. Call once per session before editing templates |
| `get_template` | Read the rendering template for a span type, scoped to a trace function key or org-global |
| `save_template` | Upsert a rendering template for a span type. Controls how span data renders in the Bitfab UI |
#### Instrumentation
| Tool | Description |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `save_trace_plan` | Create a tracing plan, or structurally update an existing plan (its root included) in place by passing its plan ID |
| `confirm_trace_plan` | Confirm a plan without the browser (the continue path, or a plan page left without saving) so `setup view`/`setup modify` can find it by key later |
| `get_trace_plan` | Read a trace plan by ID (after confirmation) or by trace function key; Modify preserves prior decisions while reconciling the plan and its root with current instrumentation |
| `list_trace_plans` | List the org's trace plans, newest first, filterable by source, status, and trace function key; used to reuse unconfirmed `analyze-repo` drafts instead of re-scanning, and to find the plan a key already has so it is updated rather than duplicated |
| `cancel_trace_plan` | Retire an unconfirmed plan nobody will act on (already instrumented, or the workflow is gone) so it stops coming back as a reusable draft |
| `get_sim_plan` | Read a trace function's sim plan (alpha): its span nodes from recent traces with type, call count, average payload, estimated monthly cost, share of traces, and whether content is captured. A node is its span name inside the trace function key |
| `save_sim_plan` | Turn content capture off or on per span node (alpha). The span itself, its name, type, timing, and errors are always recorded; content off strips only inputs and outputs. A node recorded by a framework integration, a node that is the root of its traces, a node imported from another platform, or a node named by Bitfab rather than the SDK keeps its content, and turning it off is refused with the reason. Turning a node off also turns content off for every node beneath it in the sim plan, the way untracing a span in a trace plan drops the spans under it; a node beneath it that keeps its content stays on, and a node named in the same call keeps the value given to it. Turning a node back on leaves the nodes beneath it as they are. Returns the updated sim plan |
### Skills
Amp lists every skill for the model by name and description and loads one when it becomes relevant. Ask for one by name, or invoke it from the command palette with `skill: invoke`.
| Skill | Description |
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `bitfab:setup` | Full setup workflow: authenticate, instrument, create replay registries |
| `bitfab:setup login` | Auth only |
| `bitfab:setup analyze-repo` | Non-interactively scan the repo, pick traceable workflows, and upload draft trace plans |
| `bitfab:assistant` | Build a dataset from traces, experiment with code changes, improve pass rates or cut token costs |
| `bitfab:assistant ` | Iterate on a specific trace function |
| `bitfab:assistant fix ` | Fix one failing trace: diagnose it, make the focused code change, and replay just that trace; once it passes, add it to a dataset, then choose to inspect the before/after in Studio or re-run the full dataset |
| `bitfab:update` | Update the plugin and the Bitfab SDK to the latest versions |
## Example Workflows
### Instrument a new project
```
bitfab:setup
```
The agent detects your project language, finds AI workflows, presents options, and instruments your chosen workflows, all interactively.
### Analyze the repo headlessly
```bash theme={null}
npx bitfab-cli analyze-repo --editor amp --limit 3
```
This runs Amp headless (`amp -x`) with the setup skill in `analyze-repo` mode. It ranks the AI workflows in the repository, uploads a draft trace plan for each, and prints a report. It edits no code. `bitfab:setup instrument` picks those drafts up later and reconciles each against current code.
### Diagnose and fix a failing function
Ask Amp naturally:
```
"My order-processing traces are failing. What's going wrong and can you fix it?"
```
The plugin calls `search_traces` and `get_traces` to inspect failing traces and suggests code fixes directly.
For a specific failing trace, run `bitfab:assistant fix `. The agent diagnoses the failure, confirms why the original trace is wrong before editing when the trace or conversation does not already make that clear, makes the focused code fix, and replays only that trace first. Once the fix passes, it adds that trace to a dataset with a validated failing label, then branches: inspect the before/after in Studio, re-run the full dataset (in Studio or terminal-only), keep iterating, or stop. If that full-dataset re-run reveals real regressions (previously-passing traces the fix broke), it reports them and keeps the target trace saved as a red test to revisit. If the replay still fails, it offers to keep iterating or save the trace as a failing test instead.
### Iterate on a trace function
```
bitfab:assistant memory-search
```
The agent finds failing traces, walks you through labeling them with expected outcomes, diagnoses the failure patterns in your code, then runs experiments: editing prompts or code, replaying against your labeled dataset, and reporting what improved. You stay in control at every decision point.
### Replay after a code change
After updating a function, pass your registry module to the replay command installed by the SDK:
```bash theme={null}
pnpm exec bitfab-replay --registry scripts/replayRegistry.ts extraction --limit 20
```
Or ask Amp to do it for you.
While the replay runs, Amp runs it as a background shell command and reports progress to you as it goes: one line per trace as it finishes (a pass/fail mark, the running count, and how long that trace took), with any error reason inline, plus a periodic "still running" heartbeat when a slow trace takes a while so the run never looks stuck, then a summary with the total and average time. Full per-item outputs are written under that replay run's `.bitfab/replays//items/` folder and referenced from `.bitfab/replays//events.jsonl`. If the replay command succeeds but its local result cannot be captured, the plugin reports the outcome as unverified instead of failed, then checks the server test run to recover the final result.
## Configuration
### Credentials
Credentials are stored in `.bitfab/credentials.local.json` when that project-local file exists, otherwise in `~/.config/bitfab/credentials.json` (created by `bitfab:setup login` with owner-readable permissions).
### Environment Variables
| Variable | Description |
| ---------------- | --------------------------- |
| `BITFAB_API_KEY` | Override the stored API key |
## Updating
```bash theme={null}
npx bitfab-cli update --editor amp plugin
```
This pulls the latest plugin into `~/.config/amp/plugins/bitfab`. Run `plugins: reload` in Amp afterward. `npx bitfab-cli update --editor amp sdk` starts Amp with `bitfab:update sdk` to type, which walks each workspace's SDK upgrade in Amp. Asking Amp for `bitfab:update` does both.
## What differs from the other hosts
| Capability | Amp | Claude / Cursor / Codex |
| ----------------------------------------------------------- | :----------------------------------------------------: | :---------------------: |
| Setup, assistant, update skills | Yes | Yes |
| Bitfab tools available to the host agent | Yes (registered by the plugin, bare names) | Yes (MCP) |
| Studio trace plan review, Edit-with-agent, template preview | Yes | Yes |
| Session log capture | Not yet | Yes |
| Auth and update banner at session start | Not yet | Yes (Claude, Cursor) |
| `--skip-permissions` from the CLI | No flag. Set `amp.dangerouslyAllowAll` in Amp settings | Yes |
Session log capture and the session-start banner ship in a follow-up release through Amp's `agent.start` and `agent.end` events.
## Troubleshooting
### Not authenticated
If a skill reports "Not authenticated":
1. Ask Amp for `bitfab:setup login` to authenticate via browser
2. Check that `~/.config/bitfab/credentials.json` exists and contains your API key
3. If using an environment variable, verify `BITFAB_API_KEY` is set
### Bitfab tools not available
If Amp cannot see tools like `search_traces`, the plugin did not load:
1. Run `amp plugins list`. The Bitfab entry shows whether it loaded and, if not, why
2. Run `plugins: reload` from the command palette, or restart Amp
3. Check that the plugin directory is `~/.config/amp/plugins/bitfab` (or `.amp/plugins/bitfab` in the project) and contains `index.js`
4. Make sure `node` is on your PATH. The skills' local commands run under node even though the plugin itself runs under Amp's runtime
### Plugin updates
Ask Amp for `bitfab:update`, or run `npx bitfab-cli update --editor amp plugin` and then `plugins: reload`.
# API Keys
Source: https://docs.bitfab.ai/api-keys
Create and manage API keys for authenticating with Bitfab
## Overview
API keys are used to authenticate your applications with Bitfab. Each API key is scoped to an organization and can be used to:
* Call functions via the SDK
* Send traces to Bitfab
* Access the Bitfab API
* Connect coding agents via MCP (Model Context Protocol)
## Creating an API Key
1. Click your profile avatar in the top-right corner of the Bitfab web portal
2. Select **API Keys** from the dropdown menu
3. Click **Create API Key**
4. Enter a name for your API key (e.g., "Production", "Development")
5. Copy the generated API key immediately - it will only be shown once
Store your API key securely. Never commit API keys to version control or expose them in client-side code.
## Using API Keys
### Environment Variables
The recommended way to use API keys is through environment variables:
```bash theme={null}
# .env
BITFAB_API_KEY=bf_your_api_key_here
```
### SDK Configuration
Pass your API key when initializing the SDK:
```typescript TypeScript theme={null}
import { Bitfab } from "@bitfab/sdk"
const client = new Bitfab({
apiKey: process.env.BITFAB_API_KEY,
})
```
```python Python theme={null}
from bitfab import Bitfab
client = Bitfab(
api_key=os.environ["BITFAB_API_KEY"]
)
```
### VS Code Extension
Configure your API key in VS Code settings:
1. Open VS Code Settings (`Cmd+,` or `Ctrl+,`)
2. Search for "Bitfab"
3. Enter your API key in the **Bitfab: Api Key** field
Or use the command palette:
1. Open Command Palette (`Cmd+Shift+P` or `Ctrl+Shift+P`)
2. Run **Bitfab: Set API Key**
3. Enter your API key
### MCP (Model Context Protocol)
For coding agents like Cursor and Claude Code, API keys are used to authenticate MCP connections:
**Claude Code**: Install the [Bitfab plugin](/mcp-setup#claude-code) - it handles authentication and MCP setup automatically via `/bitfab:setup`.
**Other agents**: Visit the [Bitfab setup page](https://bitfab.ai/setup) for automatic configuration with your API key embedded, or include your API key in the MCP configuration manually:
```json theme={null}
{
"mcpServers": {
"bitfab": {
"type": "streamable-http",
"url": "https://bitfab.ai/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
See the [MCP Setup guide](/mcp-setup) for detailed instructions.
## Managing API Keys
### Viewing API Keys
Click your profile avatar and select **API Keys** to view all API keys for your organization. You can see:
* Key name
* Creation date
* Last used date
* Partial key (last 4 characters)
### Revoking API Keys
To revoke an API key:
1. Click your profile avatar and select **API Keys**
2. Find the API key you want to revoke
3. Click the **Revoke** button
4. Confirm the revocation
Revoking an API key is immediate and permanent. Any applications using the revoked key will stop working.
## Best Practices
* **Use separate keys for different environments**: Create separate API keys for development, staging, and production
* **Rotate keys regularly**: Periodically create new keys and revoke old ones
* **Use descriptive names**: Name your keys clearly to identify their purpose
* **Monitor usage**: Check the "Last Used" date to identify unused keys
### MCP-Specific Security
When using API keys with coding agents:
* **Secure storage**: MCP configurations are stored locally on your development machine - ensure it's encrypted and secure
* **Team access**: Be cautious about sharing MCP configurations that contain API keys
* **Development vs production**: Consider using separate organizations or API keys for different environments
* **Review agent behavior**: Understand how your coding agent handles and logs API requests
Some MCP clients may log requests or store configuration in plaintext. Review your agent's security practices and consider the sensitivity of your traced data.
# Changelog
Source: https://docs.bitfab.ai/changelog
Product updates and announcements
## Keep people-only context with assertions
Add a human note to an assertion when reviewers need context that should not change how a replay is judged. Notes are editable from the trace page and through MCP. Coding agents can preserve them without treating them as pass or fail criteria.
TypeScript and Python assertion reads now return the note. SDK assertion saves remain focused on assessment criteria.
```typescript theme={null}
const { assertions } = await client.traces.getAssertions(traceId)
console.log(assertions[0]?.humanNote)
```
## See who ran an experiment and what code it ran
The experiments list now shows who launched each run and the branch it ran from, at every window width. Hover the marks at the end of a row for the person, their git email, the branch, and the commit. Runs launched with a shared CI key show who was at the keyboard instead of appearing anonymous.
## Experiments say when the code was uncommitted
If a run executed code that was never committed, the row marks it and the detail explains it in plain words, with the exact `git diff` command to see what changed. Asking your coding agent about an experiment returns the same thing: it now names the person who ran it and says whether the tree was clean or had uncommitted changes, rather than leaving two commit hashes to compare.
## More reliable search for large traces
Search indexing now handles traces with tens of thousands of spans more reliably, preserving searchable summaries and generated embeddings across the full trace. Experiment summaries also load with less contention during concurrent activity.
## Every experiment records who ran it and which code ran
An experiment now stores the person who launched it and the exact state of the code behind it: the branch, the commit, the tree at that commit, and the tree that actually ran. Nothing to opt into, since `replay()` records it itself, and every read is local with no network call.
The last two are the useful pair. A commit alone lies whenever the working tree is dirty, so the experiment tree is captured separately and covers uncommitted and untracked files. Two runs sharing an experiment tree ran byte-identical code, and the difference between the base and the experiment tree is the change under test:
```bash theme={null}
git diff
```
Capturing it never touches your index or your stash, so a replay is safe to run mid-edit. The person is resolved from the API key's owner on the server, so a run cannot claim to have been launched by someone else. A machine with no checkout records nothing rather than failing the replay, and `BITFAB_DISABLE_GIT_STATE` opts a process out.
## Assertion categories
Group assertions into reusable categories with a title and description. Categories appear in trace and labeling views, where assignments, renames, and removals update without refreshing the page. Manage them through MCP, `client.assertionCategories` in TypeScript, or `client.assertion_categories` in Python. Deleting a category keeps its assertions and verdicts.
```typescript theme={null}
const category = await client.assertionCategories.save({ title: "Response quality" })
await client.traces.saveAssertions(traceId, [
{ assertion: "Answers the user's question", category_assertion_id: category.id },
])
```
## Clear status for incomplete trace captures
Unfinished traces now show “Capture incomplete” after 60 minutes without activity, preserving the spans already received. Trace lists and details distinguish these unknown outcomes from failed evaluations and regressions. New activity restores the capture status on the next background check, and a recorded completion clears the warning immediately.
## Live experiment totals stay current
The experiments list no longer briefly switches back to older totals when a refresh overlaps a live update. Later refreshes can still correct the displayed totals.
## Search handles circular trace data
Trace search indexing now handles inputs and outputs with circular references, such as child objects pointing back to their parent. Searchable content is preserved while circular links are replaced with markers.
## More resilient trace ingestion
Trace and span uploads now recover automatically from brief database connection interruptions, reducing dropped telemetry during transient service disruptions. This applies to SDK ingestion across supported trace sources without requiring any SDK changes.
## Faster dataset browsing
Dataset experiment pages and dataset lists load with less work as collections grow. Dataset-filtered trace search now applies your filters before choosing its top matches.
## Page through graders
Grader lists now offer Previous and Next controls for browsing larger collections. Sorting applies across the full list, including archived graders.
## Automatic dataset paging in every SDK
TypeScript, Python, Ruby, and Go now fetch dataset lists and dataset trace IDs in pages automatically. Existing calls still return the complete result without requiring cursor handling.
```typescript theme={null}
const datasets = await client.datasets.list({ traceFunctionKey })
const { traceIds } = await client.datasets.listTraces(datasetId)
console.log(datasets.length, traceIds.length)
```
## Smoother scrolling through long experiment lists
The experiments list no longer flickers back to a loading state or jumps to the top while you scroll, and it keeps loading older runs instead of stopping partway down. Lists with many runs also settle faster after a label changes.
## Trace Ruby workflows from one root
Ruby 3.4+ can now record a workflow's project methods from one experimental `bitfab_trace` declaration. Use `bitfab_node` to configure individual calls without tracing them on their own.
```ruby theme={null}
bitfab_trace :process, trace_function_key: "orders",
max_depth: 10, max_spans: 100
bitfab_node :summarize, name: "Summary", type: "llm"
```
Keep conservative limits for discovery and prefer explicit `bitfab_span` declarations on production hot paths.
## Keep nested and lazy work in context
Nested roots with different keys keep independent traces while outer traces continue recording shared calls. Explicit spans appear once in their owning trace. Returned Enumerators are traced during consumption without forcing lazy work to run early.
## Parent spans stay in view while you scroll
Scrolling a deep span tree no longer loses track of where you are. Each parent span now stays fixed at the top of the tree for as long as any span beneath it is on screen, up to five levels, and the sim plan's code map pins its parent nodes the same way. Selecting a span still scrolls it fully clear of the pinned rows.
## Replay diffs ignore spans whose capture the sim plan turned off
Turning capture off for a sim plan node strips a span's inputs and outputs while still recording the call itself. If the plan changed between a trace and its replay, the Diff view compared a recorded side against a blank one and reported those spans as "Not called", filling the span tree with rows for code that ran normally. Spans with capture off on either side now report no diff, and a call the replay genuinely skipped still shows as not called.
## Tracing a span brings its callers with it
Turning on tracing for a span in a trace plan now turns on every span above it, so the plan always records one unbroken path from the entry point down to the span you picked. Untracing a span turns off everything beneath it, which is what the plan already claimed it did. Both hold whether you click in Studio or your coding agent adjusts the plan for you.
## Assertions stay on the original trace
Saving or archiving an assertion through a replay's trace id is now refused, and the error names the original trace to retry with. An assertion describes the case rather than one run, so a replay reads its original's assertions instead of carrying a copy. Writing onto the replay used to succeed and stop that inheritance, which left later replays of the same case with nothing to be judged against.
## Assertions are called assertions everywhere
The tools your coding agent calls now say assertion in their titles and in everything they print back, matching what the SDKs and the dashboard already called them. Nothing you call by name changed, so existing scripts and prompts keep working.
## Replay no longer runs your machine out of memory
Replaying with `primitive="process"` now starts each child only once the machine has room for it, so a large run on a busy laptop finishes with fewer replays at a time instead of being killed partway through. Bitfab checks free memory and swap before starting a child, counts the room every running child still needs rather than only what it has used so far, and learns the real figure from children that have already finished. It never goes above the `max_concurrency` you set and never holds back the last remaining child, so a run still finishes on a machine that stays under pressure.
It is on by default in process mode. Set `BITFAB_REPLAY_MEMORY_THROTTLE=off` to turn it off for a single run, or turn it off for a registration:
```python theme={null}
registry.register(
"classification",
bitfab,
classify_text,
concurrency=ReplayConcurrency(primitive="process", memory_throttle=False),
)
```
`BITFAB_REPLAY_CHILD_MEMORY_MB` and `BITFAB_REPLAY_MEMORY_FLOOR_MB` tune how much memory a child is expected to need and how much is left free. Update to Python SDK v0.54.0.
## Blank spans no longer show phantom changes
Spans that recorded no input or output no longer report a change count in an experiment's Diff view. Comparing an empty value against one that was never recorded counted as a removed line, so a blank span displayed a red removal for a difference that did not exist. The control that hides empty spans now works inside the Diff view as well, where those rows were previously stuck on screen and left out of the hidden count.
## Tracing works with proxies and computed getters
Automatic tracing now handles code that runs a function on property access, such as a database client wrapped in a proxy or an object with computed getters. Previously the SDK read those properties while it was still recording, which ran your code again and could exhaust the call stack, leaving the run with no trace at all. Update to TypeScript SDK v0.53.4.
## One reference for SDK setup
SDK setup now uses the public documentation for language and framework details. The docs also clarify Go replay support, filtering dataset replays, and tracing shared packages in TypeScript monorepos.
## Install the TypeScript transform under the Bitfab scope
The TypeScript build adapters are now available as `@bitfab/transform`. If you used `bitfab-transform`, replace the dependency and update your imports and build configuration to the scoped package. The Next.js adapter now resolves its loader from `@bitfab/transform/loader`, and plugin setup instructions use the new package name.
## Keep partial output when model streams stop
Vercel AI SDK tracing now records partial output and the error or cancellation when a model stream stops early. Once the stream ends or is cancelled, `flushTraces()` can finish delivering its span without waiting on an unfinished stream finalizer.
```typescript theme={null}
const reader = result.stream.getReader()
await reader.cancel("Response no longer needed")
await flushTraces()
```
## Capture shared monorepo packages
The Node transform now resolves its injected runtime from the SDK installation, so shared packages can be captured even when only the application depends on the SDK. Use `BITFAB_SDK_RESOLVE_FROM` to select the application's package location when needed; ESM and CommonJS are both supported.
## Turning a node off turns off the nodes beneath it
When you turn content capture off for a node on the Sim plan page, every node beneath it in the map turns off too, the same way untracing a span in a trace plan drops the spans under it. Nodes that always keep their content (recorded by a framework integration, the root of the workflow, imported from another platform, or named by Bitfab) stay on. Turning a node back on leaves the nodes beneath it as they are, so switch those on individually.
The `save_sim_plan` tool in the Claude Code, Cursor, Codex, and Amp plugins follows the same rule, and a node you name in the same call keeps the value you give it.
## Turn content off per node from the Sim plan
The TypeScript and Python SDKs now read your Sim plan and stop sending inputs and outputs for any node whose content you turned off. The span still arrives with its name, type, timing, errors, and links, marked as content off, so the Sim plan page keeps counting it while the payload never leaves your process. Turn a node off on the Sim plan page or with the `save_sim_plan` tool, and new traces pick the change up within a minute. Spans are held until the plan has loaded, so short-lived processes honor it too.
## See which SDK and integration recorded a span
Every span now carries its origin: the SDK name and version, and whether it came from `withSpan` or `@span`, `withTrace` or `@trace`, or a framework integration (Vercel AI SDK, Claude Agent SDK, OpenAI Agents, LangGraph). The trace header shows it as "Recorded by". Spans recorded before this release resolve their instrumentation from the trace source instead, and show no SDK name or version.
## Roots and framework spans always keep their content
The Sim plan refuses to turn content off for a node that is the root of its traces, a node recorded by a framework integration, an imported node, or a node Bitfab named from its model or type. The switch shows on and disabled with the reason, and the `save_sim_plan` tool returns an error that says why.
## Browse a node's recent occurrences from the Sim plan
Click any row on the Sim plan page to open that node's recent occurrences, newest first, each with its size and whether content was captured. Open one to see the span inside its full trace.
## Decide per call what a Sim plan records
Each call on a trace function's Sim plan now has a Capture switch. Turning it off marks that call as one whose inputs and outputs should stop being recorded, while the call itself, with its name, type, timing, and errors, keeps being captured so the map stays complete. The decision is stored with the plan and shows who made it. The SDKs will start honoring it in an upcoming release. Sim plan is an alpha switched on per organization.
## See what each call costs before you decide
Every call on the Sim plan shows its average recorded payload and an estimate of what it costs per month at \$2 per GB, based on how often the function ran in the last 30 days. Click the payload to open recent examples of that call in the standard trace view, inputs and outputs included, and step through them. The plan also flags a call reached from more than one parent, since a switch there affects every one of them.
## Edit the Sim plan from your coding agent
Two new MCP tools let a coding agent read and change a plan. `get_sim_plan` lists a trace function's calls with their type, call counts, payload size, monthly estimate, and capture state, and `save_sim_plan` turns capture off or on for the calls you name. The Sim plan page's Edit plan button shows the prompt to use, and a trace function's traces page now links to its plan.
## Method spans are named after their class
A span on a class method now defaults to `Class.method` in both SDKs, so two methods that share a name are two distinct spans rather than one. Plain functions, closures, and explicit names are unchanged, and the raw function name still travels separately as `function_name`.
```typescript theme={null}
class Order {
settle = bitfab.withSpan("orders", function settle(id: string) {
return finalize(id)
})
}
// recorded as "Order.settle"
```
```python theme={null}
class Order:
@client.span("orders")
def settle(self, order_id):
return finalize(order_id)
# recorded as "Order.settle"
```
A trace recorded before this release still carries the bare name, so replaying it against upgraded code will not match those method spans until the trace is recorded again.
## Jump between a nested trace and the trace that started it
When a trace function runs inside another traced function, the span tree now shows the connection. In the outer trace, the span for the nested call carries a small link naming the nested trace function, and clicking it opens that trace at its root span. In the nested trace, the root span carries a link back to the outer trace and opens it at the span that made the call. Traces recorded before this release show no link.
## Nested trace roots record both traces
When a `trace()` root runs beneath another one, both functions now get a complete trace: the outer keeps recording every call beneath it, and the nested root records its own trace under its own key, so you can label, collect, and replay one part of an agent on its own. This is new in TypeScript, where a nested `withTrace()` or `trace()` root used to take over the outer's capture; Python already recorded both traces and now links them. Each root applies its own limits, exclusions, and capture policy to its copy, and a `node()` inside the nested region keeps its name, type, test run, and finalized output in every copy.
```typescript theme={null}
const classify = bitfab.withTrace("classify-ticket", (ticket: Ticket) =>
callModel(ticket),
)
const triage = bitfab.withTrace("support-agent", (ticket: Ticket) =>
reply(ticket, classify(ticket)),
)
```
## Nested traces link to each other
The outer trace's span for the nested root now carries the nested trace's id, trace function key, and root span id, and the nested root's span carries the enclosing trace's id and span, so the two traces can be followed from either side. Inside a replay or while seeding a trace, a nested root is absorbed into the item's trace instead of starting a second one, so an experiment never gains traces under another key.
## Every trace records its commit
Every trace now carries the commit its code was running at, in all four SDKs, so you can tie a captured run to a point in your repository's history rather than only to a timestamp. The SDK reads it from your deploy platform's build variables on Vercel, GitHub Actions, Railway, Render, Heroku, Cloudflare Pages, GitLab CI, Azure Pipelines, and CircleCI, falls back to `git` in a plain checkout, and never slows the traced call. In a container built without either, set `BITFAB_COMMIT_SHA` at build time, and set `BITFAB_DISABLE_COMMIT_REF` to turn the capture off. The ref records the commit, the branch, whether the tree had uncommitted changes, and the repository as `host/owner/repo` with any credentials removed.
## An error partway through no longer erases a trace's verdict
A trace that hit an error somewhere in its span tree but still ran to completion now keeps its verdict and counts toward your experiment pass rate. Until now a single error anywhere in the tree pulled the whole trace out of the results, out of the pass rate, and out of the before and after comparison, even when graders had already scored it. Only a run that crashed outright, where the traced function threw all the way out and produced no result, is still set aside as ungradable.
## Errors read as their own signal, beside the verdict
Trace rows now carry an error badge showing how many spans threw, next to the pass rate rather than in place of it. A run that crashed at the root reads Crashed and keeps its red marker, while a run that recovered shows its error count alongside its normal verdict, so you can tell a broken run from a noisy one at a glance.
## Experiment breakdowns keep a label's baseline
The Labels and Attempts rows of an experiment breakdown no longer report work as having no baseline when the Traces row already had something to compare it against. An original labelled as a whole, or scored by different graders than its replay, carried a baseline at the trace level but not at the label level, so its labels fell into the no-baseline segment of the bar and sat outside the change in points. All three rows now read the original's verdict the same way, so that segment is left for originals that really were never labelled.
## Sim plan, an alpha code map of each trace function
A trace function now has a Sim plan: a code map of every call its traces have recorded, built from its 20 most recent runs and shown in the sidebar below Graders. Each call appears once, with its span type, how many times it ran, the share of recent runs it appeared in, and when it was last seen, so a branch that stopped running fades while the rest of the map stays put. Each call also carries the judgments the plan will drive, whether its inputs and outputs are captured and whether a reviewer needs to see it, and the root of every span tree links to its function's plan. Sim plan is an alpha and is switched on per organization, so ask us if you would like it enabled.
## Dark mode
Bitfab now has a dark theme. Flip it with the sun and moon toggle in the header, or choose Light, Dark, or System from the account menu, where System follows your operating system. Your choice is remembered, and it carries across the trace, dataset, experiment, and grader views as well as Studio and the plugin pages.
## See how an experiment moved, at three zoom levels
Opening an experiment now shows a breakdown beneath it that reads the same run three ways: by trace (a trace passes when at least 75% of its labels pass), by label (each assertion or grader rolled up across its attempts, with at least 75% of attempts passing counting as a pass), and by every label on every attempt. Each row shows the pass rate, the change against the original traces in points, and a bar splitting the run into held, fixed, regressed, and still failing, so a run that looks unchanged by trace can show where it improved underneath. Runs with several attempts per trace get a Jitter row counting the attempts that disagreed with their trace's verdict. When there is nothing to compare against, the rows show plain passing and failing instead of an empty change, and a run that mixes compared traces with ones whose originals were never labelled counts both, drawing the uncompared ones in a lighter shade.
## Filter an experiment's traces by how they changed
The trace list under an experiment now filters to fixed, regressed, still passing, or still failing traces, with an Unstable filter on multi-attempt runs for traces whose attempts disagreed. The list scrolls on its own, so the breakdown above it stays in view.
## Grader labels count the same as your labels
An experiment's pass rate, raw count, change buckets, and filters now read one verdict per trace that includes grader labels alongside labels written by people or agents. A grader's label counts whenever it is on the trace, whether or not that grader is attached to the experiment. Two things follow. A grader attached to an experiment that has not yet labelled a trace no longer holds that trace's verdict back, and a run's headline pass rate counts traces rather than individual grader checks, so grader-scored experiments will read differently than before. The graders panel keeps its own per-grader counts.
## Experiment rows show the count and the change
Each experiment's row now carries the passed-over-scored trace count and the change against the originals in points beside its pass rate, and its bar is a single fill at the pass rate. All four read the trace rollup, so a five-attempt run of 50 traces reads 36/50 rather than 180/250. The four-way split moved into the breakdown, where it has the room to be read.
## Label badges count every check a trace is held to
A trace's label badge now counts what the trace is judged against, which is its assertions when it has any and the trace as a whole when it has none. Nothing counts as passed except a label you marked pass, so a trace whose assertions nobody has labelled reads as unlabelled instead of showing a full pass. Anything still unlabelled counts on neither side of the badge, and a trace where one assertion passed and another is still open now shows the pass instead of reading as unlabelled.
## Deselect a label by clicking it again
Clicking the Pass, Fail, or Skip you already chose now clears it and returns the trace or assertion to having no label. This works the same way on trace labels and on grader labels. Clearing your own label leaves any agent or automated suggestion untouched, so the suggestion is still there when you come back to it.
## Re-seed a trace in place
When a captured trace errored, or its recorded run is stale, you can now re-seed it: run the function once more on the trace's recorded inputs and record the result under the same trace id. The trace keeps its id, labels, assertions, dataset membership, name, and metadata, so anything you stored against it still points at the same case. The previous run is kept as its own trace, linked back with `reseedOfTraceId`, so nothing is lost. Nothing is mocked and no experiment is created; a re-seed is a seed, not a replay.
```bash theme={null}
bitfab-seed --registry replay.registry.ts extraction --from-trace 3f2a...
```
`bitfab-seed` is the new home of seeding: `--from-trace` re-seeds, and `--cases cases.jsonl` does what `bitfab-replay --seed` did (that spelling still works). From code, `bitfab.reseedTrace(key, fn, { traceId })` / `reseed_trace(key, fn, trace_id=)` does the same. The Bitfab assistant's replay mode takes this path when you ask for a re-seed or the trace errored, and warns first when the function has side effects, since a re-seed runs them for real. Graders on the trace's datasets re-run afterwards, and default replay selection skips previous runs.
## Empty spans hidden in the trace view
The span tree now hides spans that recorded no input and no output, lifting their children up a level, so every row you click has something to show. A toggle at the top of the span panel says how many are hidden and brings them back, and your choice is remembered. Spans that recorded an error, and spans that differ in a replay comparison, always stay visible.
## Seeded metadata survives a streamed agent run
Metadata you record with `seedTrace` / `seed_trace` or `setMetadata` / `set_metadata` now stays on the trace even when an agent framework exports its own trace after your traced function has returned. A streamed OpenAI Agents run ends its trace when the stream drains, which happens after the function returns, so its export used to take the trace back and a later replay read the wrong values in `adaptInputs` / `adapt_inputs`. Requires TypeScript SDK v0.52.3 or Python SDK v0.52.3.
## Clashing metadata keys warn once per trace
When you and an integration both set the same metadata key to different values, your value is the one kept on the trace and the SDK warns. That warning now fires once per trace and names the trace id instead of once per process, so a large replay run shows every affected trace rather than only the first.
## Replay part of a dataset with `--limit`
`--limit` now narrows a dataset or trace ID selection instead of being ignored. `bitfab-replay --dataset-ids --limit 10` replays ten of that dataset's traces rather than all of them, so you can sample an expensive corpus while iterating and pay for the full run only when you want it. The same bound applies to an explicit `--trace-ids` list, and to a dataset your registry entry already declares.
A bounded dataset run selects the same traces every time, so two runs at the same limit stay comparable. Available in the TypeScript, Python, and Ruby SDKs.
## Seeded trace metadata survives an agent framework's own trace
Metadata you pass to `seed_trace` / `seedTrace`, or set with `set_metadata` / `setMetadata`, now always reaches a replay's `adapt_inputs` / `adaptInputs` hook. Tracing a function that also opened an OpenAI Agents trace previously replaced your metadata with the framework's own, so a seeded case's provenance went missing exactly where the replay needed it. Both sets of keys are kept now, and yours win on a conflict.
```python theme={null}
trace_id = bitfab.seed_trace(
"agent-turn",
run_turn,
kwargs=call,
metadata={"suite": "happy-paths", "case_uid": case_uid},
)
```
## Save assertions across many traces in one call
Both SDKs can now write assertions for many traces in a single request instead of one request per trace. `saveAssertionsAll` in TypeScript and `save_assertions_all` in Python take one entry per trace, so a script attaching assertions to hundreds of traces makes one call rather than hundreds.
```typescript theme={null}
await client.traces.saveAssertionsAll({
updates: traceIds.map((traceId) => ({
traceId,
assertions: [{ assertion: "Departs before 9am" }],
})),
})
```
The batch is written in one transaction, so a rejected trace saves nothing and leaves no half-written state to reconcile. A single call carries up to 500 traces and 1000 assertions in total.
## Traces are identified by name
The trace list now shows the name your code gave each trace instead of an eight-character id. Names come from `setName` in the SDK, from a seeded trace, or from the trace's name on a platform you imported from, so a run reads as something you recognize rather than a fragment of a uuid. A trace whose name only repeats its function name keeps showing the id, so no row gains a label that tells you nothing new.
## Assertions show on dataset and experiment lists
Dataset and experiment trace lists now name who authored a trace's assertions, which until now only the main traces list did. Hovering says how many assertions the trace carries, and on an experiment row it adds that they were written on the original trace rather than on the replay in front of you.
## Your coding agent can name the trace to open
Trace reads through the Bitfab MCP server now include the trace name, so an agent can tell you which trace to look at by name instead of quoting a uuid. Reading traces, searching them, and listing an experiment's traces all report it.
## Replay several datasets in one experiment
A replay could only point at one dataset, so measuring a function against several corpora meant running each of them separately and comparing the experiments by hand. You can now name as many datasets as you like. The run replays their combined traces once, scores them with the graders from every dataset you named, and files the experiment under each one, so it shows up on all of their experiments pages.
```typescript theme={null}
await bitfab.replay("checkout-agent", checkoutAgent, {
datasetIds: [checkoutDataset.id, refundDataset.id],
})
```
From the command line it is `bitfab-replay --dataset-ids ds-a,ds-b`. The existing `--dataset-id` flag keeps working and now takes a list too, so the replay scripts you already have need no changes. All four SDKs have it, as `datasetIds`, `dataset_ids`, and `DatasetIDs`.
## Read verdicts back per assertion
A verdict written against one assertion could only be read back as a passed and failed count, so an agent that scored six assertions on a trace could not say which one failed without opening Studio. `get_trace_labels` now lists each scored assertion on its own line under the trace, with that assertion's verdict, annotation, confidence, and author, keyed by the same assertion id the write used.
## Labels and grader verdicts from the SDKs
The TypeScript and Python SDKs can now read verdicts, write human-validated verdicts, and read the individual verdicts each grader recorded, the same operations available through the Bitfab MCP tools. Reading returns each trace's effective verdict plus one row per scored assertion, so a judge running inside a replay can verify what it wrote without leaving the process.
```typescript theme={null}
const [labels] = await bitfab.labels.getAll([traceId])
for (const verdict of labels.assertions) {
console.log(verdict.assertion, verdict.label, verdict.annotation)
}
const graded = await bitfab.graders.getLabels({ traceIds: [traceId] })
```
`labels.saveHuman` writes a verdict that is validated on write, for cases a person has already decided. Human and agent batches are all-or-nothing. A trace outside your organization or an assertion that is no longer active rejects the whole call and writes nothing. Python exposes the same methods as `labels.get_all`, `labels.save_human`, and `graders.get_labels`.
## Retire an assertion from your coding agent
`archive_trace_assertions` retires one or more assertions on a trace so later replays stop checking them. Archiving keeps the row for audit and leaves any verdicts already recorded against it in place. The call is all-or-nothing, so one unknown or already-archived id fails it and archives nothing.
## Replay only the traces you have assertions for
Replaying a dataset re-ran every trace in it, including the ones nobody had written an assertion for. Those runs cost a full re-execution and come back with nothing to grade. Pass `onlyWithAssertions` to narrow a replay down to the traces carrying at least one assertion.
```python theme={null}
client.replay("agent-turn", run_agent, dataset_id=DATASET, only_with_assertions=True)
```
On `bitfab-replay` the flag is `--only-with-assertions`, and it narrows `--limit`, `--trace-ids`, and `--dataset-id` alike. Paired with `--limit` it picks the N most recent traces that have assertions, instead of taking the N most recent and leaving you with however many of those turn out to be gradeable.
Available in the TypeScript and Python SDKs.
## Experiment totals now match the verdicts on your traces
When a trace carried a label on each of its assertions, its experiment totals could count it as passed even though the trace list showed it as failed. Totals and trace verdicts now follow the same rule, so any failing assertion fails the trace. Grader results count toward that verdict as well, so an experiment's pass rate reflects every label written against a trace.
## Replay diffs stop flagging repeated calls that just ran in a different order
When a replay called the same function many times, the diff matched those calls by position, so a set of identical calls that came back in a different order was reported as a wall of changes. Calls are now matched by what each one was given, so only real differences surface. On one replay with 31 repeated calls, the reported changes dropped from 30 spans to 2, and the change that mattered (a verdict moving from approve to request changes) went from buried to second in the tree.
The Diff view has a new **Diff algorithm** control. **Smart** is the default and matches calls by content. **In order** keeps the old positional matching, for when the sequence itself is what you are testing.
## Threads in the span tree
A trace that ran work on more than one thread now marks each span with the thread it ran on, as a small `T1` or `T2` chip on the row with the full thread name in the row tooltip. Traces that used a single thread are unchanged.
## Grade a replay item inside the process that ran it
`ReplayConcurrency` now takes `on_item_finish_in_child_process`, a hook that runs in the child interpreter that replayed the item rather than in the process that owns the run. Under `primitive="process"` that child is the only place the replayed run's own state still exists, so a judge can read what the run left in memory before the process exits. The item arrives with its replay trace ID already resolved, so a verdict can be keyed by `trace_id` directly instead of by lineage.
```python theme={null}
registry = ReplayRegistry().register(
"support-agent",
client,
run_agent,
concurrency=ReplayConcurrency(
primitive="process", on_item_finish_in_child_process=grade
),
)
```
The registry's own `on_item_finish` is unchanged and still runs in the process that owns the run, so both can be set at once. The child grades what only the child can see, and the parent keeps the running totals and still reports an item whose child died before finishing.
## One verdict per assertion
A trace's verdict now comes from its assertions instead of a single pass or fail. Each assertion carries its own verdict and its own reasoning, so a run that meets four of six assertions reads as 4/6 rather than one failure with the detail buried in a note. The trace verdict is derived from those labels, and any failing assertion fails the trace.
Traces with no assertions are unchanged. They keep their single whole-trace verdict, and every verdict recorded before this release reads exactly as it did.
## Labels and assertions are separate panels
The trace drawer now has a Labels panel and an Assertions panel behind a toggle, and either one collapses to give the full width back to the trace. Labels is where you record verdicts, one row per assertion with its own pass, fail, and note. Assertions stays read only and shows what the trace is expected to do when it is replayed.
Labels are counted rather than named. A badge reads passing over total, so a single verdict shows as 1/1 and six assertions with one passing show as 1/6. Traces with nothing recorded keep the dashed unlabeled pill.
## Score one assertion from the SDKs
The labels namespace takes an optional assertion id, so a replay can grade one assertion at a time. Omit it to write the trace's whole-trace verdict, which behaves exactly as before.
```ts theme={null}
await client.labels.save({
traceId,
assertionId,
label: false,
annotation: "the reply never confirmed the preference was saved",
})
```
Python takes the same argument as `assertion_id` on `save`, `save_all`, `skip`, and `archive`.
## Live in-progress trace indicator
Trace detail pages now make it clear when a trace or replay is still running, with a banner that appears while it's in flight and disappears once results are in. The page keeps checking for updates in the background and jumps to the latest span when you open a trace that's still going.
## Diff view always highlights a span
The experiment diff view now always opens with a span selected, even when every difference between the original and replayed run is a span that simply didn't run this time. Previously the view could open with nothing highlighted.
## Grade each replay attempt as it finishes
A replay registry entry can now carry an `on_item_finish` callback, and `bitfab-replay` runs it after its own progress reporter instead of replacing it. That is the earliest point a verdict can be written, because a replay verdict is keyed by the original trace plus the attempt plus the test run, and that row only exists once the attempt's own trace has been flushed.
```python theme={null}
def grade(progress):
item = progress["item"]
result = client.traces.get_assertions(item["original_trace_id"])
verdict = judge(result["assertions"], item["result"])
client.labels.save(
label=verdict.passed,
annotation=verdict.summary,
original_trace_id=item["original_trace_id"],
attempt=item["attempt"],
test_run_id=progress["test_run_id"],
)
registry = ReplayRegistry().register("support-agent", client, run_agent, on_item_finish=grade)
```
The callback is skipped under `--dry-run`, since nothing ran, and one that raises is reported on stderr with the trace and attempt instead of failing the run. Python for now, with Ruby and TypeScript to follow.
## One verdict with `save`, many with `save_all`
`save_one` is gone. `save` is now the single-verdict call and takes the same target arguments `skip` and `archive` take, and `save_all` takes a list. TypeScript is `save` and `saveAll`.
This is a breaking change. Update any call that used `save_one` or `saveOne`, and any call that passed a list to `save`.
## Faster loading for large traces
Traces with large payloads now open in about a second instead of the fifteen to twenty seconds they could take before. The trace page used to download the full input and output of every span before it could show you anything, which made big agent traces slow to open. Span content now loads when you select a span, so the span tree and timings appear right away.
## Automatic capture policy is in Alpha
The capture control on a TypeScript SDK function trace is now labeled Policy with an Alpha badge. It still opens the Studio capture policy for that function, but the page is read-only for now: you can see which functions include content on future traces, and you cannot change or save the selection while the feature is in Alpha.
## Say what a trace should do the next time it runs
A grader checks something that must hold for every trace of a function. An assertion is the other half: what the right answer is for one specific input. Write it against the trace you are looking at, and it is checked against the replay of that trace rather than against the run you wrote it on.
Your coding agent writes them through two new MCP tools, `save_trace_assertions` and `get_trace_assertions`, and both SDKs read and write them directly:
```python theme={null}
result = client.traces.get_assertions(item.original_trace_id)
verdict = judge(result["assertions"], replay_output)
client.labels.save(
label=verdict.passed,
annotation=verdict.summary,
original_trace_id=item.original_trace_id,
attempt=item.attempt,
test_run_id=run.test_run_id,
)
```
`client.labels` is new too. Verdicts from a replay process used to be reachable only through MCP, which meant a judge running inside your own replay script had nowhere to write. It now writes the same rows, keyed the same way, and `skip` withholds a verdict on an attempt that crashed or an assertion whose target could not be found, so neither one is recorded as a behavior regression.
Assertions show up read-only on the trace page and in the labeling panel. Fill them in from your agent or the SDK for now.
An assertion takes the same `assertion`, `passCriteria`, and `failCriteria` a grader takes, so one that proves out across many traces is promoted into a grader by copying its fields.
## Trace outlines on every replay item
Every replay result item now carries two trace outlines: `originalTraceOutline` for the trace that was replayed and `traceOutline` for the trace the replay produced. An outline is the span tree without any inputs or outputs, so it stays small: each span's name, type, order, duration, tokens, model, errors, and whether it was served from a recording. Use them to grade how a replay reached its output and not only what it produced, such as whether it called the same tools in the same order or leaned on a mocked span that used to run real code.
```ts theme={null}
const result = await bitfab.replay("book-flight", bookFlight)
for (const item of result.items) {
const before = item.originalTraceOutline?.spans.map((span) => span.name)
const after = item.traceOutline?.spans.map((span) => span.name)
}
```
Python and Ruby expose the same fields as `trace_outline` and `original_trace_outline`, and Go as `TraceOutline` and `OriginalTraceOutline`. Both are filled in when the replay completes, so they are `null` on the per-item finish callback and against older servers.
## The assistant grades replays by their path
The Bitfab plugin now writes both outlines into each replay run's per-item files and compares the two span trees when it judges a replay, so a replay that produced the right text by skipping a required tool call, erroring in a child span, or leaning on a mocked span no longer passes on output alone. Items judged while the replay is still running are re-checked against their outlines once the run completes.
## Replay each trace in its own process
Python replay can now give every work item its own process, which makes it possible to replay code that keeps its world in process-global state: a settings module read once at import, a database provisioned per item, or a module-level registry. Previously all items shared one process, so anything that could not be set up twice in the same interpreter only ever replayed its first trace.
```python theme={null}
from bitfab import ReplayConcurrency
registry.register(
"my-pipeline",
client,
run_turn,
concurrency=ReplayConcurrency(attempts=3, primitive="process"),
)
```
`ReplayConcurrency` also carries `attempts` and `max_concurrency`, so a run's repeat count and its parallelism are set in one place and cannot disagree. The fan-out unit is the work item rather than the attempt, so 40 traces at 3 attempts is 120 processes drawn from one queue, bounded by `max_concurrency` (4 by default in process mode). The default primitive stays `"async"`, so existing replays are unchanged. Process mode runs through the `bitfab-replay --registry` command, which is what re-runs your pipeline once per item.
## The Amp plugin, at parity
Amp support graduates from the Alpha skill pack to a native plugin. It registers the same three skills the other hosts get, `bitfab:setup`, `bitfab:assistant`, and `bitfab:update`, and every Bitfab tool, so Amp runs the full workflow itself instead of handing off to a second agent in your terminal.
```bash theme={null}
npx bitfab-cli init --editor amp
```
What this unlocks on Amp: the improvement loop (datasets, labeling, graders, experiments, replay, cost optimization), Studio review for trace plans and template previews, SDK updates through `bitfab:update sdk`, and Bitfab tools you can call directly. Ask Amp to search your traces and it can. The tools are registered by the plugin under their bare names (`search_traces`, `get_traces`, `save_trace_plan`) rather than through an MCP server.
`bitfab-cli` installs the plugin into `~/.config/amp/plugins/bitfab` and confirms Amp loaded it. `analyze-repo --editor amp` runs headless through `amp -x`.
Still to come on Amp: session log capture and the auth and update banner at session start. Both ship through Amp's `agent.start` and `agent.end` events in a follow-up. Details are in the [Amp plugin docs](/amp-plugin).
## Bitfab in Amp, in Alpha
Amp is now a supported host, in Alpha. `bitfab init` installs Bitfab for Amp the same way it does for the other editors:
```bash theme={null}
npx bitfab-cli init --editor amp
```
Because Amp has no Bitfab plugin yet, this installs a skill pack and then runs the setup flow in your terminal rather than handing off to an editor agent. Setup decisions, plan review, and edit approvals all stay where you are. `--editor amp` works for `plugin-install`, `setup`, and `analyze-repo` too, and the CLI offers Amp automatically when it is the editor on your PATH.
Three skills. `bitfab-setup` instruments workflows and covers modify, inspect, replay, database snapshots, and templates. `bitfab-analyze-repo` scans the repository and uploads draft trace plans without prompts or code edits. `bitfab-account` handles sign-in, organization switching, and health checks.
Alpha means the improvement loop is not there yet. Datasets, graders, experiments, trace labeling, and Studio review all need the native Amp plugin, which is planned. Amp also registers no Bitfab MCP tools of its own, so you cannot ask Amp to query your traces directly, though the setup flow itself has the full tool set. Run the improvement loop in Claude Code, Cursor, or Codex for now. Full limits are in the [Amp plugin docs](/amp-plugin).
## Mixing the two tracing styles now fails fast
`withSpan` records exactly the functions you wrap. `withTrace`, `trace`, and `node` record a root plus the first-party calls beneath it. Mixing them in one call stack used to record both surfaces and log a warning. It now throws `MixedTracingError`, matching the Python SDK. A blended stack produced a trace whose shape misrepresented how your code was instrumented and left replay boundaries unpredictable, so it is better caught at the call that made the mistake.
```typescript theme={null}
const loadContext = bitfab.withSpan("fetch-context", fetchContext)
// Throws MixedTracingError. Use node() or withNode() inside a subtree trace.
const runTurn = bitfab.withTrace("agent-turn", async () => loadContext())
```
Pick one style per workflow. Inside a subtree trace, reach for `node()` or `withNode()` when a discovered call needs its own name, type, or replay-mocking policy. Two things are deliberately exempt: framework integrations open their spans on whichever surface surrounds them, and the root that `replay()` and `seedTrace()` wrap around an undecorated function still accepts a subtree trace inside it.
## A clearer span tree in the trace view
Span icons are now filled in their type color instead of drawn as outlines, so the kind of each span reads at a glance. Rows are tighter and every row is the same height, which fits more of a trace on screen before you scroll. The detail pane sits on a light grey ground against the white span list, so the two halves of the page no longer blend into one sheet.
## The collapsed sidebar opens when you click a function
Clicking a function name or All Functions while the sidebar is collapsed now expands it, instead of only navigating. The section links under a function name still navigate without expanding, so you can jump straight to Traces or Datasets and keep the sidebar narrow.
## Seed a replayable trace by running your code once
`seedTrace` can now run a function once and record that execution as a replayable original. Pass the function instead of a case, and the SDK records the real input, the real output, and the first-party subtree beneath it, while capture stays off for everything else. A corpus you hold outside Bitfab becomes traces that `replay` can select, without waiting for production to produce them.
```typescript theme={null}
const bitfab = new Bitfab({ captureEnabled: false })
const traceId = await bitfab.seedTrace("agent-turn", runTicket, {
args: ["T-1"],
metadata: { caseUid: "c-1" },
})
```
The existing case form still writes a trace without running anything. A later replay's `adaptInputs` hook now receives the seeded trace's stored metadata on `ctx.metadata`, so a case keeps its provenance instead of smuggling it through the recorded inputs. The replay CLI gained `--run` for seeding a whole cases file through the registered function.
## captureEnabled replaces enabled
The client option that turns capture off is now `captureEnabled`. The old `enabled` still works and warns once. Turning capture off no longer removes the tracing wiring, so wrapped functions keep their trace function key and both `replay` and `seedTrace` still record against a capture-off client. If you relied on `enabled: false` handing back your original function, you now get a wrapper that runs it untraced.
## A warning when both tracing styles meet
`withSpan` records exactly the functions you wrap. `withTrace`, `trace`, and `node` record a root plus the calls beneath it. Both still work when they meet in one call stack, and the spans nest where they were called, but the SDK now warns once per trace function key so a mixed setup is visible rather than silent.
## TypeScript subtree tracing is ready to install
Marking one function with `trace()` or `withTrace()` records every first-party call beneath it, at any depth, with no wrapper on each one. The build adapters that make those nested calls visible now ship as `bitfab-transform`. Install it next to the SDK and add one adapter where your server code is compiled.
```bash theme={null}
pnpm add -D bitfab-transform
```
Adapters are available for Next.js, NestJS, Vite, Rollup, Rolldown, esbuild, Bun, webpack, Rspack, Rsbuild, Babel, the TypeScript compiler, SWC, and direct Node or `tsx`. The August 25 entry called this package `@bitfab/transform`. It is published as `bitfab-transform`, and the docs now match.
## The Diff view collapses to the spans that changed
A replay's Diff view now folds the spans that match the original trace out of the span tree. Any run of two or more neighboring spans with no change, their children included, collapses into one row reading how many spans it hides, which opens and closes on click. A span whose child changed stays on screen, so nothing that moved is ever hidden behind a fold.
Because the fold follows the diff, loops no longer group the tree here: a loop with one changed iteration reads as that iteration between two folded runs rather than as a single loop row. Each changed span's `+N −M` line counts now sit at the right edge of its row, so they line up down the tree.
## Archive trace functions you are done with
You can now archive a trace function from the sidebar, so a workflow whose code is gone stops crowding the function list. Archiving deletes nothing. Its traces, datasets, experiments, and graders all stay, and the function comes back on its own the next time a trace arrives for it. Archived functions collect in an Archived section at the foot of the sidebar, where you can restore one at any time.
## See whether a traced function still exists in your code
Clicking a function in the sidebar now shows when it last traced, when an agent last confirmed its trace plan, and how likely it is that the code producing it still exists. That likelihood reads as a plain label rather than a number, because it is inferred from those two signals rather than measured.
## LangGraph tracing works inside `trace()` subtrees
A LangGraph graph wrapped with `get_langgraph_integration()` can now run inside a Python `trace()` subtree. Previously the integration's tool and invoke spans raised `MixedTracingError` and stopped the run, so opt-out tracing and the LangGraph tool replay hooks could not be used together. Put `@trace` on the function that calls the graph and everything beneath it is recorded, with tool spans attaching under the call that ran them, counting toward the trace's span budget, and following its `mock_on_replay_default` policy. Outside a `trace()` subtree the integration behaves exactly as before.
## Framework spans nest where they were called
Inside a Python `trace()` subtree, spans from the OpenAI Agents tracing processor, the Claude Agent SDK handler, and the LangGraph callback handler now attach to the function that invoked them instead of collapsing onto the trace root. An agent run started deep inside a workflow now appears at that depth in the span tree, so you can see which step made the call. Span types are unchanged.
## Replay every trace K times in one experiment
Replays can now run each trace several times inside a single experiment, so a flaky result reads as flaky instead of as a regression. Pass `attempts` to `replay` in the TypeScript or Python SDK, or `--attempts N` to `bitfab-replay`, and every trace replays that many times, each attempt with its own verdict, tokens, and database branch.
```python theme={null}
client.replay(book_flight, dataset_id=dataset_id, attempts=3)
```
The experiment page folds the attempts into one row per trace with a passed ratio, calls out traces whose attempts disagree as inconsistent, and keeps every total a real sum over the attempts that ran. Agent verdicts can target a single attempt by passing `attempt` alongside `originalTraceId` in `save_agent_labels`.
## Loops fold up in the span tree
When an agent or batch run repeats the same call, the trace viewer now folds those repeats into one row showing the iteration count, how many iterations failed, and the loop's total duration. Expand it to see the first, last, and failed iterations, and reveal the rest ten at a time. Repeating cycles fold too, such as an LLM call followed by a tool call, and arrowing into a folded loop opens it on the span you land on.
## A cleaner span tree
Each span type now has its own icon color, so LLM calls, agents, and tool calls read at a glance, and connector lines fold into each span instead of running as bare borders. The span panel in the trace view can be dragged wider or narrower, and folds away behind a "Spans" button in the header when you want the full width for the span you are reading.
## Name a trace by the record it is about
Every trace now has a `name`: the title Bitfab shows for it, and a field you can search and filter on. Set it from inside any traced function with `getCurrentTrace().setName(...)` (`set_name` in Python and Ruby, `SetName` in Go), pass `name` when seeding a trace, or set it later through the detached trace handle. Give it the ticket, order, or dataset row the run handled, and the trace list filter and `search_traces` will find that run by its label.
```typescript theme={null}
const triage = bitfab.withSpan("triage", async (ticketId: string) => {
getCurrentTrace().setName(`Ticket ${ticketId}`)
return await handle(ticketId)
})
```
Traces imported from Langfuse, Braintrust, and Keywords AI keep the name they had on the source platform. A trace with no name is titled by its trace function key, as before. The `workflow_name` field the SDKs used to send is now `name`; older SDKs and direct HTTP integrations that still send `workflow_name` keep working, and an explicit `name` always wins over it. The CSV export's `workflowName` column is now `name`.
## Seed originals by running a function once with capture off
`seed_trace` in the Python SDK now runs your function once and records that run as an original trace, with capture still off. The recorded trace carries the root span, the full first-party subtree, and the real inputs and output, so `replay` selects it like any captured trace and every replay of it links back as `original_trace_id`. Pass `metadata` to store a case's provenance on the trace.
```python theme={null}
bitfab = Bitfab(capture_enabled=False)
trace_id = bitfab.seed_trace(
"agent-turn",
run_ticket,
kwargs={"ticket_id": "T-1"},
metadata={"case_uid": "c-1", "suite": "smoke"},
)
bitfab.replay(run_ticket, trace_ids=[trace_id])
```
Async functions run to completion on their own event loop, so call `seed_trace` from synchronous code. With `trace_across_threads=True`, spans from worker threads nest under the seeded root, and an exception is recorded on the root span and re-raised. `bitfab-replay --seed cases.jsonl` runs each case the same way: a case's `input` and `kwargs` are the call itself, and `expected` is no longer accepted because the output is what the run produced. This replaces the earlier Python `seed_trace`, which wrote a root-only trace from a case without running it. The TypeScript `seedTrace` keeps that behavior for now.
## Original trace metadata in the replay input adapter
The `adapt_inputs` hook's context now includes `metadata`, the original trace's stored metadata, so a table-driven adapter can key off a case id or suite without smuggling it through the recorded inputs. It is fetched only when an adapter is registered. In the trace list, a seeded trace that is still running or threw now shows its running or error state instead of the seeded icon.
## Manage datasets from the SDKs
You can now create, read, and modify datasets programmatically, without a coding agent in the loop. Every SDK exposes a datasets namespace (`client.datasets` in TypeScript, Python, and Ruby, `client.Datasets` in Go) with the same operations the Bitfab agent tools offer: save a dataset, list datasets, fetch one, list its trace ids, add or remove traces, assign or unassign graders, and re-run graders over the dataset.
```typescript theme={null}
const { dataset } = await bitfab.datasets.save({
traceFunctionKey: "checkout-agent",
name: "Refund failures",
})
await bitfab.datasets.addTraces(dataset.id, traceIds)
const { run } = await bitfab.datasets.rerunGraders(dataset.id)
```
Saving is an upsert on the dataset name within its trace function, so a script can run repeatedly without creating duplicates. Adding traces or graders reports the ids it skipped instead of failing the whole call, and removing a trace only drops it from the dataset. The same operations are available over HTTP under `/api/sdk/datasets` for anything not using an SDK.
## Opt-in and opt-out tracing never mix
The Python SDK has two tracing surfaces. `@span` is opt-in: only the functions you decorate are recorded. `@trace` with `@node` is opt-out: one root records every first-party function beneath it, and `@node` configures a discovered call without creating a boundary. They were never meant to share a call stack, and a `@trace` root called beneath a `@span` used to detach into a trace of its own, or, inside a replay item, into a second parentless root. The SDK now raises `MixedTracingError` at the boundary in either direction, naming which decorator was entered inside which.
```python theme={null}
@bitfab.span("ticket-detail")
def build_ticket_detail(ticket): ...
@bitfab.trace("ticket-workflow")
def process_ticket(ticket):
return build_ticket_detail(ticket)
process_ticket(ticket)
# MixedTracingError: Opt-in and opt-out tracing can't be mixed: @span (opt-in)
# was entered inside a @trace call (opt-out). ...
```
Inside a `@trace` subtree, configure a step with `@node`. The root span that `replay("key", fn)` wraps around an undecorated callable belongs to neither surface, so an entrypoint that calls a `@trace` root now nests that root beneath the item root instead of leaving it parentless. Nested `@trace` roots still record independent traces.
## Replay with capture off in the Python SDK
`Bitfab(enabled=False)` used to hand your functions back undecorated, so a client with tracing off could not replay them: `replay(fn)` and the replay registry had no trace function key to find. The flag is now `capture_enabled`, and it only controls capture. Decorated functions keep their wrapper, a replay item always records its trace, and `seed_trace` always writes, so one client serves both the environments where you do not want capture and the replay runs you start on purpose.
```python theme={null}
bitfab = Bitfab(capture_enabled=False)
@bitfab.span("support-agent")
def support_agent(message): ...
bitfab.replay(support_agent, limit=20)
```
Every replayed item above records a trace even though capture is off. `enabled` still works as a deprecated alias and logs a one-time warning; move to `capture_enabled` when you upgrade to Python SDK v0.39.1. TypeScript, Ruby, and Go are unchanged for now.
## Seed replayable traces from cases you already have
Replay used to require a captured trace, so a corpus of test cases could not be replayed until the code had run against every one of them. `seedTrace` / `seed_trace` writes a replayable trace directly from a case, recording its arguments as the input and the value you expect as the output, with no execution at all.
```typescript theme={null}
const traceId = bitfab.seedTrace("support-agent", {
input: [{ message: "Cancel my Tuesday booking" }],
expected: { intent: "cancel" },
fn: supportAgent,
})
```
Passing `fn` checks the case against the function's real signature, so a case that could never run is rejected while you seed rather than failing later during replay. The replay registry seeds too, with `bitfab-replay --registry --seed cases.jsonl`, which binds every case to the exact function that replay will select.
A seeded trace records no inner calls, so replay mocking has nothing to substitute and database snapshots are refused rather than silently pinned to the wrong moment. Replays of seeded traces report each item as matched or missed against its expected value instead of same or changed against a previous run, and the trace list marks them with their own icon.
## Keyword arguments now replay as they were called
In the Python SDK, a traced function called with keyword arguments recorded them correctly but replayed them as a single trailing positional argument, so a keyword-only signature raised a `TypeError` during replay. Whether it broke depended on the values, not the signature: a `datetime` or `UUID` anywhere in the call happened to preserve the shape, and plain JSON did not.
Recorded inputs now keep the positional and keyword split whenever keyword arguments are present. If you worked around this with an `adapt_inputs` hook that re-splits a trailing dict, remove it when upgrading to Python SDK v0.39.0, as replay now hands your function the arguments it was originally called with.
## Preview what a replay will run
`--dry-run` resolves every selected trace's inputs, applies any input adapter, and prints the exact arguments your function would receive without calling it. It is the quick way to confirm that recorded inputs still fit a signature you have changed since capture, and no database branches are provisioned for a run that executes nothing.
A replay whose selection matched no traces now exits non-zero instead of reporting a clean run of zero items.
## Setup now surfaces subtree tracing
Setup treated spans as the only instrumentation primitive, so it reached for `withSpan` and `@span` and never surfaced the newer subtree API. It now names all three primitives when it reads the SDK reference, so `trace` (a root plus every first-party call beneath it, with no decorators on those calls) and `node` (naming, typing, capture, and replay-mock policy for a single call inside that subtree) are both on the table while your code is instrumented. The Reference overview in the docs gained a table showing which primitives each SDK supports.
## Setup entirely from the CLI
Bitfab’s experimental setup can now plan, instrument, and verify AI workflows without leaving your terminal. Run `bitfab init --v2` for onboarding or `bitfab setup --v2 [mode]` for a specific setup workflow; repository edits, commands, and setup decisions stay behind terminal approval. Use `--diagram` to print the flow’s state diagram without starting the agent.
## Replay production traces from Go
Go SDK v0.37.0 can now run a trace's historical inputs through your current Go code with typed arguments and results. `GetFunction(...).Replay(...)` supports replay lifecycle callbacks, replay mocking, mock overrides, input adapters, code-change capture, and database snapshots, while waiting for each replayed trace to reach Bitfab before finishing the experiment.
```go theme={null}
result, err := client.GetFunction("support-agent").Replay(
ctx, supportAgent, &bitfab.ReplayOptions{Limit: 10},
)
```
## Stable call ordering across every SDK
TypeScript, Python, Ruby, and Go now record strictly increasing microsecond timestamps, preserving call order even when several calls start within the same millisecond. Go replay also uses delivery acknowledgments from the SDK transport before finalizing an experiment, matching the other SDKs; server polling remains a fallback when delivery is ambiguous.
## Opt a traced subtree into default replay mocking
`@trace(..., { mockOnReplayDefault: true })` / `@trace(..., mock_on_replay_default=True)` now establishes replay mocking as the default for nodes under that trace. Use `@node({ mockOnReplay: false })` / `@node(mock_on_replay=False)` to override the default for code that should keep running live.
This behavior is opt-in per trace. Existing traces and replay strategies are unchanged, including `mock: "all"`, which still mocks every matched recorded descendant. TypeScript transformed descendants inherit the trace policy automatically; Python requires `@node()` on descendants that need replay boundaries.
## More reliable large trace uploads
Large span payloads now go directly to durable storage as they arrive, reducing duplicate work during trace ingestion. If that upload fails, Bitfab temporarily retains the payload and retries automatically so the span is not lost. TypeScript SDK v0.38.8 also sends OpenAI Agents span data only when the span is complete, giving Bitfab one authoritative snapshot with its output, error, and timing.
## Replay LangGraph tools without running them again
LangGraph `ToolNode` calls can now use output mocks during replay. Selected tools return their recorded or overridden output without executing, so unsafe side-effects such as sending an email or charging a card are not triggered again.
`getLangGraphIntegration()` in TypeScript wraps the tool array passed to `ToolNode`. `get_langgraph_integration()` in Python exposes the native `wrap_tool_call` and `awrap_tool_call` hooks accepted by `ToolNode`. `createInvoker()` / `create_invoker()` then returns the normal graph entry point with the callback handler and replayable root already connected. Both integrations preserve native `ToolMessage` and `Command` results and refuse to execute a live tool when replay expected a recorded result but none exists.
This API is experimental (alpha). Calls are currently matched by tool name and occurrence order, so start with deterministic tool flows and test replay against non-production dependencies before relying on it for unsafe side-effects.
## Live experiment grading stays responsive
Experiments now apply grader results smoothly as they arrive, without freezing the page or repeatedly refreshing summaries and statistics. Run headers and grader statistics reconcile once grading finishes, so final results remain accurate even when checks complete together.
## Faster, more reliable trace ingestion
Trace ingestion now completes without waiting for search indexing, reducing timeouts when applications upload large or highly concurrent traces. Newly completed traces may take a brief moment to appear in dashboard search while their lexical and semantic indexes finish in the background.
## Route replay mocks by traced function
You can now register replay mock overrides once on a TypeScript, Python, or Ruby client and scope them to a trace function key, so large replay suites can share mocks without re-declaring them on every call. Resolvers can inspect each span, return a replacement, or return `NO_MOCK_OVERRIDE` (`Bitfab::NO_MOCK_OVERRIDE` in Ruby) to fall through to lower-priority overrides and the replay's base mock strategy; null and undefined remain valid mocked outputs.
```typescript theme={null}
client.registerMockOverride("support-agent", ({ node }) =>
node.spanName === "Summarizer"
? { summary: "stubbed" }
: NO_MOCK_OVERRIDE,
)
```
## Capture complete TypeScript trace subtrees
`@bitfab.trace` and `withTrace` now capture every discovered call's full inputs, output, and thrown error by default. No `node()` annotation or Configure capture step is required. Source exclusions and `capture: false` still omit calls, and an optional Studio policy can narrow later traces to selected function IDs.
Use `@bitfab.node(options)` on transformed methods or `bitfab.withNode(options, fn)` around named standalone functions only when one call needs an explicit name, type, finalizer, omission, or replay policy. Outside an enclosing trace they execute normally without emitting a span. Omitted nodes keep captured descendants attached to the nearest captured parent, and replay-mocked nodes return their recorded outputs.
```typescript theme={null}
@bitfab.node({ type: "llm", mockOnReplay: true })
async generate(message: string) {
return callModel(message)
}
```
## Experiment updates keep your place
Experiment pages now keep open modals and expanded run details in place while new experiment results arrive. Live updates refresh the list and its statistics without flashing the page or closing what you had open.
## Replay with one SDK-owned command
TypeScript SDK v0.38.3, Python SDK v0.38.3, and Ruby SDK v0.38.1 now install `bitfab-replay`, so your project owns only a replay registry mapping short names to the exact traced functions production calls. Pass the registry path and registered name to the command; common flags, progress reporting, and result output stay current when you update the SDK.
```bash theme={null}
bitfab-replay --registry ./scripts/replayRegistry.ts classification \
--limit 10 --param forcedLabel=positive
```
## Parameterize custom replay behavior
Registry entries can combine defaults with an options factory that receives values from `--param` or `--params`, so a replay can configure mock overrides and input adapters without editing the registry. Command-line values override overlapping defaults, while source-conflict validation catches incompatible trace and dataset selections before running.
## Know which function every row came from
The All Functions views for traces, datasets, graders, and experiments pool work from every traced function, and until now nothing on a row said which one produced it. Every row now names its function. Traces, datasets, and graders give it a column of its own, so the names line up and you can scan down them to see where one function ends and the next begins. Lists already scoped to a single function are unchanged, since repeating the same name on every row tells you nothing. Dataset names also stopped truncating early, because the pass rate column was holding on to width it never used.
## Trace complete TypeScript call paths from one root
Mark one TypeScript workflow with `@bitfab.trace(key, options)` or `bitfab.withTrace(key, options, fn)` to record its nested repository calls as lightweight spans. Add `bitfab-transform` through your build tool, then use Configure capture in the trace view to choose which functions include inputs, outputs, and errors on future traces. This experimental release supports Babel, the TypeScript compiler, SWC, common bundlers, direct Node execution, Next.js, and NestJS.
```typescript theme={null}
class SupportAgent {
@bitfab.trace("support-agent")
async run(message: string) {
return this.generate(message)
}
}
```
## Labels update everywhere without a refresh
Labeling a trace now updates its trace list row, open trace, dataset bucket, and related experiments together without refreshing the page or refetching whole lists. Trace-function counts also update immediately, while older sessions and uncertain filter matches still reconcile safely.
## An interactive tour of how Bitfab works
There is a new tutorial page at `/tutorial` that walks through Bitfab in two steps: a small traced workflow you can read in TypeScript, Python, or Ruby, and a live replay of it. Every call in the replay plan has a Re-run or Mock toggle, and flipping one restarts the animation, so you can watch what a replay actually does: which calls execute against your current code, which answer from the recording, and which are skipped because they sit inside a mocked call. Hovering a call links it to the lines of code that created it, and back.
## Runs nobody labels no longer look unfinished
An experiment run with no graders attached now reads as finished once its replays land. Until now the pass rate kept circling and Done never showed the run as complete, because a replay waiting on a person or an agent to label it counted as work still in flight. Runs that do have graders attached now circle until the last grader check reports, instead of reading as done the moment their replays finish.
## See how much each span changed
Comparing a replay against its original trace now shows a +N −M line count on every span in the list, the way a diff counts changed lines. The Diff view already marked which spans differed, but working out which one moved the most meant opening each span in turn. Spans whose content is identical, and spans that ran on only one side of the comparison, show no count.
## Filters stay put when you open a trace
Opening a trace from the traces, experiments, or dataset lists now slides the detail panel in directly beneath the page header, with the filter bar behind it left where it is. You can change Group by, switch between original and replay traces, or switch organizations without first closing the trace you are reading. Regrouping experiments keeps the open trace open and leaves expanded runs expanded, instead of resetting the list underneath you.
## Links to an experiment trace open that trace
A link to a trace inside an experiment now opens straight to it, and reloading the page keeps it open. The run holding that trace opens with it, so the arrows in the panel still step to the next and previous traces in the run.
## Configure traced Python calls without another span
Python SDK v0.38.2 adds `@client.node` to configure a function only when it runs inside an enclosing `@client.trace`, without turning that function into a standalone span. Set `capture=False` to omit the node while keeping its captured descendants connected to the nearest captured parent, or set `mock_on_replay=True` on a captured node to return its recorded output during replay.
```python theme={null}
@client.node(mock_on_replay=True)
def call_model(prompt: str) -> str:
...
```
## Name what to instrument instead of picking from a list
`/bitfab:setup` now takes a file, function, or directory directly when you already know what you want traced. Type it at the prompt and setup reads only that location, instead of scanning your codebase and handing you a list of candidates. If you would rather it find the workflows for you, that option is still there and still the recommended default.
## Setup stops asking questions you already answered
When your project already has instrumentation, setup asks once what you want to do and carries that answer forward. Naming a workflow at the first prompt survives the SDK install, so you are not asked the same thing again a few steps later, and naming an existing trace function key takes you straight to changing what it captures. Setup now also tells you before it installs the SDK and writes your API key, rather than doing it silently.
## Reliable experiment startup
New experiment pages now open into a preparation state while the first replay starts, rather than failing because the experiment group has not been created yet. If setup takes longer, the page switches to a neutral waiting message; genuine request and authorization failures remain distinct and provide diagnostics the Bitfab team can investigate.
## Experiment history appears sooner
Experiment history now shows its first page with the initial page load, so organization-wide and function-specific lists no longer wait for an extra browser request before appearing. Statistics still fill in asynchronously, and pagination and live updates continue working as before.
## Faster, more reliable live updates
Live trace, label, dataset, and experiment updates now arrive with less duplicate work across open Bitfab dashboard tabs. Experiment summaries update immediately when Bitfab has the complete result, while targeted refreshes keep moved runs and labels on replayed or original traces accurate.
## Closely timed calls keep their captured order
Trace views and replays now preserve the original order of calls captured within the same millisecond. Bitfab keeps the full timestamp precision from supported SDKs and trace imports, so closely timed calls no longer swap positions between rendering and replay.
## Faster experiment history
Experiment history now shows each run as soon as its metadata arrives, then fills in verdicts, trace counts, and token costs without blocking the page. Loading rows keep their final size while those statistics arrive, so the list stays stable as it becomes interactive.
## Setup restores a disabled Bitfab MCP server
Running `bitfab init` now re-enables the Bitfab MCP server if it had been switched off for that project. Coding agents remember that setting per project and it survives reinstalling the plugin, so the Bitfab skills would come back while every Bitfab tool stayed missing.
## Clearer recovery when the Bitfab tools are missing
When setup cannot reach the Bitfab tools, it now points you at your editor's MCP settings and stops, rather than continuing into steps that depend on those tools. In Claude Code, enable Bitfab from `/mcp` and re-run setup: the tools load into the running session, and restarting does not help because the setting is stored per project.
## Nested Python traces capture complete trees
Nested `trace()` roots in Python now each record a complete independent trace, matching the result you would get from running either root alone. Shared work appears in both traces with distinct span IDs, while `span()`-decorated functions remain in their active trace and record only once.
Async-generator traces also release capture between pulls and still emit their span if closing raises, preventing unrelated work from leaking into a suspended generator's trace. Nested trace regions intentionally record each session separately, so their shared region produces twice the span volume.
## Re-run graders on a single trace
You can now re-run graders on one trace instead of re-grading a whole dataset. Open a trace and use the Graders panel: each grader row has its own re-run control, and the panel header opens a picker for running several at once. A re-run also takes over a grader that never finished, so a trace left mid-grade can be restarted rather than waited out.
## Grading progress reads as a live pass rate
While a trace is being graded, the score counts passes over the graders that have answered so far, so "3/4" means three of the four back so far passed. A trace whose graders were all just queued shows a grading indicator instead of a zero score.
## Replay items report their own duration, and name the original's fields
Each replay result item now reports how long **that replay** took under `durationMs` / `duration_ms`, measured around the replayed call. It previously carried the duration of the original trace, which meant a live progress line could show a number from a run that happened weeks earlier.
Everything describing the trace being replayed now carries an `original` prefix: `originalDurationMs`, `originalModel`, and the new `originalTokens`, which was already computed server-side but never surfaced. With `tokens` reporting the replayed run, one item now holds both halves of a cost or latency comparison. `model` / `:model` stays as a deprecated alias for the original's model, since there is no replay equivalent.
Replay branch provisioning is also measured now: `dbBranchTimings` / `db_branch_timings` breaks down how long resolving the project, creating the branch, connecting the compute, proving it serves, and running your warm-up SQL each took, on success and on failure alike.
**If you read `durationMs` today, it now means the replay's duration.** Switch to `originalDurationMs` to keep the previous value.
## Replays stop before an unsafe mocked call can run
Selected replay mocks now fail closed across TypeScript, Python, and Ruby. If a replay cannot load the historical span tree, match the recorded occurrence, or retrieve its output, the item returns an error instead of falling through to the real child call. The Claude, Cursor, and Codex plugins also audit unsafe actions and execution-context limitations before running a replay, and database-snapshot checks target the exact captured trace.
## Trace a whole call tree from one decorator
The Python SDK can now record an entire call tree from a single annotation. `trace()` captures the function you decorate plus every function in your own code that it calls, at any depth, with none of them decorated.
```python theme={null}
@bitfab.trace("ticket-triage", type="agent")
def triage(ticket: dict) -> dict:
normalized = normalize_ticket(ticket) # recorded, not decorated
signals = extract_signals(normalized) # recorded, not decorated
return build_response(normalized, signals)
```
Capture is scoped to the traced call, so the rest of your application is unaffected. Your own package is recorded; the standard library, installed dependencies, lambdas, and decorator wrappers are not. `max_depth` and `max_spans` bound each subtree and warn once if they truncate it.
Nested `trace()` roots produce complete independent traces, each with its own span IDs; two roots double span volume in their shared region. A `span()`-decorated call behaves differently: it stays in the active trace, records once, and parents the automatically captured calls beneath it. Async-generator capture is released between yielded items, even when the caller stops consuming without closing the generator.
Experimental, and requires Python 3.12 or newer. On older versions the decorated function still records its own span exactly as `span()` does.
## Replay diffs show every line
Comparing a replay against its original trace now shows the input and output in full. Unchanged lines that used to sit behind an "N unchanged lines" toggle are always visible, so you read a change in the context it happened in without expanding anything.
## See how each replayed span was produced
The diff view now carries the same header as the span view, so a span keeps its name and any error while you are reading its diff, alongside how the replay produced it. A mocked span says whether it returned the output recorded in the original trace or a value your replay code supplied, and a replay that ran against a database snapshot says which database it branched from and when the snapshot was pinned.
Reporting the kind of mock needs the TypeScript, Python, or Ruby SDK version above. Spans captured by earlier versions keep the general "Mocked" label.
## Reach every organization in the switcher
The organization switcher scrolls now. Long organization lists were cut off at the bottom with no way to scroll, which put the organizations latest in the alphabet out of reach.
## More resilient concurrent Studio sessions
Studio is more resilient when multiple trace sessions are active at once. Live event polling now uses backend capacity more efficiently, reducing interruptions while traces are updating.
## Replay diffs recognize renamed files
Replay code diffs now recognize files you renamed, so experiments show the edits inside a moved file instead of counting the whole file as deleted and added. Automatic code-change capture applies this behavior across the TypeScript, Python, and Ruby SDKs and ignores moves with no content changes.
## Replay surfaces each trace's server ID as it finishes
Replay now gives you the server `traceId` for each item the moment that item finishes, instead of only after the whole run completes. The id arrives on both the returned `ReplayItem` and the `onItemFinish` callback, so you can link straight to the trace in Bitfab or start per-item work while the rest of the run is still going. Available across the TypeScript, Python, and Ruby SDKs.
```typescript theme={null}
await client.replay("my-fn", myFn, {
onItemFinish: (progress) => {
// progress.item.traceId is the server trace id, in hand as this item finishes
console.log(progress.item.traceId)
},
})
```
## Function counts arrive with the sidebar
Function names and their exact trace counts now appear together when the Bitfab dashboard loads. The function sidebar no longer fills in totals after navigation is already available, while counts still stay current as traces arrive.
## Trace ingestion survives count update failures
Trace ingestion now completes even if Bitfab encounters a temporary problem updating dashboard function counts. Any resulting count drift is repaired automatically, so newly ingested traces remain available without sacrificing accurate sidebar totals.
## Graders stay on the model you chose
Graders using Gemini on Vertex now retry temporary connection failures without switching away from the model you selected. If a run still fails, Bitfab records safer diagnostic context for faster investigation while keeping credentials out of error reports.
## TypeScript replays finish reliably
TypeScript replays now keep the process alive until their final traces are safely stored or the persistence deadline is reached. This prevents a locally completed replay from leaving its experiment unfinished while the SDK is still waiting for confirmation.
## Faster trace function counts
Trace counts in the function sidebar now load with the page instead of waiting for Bitfab to scan your trace history. Counts stay current as new traces arrive and when traces move between functions, so navigating large projects remains fast without showing stale totals.
## Grader results stay current during re-runs
Re-running graders now shows grading as soon as work is queued, then shows the latest verdict when each check finishes. Past automated results remain available in history, while counts and verdict pills use only the newest result and any human label stays authoritative. Dataset and experiment views update across open browser windows throughout the re-run.
## Live traces keep up with busy runs
The Traces page now stays responsive and up to date when many traces arrive at once. High-volume arrivals refresh in batches, and newly created traces remain visible instead of being lost among follow-up updates.
## One trace plan per function, kept up to date
Bitfab's coding plugins now revise the trace plan you are already looking at instead of posting a new one beside it. Asking for a change while the plan is open in Studio updates that page in place, so the plan you are reviewing is the plan that gets saved, and the window stays where you left it.
The same rule now covers the plans an earlier repo analysis drafted. A draft whose code has moved or been renamed since it was written is rebuilt against your current code and saved back onto the same plan, its trace boundary included, rather than being set aside for a fresh scan to duplicate. A draft for work that is already instrumented, or for a workflow that no longer exists, is retired so it stops being offered every time. Re-analyzing a repository refreshes the drafts it made before instead of adding a second draft per function, and leaves alone any function whose plan you are in the middle of working on.
Every path that would post a plan now checks whether the function already has one, so instrumenting or modifying a function picks up the plan waiting for it instead of starting a rival. Viewing a function's plan shows an unconfirmed one where before it reported nothing to view.
## Trace function pages open instead of erroring
Opening a trace function's traces, or one of its versions, now loads rather than showing an error page. Those two pages worked out which function you meant differently from every other page in that group; they all share one answer now.
## A color means one thing
Language and agent chips, status badges and cautions are now easy to tell apart at a glance. A Python chip used to render in exactly the color that means "passed" and a Go chip in the color that means "mocked", so a language could read as a result. Cautions shared the brand color, so a warning looked like an ordinary highlight. Each of these now has a color of its own, and hover labels that had gone missing from the session timeline are back.
## Graders that error or are still running are now visible
When a grader throws while scoring a trace, or is still working on it, that state now shows on the trace verdict pill, the experiment bars, and the labeling panel. Previously a broken grader looked the same as one that had never run. Neither state counts toward the pass rate, so a grader that failed reads as failed rather than as a lower score, and you can still label a trace by hand while its graders are mid-run.
## One failing grader no longer stops the others
If a grader errors while scoring a trace, the remaining graders now finish instead of being skipped. Re-running graders also picks up the ones that errored, rather than treating them as already scored.
## Database snapshots replay from their original source
Replays now restore database snapshots from the exact source recorded when each trace was captured, even when both a direct Neon connection and a mirrored connection are enabled. If that source has since been replaced or disconnected, replay stops safely instead of restoring from a different database.
## Replay completion stays fast under heavy span traffic
Python and Ruby replays now keep their fast completion path even when applications submit spans faster than the local export queue can drain. A dropped span no longer leaves acknowledgment state behind, and sustained traffic no longer displaces acknowledgments for spans still waiting to be sent, so completed replays avoid falling back to slower server polling.
## Faster trace function navigation
The trace functions sidebar now loads function names before waiting for exact trace counts, so you can navigate as soon as the sidebar appears. Trace totals fill in shortly afterward and remain scoped to your selected environment, without showing placeholder zeros while data is still loading.
## Replays finish as soon as their traces are safely stored
Replays no longer wait on a fixed delay before checking whether their traces were saved. Each SDK now tracks the acknowledgment the server already returns when it accepts a batch, so a replay finishes the moment its traces are durable instead of pausing and then polling. Runs that used to sit waiting now complete as quickly as the data lands.
## Steadier span delivery when ingestion is busy
All four SDKs now retry span delivery the way the OpenTelemetry protocol prescribes. When the server asks for a pause, that pause is respected exactly rather than cut short, and the whole client holds off rather than only the one request that was turned away. Without such a request, retries back off with jitter so many clients do not return in lockstep. Spans are also retried in more of the cases worth retrying, so a brief hiccup during ingestion is less likely to cost you data.
## More reliable live trace updates
Live trace updates in the Bitfab dashboard now stay connected when trace details cannot be loaded, so a temporary database problem no longer interrupts the rest of the stream. The dashboard also stops pending trace loads when you disconnect and fails stalled database connections promptly instead of leaving the stream hanging.
## Large spans stay complete
Bitfab SDKs now preserve highly compressible span payloads up to 7.8 MB instead of trimming them at the normal 2.8 MB request budget, keeping large documents, message histories, and agent state available for replay and evaluation. The TypeScript, Python, Ruby, and Go SDKs send an oversized single span intact when it fits within the 3 MB wire target; payloads above the 8 MB ceiling or those that cannot fit still use the visible trimming fallback.
## Know when every replay item starts and finishes
Replay lifecycle callbacks now identify exactly when each item starts and finishes, so integrations can distinguish queued work from in-flight work and handle each result as it arrives. Use `onItemStart` and `onItemFinish` in TypeScript, or `on_item_start` and `on_item_finish` in Python and Ruby; every callback includes the replay item, including finishes with an error. The deprecated `onProgress` and `on_progress` callbacks remain compatible for existing integrations.
```typescript theme={null}
await bitfab.replay("my-fn", fn, {
onItemStart: ({ item }) => console.log("started", item.originalTraceId),
onItemFinish: ({ item }) => console.log("finished", item.originalTraceId),
})
```
## Trace class methods with TypeScript decorators
TypeScript 5+ projects can now trace class methods with standard ECMAScript decorators. Decorators are an optional shorthand; `withSpan` remains the recommended default for TypeScript 4.x, standalone functions, class fields, accessors, and legacy decorator projects.
```typescript theme={null}
const pipeline = bitfab.getFunction("document-pipeline")
class DocumentService {
@pipeline.span({ type: "agent" })
async process(text: string) { return text.trim().toLowerCase() }
}
```
## Replay counts on the usage page
The usage page now reports replays alongside the traces you send. A Replays card sits next to Total traces, the chart adds a Replays metric that works in both the total and by-function views, and the usage-by-period table and CSV export each carry a Replays column. Replays are counted within your total traces, not on top of them.
## Move experiments between groups
You can now move an existing experiment into another experiment group, or remove it from its group, from your coding agent. Use `save_experiment` with a group ID to move it or `null` to ungroup it; open experiment views update automatically.
## Experiment history your agent can organize
Coding agents can now name and annotate individual experiments and experiment groups, then list recent groups or retrieve one group with its member experiments. This keeps related replay runs discoverable and gives your agent the context behind each iteration when it returns to an experiment later.
## Every replay failure stays visible
Replay now returns one item for every attempted trace, even when setup fails before your function runs or prevents a replay trace from being created. The TypeScript, Python, and Ruby SDKs distinguish trace errors from replay errors, preserve the original exceptions and database snapshot failure details, and retain collected items if the whole run throws. New replay-result serializers keep those structured errors intact in direct stdout and SDK-managed result files.
## More reliable background jobs
Scheduled jobs and other durable background work now keep reaching Inngest when optional authentication or request-logging services are unavailable. This prevents unrelated dependency disruptions from interrupting grading, imports, enrichment, and other workflows that run in the background.
## Trace plans stay current with your code
When you modify tracing for an existing function, Bitfab's coding plugins now reread the current instrumentation before revising its trace plan. The stored plan preserves your earlier capture and replay choices, while newly added, removed, moved, renamed, or unwrapped calls are reconciled against the code so the plan does not silently omit current work. If the trace boundary itself changed, the existing plan moves to the current root instead of preserving an obsolete one. Newly discovered calls also receive samples and replay analysis before the updated plan is saved.
## Grading progress across a whole dataset
Re-running graders on a dataset or experiment now shows every trace in it as grading right away, instead of lighting up a handful of rows at a time as the work moves through them. Each row settles back to its pass rate as its own graders report, and a re-run that fails part-way no longer leaves rows stuck looking like they are still being graded.
## Trace rows say what they are waiting on
A trace whose graders are still scoring now counts up from zero, so a re-grade no longer shows the previous run's pass rate as though it were current. Rows still waiting on the trace itself read "Trace" rather than "Label", and anything in flight carries a moving marker around its pill, so you can tell at a glance which rows are still working.
## Live grading recovers from stalled checks
Live grading now stops individual grader checks that stay stuck too long, then retries a timed-out check once without consuming the retry budget for other temporary failures. The deadline covers the full check, including its own model retries and failure explanation, so one stalled grader no longer keeps an entire live-grading run open indefinitely.
## All Functions keeps the section you are in
Switching from a single function to All Functions in the sidebar now keeps you in the section you were already viewing. Previously it always returned you to Traces, so going from one function's Experiments to every function's Experiments took an extra click. This applies to Traces, Datasets, Experiments, and Graders.
## Simpler trace plan saves
Bitfab's coding plugins now use one `save_trace_plan` tool to create new plans and revise existing ones. Older plugin installations that call `create_trace_plan` or `update_trace_plan` remain compatible, so trace setup and re-instrumentation continue working during upgrades.
## More reliable live grading
Live grading now preserves completed grader checks when a later check needs to retry, avoiding repeated evaluations on traces with multiple graders. Longer grading runs also get more time to finish, while successful results remain authoritative if overlapping runs settle in a different order.
## Trace plans stay within the workflow you are tracing
When the Bitfab plugin drafts a trace plan, it no longer attaches callers from above your workflow's root function. Those nodes counted toward the plan's node total but were never drawn, so the count could be higher than the tree you actually saw. Plans now cover the root and the code beneath it, and the node count matches what is in front of you.
## Faster replay mocking
Replays now transfer only the recorded input and output data needed to reconstruct calls, reducing payload size for traces with large captured context. Eager `mock: "all"` runs also avoid fetching the root output twice, while marked and selective mocking continue to load recorded outputs only when they are used. This improvement is available in the TypeScript, Python, and Ruby SDKs.
## Neon databases connect on every plan
Connecting your own Neon database now works whichever Neon plan you are on. Bitfab used to ask Neon for a custom scale-to-zero delay on the branches it creates, which Neon accepts only on its Scale plan and rejects outright on Free and Launch, so the connection check failed at the create-branch step. Bitfab now leaves that setting alone on databases you own, and the branches it creates for replay follow your own project's scale-to-zero.
## Neon connection errors say what actually went wrong
When connecting a Neon database on the Integrations page fails, the error now quotes Neon's own explanation instead of a bare HTTP status. A refusal that used to read "Request failed with status code 412" now names the cause, such as reaching your project's branch limit, alongside Neon's error code and request id so their support can trace it. Each failed step is still listed separately, so you can see exactly how far the check got.
## Trace plans update in place
Your coding agent now revises an existing trace plan instead of creating a new one each time it changes what gets captured. Each traced function keeps a single authoritative plan, so coming back to it later shows the current capture and replay decisions rather than a stack of competing drafts. Every revision is recorded, so the earlier version is still there to compare against.
## The trace plan page updates while you watch
The trace plan page in Studio now reflects your agent's changes as they happen, with no reload. Spans it captures, uncaptures, or switches between replay and mock update in place, keeping the groups you expanded and your place on the page. The primary button reads Save once the plan differs from the one you were first shown.
## Codex setup launches autonomously again
Codex users can once again run `bitfab init` without permission prompts. The CLI’s `--skip-permissions` option now uses Codex’s supported autonomous mode instead of passing a removed flag that stopped the agent before setup began.
## Replay traces always open in the comparison view
Opening a replay trace now shows the same comparison view experiments use: a verdict chip and a Diff | Original | Replayed toggle that flips between a per-span diff against the original trace, the original itself, and the replayed run. This applies everywhere a replay opens, including the traces list and direct links, so replays no longer fall back to the plain trace view. Labels and a code change are optional; the view shows whatever the replay has.
## Fixed invisible tooltip text
Fixed an issue where tooltips across the dashboard rendered their text in a dark color on the tooltip's dark background, making them unreadable. Tooltips on the sidebar navigation, trace status icons, and elsewhere now show their labels clearly.
## Span-by-span diffs in the replay comparison view
The Diff view now compares a replay to its original span by span instead of only the trace's overall input and output. Spans whose content changed open a line diff, spans the replay never ran appear as faded red rows in the span tree, and new spans the change introduced are tinted green with their full content one click away. Spans that match the original stay selectable and render their content as unchanged lines, so you can still inspect what didn't move.
## Span timing at a glance in the trace view
Every span row in the trace tree now carries a timing bar showing how long that span took as a share of the whole trace, offset by the time that had already elapsed when it started. Durations sit inline next to each span name, index badges are gone, and nesting is tighter, so deep traces fit in the sidebar without truncating span names as early.
The trace and span views also move to a calmer, more consistent color treatment: the selected trace and span read as one highlight color everywhere, span types are identified by their icons, and error, replay, and mock states keep their own distinct colors so problems still stand out.
## Trace across threads in the Python SDK
Functions dispatched to worker threads (`ThreadPoolExecutor`, `loop.run_in_executor`, or `threading.Thread`) previously recorded their spans as separate single-span traces, and replay mocking could not serve them. Construct the client with `trace_across_threads=True` (or set `BITFAB_TRACE_ACROSS_THREADS=1`) and those spans now nest under the trace that submitted the work, with `mock="marked"` replay serving their recorded outputs on the worker thread. Plain `asyncio` code needs no flag.
```python theme={null}
client = Bitfab(trace_across_threads=True)
```
## See where every span ran
Every Python SDK span now records a `runtime` block in its raw data: the thread it executed on, the thread its parent span ran on, and, for work dispatched across threads, the thread that submitted it. Open any span's raw view in the dashboard to inspect it.
## Replay warns when a mock cannot be served
During replay, a span marked `mock_on_replay=True` with nothing to serve now logs a warning saying it ran live, and whether that is because the replayed trace recorded no matching span or because all recorded occurrences were already consumed by earlier calls.
## Faster experiment history
Experiment history now stays responsive in organization, dataset, function, and multi-run views by loading trace details only when you expand an experiment. Single experiments and experiment groups still preload their traces, so focused drill-down remains immediate.
## More reliable Neon database snapshot setup
Connecting a Neon project for database snapshots no longer depends on a restore point from the last minute, so valid projects are not rejected when restore history is still catching up. The Integrations flow now validates the connection against the project's current state and clearly tells you when point-in-time restore is disabled and a restore window needs to be enabled.
## Cancel database snapshot setup while it runs
Setting up per-trace database snapshots can take a while, and you can now stop it partway. The Database section of Integrations shows which step is running (checking your database, preparing branching, validating branches) instead of a single spinner, and offers Cancel setup, which stops the work and removes whatever had been built.
## Snapshot setup failures say what actually went wrong
A failed setup no longer always points at your connection string. Bitfab now tells apart a problem with your own database, a setup it could not finish on its side, and a database that is already connected elsewhere in Bitfab, and gives each its own advice. A status Bitfab could not read is reported as unverified and keeps re-checking itself instead of being shown as a failure.
## Disconnecting a database removes its snapshot copy
Deactivating snapshots now deletes the snapshot copy Bitfab built for that database, so nothing of yours keeps being replicated afterwards. The confirmation spells out what that means first: existing snapshots go with it, traces captured earlier will no longer replay against a snapshot, and connecting again rebuilds from scratch.
## Connecting a Neon project one step at a time
The Neon connection flow now asks one question at a time and collapses each answer into a line you can change, so the key, project, and database you picked stay visible as you go. Start over is always available, and leaving the page and coming back mid-setup no longer clears what you had entered.
## Replay branches tell you what they are pinned to
The database branch handed to your code during a replay now carries the moment it was pinned to, so you can confirm a replay is really reading your data as it stood when the original trace ran. It also carries the branch's own id and the name of the environment variable your app reads. Reading the connection string is still the only thing that counts as using the branch, so inspecting these costs nothing.
```typescript theme={null}
const branch = getCurrentReplayBranch()
console.log(branch?.snapshotTimestamp) // 2026-07-31T20:11:05.688Z
```
Every field the service puts on a branch is now passed through in the TypeScript, Python, and Ruby SDKs, so a field added later reaches your code without an SDK upgrade.
## Traces record the region a replay branch ran in
A replayed trace now records the region its database branch lived in, and the snapshot badge on the trace names it. When one replay looks slower than another, that tells you whether the difference is your code or a database in another region paying a round trip on every query.
## Ruby replay branches keep the database URL out of your logs
In the Ruby SDK, turning a replay branch into JSON no longer includes its database connection string or the internal replay state. Under Rails, serializing one of these objects into a log line or an API response previously carried the connection string with it. It now serializes only the descriptive fields, while code that asks for the connection string directly still gets it.
## Experiment results count every graded replay
Experiments now show a grader verdict on every replay the graders scored, including replays that hit an error partway through. Those rows used to show no result at all and their checks were left out of the run's totals, so the pass rate could disagree with the traces listed under it. Grader progress now reads passed, failed, and grading, and in label mode the counts add up to the number of traces in the run.
## Labels show what is still waiting on you
A verdict written by an agent now stays marked as unreviewed until someone approves it, while keeping its pass or fail color, so you can scan a list and see which traces still need a look. Once approved it reads as settled while still showing that a machine wrote it.
## Spans capture much more of your data
Each span now records up to 2.8 MB of its inputs and outputs, up from 512 KB. A large document, a long message history, or an agent state that used to be replaced with a placeholder is captured in full, so traces stay complete enough to read and replay. This applies to the TypeScript, Python, Ruby, and Go SDKs.
## Large spans arrive instead of going missing
A span carrying more than the limit now ships with its largest fields replaced by placeholders, keeping its name, timing, and remaining fields intact. Previously a span that big could be discarded on the way out, leaving a hole in the trace with nothing to explain it. When a span is trimmed the SDK warns once in your logs and marks the trace as incomplete, so a partial capture is never mistaken for a faithful one.
## Steadier trace list when opening a trace
Opening a trace from the trace list no longer shifts the list sideways. The list used to slide left and drift back as the detail panel slid in, which made it easy to lose your place while working through a run of traces. The list now stays put while the panel opens, and the span you select still scrolls into view inside the panel.
## Long code change descriptions no longer cut off the window
Opening the code change behind an experiment whose description ran several lines used to push the bottom of the window off screen, with no way to scroll back to it, leaving the file diffs underneath unreachable. The window now stays on screen, with the description pinned at the top and the diffs scrolling beneath it.
## Confirm your database snapshot connection
Your coding agent can now tell whether database snapshots use a direct Neon project or a Bitfab-managed Postgres mirror. The `get_database_connection_status` tool also reports the pinned Neon project name and ID for direct connections, making it easy to confirm which project replay uses.
## SDKs gzip the traces they send
Every SDK now compresses request bodies of 8,192 bytes or more with gzip, which is where most of a trace's bandwidth goes. Smaller requests are unchanged, because compressing them would cost more than the bytes it saves. Nothing to configure, and no new dependencies: each SDK uses its language's standard library.
Compression is best-effort. If it ever fails, the SDK sends the original body rather than dropping the recorded call. Set `BITFAB_DISABLE_COMPRESSION` to turn it off, and note that browsers without `CompressionStream` fall back to uncompressed automatically.
## Shareable links to any trace
Opening a trace now puts it in the page address everywhere you can open one: the trace list, inside a dataset, and inside an experiment. Copy that link to send someone the exact trace you are looking at, and it opens straight to it. Back and Forward now step through the traces you opened instead of leaving the view.
## A clearer sidebar
The sidebar runs the full height of the window and lists Traces, Datasets, Experiments and Graders under whichever function you are working on, so you can move between them directly instead of through a menu. Each function shows how many traces it has. Collapsing it leaves a strip of icons rather than hiding it, so it is always one click back.
## Reliable trace delivery
SDK trace uploads now complete reliably during concurrent ingestion instead of timing out. This keeps traces flowing into the dashboard even when many spans for the same trace arrive together.
## Traces with self-referencing data are readable again
Opening a trace whose recorded input or output contained a self-referencing value used to fail. Because traces are read in batches, a single affected trace could also stop the others in the same request from loading. These traces now open normally, with the repeating value shown as a placeholder, and one unreadable trace no longer affects the others in the same request. This applies to traces you have already recorded, so nothing needs to be re-run.
## Ruby spans containing NaN or Infinity now send
The Ruby SDK now records non-finite numbers such as `NaN` and `Infinity` as a placeholder instead of failing to encode them. Previously a single one of these values could stop an entire batch of spans from reaching Bitfab, so unrelated spans were lost alongside it. This matches how the Python SDK already handles the same values.
## Sharper cutoff for snapshots from a replaced database connection
The date Bitfab uses to decide which traces predate a replaced database connection now comes from the connection that actually holds your snapshots, rather than the oldest database connection on your account. If you connect a second snapshot provider after running on a first one, traces captured before that switch now correctly show "Snapshot on old branch" instead of replaying against a project that never held their data.
## Grader reruns recover from interrupted jobs
Dataset and experiment grader reruns no longer stay blocked forever when an earlier run is interrupted. After a stale run expires, you can start the graders again instead of remaining stuck on an in-progress state.
## More reliable database snapshot replay
Database snapshot metadata now fills in reliably across large trace histories, avoiding stalls that could leave older traces incomplete for replay. This improves replay consistency for organizations with extensive trace histories.
## Re-run graders from your coding agent
Your coding agent can now re-score existing traces with the `rerun_graders_on_dataset` and `rerun_graders_on_experiment` tools. Reach for them when traces were added to a dataset before a grader existed, or after you change a grader's criteria and want everything scored again, since attaching a grader records the assignment without grading anything on its own. Both default to every grader attached to the dataset or experiment, and report how many traces were graded once the run finishes.
## Grading progress on dataset and experiment rows
Traces now show that grading is under way while a re-run is in flight, instead of holding the previous run's scores until new results arrive. Each row's pass-rate pill switches to a pending state when its graders start and settles on the new score when they finish.
## Open Studio resources across organizations
You can now open Studio datasets, experiments, traces, graders, functions, and trace plans from any organization you belong to without switching your active organization first. When a resource belongs to a different organization, Studio shows its organization in the header so you can see which workspace you are working in.
## Leaner plugin skill discovery
Bitfab's Claude, Cursor, and Codex plugins now load leaner skill descriptions, leaving more of the agent's startup context available while preserving natural-language routing. Command usage hints are generated from the same declared modes the flows execute, so setup, update, and assistant guidance stays aligned as those modes evolve.
## Setup reuses trace plans you already drafted
When you instrument your app with the Bitfab plugin, setup now picks up a trace plan you drafted earlier while analyzing your repository instead of rebuilding one from scratch. If a saved draft already covers the workflow you're instrumenting, the plugin reuses it, so the analysis work carries straight into instrumentation.
## Discover existing trace plans from your coding agent
A new `list_trace_plans` tool lets your coding agent see the trace plans already in your workspace, filtered by status or how they were created. This is what lets setup find and reuse an earlier draft rather than starting over.
## Snapshots from a replaced database connection
Traces whose database snapshot was captured against a connection you have since replaced now show "Snapshot on old branch" instead of "Snapshot captured", and replaying one is refused with an explanation. Those replays previously ran against your current database without saying so, returning results that looked valid but restored the wrong data. The check applies whether Bitfab maintains the snapshot copy for you or you connect your own Neon project.
## Faster trace export in every SDK
Each carrier is now encoded exactly once and the request body is assembled from those encodings. Packing a batch used to re-encode the whole request for every span it considered, which re-escaped the captured inputs and outputs over and over. A full export window is now roughly 10x cheaper to prepare, which in Node.js is time given back to your event loop.
## OpenTelemetry Collector delivery has been removed
`BITFAB_OTEL_EXPORTER_ENDPOINT` no longer routes SDK traces through your own OpenTelemetry Collector; setting it has no effect and SDKs deliver to Bitfab directly. Removing it drops `@opentelemetry/exporter-trace-otlp-proto` from the TypeScript SDK, `opentelemetry-exporter-otlp-proto-http` (and protobuf) from the Python SDK, and gRPC, protobuf, and grpc-gateway from the Go SDK's dependency graph entirely.
If you route telemetry through a Collector today, point its `otlphttp` exporter at Bitfab instead. Bitfab's ingress accepts standard OTLP/JSON, so a Collector can still forward to us:
```yaml theme={null}
exporters:
otlphttp/bitfab:
endpoint: https://bitfab.ai/api/sdk/otel
encoding: json
headers:
Authorization: Bearer ${env:BITFAB_API_KEY}
```
## More reliable TypeScript trace delivery
TypeScript SDK traces now use OpenTelemetry's batching and lifecycle machinery behind the existing `withSpan` wrapper and framework integrations, so no instrumentation changes are required. Large captured inputs and outputs are packed beneath request-size limits, completed requests are delivered concurrently, and `flushTraces()` now reports whether delivery actually succeeded instead of only that the queue drained.
## Route TypeScript traces through an OpenTelemetry Collector
TypeScript applications can send Bitfab traces through an OpenTelemetry Collector by setting `BITFAB_OTEL_EXPORTER_ENDPOINT` to the Collector's OTLP/HTTP base URL. Collector delivery replaces direct delivery, so enabling it does not duplicate traces. The protobuf exporter ships with the SDK and is code-split, so it loads only when you set an endpoint.
## Replay waits for every TypeScript span
TypeScript replay now verifies that each trace is complete and every expected span has persisted before finalizing the run. A successful queue flush or Collector acknowledgment can no longer make an incomplete replay appear ready.
## BAML `call()` traces ride the same pipeline
Traces from locally executed BAML functions now travel through the same batching transport as spans and trace completions, instead of posting one request each. Nothing changes in how you call `call()`.
## Detached trace updates now confirm they applied
`client.getTrace(id).addContext()`, `.setMetadata()`, and `.setSessionId()` now wait for the server and reject if the update is refused. Previously they resolved regardless and a rejected update, such as one naming a trace ID that does not exist, was only written to the log. If you were ignoring the returned promise, await it.
```typescript theme={null}
const trace = client.getTrace(traceId)
await trace.addContext({ refundStatus: "approved" })
```
## Close a client when you are done with it
`client.close()` flushes and releases the trace transport shared by a client's wrapped functions and framework handlers, which matters for long-running processes that create short-lived clients.
See [OpenTelemetry Transport Architecture](/otel-architecture) for the full design.
## More reliable Ruby trace delivery
Ruby SDK traces now use OpenTelemetry's batching and lifecycle machinery behind the existing `bitfab_span` and `Bitfab::Traceable` APIs, so no instrumentation changes are required. Large captured inputs and outputs are packed beneath request-size limits, completed requests are delivered concurrently, and `Bitfab.flush_traces` now reports whether queued traces actually landed instead of only that the queue drained. A new `client.close` releases a transient client's batch worker when a long-running process builds clients it later discards.
## Route Ruby traces through an OpenTelemetry Collector
Ruby applications can now send Bitfab traces through an OpenTelemetry Collector by adding the `opentelemetry-exporter-otlp` gem and setting `BITFAB_OTEL_EXPORTER_ENDPOINT` to the Collector's OTLP/HTTP base URL. Collector delivery uses the official OpenTelemetry exporter and replaces direct delivery, so enabling it does not duplicate traces. If the gem is missing, the SDK warns once and keeps delivering directly rather than dropping spans.
## Bring your own OpenTelemetry version
The gem accepts `opentelemetry-sdk` from 1.2 up to but not including 2.0, verified across that range, so an application already pinning its own OpenTelemetry version does not have to move it to upgrade Bitfab.
## Replay waits for every Ruby span
Ruby replay now verifies that each trace is complete and every expected span has persisted before finalizing the run, instead of waiting on per-item upload threads. A successful queue flush or Collector acknowledgment can no longer make an incomplete replay appear ready.
## Faster trace export in the Python SDK
Preparing a batch of spans to send is now about 2.5 times faster, continuing the encoding work in v0.33.8. Sending 64 spans spends 4.9ms on preparation instead of 12.8ms, and the saving grows with batch size, so the busiest traced functions benefit most. Nothing changes in how you instrument your code.
## Go SDK spans now ship over OpenTelemetry
The Go SDK now batches and delivers spans through the OpenTelemetry trace SDK instead of one HTTP request per span. Nothing changes in how you instrument: `Span`, `Start`/`End`, and `GetFunction` still produce Bitfab spans with Bitfab trace IDs, and no OpenTelemetry type appears in the Bitfab API. Under load this means far fewer, larger requests, and a bounded queue that will not pile up unbounded work behind a slow network.
Each client lazily starts one background worker on its first span, so a client that never traces starts nothing. The SDK keeps its own private tracer provider and never touches your application's global OpenTelemetry provider, so your traces and Bitfab's stay separate.
## Close a client when you are done with it
`Close` flushes pending spans and shuts the worker down for good. `FlushTraces` now also reports whether delivery actually succeeded.
```go theme={null}
client := bitfab.NewClient(os.Getenv("BITFAB_API_KEY"))
defer client.Close(5 * time.Second)
```
## Send Go spans through your own collector
If you already run an OpenTelemetry collector, set `BITFAB_OTEL_EXPORTER_ENDPOINT` to it (for example `http://localhost:4318`) and Bitfab spans flow through your existing telemetry pipeline instead of going straight to us. Configure that collector to forward to Bitfab. Leave the variable unset and the SDK talks to Bitfab directly, which is the default.
## Go 1.25 is now required
The OpenTelemetry Go modules require Go 1.25, so the Bitfab Go SDK does too. The official OTLP/HTTP exporter is linked into every build, because Go has no dynamic import to defer it behind the collector opt-in.
One note on v0.12.7's encoding change: spans are still encoded only once, but that encode now happens on the thread that ran your traced function rather than in a per-span background sender, because queueing a span no longer starts a goroutine. Queueing is non-blocking, so the delivery itself never sits on your thread.
See [OpenTelemetry Transport Architecture](/otel-architecture) for the full design.
## Lower tracing overhead in the Python and Go SDKs
Both SDKs were encoding each span twice on its way out: once to check the data could be serialized, then again to build the request that ships it. They now reuse the first encode, which roughly halves the encoding work a span costs in Python and cuts it by about a third in Go. The saving scales with how much your spans capture, so the largest inputs and outputs benefit most.
The Go SDK also moves the span's final encode into its background sender, so less work happens on the thread that ran your traced function.
## Plugin updates work wherever the plugin is installed
Claude Code can install the Bitfab plugin for a single project instead of your whole account, and updating a project-scoped install now works. Previously `npx bitfab-cli init` and `/bitfab:update` assumed the account-wide install, so they failed with a "not installed at scope user" error and quietly left you on the old version. Both now update the install that belongs to the project you are in, and if you have the plugin installed in more than one place, a failure on one of them is reported instead of being hidden behind a success message.
## Database connection changes now take effect
Updating the connection string for database snapshots now applies to the connection you already have, instead of being saved without changing anything. Rotating a password or username updates in place, with no interruption to snapshots and no loss of history.
## Confirmation before replacing a connected database
Pointing snapshots at a different database now asks you to confirm first. The existing snapshot connection cannot be moved across, so it has to be torn down and rebuilt, and snapshots captured before the switch are lost. Bitfab checks the new database is set up for replication before anything is torn down, so a database it cannot mirror is refused while your existing connection stays untouched.
## More accurate snapshot status
The database snapshot status now reflects the live state of your connection. A temporary outage on our side no longer shows a working connection as failed, and a connection that stalled partway through setup now reports a failure you can act on instead of showing as still activating.
## Diff view opens on the first real change
The experiments Diff view now lands directly on the first span whose input or output actually differs from the original run, instead of opening on the trace root. Spans with no differences (and the trace root itself) are dimmed and unselectable, and when a replay matches the original everywhere, the view says "No diff available" instead of showing an empty comparison.
## Detached trace updates now confirm they applied
In the Python SDK, `client.get_trace(id).add_context()`, `.set_metadata()`, and `.set_session_id()` now wait for the server and raise if the update is rejected. Previously they returned a `threading.Thread` you could join, and a rejected update, such as one naming a trace ID that does not exist, was only written to the log. If you were joining the returned thread you can drop that call, because the update has already been applied by the time the method returns.
```python theme={null}
trace = client.get_trace(trace_id)
trace.add_context({"refund_status": "approved"})
```
## Compressed trace ingestion
Bitfab's ingestion endpoints now accept gzip-compressed request bodies. If you send traces through an OpenTelemetry Collector, its default compression now works as-is, so there is no need to turn compression off on the exporter. Requests compressed with deflate work too, and the accepted encodings are listed in the HTTP endpoints reference.
## A connected Neon database now counts everywhere
If you connect a Neon project directly, Bitfab treats it as your database connection across the product: traces show the "Snapshot captured" badge, and the connection status shown on the Integrations page and reported to your coding agent reflects the Neon project you pinned. Those signals previously only recognized a connection made through the Database section, so a Neon-connected workspace could look unconnected while replay was already branching its database.
## A brief Neon outage no longer looks like a broken connection
When Neon cannot be reached for a moment, your connection is now marked unverified and re-checked on its own instead of being reported as broken. You are only asked to reconnect when the API key or the project has actually changed.
## Setup teaches the primitives before it touches your code
Setup now opens by explaining the two primitives you instrument with, `withSpan` and `replay`, including the five ways replay can change a method when you re-run a captured trace: run it normally, feed it the recorded inputs, feed it modified inputs, or skip it and return its recorded or modified outputs. Instrumenting asks you to make that choice per method, so the explanation now comes first, right after you sign in and before any part of your repository is read. Running `/bitfab:setup explain` on its own shows the same thing without signing in.
## Choose guided setup, or instrument it yourself
After signing in, setup asks whether you want to be walked through instrumenting or would rather do it yourself. Guided is the default and drives the whole thing, checking with you at each decision. Choosing to do it yourself hands over the SDK guide for your language and where to fetch your API key, then stops without scanning your codebase or changing a line.
## Know what setup will do before it starts
Once you pick the guided path, setup says what is about to happen before it reads anything: it analyzes your repository, instruments the AI features you choose, and writes a replay script. It also notes that it will prompt you whenever it needs input, and that the whole thing usually takes 10 to 17 minutes depending on how many features you instrument.
## Close or update a trace plan and keep going
The trace plan page in Studio now has a single button. It reads Close when you have not changed anything, and Update once you have toggled which spans get captured or how they replay. Either one saves the plan and hands straight back to your coding agent, which carries on writing the instrumentation. Closing the window does the same thing, so opening a plan to look at it is never a dead end.
## Skip the browser without losing your edits
While the plan is open, your agent asks in the terminal whether to continue instrumenting, so you can skip the review entirely and it will close the plan page for you. Answer it whenever you like: a capture set you saved in Studio always takes precedence over that prompt, so the spans you chose are the spans that get traced.
## Connect a Neon database directly
If your production database runs on Neon, you can now connect it directly from the Integrations page instead of provisioning a replicated copy. Bitfab branches your own project at each trace's timestamp, so replay reads the database as it was when the trace was captured. Paste a Neon API key and Bitfab handles the rest, asking which project or database only when there is more than one to choose from.
Before anything is saved, the key is checked against the real workflow: Bitfab creates a branch at a past point in time, reads its connection string, and deletes it again. A key that cannot do all three is rejected with the step that failed, so a broken connection surfaces during setup instead of during your first replay.
## More reliable Python trace delivery
Python SDK traces now use OpenTelemetry's batching and lifecycle machinery behind the existing Bitfab decorators and framework integrations, so no instrumentation changes are required. Large captured inputs and outputs are packed beneath request-size limits, completed requests can be delivered concurrently, and shutdown reports when queued traces could not finish instead of silently claiming success.
## Route Python traces through an OpenTelemetry Collector
Python applications can now send Bitfab traces through an OpenTelemetry Collector by setting `BITFAB_OTEL_EXPORTER_ENDPOINT` to the Collector's OTLP/HTTP base URL. Collector delivery uses the official OpenTelemetry exporter and replaces direct delivery, so enabling it does not duplicate traces.
## Replay waits for every Python span
Python replay now verifies that each trace is complete and every expected span has persisted before finalizing the run. A successful queue flush or Collector acknowledgment can no longer make an incomplete replay appear ready.
## Clearer errors when replay warm-up SQL fails
When a replay provisions a database branch and your warm-up SQL fails, the error now names the actual failure instead of reporting it as a four-minute timeout. A statement the database rejects, such as one referencing a table or index that is not in the snapshot, comes back immediately as `branch_warmup_invalid`, while a warm-up that genuinely runs out of time still reports as `branch_warmup_failed`.
## Faster failure when a database branch cannot start
A replay database branch that never accepts a connection now fails in about ten seconds instead of holding the request for up to four minutes.
## Collapse the labeling panel on any trace
The labeling panel can now be collapsed on every trace you open, including the traces list and dataset review, where it was previously pinned open and always held a column of the drawer. Collapsing hands that space back to the trace itself, and your choice carries as you step between traces and reopen the drawer. The toggle sits in the trace header beside the previous and next controls, and names what it opens: Graders on grader-scored traces, Label everywhere else.
## Two ways into every empty rail section
A dataset's Graders and Experiments sections now stay on the page even when they're empty, and each teaches both ways to fill it: ask your coding agent (a card with a copyable prompt and the MCP endpoints it uses) or do it yourself with the Manage modal. Editing a dataset moved into an Edit popover in the header, and Manage and Re-run live in a compact menu on the Graders section.
## Bulk approval back in the grader rail
When your coding agent has labeled traces that await your review, the Graders section shows "Approve all pending" with a live count, opening the same confirmation flow as before. This also fixes the Re-run action in the section menu, which previously closed without opening the grader picker.
## Studio keeps experiments in reach
Finishing a labeling session in Studio brings back the list of experiments run against that dataset, so the natural next step after labeling stays one click away.
## Clearing a filtered experiments view
Opening experiments scoped to a dataset or a single run now shows that scope as a pill in the list header, with an x to clear it. Clearing drops you back to every run for the function, so leaving a filtered view no longer means editing the URL or navigating away.
## Settings pages scroll again
Content that ran past the bottom of the window on the Integrations, API Keys, and Export Traces pages was being cut off with no way to scroll down to it. These pages scroll normally now, so the connection string form on Integrations and the full list on API Keys stay reachable whatever the height of your window.
## The MCP endpoints behind graders, datasets, and experiments
Each of these pages now lists the MCP endpoints your coding agent calls to read and manage that resource, so the surface is visible from the page itself instead of living only in your agent's tool list. The list sits under the empty state on a function with nothing to show yet, and the "New" button on each of these pages carries the same list in a section you can expand, next to a copyable prompt scoped to the function you're on.
## Graders explain how they get made
A trace function with no graders now shows what a grader is and a prompt you can copy into your coding agent to create one, matching what datasets and experiments already did. A grader's own page adds an "Edit" button that hands you a prompt naming that grader, for when you want to tighten its criteria or rename it.
## Experiments list, grouped and one line per run
The experiments list now gives each run a single line, gathered under the experiment group that produced it, so pass rates, before and after deltas, and code changes line up in columns you can read straight down. A "Group by" control in the header switches between grouping by experiment group and a flat list, and your choice travels in the URL, so a link you share opens on the same view you were looking at.
Opening a run is quicker as well. Its traces show a placeholder the moment you click, then fill in, and they stay put when you collapse the run, so reopening it is immediate.
## Framework-observed spans can't be mocked in replay plans
Replay plans no longer let you mock spans that a framework's instrumentation only observes, such as LangChain or LangGraph calls. Because the SDK can't intercept these spans, a mock would silently have no effect and the call would re-run anyway, so the trace plan now locks them with a clear explanation instead of showing a warning you could override. Spans that genuinely support mocking, including Vercel AI SDK model calls, are unaffected.
## Turn on replay database branching with a boolean
Replaying against a database branch restored to each trace's original state now reads as the switch it is. Pass `dbBranch: true` in TypeScript, `db_branch=True` in Python, or `db_branch: true` in Ruby, and every replay item runs against a branch sized by your mirror project's own defaults. Passing an options object still works and is now only for tuning the branch's compute and warm-up, and `false` turns branching off explicitly.
```typescript theme={null}
await bitfab.replay("checkout-agent", runCheckout, {
limit: 10,
dbBranch: true,
})
```
## Trace helpers only when nested
Reusable helper functions can now appear as child spans inside an existing Bitfab trace without creating standalone root traces when called on their own. Set `captureWhen: "nested"` in TypeScript, `capture_when="nested"` in Python or Ruby, or `WithCaptureWhen(CaptureWhenNested)` in Go. Your helper still runs normally without a parent span, and unknown values warn once before falling back to the existing always-capture behavior.
```typescript theme={null}
const helper = bitfab.withSpan("workflow", {
captureWhen: "nested",
}, async (input) => transform(input))
```
## Control replay code-change capture
Replay scripts now distinguish omitted code-change files from an explicit null value, so you can keep automatic git-diff capture by default or turn it off for a specific run. Supplying only a custom code-change description preserves that description while Bitfab still captures the changed files automatically. Pass `codeChangeFiles: null` in TypeScript, `code_change_files=None` in Python, or `code_change_files: nil` in Ruby to opt out.
```typescript theme={null}
await bitfab.replay("my-function", updatedFn, {
codeChangeFiles: null,
})
```
## Clearer experiment completion and grader progress
Completed experiment replays without labels now appear as unlabeled and count as complete, instead of carrying a separate awaiting-label status. While graders are still running, experiment and trace progress bars now separate finished pass/fail results from pending grader checks in purple, so replay completion and grader completion remain distinct.
## The usage page opens right away
The usage page now renders as soon as you open it, instead of waiting on a scan of your whole trace history before showing anything. For organizations with a lot of history that wait ran past ten seconds, long enough that the Usage link looked like it did nothing at all. The heading, controls, and layout appear immediately now, and the numbers fill in as they arrive.
## Loading placeholders instead of misleading empty states
While usage data is still loading, the stat cards, chart, and period table now show placeholders rather than dashes and a spinner. The period table also no longer says "No usage in this period yet" before its data has arrived, and the CSV download stays disabled until there is something to export.
## Consistent names across every MCP tool
Bitfab's MCP tools now follow one naming rule: `list_` for collections, `get_` when you already have the id, `create_` for things that are new every time, `save_` for create-or-update, and `add_`/`remove_` for attaching and detaching. Ten tools were renamed under it, including `read_traces` to `get_traces` and `update_agent_labels` to `save_agent_labels`. Nothing breaks if you are on an older plugin version: the previous names are still accepted, so your coding agent keeps working until you update.
## Datasets stop duplicating when a flow re-runs
`save_dataset` (previously `create_dataset`) now updates the existing dataset when you save one whose name already exists on the same traced function, instead of adding a second copy. Re-running a labeling flow, or picking one back up after your coding agent loses context, no longer leaves duplicates behind to clean up. The same name under a different traced function still creates its own dataset.
## Grader changes across a whole experiment group
`add_graders_to_experiment_group` now takes several graders in one call, and the new `remove_graders_from_experiment_group` detaches them again from every experiment currently in the group. A grader archived after it was assigned can still be removed, and ids that were never assigned are reported back instead of failing the call.
## Replay concurrency now bounds database branches
Replay now creates a historical database branch only when an item enters a worker, so `maxConcurrency` and `max_concurrency` limit both active work and live branches. Larger runs avoid provisioning every branch at startup, and older SDKs remain compatible through the eager server path. If a branch cannot be created, the item reports the resolver's specific reason instead of silently running without historical data.
## Warm-up SQL gets 240 seconds
Warm-up SQL passed with a replay's `dbBranch` settings now gets 240 seconds to run, rather than sharing the 10 seconds the branch readiness check allows. Warming a real working set takes far longer than proving a branch answers a query, so warm-ups that used to fail the lease now finish. If warm-up SQL does fail or outrun its budget, the error says so specifically instead of reporting the branch as never having come up.
Starting a replay with database branching on now also waits up to 300 seconds rather than 180. A warm-up longer than about three minutes needs an SDK carrying that longer wait, so upgrade the TypeScript, Python, or Ruby SDK before relying on one.
## Keyboard navigation in the trace list
Arrow keys now move through traces one row at a time instead of jumping to the end of the list. Up and down (or `j` and `k`) walk the list, left and right (or `h` and `l`) step to the previous or next trace, and Enter opens the one under the cursor. With a trace open, up and down move between its spans while left and right keep moving between traces.
## Save & Next keeps you moving through a dataset
Saving a label from a dataset's trace panel now opens the next trace in the list instead of closing the panel. This previously only worked for traces waiting on your approval, so labeled and unlabeled traces dropped you back to the list and made you pick the next one by hand.
## Database-snapshot setup wires the new replay branch
`/bitfab:setup db-snapshot` in Claude Code, Cursor, and Codex now wires replay using the `dbBranch` option and the replay-branch accessor introduced in SDK v0.33.0, in place of the removed `ReplayEnvironment`.
## Breaking: `ReplayEnvironment` is replaced by a replay-branch accessor
Database branching for replay is now turned on by the `dbBranch` replay option, and the resolved branch is read through an accessor rather than an object you construct and pass in. `ReplayEnvironment` is removed.
The old shape asked two unrelated questions with one signal: constructing an environment object was both "give me a database branch" and "here is the thing I will read inside my function". Since the branch was always resolved through the replay context, never from the object you passed, two different instances behaved identically and the object could not actually be used for dependency injection. Making `dbBranch`'s presence the switch and the branch an accessor states what was already true.
Before:
```typescript theme={null}
const env = new Bitfab.ReplayEnvironment({ minCu: 2, maxCu: 2 })
const runCheckout = pipeline.withSpan({ name: "CheckoutAgent" }, async (id) => {
const url = env.active ? env.databaseUrl : process.env.DATABASE_URL
return decideRefund(await makeDbClient(url).orders.find(id))
})
await client.replay("checkout-agent", runCheckout, { environment: env })
```
After:
```typescript theme={null}
import { getCurrentReplayBranch } from "@bitfab/sdk"
const runCheckout = pipeline.withSpan({ name: "CheckoutAgent" }, async (id) => {
const branch = getCurrentReplayBranch()
const url = branch?.databaseUrl ?? process.env.DATABASE_URL
return decideRefund(await makeDbClient(url).orders.find(id))
})
await client.replay("checkout-agent", runCheckout, {
dbBranch: { minCu: 2, maxCu: 2 },
})
```
The same move applies in Python (`db_branch={"min_cu": 2}` with `get_current_replay_branch()`) and Ruby (`db_branch: {min_cu: 2}` with `Bitfab.current_replay_branch`). Passing `dbBranch` at all is what enables branching, so `dbBranch: {}` turns it on with your mirror's own sizing. The accessor returns `null` / `None` / `nil` outside a replay item and for items with no branch, which replaces the old active flag: gate on it and fall back to your normal connection string. Branch sizing and warm-up, and every size the server accepts, are unchanged.
## Breaking: database branch settings move onto `ReplayEnvironment`
The branch sizing and warm-up settings introduced in v0.31.0 are now constructor arguments on `ReplayEnvironment`, not a separate replay option. As a sibling of `environment` they could be passed on their own, where they were silently discarded: the settings only ever applied to a replay that had an environment. On the constructor that state cannot be expressed at all.
Before:
```typescript theme={null}
await client.replay("checkout-agent", runCheckout, {
environment: env,
dbBranch: { minCu: 2, maxCu: 2, warmupSql: "SELECT 1;" },
})
```
After:
```typescript theme={null}
const env = new Bitfab.ReplayEnvironment({
minCu: 2,
maxCu: 2,
warmupSql: "SELECT 1;",
})
await client.replay("checkout-agent", runCheckout, { environment: env })
```
The same move applies in Python (`ReplayEnvironment(min_cu=2, max_cu=2, warmup_sql="SELECT 1;")`) and Ruby (`Bitfab::ReplayEnvironment.new(min_cu: 2, max_cu: 2, warmup_sql: "SELECT 1;")`). The `dbBranch` / `db_branch` replay option is removed rather than deprecated. All three fields stay optional, so an environment you construct with no arguments keeps working and leaves the mirror's own defaults in place. Everything the settings do, and the sizes the server accepts, are unchanged.
Superseded by v0.33.0 above: `ReplayEnvironment` is gone, and these settings are back on the `dbBranch` replay option.
## Clear replay outcomes when local capture is unavailable
When a replay command finishes successfully but the plugin cannot read its local result, Claude Code, Cursor, and Codex now report the run as unverified instead of showing a misleading background-command failure. The assistant checks the server for the actual experiment outcome before evaluating results, while genuine replay-script crashes still appear as failures.
## Quoted numbers no longer break read\_span\_field over MCP
Some MCP clients send numeric arguments as strings. When a client connected directly to `https://bitfab.ai/mcp` called `read_span_field` with a quoted `maxChars` (for example `"200000"`), the request failed with an invalid-arguments error, even though the identical call succeeded through the Bitfab editor plugins. The endpoint now accepts either form, so reading a full span field works the same way from any client.
## Size and warm the database branch your replay runs against
When you replay traces against a historical database branch, that branch used to start at whatever compute the snapshot mirror happened to be provisioned with, and it served its first query from a cold cache. Both made replay latency reflect the branch instead of your code. Pass `dbBranch` to size the compute and warm it before your function runs:
```typescript theme={null}
await client.replay("checkout-agent", runCheckout, {
environment: env,
dbBranch: {
minCu: 2,
maxCu: 2,
warmupSql: "SELECT count(*) FROM orders;",
},
})
```
Setting `minCu` equal to `maxCu` pins the size, so every item in a run is measured against identical compute rather than one that warms up as the run proceeds. `warmupSql` runs inside the branch's readiness check, before your function sees the lease, so warm-up time is never charged to the replayed call. The same options are available as `db_branch` in the Python and Ruby SDKs.
## Replay tells you when it could not reach a database branch
When a database branch could not be provisioned, replay used to run your function against your live database and hand back a result that looked normal. It now fails that item and reports why, so a run that could not use the historical data you asked for never reports a passing result. Traces captured before snapshot support still replay against your live database as before, since no branch was requested for those.
## See which grader failed a trace, and why
Your coding agent can now read the individual verdict each automated grader recorded on a trace, instead of only the overall pass or fail. In Claude Code, Cursor, or Codex, `read_grader_labels` returns each grader's name, its verdict, the reasoning behind it, and its failure diagnostic, so you can ask which specific check a trace failed. Query it by trace to see every grader's verdict on those traces, or by grader to review that grader's most recent verdicts across traces when you are working out why a grader keeps misfiring.
## Name experiment groups from your coding agent
You can now give experiment groups a name and rename them later, making related runs easier to identify across replay and grading workflows. In Claude Code, Cursor, or Codex, `save_experiment_group` creates a group from existing experiments or updates an existing group name without changing its membership.
## Browse workflows across every trace function
All Traces now includes Datasets, Experiments, and Graders in the same workflow dropdown used inside each trace function. The organization-wide dataset and experiment views let you browse and open work across functions from one place, with function context on dataset rows, pagination for larger histories, and live experiment progress and results.
## Read a single experiment's grader results from your coding agent
The Bitfab MCP tools now include `get_experiment`, which returns one experiment (replay test run) by id: its status, pass/fail totals, replay-versus-original delta, experiment group, and grader results, including a per-grader passing and failing breakdown. Use it when you already have an experiment id and want just that run's results, instead of listing a function's whole experiment history. The `list_experiments` results now also include each run's experiment group id.
## Consistent MCP tools across local and remote connections
Coding agents now receive the same 32 Bitfab tools, descriptions, validation schemas, and metadata whether they connect through the local plugin or the remote MCP server. Trace searches now validate real calendar dates and integer limits consistently, and trace-plan creation derives initial replay mocking decisions server-side so agents cannot accidentally override them before confirmation.
## Attach the first grader from an experiment
You can now add or remove graders from an experiment even when none are currently attached, as long as its trace function has graders available. The Graders section remains visible in this state, so you can open Manage without needing an existing assignment first.
## Choose the model that judges your graders
You can now pick which model runs each LLM-as-judge grader instead of every grader using the same one. On a grader's detail page, the new Model dropdown lets you choose Gemini 2.5 Flash (the default), GPT-5.6 Sol, GPT-5.6 Terra, Claude Opus 4.8, or Gemini 3.1 Pro, and the choice takes effect the next time that grader evaluates a trace. You can also set it from your coding agent: the `save_grader` plugin tool now takes a `model` option, so a request like "have this grader use Opus 4.8" sticks. Graders you do not change keep running on the default.
## Arrow keys move between spans inside an open trace
With a trace open on the experiments, dataset, or traces pages, up and down (or j and k) now step through that trace's spans instead of jumping you to a different trace. Left and right (or h and l) still move between traces, and when no trace is open, up and down keep moving the cursor through the list.
## Incremental summaries for resumed coding sessions
Coding sessions resumed under the same session now produce a summary of only the new work instead of repeating earlier turns. Sessions that never include both a user request and an assistant response no longer generate a summary, reducing noise from open-and-close sessions.
## Grader results in the experiment comparison view
Opening a trace in a grader-scored experiment now shows its grader pass and fail results in the before-and-after comparison, matching what the trace list already shows, instead of an "Awaiting label" placeholder. You see the grader pass rate move from the original run to the replay, or a live grading indicator while scoring is still in progress.
## Filter traces by original vs. replay
The traces list now has an Original / Replays / All filter so you can separate live production traces from replay traces generated by experiments and test runs. It defaults to Original, so you see real production activity first, and you can switch to Replays or All whenever you need to inspect experiment re-runs.
## Complete grader set on experiment panels
The graders panel on an experiment now lists every grader scoring that run, including the ones inherited from its dataset, so it matches the pass and fail results shown on each trace. Graders that come from the dataset are marked with a small icon you can hover to confirm, and re-runs stay limited to the experiment's own graders.
## Reliable grader reruns
You can now see live progress when re-running graders on a dataset or experiment, and that progress resumes if you refresh or return while the run is still active. Starting the same rerun again reconnects to the existing work instead of creating a duplicate, while a different selection is blocked until the current run finishes. Completion updates recover even if a live update is missed, so results do not stay stuck in a running state.
## Live grading progress on trace rows
When graders re-score a dataset or experiment, each trace row now shows its grading as it happens instead of going blank. A loading ring appears the moment scoring starts and the pass rate fills in live as each grader reports, so you can watch results arrive. Finished rows keep their green or red verdict, with the pill's border showing the pass/fail split.
## Per-grader labeling on the trace list
When you open a trace that has graders from the Traces page, the labeling panel now shows each grader's verdict so you can review and label them one by one, the same per-grader view you get inside a dataset. Traces without graders keep the standard pass/fail labeling panel.
## Related experiments on the dataset page
Each dataset now shows the experiments run against it, right in the dataset view. See each run's pass rate at a glance and jump straight to a single experiment, or open the full list filtered to that dataset, without leaving your workflow.
## Graders now have their own section
Every trace function has a Graders section alongside Traces, Datasets, and Experiments. It lists your graders with their pass rate and the number of datasets and experiments each one is attached to, and you can sort by status, pass rate, evaluation volume, or recency, and archive or unarchive a grader without leaving the list.
## Grader overview with dataset and experiment footprint
Opening a grader now shows an overview of what it does and where it runs: its overall pass rate, the criteria it evaluates, and every dataset and experiment it is attached to with that grader's pass rate on each. Pass rates combine the grader's automated results with your own human labels.
## Automatic code-change capture on replay
When you replay traces after editing your code, the SDK now attaches the diff to the resulting experiment automatically, so you can see exactly what changed alongside the results with no extra arguments. If you don't pass `codeChangeFiles` to `replay()`, it captures your working-tree changes against your trunk and includes them; passing an explicit code change still takes precedence when you want a precise per-edit before/after.
Available in the TypeScript, Python, and Ruby SDKs. Opt out with `BITFAB_DISABLE_CODE_CHANGE_CAPTURE`.
## Attach graders to a single replay
`replay()` now takes a `graderIds` / `grader_ids` option so you can grade one experiment with specific graders without permanently adding them to the dataset. The run is graded by the union of the graders you pass and the dataset's own graders, so a one-off check runs alongside your standard ones.
```ts theme={null}
await client.replay("my-function", processInput, {
datasetId: "",
graderIds: [""],
})
```
Available in the TypeScript, Python (`grader_ids`), and Ruby (`grader_ids`) SDKs.
## Accurate experiment pass rates
Completed experiments without assigned graders now show a pass percentage from their trace labels instead of `--%`, so the summary reflects the results already available below it. Experiment rows use grader results when graders are assigned and label results otherwise, while runs with labels still outstanding remain pending.
## Honest experiment comparisons
Experiment trace rows now omit the before-to-after grader badge when the original trace has no compatible grader results. The current grader score remains visible, so you can see the replay outcome without a misleading missing baseline.
## Graders on the experiment page
Experiments now show the graders attached to each run right beside the traces, with a per-grader pass/fail breakdown for the run. Open a grader to read its criteria, use Manage to attach or detach graders, and Re-run to re-grade the experiment's traces. The pass and fail bars update live as grading runs.
## Group and grade experiments from your coding agent
You can now organize existing experiment runs into a shared group and attach a grader across every current experiment using the new `create_experiment_group` and `add_grader_to_experiment_group` MCP tools. Completed experiments queue only missing grader results immediately, while pending experiments use the grader when they finish. Grader assignments stay on the experiments already in the group, so experiments added later do not inherit them automatically.
## Replays for up to 5,000 recent traces
Replay runs can now select up to 5,000 recent traces at once, up from 100, so larger evaluations no longer need to be split into manual batches. Starting a replay now avoids loading every child span up front, while dataset-backed replays continue to use the dataset's full trace list regardless of the recent-trace limit.
## Guidance for creating datasets and experiments
Datasets and experiments are created by asking your coding agent, and the dashboard now shows you how. When a datasets or experiments page has none yet, it explains what the primitive is and gives you a ready-to-copy prompt to paste into your coding agent. The "New" button on those pages opens the same guidance with an example, so you are never left on a blank screen wondering how to start.
## Paginated grader lists in coding agents
`list_graders` now returns manageable pages for functions with large grader collections, with name search and a cursor for fetching the next page. It returns active graders by default, can include archived definitions when requested, and leaves grader-training pipeline entries out of the results.
## Assistant runs your plan without pausing to ask
The Bitfab assistant no longer stops after each experiment to ask whether to keep going or to revert a fix. It now reports the results and automatically continues through the experiments you already approved, wrapping up on its own once the plan is done. A multi-experiment run finishes in one pass instead of prompting between rounds.
## Attach graders to experiments from your coding agent
You can now attach graders directly to an experiment so they run against its replay traces, using two new Bitfab MCP tools, `add_graders_to_experiment` and `remove_graders_from_experiment`. Your coding agent can manage an experiment's grading scope without leaving its workflow, the same way it already manages dataset graders. At completion an experiment is scored against the union of these direct attachments and its dataset's current graders.
## Experiments grouped by run
Your experiments page now groups a trace function's runs into groups, so each iteration reads as one unit rather than a flat list of runs. Groups are ordered by their most recent run, each header links back to the dataset it was run against, and you can collapse a group to focus. Runs launched without a group are gathered under an "Ungrouped" heading.
## Live dataset grader pass rates
Dataset grader pass-rate pills now stay current as new grader results arrive or grader assignments change. The datasets list refreshes automatically, so you can monitor evaluation progress without reloading the page.
## Experiment labeling shows only the run's graders
When you open a trace inside an experiment, the labeling panel now shows only the graders that experiment was scored against: its dataset graders plus any graders attached to the run, rather than every grader defined for the function. Experiments with no graders open straight to the pass/fail labeling panel.
## Grader definitions after saves
After you create or update an automated grader through Bitfab MCP tools, your coding agent now presents the full saved definition instead of a generic success message. The response includes the grader function, status, evaluation focus, and any passing or failing criteria so you can immediately verify what will be evaluated.
## Reliable Studio links
Links that open Studio now preserve all handoff parameters, including repeated values, across sign-in, datasets, experiments, trace plans, and template previews. Dataset-linked experiments also switch to the dataset organization automatically, so shared links open in the correct context.
## Manage graders from a dataset
You can now attach and remove graders directly from a dataset. Open Manage in the Graders panel on a dataset page to move graders between Available and Attached, with search and sorting to find the right one. Attached graders score every trace in the dataset, and changes take effect immediately.
## Experiment graders run across every replay
You can now attach graders directly to an experiment, and Bitfab runs them together with the dataset’s assigned graders across every trace in the completed replay. The finalized grader set stays with the experiment, so late-arriving replay traces are evaluated consistently and completed experiment results retain the exact grader coverage that ran.
## Framework labels on trace plans
Trace plan headers now show the instrumentation framework Bitfab detected for your workflow, next to the function name and language. Plans that span multiple frameworks list each one, so you can see at a glance how a workflow is instrumented.
## Consistent labels across trace lists
Workflow, dataset, and experiment trace lists now use the same current label state, so pass, fail, and skipped results stay consistent across views. List loading also avoids fetching unused trace metadata, making these pages more efficient without changing how traces are managed.
## Keep your place when switching functions
Switching trace functions from the sidebar now keeps you in the current section, such as Traces, Datasets, Graders, or Experiments, instead of sending you back to Traces. When you switch from a specific trace, dataset, or grader, Bitfab opens the matching section list for the new function so resource IDs are not carried across functions.
## Live grader results across review views
Dataset reviews, experiment cards, experiment trace rows, and trace lists now update immediately as grader results and dataset grader assignments change. Pass rates and per-grader verdicts stay current during grading and review without a manual refresh.
## More reliable dataset grader labeling
Dataset grader labels now stay consistent when automated evaluations and human reviews overlap, so one source no longer overwrites the other. Newly added completed traces are automatically graded by their assigned dataset graders, and the labeling panel preserves in-progress selections when a save or refresh fails.
## Grader pass rates on trace lists
When a dataset has graders assigned, trace rows now show how many graders passed as a pass-rate bar (for example, 3 of 4) instead of a single Pass or Fail. It appears on the dataset review page, on experiment rows, and in the experiment header, so you can see at a glance how each trace and each run scored across all of its graders. The main traces list shows the same pass rate for grader-evaluated traces.
## Run graders without tuning
Active graders now evaluate traces directly from their saved criteria, so dataset grader runs work immediately without tuning. If a tuned prompt exists, Bitfab still uses it; empty or stale prompts fall back to the grader criteria.
## Live dataset review updates
Dataset pages, trace lists, and labeling panels now stay in sync as graders are assigned or removed and labels are added. Reviewers see the current dataset grader order immediately, while removed graders disappear from active labeling workflows without deleting their historical labels.
## Label traces while reviewing experiments
You can now label traces without leaving the experiments view. Open the before/after comparison for any trace and use the new Label toggle, next to the Diff / Original / Replayed switch, to score it. When the dataset has graders, the panel shows each grader for per-grader approval or override; otherwise it is a simple pass/fail verdict with notes.
## Smoother dataset grader reruns
Dataset grader reruns now keep traces in a grading state through transient evaluation failures, so temporary issues no longer appear as permanent errors. If every retry fails, the trace still moves to an error state instead of remaining stuck in grading.
## Compare individual spans in the Diff view
When you compare a replayed trace against its original, the experiment Diff view now lets you open any span that changed, not just the whole trace. The span tree greys out spans whose input and output stayed the same and keeps the changed ones selectable, so you can jump straight to what your code change actually affected.
## Clearer replay diffs
The Diff view now shows the input side fully expanded, so you can read exactly what the model received, and hovering a highlighted line tells you whether the replayed run excluded or included it.
## Trace details open in a side panel
Clicking a trace on the traces page now opens a slide-in detail panel next to the list, matching how dataset and experiment review work, instead of navigating to a separate page. You can label the trace Pass, Fail, or Skip right from the panel and move through traces with Save & Next. Links to individual traces still work: sharing or reloading a trace URL opens the same list with the panel already open, and your active filters stay in the URL.
## Redesigned trace list rows
Trace rows across the traces, dataset, and experiment lists now share one design that separates how the run went from how it was judged. A leading icon shows the run state (running, completed, errored, or a replay), while the verdict pill shows Pass, Fail, Skip, or an agent suggestion awaiting review, with a robot glyph marking machine verdicts. Replayed traces are tinted indigo so re-runs stand out, unlabeled rows show an input and output preview, and reviewer notes appear inline on labeled rows.
## Re-run graders on a dataset
When a dataset has graders attached, you can now re-run them across every trace in the dataset directly from the dataset page. Open the Graders panel, choose which graders to run, and follow the run from running to finished. You can also click any grader to view its evaluation criteria and prompt.
## Label datasets grader-by-grader
When a dataset has graders assigned, reviewing a trace now shows a labeling panel with one row per grader instead of a single pass/fail. Approve an automated grader's suggested verdict in one click or override it, add a note, and move through the dataset trace by trace. Human labels always take precedence over the automated suggestions.
## Reliable experiment history pagination
Experiment histories now load every run when several experiments start at nearly the same time. Infinite scrolling no longer skips or repeats experiments that share the same timestamp.
## Scroll through complete experiment histories
Experiment histories now keep loading as you scroll, so older runs remain available instead of stopping after the newest 50. This works across function, dataset, and experiment-group views. The loading footer stays visible while more runs remain, with a spinner while the next page arrives.
## Dataset graders now grade automatically
The graders you assign to a dataset now run automatically on that dataset's traces, and on the replayed traces when you run an experiment against the dataset. You get grades on your dataset and experiment results without kicking off anything by hand.
## Create and edit graders from your coding assistant
Claude Code, Cursor, and Codex can now create and edit automated graders for a traced function with the new `save_grader` and `list_graders` tools. Ask for a check like "the reply never invents order numbers" and your coding agent defines the grader, then renames, updates, archives, or restores it on request without leaving your workflow. Graders are saved as definitions for now; nothing runs them against new traces automatically yet.
## Diff view for replay comparisons
When you replay a trace to test a code change, the trace comparison drawer now opens on a new Diff view that shows the replayed run against the original side by side, input on the left and output on the right, with changed lines highlighted. You can tell whether the change helped without flipping between the Original and Replayed panes and holding both in your head. Use the Diff, Original, and Replayed toggle at the top of the drawer to switch views; your choice sticks as you step through the run.
## Assign graders to datasets from your coding assistant
Claude Code, Cursor, and Codex can now assign graders to datasets with `add_graders_to_dataset` and `remove_graders_from_dataset`. `list_datasets` now includes assigned graders, so you can inspect evaluation coverage and update it without leaving your coding workflow.
## See experiment annotations in trace comparisons
Experiment trace comparisons now show the replay annotation beside the pass/fail transition, so you can see the labeler's reasoning without returning to the trace list. Hover over a truncated annotation to read its full text in a tooltip.
## Keyboard navigation in dataset review and experiments
You can now move through traces and spans with the keyboard while labeling a dataset or comparing experiment results. Use the arrow keys or Vim keys (`h`, `j`, `k`, `l`) to step between traces and their spans, and press `Esc` to close a trace's detail view.
## Start setup with a specific request
You can now pass `--prompt` (or `-p`) to `bitfab init` and `bitfab setup` to tell the setup agent what you want instrumented from the start. The prompt is forwarded into setup in Claude Code, Codex, and Cursor, so onboarding can begin with the workflow you already have in mind.
## Lower tracing overhead in the Python SDK
Traced functions in the Python SDK now return as soon as their trace data is captured, instead of waiting for spans to finish uploading to Bitfab. For latency-sensitive code, this takes a network round-trip out of your own request path while your traces keep uploading in the background.
## Browse past experiments from the dashboard
Every trace function now has an Experiments tab next to Traces and Datasets, listing the experiments that have run against it, newest first. Each run shows its pass rate and how many traces were fixed, regressed, still passing, or still failing versus the original, so you can tell at a glance whether a change helped. Expand a run to inspect its traces, compare the original and updated output, and view the code change that produced it.
## Replay verdicts persist by the original trace
When your coding agent evaluates a replay run, its pass/fail verdicts now persist against the trace each item was replayed from: pass `testRunId` plus the item's `originalTraceId` to `update_agent_labels` and Bitfab resolves them onto that run's replay traces. Verdicts reliably reach the experiments page without the agent ever needing a server-generated replay trace id.
## Replay items rename source to original
Replay items, progress events, and adapt-inputs context now name the replayed-from trace `originalTraceId` and `originalSpanId` (snake\_case in Python and Ruby); the previous `sourceTraceId`/`sourceSpanId` names keep working everywhere as deprecated aliases, so existing scripts are unaffected. An item's `traceId` is now `null` while the run streams and is filled in with the server replay id when the run completes.
## Replay verdicts are saved automatically
When you replay a single trace to check whether a fix worked, the assistant now saves its pass/fail verdict onto that replay trace instead of only showing it in chat, so your conclusion sticks and appears alongside the trace. The replay path stays lightweight, with no Studio, dataset, or experiment setup. If your SDK is too old to return a replay trace ID, the verdict stays in chat with a prompt to upgrade the SDK.
## Inject custom values into specific spans during replay
When you replay a trace, you can now override the output of a chosen span instead of running its real code or replaying its recorded output. Match a span by its name, function key, or type, then return a fixed value or one computed from the span's live inputs and its original recorded output. Available in the TypeScript, Python, and Ruby SDKs.
```typescript theme={null}
await client.replay("my-workflow", runWorkflow, {
mockOverride: {
match: (node) => node.spanName === "Summarizer",
value: async (ctx) => ({
...(await ctx.getOriginalOutput()),
score: 1,
}),
},
})
```
## Long trace results stay visible
Large arrays in trace Input and Output views now open automatically, so you can see long result lists without an extra click. When those lists contain nested objects or arrays, each entry stays collapsed to keep the trace readable; compact scalar values remain visible.
## Read one span without loading the full trace
Fetch a single persisted span directly from a trace in the TypeScript, Python, Ruby, and Go SDKs. Select it by canonical `id` or by name; repeated names return the last occurrence by default, with options for the first or a zero-based occurrence.
```typescript theme={null}
const span = await bitfab.getTraceSpan(traceId, {
name: "generate",
})
```
## Reliable nested traces from the first Node.js call
The TypeScript SDK now preserves nested span context from the first traced call in Node.js, including ESM and CommonJS applications. For async-generator streams, wrap the controller that owns the iteration in an outer `withSpan` call so service-side production and consumer-side work appear under one trace.
## Trace plan warnings for spans that won't replay
Trace plans now flag spans that won't replay cleanly before you confirm. If the entry point's input can't be serialized, the plan shows a "Root not replayable" warning; spans mocked despite output that can't be serialized are called out the same way. When a plan carries any of these warnings, Confirm opens a confirmation step so you accept them deliberately rather than by accident. The warnings show on the trace plan review page, and your coding agent applies the same rules when it drafts a plan.
## Structured Map and Set trace output
Trace outputs containing JavaScript Maps and Sets now render as navigable structured data instead of collapsed string representations. Maps preserve distinct keys even when their string forms collide, and nested Maps and Sets remain expandable in the trace viewer.
## More reliable dashboard startup
The Bitfab dashboard now avoids analytics initialization crashes in browsers where cookie access is unavailable or blocked. Analytics stays inactive until it is ready, so affected sessions can load the dashboard normally.
## Edit span templates without leaving the trace you're viewing
When you ask your coding agent to change how spans render, Bitfab no longer pulls you onto a separate template-preview page. If you're already looking at a trace of that function, it now offers to edit the templates in place, and your open trace re-renders live with each change you make. The dedicated preview page, with click-to-target editing on the function's most recent trace, is still one option away when you want it. Available in the Claude, Cursor, and Codex plugins.
## Fix flow reverts changes that cause regressions
When you use the Bitfab assistant's fix flow and re-run your full dataset to lock in a fix, it now checks whether your change broke traces that were passing before. If it finds real regressions, the assistant recommends reverting the fix and starting a fresh attempt, so you never ship a change that trades one fixed trace for several broken ones. The trace you were fixing stays saved in your dataset as a regression test to revisit. Available in the Claude, Cursor, and Codex plugins.
## Choose where fixed traces get saved
When you fix a failing trace with the Bitfab assistant (`/bitfab:assistant fix`), it now asks which dataset to save the fixed scenario to, instead of silently adding it to whichever dataset already existed. Pick an existing dataset, create a new one, or continue without saving. When you have several datasets it recommends the most recently used one and keeps the list short, so the choice stays quick.
## Trace plans stay available longer
Trace plans now remain available for 365 days, giving you much more time to return to an instrumentation plan before confirming it. The longer review window applies to newly created trace plans.
## Session length in chat summaries
Chat session summaries now show how long each coding session lasted, so you can put the product feedback in context at a glance. Slack notifications show the readable duration, while generic webhooks include the exact start time and duration in milliseconds.
## Redesigned trace input and output view
Trace spans now show Input, Output, Context, and Error as distinct, color-coded zones with sticky headers, so you always know which part of a span you're reading as you scroll. Input and Output sit side by side and stack automatically when the panel is narrow. When a span has an error, the Output header shows a control that jumps straight to the error details.
## Collapsible JSON for trace payloads
Trace input and output now render as an interactive tree you can expand and collapse, instead of a flat text dump. Deeply nested objects, large arrays, and long embedding vectors stay collapsed by default, so you can drill into just the parts you care about.
## A clearer trace plan review
The trace plan review page now speaks the same visual language as its "What is a replay?" explainer. Each span shows a type-colored dot with an icon for what happens on replay (re-runs live, mocked from the recording, or skipped inside a mock), its classification written out beside the name, and a Re-run | Mock toggle on the right. Every control has a tooltip, and the tree navigates with the arrow keys.
## The replay entry point always re-runs
A replay starts by re-running the top traced span, so its Mock control is now disabled with an explanation instead of silently having no effect. If you untrace spans above a mocked one, the newly promoted entry point switches to re-run and its children are no longer marked as skipped. Run/mock choices also survive untracing and re-tracing a span, and closing the replay explainer with Escape no longer cancels the whole plan.
## See which spans were mocked on a replay
When you replay a trace, spans set to mock on replay are served from the original trace's recorded output instead of re-executing. The trace view now marks those spans with a badge, both in the span tree and on the span header, so you can tell at a glance which nodes were replayed from history and which re-ran live. Recording that disposition requires the latest TypeScript, Python, or Ruby SDK.
## A replayability check before you accept a trace plan
When you instrument an AI workflow, Bitfab now verifies the traced function's root can be replayed before it proposes the trace plan, instead of letting you accept a plan and only then discover the root cannot be replayed. If the root's inputs are not serializable, setup resolves it up front by moving the trace boundary inward, using a framework handler, or refactoring, so the plan you confirm is one you can actually replay against later. Available in Claude Code, Cursor, and Codex.
## A faster path through instrumentation setup
Bitfab setup now moves directly from finishing one instrumented workflow to choosing the next workflow, selecting another target, or finishing setup. It shows how to exercise the workflow and run its generated replay command, while replay coverage checks remain available as an explicit action. When you choose the next workflow, Bitfab refreshes its workflow scan so targeted setup runs do not miss other candidates. Available in Claude Code, Cursor, and Codex.
## Studio opens in your normal browser
Bitfab Studio now opens in your usual browser as a regular tab, so you keep the address bar, tab controls, and the rest of your browser workflow. When a Studio session ends on macOS, the plugin closes only its matching tab and leaves your other browser tabs untouched.
## Honest Studio launch reporting
When a plugin command opens Bitfab Studio, it now always surfaces a clickable link in chat, and the message no longer claims a window opened before one actually did. If a browser could not be launched at all (common on remote or SSH sessions, or when no supported browser is available), the command reports why and the surfaced link still connects the session when you click it.
## Automatic recovery when a Studio window never appears
Previously, if a Studio window failed to surface, commands could wait indefinitely and later opens kept pointing at the dead session until it was cleared by hand. The plugin now detects a window that never connected, ends the wait with a clear reason so your coding agent offers a retry instead of treating it as a cancel, and clears the session automatically so the next open starts a fresh window. Stale or unresponsive Studio background processes are also detected and restarted on their own, so Studio commands always run the version of the plugin you have installed.
## Animated replay explainer on the trace plan page
The trace plan's "What is a replay?" modal now teaches by showing. A side-by-side animation plays the original run recording each call's input and output, then a replay re-running it: injecting the recorded input, answering mocked calls straight from the recording, and skipping everything nested under a mock. Each call is annotated with its replay plan (runs live, from recording, or skipped) so you can see why the replay behaves the way it does before confirming your plan. With reduced motion enabled, the modal shows the final annotated diagram as a static picture instead.
## Recover a stuck Studio session with `bitfab login --force`
If `bitfab login` reported that a Studio window was recorded as open but was not responding, there was previously no way to clear it from the command line. Running `bitfab login --force` now clears the stale Studio session before opening a fresh window, so you can get straight back to signing in. The error message also points you to the flag whenever you hit that state.
## More reliable trace ingestion
Bitfab now handles traces and spans that contain Postgres-incompatible text in their raw payloads, so ingestion can continue instead of failing the write. This improves reliability for SDK uploads that include null bytes or malformed Unicode from upstream tools.
## Readable analyze-repo summaries
`bitfab analyze-repo` now prints a compact terminal report after uploading draft trace plans, so you can see the selected workflows, frameworks, instrumentation effort, suggested capture methods, replay mocks, and real-data value without opening plan links. The same summary output works across Claude Code, Codex, and Cursor runs, with long lines wrapped for terminal readability and skipped candidates kept in the report.
## More reliable trace plan setup
Asking Bitfab to create a trace plan now reliably opens it in Bitfab Studio for review, instead of occasionally rendering the plan inline in the chat. Requests like "create a trace plan" or "instrument the next function" route directly into the setup flow, and following up to instrument another function reopens the Studio confirmation UI automatically.
## More reliable trace search indexing
Bitfab now recovers from temporary upstream interruptions while preparing trace search summaries, reducing cases where newly ingested traces fail to become searchable. The retry behavior covers rate limits, provider-side failures, and network transport failures while still stopping on deterministic request errors.
## Steer `analyze-repo` with a prompt
`bitfab analyze-repo` now takes free-text guidance so you can point it at the parts of your codebase you care about. Pass `--prompt` (short form `-p`, or just a trailing quoted argument) with something like "focus on the billing and checkout flows" and the scan biases toward those areas when picking which AI workflows to draft trace plans for, topping up any remaining slots from the rest of the repo. Combine it with `--limit` to cap how many plans it uploads.
## Go SDK: `drop()` is safe to call across goroutines
Calling `drop()` on a trace from one goroutine while another goroutine finishes a span on the same trace no longer races on the trace's dropped state. If your Go service drops traces from a different goroutine than the one running the traced work, that path is now safe.
## More reliable CLI sign-in handoff
Bitfab now waits for the browser sign-in handoff to confirm that the CLI received its credentials before showing success. The Studio close page stays in a finishing state while sign-in completes, and if delivery fails it leaves the page open with a retry option instead of making the terminal wait silently.
## `drop()` now stops later spans from being sent
Calling `drop()` on the current trace now prevents any span that finishes afterward from being uploaded at all, so dropping a run that carries sensitive data keeps that data local instead of sending it and clearing it server-side. Spans already sent before the `drop()` call are still cleared, and the trace is still marked dropped. Available in the TypeScript, Python, Ruby, and Go SDKs.
## Discard an in-flight trace with `drop()`
You can now discard the current trace at runtime from your own code, when you decide it shouldn't be recorded (a health check, a cache hit, any path with no useful signal). Call `drop()` on the current trace and Bitfab skips it: its inputs, outputs, and spans are never stored, and anything already uploaded for that trace is cleared.
```typescript theme={null}
import { getCurrentTrace } from "@bitfab/sdk"
// inside a traced function, when you decide this trace isn't worth keeping
getCurrentTrace().drop()
```
Available in the TypeScript, Python, Ruby, and Go SDKs (`getCurrentTrace().drop()`, `get_current_trace().drop()`, and `GetCurrentTrace(ctx).Drop()` in Go). The call is always safe: if there's no active trace it does nothing, and it never throws or interrupts your code.
## More reliable analyze-repo from the CLI
`bitfab analyze-repo` now runs non-interactively through Claude Code, Codex, and Cursor Agent, so you can scan a repository and upload draft trace plans without opening an editor UI. Use `--editor` to choose the agent and `--limit` to cap how many plans are drafted.
The CLI also gives these headless runs clearer outcomes: uploaded plans share a Bitfab run identity, logs redact API keys safely, and terminated agent processes fail with a clear error instead of looking successful.
## Analyze-repo runs link plans and session logs
The `bitfab analyze-repo` command now gives each scan a shared run identity, so the draft trace plans it uploads and the optional captured Claude Code session are tied together in Bitfab. This makes it easier to audit what the agent found, which plans it created, and the conversation that produced them.
## Codex can run analyze-repo headlessly
`bitfab analyze-repo --editor codex` now runs through `codex exec` without opening the Codex TUI, using the same non-interactive scan and draft trace-plan upload flow that was already available through Claude Code. Cursor Agent is now supported too through its `--print` headless mode.
## Analyze-repo can sign in before scanning
The `bitfab analyze-repo` command now opens the Bitfab sign-in flow when an interactive run is not authenticated, then continues the scan after login succeeds. Non-interactive runs still stop with clear instructions, and if a stale environment API key blocks verification the CLI explains how to fix it.
## Cleaner plugin login and logout in local projects
Plugin login now keeps the Studio sign-in URL visible and reports success as its own status line, which makes local and dev sign-ins easier to follow. Logout now clears project-local credentials before falling back to global credentials, so worktree-specific logins can be reset without affecting other projects.
## Editor sign-in checks before setup
The `bitfab` CLI now checks whether Claude Code, Codex, or Cursor is signed in before it launches setup, assistant, SDK update, or `analyze-repo`. If the editor agent is logged out or the editor CLI is missing, Bitfab stops early with the login command to run instead of opening an agent session that fails later.
## Command-specific help in the Bitfab CLI
The `bitfab` CLI now shows help for individual commands, so you can check the right flags and usage without triggering the command itself. Use `bitfab help `, ` --help`, or ` -h` to inspect commands before running onboarding, login, install, or other workflows.
## More accurate natural-language routing in the plugin skills
The setup and assistant skills now route free-form requests to the right mode more reliably. Phrases like "trace a new workflow," "why aren't my traces showing up," or "did my fix work on this trace" land in the correct mode without you having to name it. Each skill also lists its modes and what they do up front, so the full set of things it can do is visible at a glance.
## Clearer plugin setup status
Bitfab plugin setup now makes local and dev authentication easier to verify. Login success messages include the non-production endpoint when the plugin is pointed away from production, and plugin update checks show the version that is already installed or was just updated.
## Trace plans isolate external calls by default
Setup-generated trace plans now mark mockable external reads and side effects, such as database queries, HTTP calls, and writes, to return their recorded output during replay. LLM calls and local code stay live by default, so replay still tests the model behavior you are trying to improve. If an external parent span would skip live child spans, the server now keeps that parent live and expects the smaller external boundary to be mocked instead.
## Replay scripts stay aligned with production roots
Bitfab setup and assistant workflows now require generated replay scripts to call the same production root wrapper for traced functions, instead of introducing a replay-only helper that can drift from runtime behavior. For handler-based integrations, replay guidance now points back to the same production framework entrypoint and includes a root-parity checklist before setup finishes.
## More reliable replay results
Replays run through the Bitfab plugin no longer fail to report their result when the replay script prints extra output (framework logs, env-loader noise). The TypeScript, Python, and Ruby SDKs now write the full replay result to a file the plugin reads directly, so a passing replay is never mistaken for a failed one.
## Scan a repo for AI workflows from the terminal
Run `npx bitfab-cli analyze-repo` to headlessly scan a repository for its AI workflows and upload a draft trace plan for each of the top candidates, with no prompts and no code changes. Cap how many plans it uploads with `--limit` (default 5), then review and confirm the drafts in Studio. Available for Claude Code.
## Setup asks before rewriting your code
When `/bitfab:setup` instruments a function, it now pauses and asks for approval before restructuring an existing framework or SDK call to attach a trace, instead of rewriting it silently. Purely additive instrumentation (wrapping an unchanged call) proceeds as before; only a change that would modify existing code stops for your confirmation, and any rewrite you approve preserves the original behavior exactly.
## Trace plans stay valid longer
Trace plans no longer expire after 30 minutes. When you set up tracing and step away before confirming a plan, it now stays valid for 7 days, so you can pick up where you left off instead of recreating it.
## More reliable Studio sessions
Studio now stays put across restarts. If Studio's background process reloads, your active session is restored and reopens on the page you were last viewing instead of being lost or opening a duplicate window. Switching between pages (a dataset, a trace plan, the experiments view) now reuses the open Studio tab instead of closing and reopening it.
## Verified replay labels in assistant runs
Bitfab plugins now verify replay labels immediately after persisting them, so benchmark scorecards only continue once the server reports the expected effective PASS, FAIL, or skipped state. During assistant and replay workflows, `persistReplayLabels` parses the `update_agent_labels` response and stops with `verification-failed` if any trace label is missing or mismatched, giving the agent a clear retry path instead of reporting partial results.
## Framework-aware replay mocking
The trace planner now decides which spans can be mocked on replay based on how your framework captures them. Spans your code wraps directly can be mocked; spans a framework observes from the outside (LangChain, LangGraph, and similar callback-based integrations) re-run live on replay instead, while Vercel AI SDK model calls stay mockable. This keeps model calls from being dropped or wrongly mocked when you set up tracing on a framework app with `/bitfab:setup`.
## Mock a span the planner flagged
When the trace planner marks a span as not mockable, you can now override it and mock it anyway from the trace plan page in Studio. The choice is kept as a warning rather than blocked, and the plan footer shows a warning count so you can review these before confirming.
## Batch-analyze a repo for what to trace
The Bitfab setup plugin can now scan a whole codebase and draft trace plans in one non-interactive pass. Run `/bitfab:setup analyze-repo` and it finds your AI workflows, picks the top few worth tracing, and uploads a draft trace plan for each, without prompting or changing any code. Review the drafts in Studio, then run `/bitfab:setup instrument` on the ones you want to wire up.
## Replay status is available as an MCP tool
The Claude, Cursor, and Codex plugins now expose `get_replay_status` directly through the local Bitfab MCP server. During replay-based assistant runs, agents can map local replay trace IDs to server trace IDs while the run is still in progress, so they can persist per-trace verdicts incrementally without relying on a separate command wrapper.
The plugin MCP tool list has also dropped the deprecated grader tools, keeping trace inspection, datasets, labeling, experiments, setup, and replay status aligned across the editor plugins and Bitfab MCP endpoint.
## Clearer replay label errors
Replay label persistence now fails loudly when a verdict batch includes trace IDs that do not exist in the active organization, instead of saving only part of the batch and treating the rest as skipped. The Claude, Cursor, and Codex plugins now guide agents to remap replay results to server replay trace IDs and retry, so experiment labels are less likely to disappear behind a misleading success message.
## Trace plans analyze context nodes before review
Bitfab setup now asks Claude, Cursor, and Codex to classify replay behavior for every trace-plan node, including surrounding context nodes that are not initially captured. When you toggle those context nodes into capture from the trace-plan review, they already have a replay decision instead of needing a second classification pass.
Modify flows now backfill missing analysis on older trace plans before opening review, so expanding an existing plan uses the same per-node replay guidance.
## Clearer dataset wording in assistant fixes
The Bitfab assistant fix flow now describes the final step as adding the trace to a dataset with a validated failing label, instead of using capture language for dataset membership. In Claude, Cursor, and Codex, the flow still replays the target trace first, then saves it to a dataset only after the replay passes and lets you choose Studio, a full dataset rerun, another iteration, or stop.
## Assistant fixes ask before guessing
The Bitfab assistant fix flow now confirms why a trace is wrong before changing code when the trace or conversation does not already provide a clear failure reason. In Claude, Cursor, and Codex, targeted trace fixes reuse an existing failing label, a user-stated defect, or obvious trace evidence; otherwise the agent asks what correct behavior should be before replaying or saving the trace as a regression test.
## See when experiment results are still settling
When you open the code-change summary for an experiment that is still running, it now shows how many replays are still running or awaiting agent labels, with a note that the breakdown is provisional and will keep updating as they finish. Before, the summary presented partial results as if they were final.
## More accurate unpaired counts
Replays that are still running, awaiting labels, or errored are no longer counted as "unpaired" in the experiment breakdown, and each now appears as its own segment in the run progress bar. The unpaired count now reflects only replays that genuinely could not be matched to an original.
## Experiment replays update as traces finish
Experiment pages now refresh replay progress as each trace finishes, so results appear while the replay is still running instead of waiting for later labeling work. Runs outside an experiment group no longer trigger experiment updates, keeping grouped experiment views focused on the runs they are showing.
## Trace descriptions route to assistant fixes
The Bitfab assistant now treats requests like "find and fix the trace where..." as targeted fixes, even when you describe the bad output instead of pasting a trace ID. In Claude, Cursor, and Codex, the fix flow first verifies that the request is trace-backed, then uses local instrumentation and trace search to find the matching failing trace before making changes.
## Expired trace plans show their final state
Trace plan review now shows when an awaiting plan has expired instead of letting you try to confirm it and then showing an error. The review bar switches to a disabled `Plan expired` action as soon as the plan is stale, including when an already-open review page crosses its expiry time.
## More reliable BAML client execution
BAML client execution is more reliable across the Bitfab dashboard and TypeScript SDK. Generated OpenAI clients no longer send unsupported temperature options to GPT-5 and o-series models, while supported OpenAI, Claude, and Gemini clients keep deterministic sampling where the provider accepts it.
Shared Bitfab BAML fallbacks now use current tested Gemini and Claude models, so duplicate detection, entity extraction, trace summaries, and custom BAML prompts avoid stale model endpoints that could fail before the prompt ran.
## Trace plans show when spans cannot be mocked
Trace plan review now distinguishes spans that can return recorded output during replay from spans that must re-run live. When Bitfab cannot mock a span because the recorded output is not serializable or the span comes from library instrumentation, the plan keeps it on re-run and shows the reason in the replay control tooltip, so reviewers know why the mock toggle is unavailable.
The Claude, Cursor, and Codex plugins now pass that mockability metadata through `create_trace_plan`, so generated plans can mark those spans before you confirm instrumentation.
## Replay mocks are marked by default
Replay now uses `mock: "marked"` by default in the TypeScript, Python, and Ruby SDKs. A span tagged with `mockOnReplay: true` / `mock_on_replay: true` now returns its recorded output during replay without also passing a mock option. Pass `mock: "none"` when you explicitly want every child span to run real code, or `mock: "all"` when you want every child span to return historical output.
The docs now treat replay mocking as a first-class workflow, with a dedicated Replay Mocking guide linked from the introduction and SDK pages.
## Redesigned trace plan review
The trace plan you confirm when setting up tracing has a cleaner, more informative review screen. Each captured span is now labeled with what it does, `code`, `llm call`, `read`, or `write`, and whether replay re-runs it live or serves its recorded output, so you can see at a glance how your workflow will replay. The span tree is easier to scan: you can toggle spans in or out of the capture set and mark which ones to mock on replay, and a "How to review" guide walks you through it the first time.
## Manual Studio links in plugin flows
Bitfab plugins now print a copyable Studio link whenever they open a fresh Studio page, so you can still get to login, setup, datasets, experiments, and trace-plan flows if your editor hides the browser launch. The agent instructions for Claude, Cursor, and Codex now tell the agent to surface that link in chat as `Studio opened: `, while the command output also includes `Studio opened at: ` for terminal visibility.
## Setup signs you in before instrumenting
Running `/bitfab:setup instrument` now signs you into Bitfab before it starts analyzing your code, so instrumenting a new AI workflow no longer stalls partway through when you aren't logged in. Previously the flow could begin and then fail later at the trace-plan step; it now authenticates up front, matching how the other setup modes already work.
## Safer failed-fix regression capture
The Bitfab assistant now only marks an unresolved fix as saved after the failing trace is actually attached to the selected dataset. If a dataset attach is skipped, the Claude, Cursor, and Codex plugins keep the fix unsaved and guide the agent to choose a function-scoped dataset instead, so a failed fix cannot disappear without a real dataset entry.
Codex Studio navigation also returns control to the conversation once the Studio page is open and ready to report events, instead of waiting in the foreground while you inspect the page.
## More reliable Codex session capture
The Codex plugin now captures active Bitfab sessions more reliably while you work, so chat-session history is less likely to miss turns during tool-heavy workflows. Final session uploads still complete when Codex stops, and late background capture work can no longer revive stale local progress after a session has closed.
## Coding-agent session summaries
Bitfab now turns ended coding-agent chat sessions into structured product-feedback summaries, so teams can see what the agent tried, where Bitfab helped, where the plugin or service got in the way, and what should be improved next. Idle sessions are closed automatically, reopened sessions are summarized again after new work, and configured webhooks can send the summary to Slack with a link back to the chat session.
## Trace environment in `read_traces`
`read_traces` now shows each trace's environment alongside the trace id, function, status, and timing details, so agents can distinguish production, staging, local, and unset traces during investigations. When a trace has no stored environment tag, the tool prints `unset` instead of hiding the field.
## Test-first fixes with the assistant's `fix` command
The assistant's `fix` command now proves a fix before it saves anything. It diagnoses the failing trace, makes the change, and replays just that one trace; only once the replay passes does it add the trace to a dataset with a validated failing label, then offers to re-run the whole dataset (in Studio or your terminal). If the fix does not land, you can save the trace as a failing test to revisit later. When you open the fixed trace in Studio, the experiments page now lands directly on that trace's before/after comparison instead of a list you have to click into. Available in the Claude, Cursor, and Codex plugins.
## Replay your first captured trace during setup
When you instrument a workflow with `/bitfab:setup`, Bitfab now waits for your first trace to land and then offers to replay that exact trace right away, so you can confirm your replay script works before moving on. Available in the Claude, Cursor, and Codex plugins.
## Codex uses project-local Bitfab credentials from cached launches
The Codex plugin now resolves its active worktree even when the MCP server starts from Codex's plugin cache without a recorded session id. That keeps setup, login, and plugin actions pointed at the project's local Bitfab connection instead of falling back to stale global credentials.
## More dependable coding-agent session capture
Bitfab now keeps coding-agent chat sessions in order across transcript compaction and retry recovery, so older turns stay attached to the same session instead of shifting sequence positions. Sessions that stop without new turns also preserve their real activity and end times, and the dashboard now recognizes Bitfab slash-command activity in the same session activity view.
## More reliable replay mocks
Replay mocking in the TypeScript SDK now matches older historical spans even when the recorded span tree does not include a span name, falling back to the trace function key for the mock lookup. This keeps `mock: "all"` and `mock: "marked"` replay runs working on older traces instead of rerunning child spans that were meant to use their historical outputs.
The TypeScript and Python replay examples now show both mock modes with `mockOnReplay` and `mock_on_replay`, so you can verify that expensive child steps are skipped before running a full experiment.
## Cleaner final replay results
The Bitfab assistant now keeps the final replay event log focused on one server-backed item reference per replayed trace, even when progress events already wrote per-trace payload files. This prevents assistant evaluations from scoring the same replayed trace twice in mixed-payload runs, while live progress rows still support incremental evaluation as traces finish.
## Live per-item replay evaluation
The Bitfab assistant now scores and records each replayed trace as soon as it finishes, instead of waiting for the entire replay run to complete. On long replays, Studio's experiment view fills in pass/fail verdicts trace by trace while the run is still going, so you see results as they land instead of all at once at the end.
The replay `onProgress` callback in the TypeScript, Python, and Ruby SDKs now reports per-item detail for each trace as it settles, including the deserialized inputs, the replayed output, the original output, and token usage, so tools watching a running replay can evaluate results mid-run rather than waiting for the final `ReplayResult`.
## Reliable tracing regardless of API key load order
The Bitfab SDKs now resolve your API key the first time a traced function runs, instead of when the client is constructed. If your key (or `.env` file) loads after the client is created, tracing still activates, so a client built before your environment is ready no longer silently drops every trace.
All four SDKs (TypeScript, Python, Ruby, Go) also gain a `BITFAB_API_KEY` environment fallback when you don't pass a key explicitly, a callable key form for deferred resolution (e.g. `new Bitfab({ apiKey: () => process.env.BITFAB_API_KEY })`), and an opt-in `strict` mode that fails loud on a missing key instead of disabling tracing quietly.
## Trace-first assistant investigations
The Bitfab assistant now grounds traced-function investigations in actual trace evidence before recommending a fix or revert. When it investigates an AI workflow failure, it first derives the function key from local instrumentation, searches the matching traces, and separates what the trace proves from code-based inference, so debugging starts from the run that failed instead of only from static code.
## Targeted trace fixes before dataset runs
The assistant fix flow now replays only the target failing trace first, then asks whether to inspect the before/after in Studio, run the full dataset, keep iterating, or stop. This makes `/bitfab:assistant fix `, `/bitfab-assistant fix `, and `$bitfab:assistant fix ` faster to validate and prevents a dataset run from hiding whether the original bug is green. If you choose the full dataset run, Bitfab opens Studio as an experiment so you can catch regressions after the target fix is proven.
## Clearer LangChain setup guidance
Bitfab setup guidance now recognizes LangGraph and LangChain projects earlier and recommends the callback handler instead of manually wrapping graph nodes, tools, retrievers, or model calls. The docs and setup prompts also clarify that the handler already records a replayable framework root, so you only need a same-key outer span when there is meaningful application work around the graph or chain invocation.
## In-progress LangGraph and LangChain traces
Bitfab now shows LangGraph and plain LangChain runs as in progress as soon as the framework callback root starts, so long-running agents appear in the dashboard before their final output is available. The TypeScript and Python callback handlers also keep configured LangChain run names on chain, model, tool, and retriever spans, making trace trees easier to line up with your graph or chain config.
```typescript theme={null}
const handler = bitfab.getLangGraphCallbackHandler("support-agent")
await agent.invoke(input, { callbacks: [handler] })
```
## Cleaner framework replay inputs
Python tool spans now preserve empty structured tool inputs as `{}` instead of falling back to raw text, and TypeScript chain callbacks handle both LangChain callback argument orders for run names and parent IDs. This keeps handler-captured traces more accurate for replay and avoids nested framework callbacks marking the outer trace complete too early.
## Project-local plugin credentials from cached launches
Bitfab plugins now keep using your project-local connection settings when Claude Code, Cursor, or Codex launches the plugin from a cache or nested workspace. Local `config.local.json` and `credentials.local.json` files are resolved together from the same project search path, so agents are less likely to fall back to the wrong workspace or global credentials.
## Consistent Studio navigation across plugins
Studio navigation commands now work consistently in the Claude Code, Cursor, and Codex plugins when a Bitfab skill opens a specific Studio page. The plugins now ship matching command wrappers, so agent workflows can rely on the same Studio actions regardless of which editor plugin you use.
## Name replay experiments
You can now give Bitfab experiments a readable name when you create a dashboard test run or start a replay. Experiment cards show the name next to the short run ID, making it easier to compare baselines, prompt edits, and code-change runs without opening each result.
## Replay names in SDKs and plugins
TypeScript, Python, and Ruby replay calls now accept a `name` option, and replay scripts can forward it with `--name`. The Bitfab plugins detect whether the installed SDK and script support experiment names before using the flag, and recommend an upgrade when a project is too old.
## Live replay progress in setup and assistant runs
Bitfab replay runs can now stream per-trace progress in Claude, Cursor, and Codex while the replay keeps running in the background. The plugins show each trace as it settles, write a per-run event log under `.bitfab/replays//events.jsonl`, and store full per-item outputs under `.bitfab/replays//items/` so agents can review complete outputs without mixing progress logs into the result JSON.
## Replay progress reporters for SDK scripts
TypeScript, Python, and Ruby SDKs now include ready-made replay progress callbacks: `reportReplayProgress`, `report_replay_progress`, and `Bitfab.report_replay_progress`. Pass them into replay scripts to emit plugin-readable progress events while stdout stays reserved for the final ReplayResult JSON.
## Capture warnings stay out of execution errors
SDK spans that have to degrade serialization now mark those payload warnings as SDK-sourced capture notices. The dashboard shows them as a bordered "Capture incomplete" tag on the span instead of mixing them into execution errors, so real runtime failures stay distinct from lossy capture metadata.
## Studio opens in an app-style window from Codex
The Bitfab Codex plugin now opens Studio in the same focused app-style browser window used by the other editor plugins when Chromium supports it. If the app-window launch cannot start, the plugin falls back to your normal browser, and `BITFAB_DISABLE_CHROME_APP_WINDOWS=1` still forces the tabbed fallback.
## Chat-session capture handles unsupported text characters
Bitfab now keeps coding-agent chat-session capture working when terminal output includes characters that databases cannot store in JSON fields. Unsupported characters are safely replaced so the rest of the transcript, tool calls, and usage details remain available in the dashboard.
## More reliable chat session capture
Bitfab now captures coding-agent chat sessions more reliably when turns include token usage details. This keeps session histories and usage metadata flowing into the dashboard instead of dropping affected turns during ingestion.
## Fix a failing trace end to end
The Bitfab assistant has a new `fix` mode that takes one failing trace and drives it to green. Point it at a trace (`/bitfab:assistant fix ` in Claude Code, `/bitfab-assistant fix ` in Cursor) and it adds the trace to a dataset with a validated failing label, edits the code, then replays until the trace passes, flagging any regressions and offering to fix the other failing traces too. It only engages when you point it at a real Bitfab trace, so an ordinary "fix this bug" on untraced code still goes through normal coding. Works across the Claude, Cursor, and Codex plugins.
## Plugin login reuses its sign-in window
Signing in to the Bitfab plugin no longer opens duplicate Studio windows when a login is retried before it finishes. The plugin now reattaches to the window that's already waiting for you, so a sign-in that looks slow won't stack up extra browser windows.
## Filter trace search by database snapshot
The `search_traces` tool now accepts a `hasDbSnapshot` filter, so you can scope a search to traces that captured a database snapshot, or only those that didn't. Use it to quickly find the traces that can be replayed against their historical database state. Works across the Claude, Cursor, and Codex plugins.
## Plugin sign-in with Google no longer hangs
Signing in to the Bitfab plugin with Google now completes reliably and connects your coding agent. Previously a Google sign-in could leave the plugin waiting on a login that never registered.
## Faster usage page
The usage page loads noticeably faster, especially for organizations with high trace volumes, and switching the function filter now applies instantly instead of pausing while the page catches up.
## Shareable links for usage filters
The usage page now keeps your function filter in the page URL, so a filtered view can be bookmarked or shared as a link and it stays put as you move around. Your browser's back and forward buttons also step through filter changes.
## Accurate "Snapshot captured" badge on traces
The "Snapshot captured" badge now appears only on traces that actually captured a database snapshot, instead of on every trace once your organization connected a source database. The trace list and trace view now give an accurate at-a-glance signal of which traces can be replayed against their database state from when they ran.
## Break down usage by function
The usage page can now be scoped to a single traced function: pick a function and the totals, chart, periods table, and CSV export all narrow to just that one. A new "By function" chart mode plots one line per function so you can compare volume across functions over time. The function picker and chart mode appear once your organization has more than one traced function.
## Bind framework handlers to your trace key once
The `getFunction()` handle now hands you framework handlers and middleware already bound to its trace key, so an outer span and the handler share one key without repeating the string. Available for the Claude Agent SDK, LangGraph/LangChain, and (TypeScript only) the Vercel AI SDK.
```typescript theme={null}
const pipeline = bitfab.getFunction("my-agent")
const runAgent = pipeline.withSpan({ type: "agent" }, async (prompt) => {
const handler = pipeline.getClaudeAgentHandler() // same key, no retyping
// the handler-traced agent run nests under this span
})
```
Python exposes the same on `get_function()`: `pipeline.get_claude_agent_handler()` and `pipeline.get_langgraph_callback_handler()`.
## Live progress callbacks for replay
Replay now takes an optional progress callback, so you can show live progress while a run is in flight instead of waiting for it to finish. It fires once per trace as each one settles, with running totals you can render however you like.
```ts theme={null}
await bitfab.replay("my-function", fn, {
onProgress: ({ completed, total, errored }) => {
process.stderr.write(`\rReplaying ${completed}/${total} (${errored} errored)`)
},
})
```
It's `onProgress` in the TypeScript SDK and `on_progress` in Python and Ruby. The totals report how many traces have finished, how many ran without error, and how many threw; pass/fail verdicts are assigned after the run, so the live totals split ran-ok versus errored. Upgrade to `@bitfab/sdk` v0.25.0, the `bitfab` Python package v0.24.0, or the Ruby gem v0.21.0.
## Clearer experiment run summaries
Experiment run rows now report accurate counts: a replay that finished with an error shows as "errored" instead of "awaiting labels", and a trace still awaiting an agent label is no longer also counted as "unpaired". Hover any status pill to see the full breakdown of passing, failing, awaiting, and errored traces. The row also stays readable at any panel width, collapsing to the essentials when space is tight.
## Replay results report the replayed run's token usage
When you replay traces, each result item's `tokens` now reflects the token usage of the **replayed** run instead of the original trace. Compare it against the original trace's recorded usage to see how a code change moved cost. `durationMs` and `model` stay as the original trace's reference values, and the token counts match what the experiments view shows. Upgrade to `@bitfab/sdk` v0.24.1, the `bitfab` Python package v0.23.2, or the Ruby gem v0.20.2.
## See database snapshot status on your traces
Traces now show whether they can be replayed against the database state from when they ran, and whether a replay actually used that historical state. A "Snapshot captured" badge marks an original trace you can replay against its past database; on replays, "Snapshot replayed" means your code read the historical database branch and "Snapshot unused" means it fell back to the live database. The indicator appears on the trace list, the trace and span detail headers, and in dataset and experiment rows, and only shows for organizations with a connected source database.
## Watch experiment results appear live during replay
When you start a replay, its experiment now appears right away and fills in trace by trace as results come back, instead of staying on "Waiting for experiments to start..." until the whole replay finished. You can watch the pass and fail counts climb in real time as each trace completes.
## The TypeScript SDK never crashes or hangs your app
Tracing is now fully fail-open. If anything in the SDK's instrumentation goes wrong (a runtime without a usable `crypto`, an oversized payload, a serialization edge), your traced function still runs and returns its real value, and your Node process still exits cleanly instead of being held open by a pending timer. When tracing has to degrade, the SDK logs a one-time `[bitfab]` warning so a dropped span is visible rather than silent. Upgrade to `@bitfab/sdk` v0.24.0.
## Type-checking no longer needs the OpenAI Agents package
The SDK's published type definitions no longer reference `@openai/agents`, so projects that don't use the OpenAI Agents integration type-check cleanly without installing that package (previously `tsc` could fail with "Cannot find module '@openai/agents'" under `skipLibCheck: false`). If you use `getOpenAiAgentHandler(key).wrapRun(...)`, its result is now typed structurally: read `finalOutput` and cast it to your agent's output type.
## More reliable trace ingestion under load
Traces now ingest reliably during high-volume bursts. We fixed a case where ingesting a trace could time out and fail when its search index was being built inside the same database operation that saved the trace. Search indexing now runs after the trace is safely stored, so a slow index can never block or fail ingestion.
## TypeScript SDK: builds no longer fail on unused optional integrations
If you used the TypeScript SDK with one integration (like the Vercel AI SDK) but not others, your bundler could fail at build time with `Module not found: Can't resolve '@openai/agents'`, even though you never used the OpenAI Agents integration. The SDK no longer references its optional peer dependencies (`@openai/agents`, `@boundaryml/baml`) in any way a bundler tries to resolve up front, so your app only needs to install the integrations it actually uses. Upgrade to `@bitfab/sdk` v0.23.3.
## Vercel AI SDK tracing
Bitfab now traces the Vercel AI SDK out of the box. Wrap your model with `getVercelAiMiddleware` and every `generateText`, `streamText`, `generateObject`, and `streamObject` call is captured as a traced LLM span, including streaming responses and which provider answered each call (handy when you fall back between models). The Bitfab plugin also detects and instruments Vercel AI SDK projects automatically during setup.
```typescript theme={null}
import { openai } from "@ai-sdk/openai"
import { streamText, wrapLanguageModel } from "ai"
const model = wrapLanguageModel({
model: openai("gpt-4o"),
middleware: bitfab.getVercelAiMiddleware("chat-turn"),
})
streamText({ model, messages }) // traced; your live stream is untouched
```
## Ruby SDK guards against replay key mismatches
`replay()` in the Ruby SDK now raises an `ArgumentError` when the `trace_function_key:` you pass does not match the key the method is actually traced under, instead of silently fetching one function's history and recording the run under another. This brings Ruby in line with the TypeScript and Python SDKs.
## Replayed traces match the original trace structure
Replaying a handler-instrumented run (OpenAI Agents, Claude Agent SDK) now produces a root span named after your trace function key, matching the original production trace instead of the replayed function's name. For OpenAI Agents, a replayed run also nests under a single root span instead of an extra duplicated one, so a replay's span tree lines up with the trace it replays.
## Closed Studio windows stay closed
When you close a Studio window, it now stays closed. Previously the plugin could reopen a Studio window on its own shortly after you dismissed it; that no longer happens. Studio windows open only when you or your coding agent explicitly ask for one.
## Recovering a stalled Studio session
If a Studio window crashed, was closed, or your machine went to sleep, reopening Studio could stay stuck reporting that the existing session was unreachable, even after you cleared it. Clearing a stalled session now reliably reopens a fresh Studio window.
## More reliable Studio sessions
Fixed an issue where leftover Studio background helpers from earlier plugin versions could pile up and cause a Studio session to hang or fail to open. The plugin now clears out stale helpers before starting a session, so opening Studio stays reliable across plugin updates.
## LangGraph and LangChain retriever steps are now traced
Retriever calls in your LangGraph or LangChain graphs now appear as spans, capturing the query and the documents returned, so retrieval shows up alongside the rest of your agent's work. No code change is needed beyond the callback handler you already pass.
## Trace streamed OpenAI Agents runs
You can now trace streamed agent runs. In TypeScript, pass `{ stream: true }` to `wrapRun`; in Python, use the new `wrap_run_streamed` async generator as a drop-in for `Runner.run_streamed`. The run's input and final output are recorded on the root span once the stream finishes, so streaming runs are traced without changing how you consume the events.
```python theme={null}
async for event in handler.wrap_run_streamed(agent, "What's the weather?"):
... # handle each streamed event
```
## Read the BAML Collector after a call
`wrapBAML` / `wrap_baml` now expose the BAML `Collector` from the most recent call through a `.collector` attribute, plus an `onCollector` / `on_collector` callback that runs after each invocation. Use it to inspect the prompt, model, and token usage yourself, in addition to the metadata Bitfab already captures.
```python theme={null}
classify = bitfab.wrap_baml(b.ClassifyText, on_collector=lambda c: log(c))
```
## Faster trace lists for high-volume functions
The traces list now loads quickly for functions with very large trace volumes, where it could previously take several seconds or fail to load. The speedup is automatic, with nothing to configure.
## Replayable OpenAI Agents runs with one line
If you trace the OpenAI Agents SDK with Bitfab, you can now make agent runs replayable with a drop-in change. The new run wrapper records the run's input as a replayable root, with the tracing processor's spans nested underneath, so the trace replays by key with no hand-written wrapper. Keep registering the processor for the internals and swap your run call for the wrapper.
```typescript theme={null}
const handler = bitfab.getOpenAiAgentHandler("my-agent")
const result = await handler.wrapRun(agent, "user input")
```
In Python, use `get_openai_agent_handler("my-agent").wrap_run(agent, "user input")`.
## More reliable organization switching
Switching between organizations now reliably loads the selected organization's data. Previously, changing orgs (including opening a Studio page for a resource that belongs to another organization you're a member of) could leave the view stuck on your previous organization and show missing or "not found" data until you refreshed. The switch now finishes applying before the page reloads, so the correct organization's data loads the first time.
## Replayable instrumentation for OpenAI Agents
When you set up Bitfab tracing, it now makes sure each traced workflow can be replayed, not just observed. For OpenAI Agents, that means wrapping the agent run in a root span that captures its input, so you can re-run the trace against your current code. The setup health check also flags any instrumented function whose root can't be replayed and points you to the fix.
## Full Claude Agent SDK tracing in TypeScript
Tracing for the Claude Agent SDK now captures complete agent runs in TypeScript: every LLM turn, tool call, and subagent becomes a span, with token usage included. Get a handler with `getClaudeAgentHandler`, inject the hooks with `instrumentOptions`, and wrap the `query()` stream:
```typescript theme={null}
const handler = bitfab.getClaudeAgentHandler("my-agent")
const options = handler.instrumentOptions({ model: "claude-sonnet-4-5" })
for await (const message of handler.wrapQuery(query({ prompt, options }), { input: prompt })) {
// your normal message handling
}
```
## Replay handler-traced agent runs
Agent runs captured by the Claude Agent SDK handler are now replayable without wrapping your code in an extra span. Pass the prompt as `input` to the wrap call and the handler records it as the run's root, so `replay()` can re-run each historical prompt against your current code. Works in both the TypeScript and Python SDKs.
## Trace and dataset labeling panels adapt to narrow widths
The dataset review and trace detail views now reflow to fit tight spaces, so labeling traces in the Studio side panel stays comfortable even when the panel is narrow. The labeling toolbar collapses onto a compact top row, and a trace's input, output, and context sections stack cleanly instead of overflowing or leaving uneven gaps between them.
## Compact span tree with hover details
When the trace panel is narrow, the span tree collapses to an icon rail that keeps the full call hierarchy while handing the freed space to the span content. Hover any span to see its name, position, duration, and token count, and the token figure follows your selected token view (all or uncached).
## Cost-optimize mode for the assistant
The Bitfab assistant has a dedicated cost-optimize mode for cutting token spend. Run `/bitfab:assistant cost-optimize ` and it first profiles where a dataset's tokens go (prompt size, redundant context, output shape, model choice), then edits and replays against your labeled dataset to lower cost while holding the pass rate. The token-cost view turns on automatically, reporting token deltas next to pass/fail. Available across the Claude, Cursor, and Codex plugins.
## Before and after on every experiment trace
In the experiments view, any replay trace paired with an original now always shows the before and after comparison: the verdict change, the Original/Updated toggle, and the token trend. Values that aren't known render as blanks instead of hiding the comparison, so still-passing and still-failing traces are as easy to inspect as fixed and regressed ones.
## More accurate OpenAI Agents trace capture
Traces from the OpenAI Agents SDK integration now capture tool-call and generation inputs correctly. Previously the TypeScript SDK could drop these inputs and record them as empty, and both SDKs added empty input and response fields to agent and other non-LLM spans. Tool calls, agent steps, and other spans now show exactly the data the run produced. Update to TypeScript SDK 0.21.1 or Python SDK 0.21.2 to pick up the fix.
## Faster labeling and evaluation on large datasets
When the assistant labels a fresh batch of traces or evaluates an experiment, it now judges the traces in parallel on larger datasets, so big batches finish noticeably faster. The speed-up kicks in automatically above roughly 20 traces; smaller batches are unchanged.
## Read a span field in full when a trace is truncated
When Bitfab shows you a trace, large span fields (a long input or a big tool output) are truncated so the response stays readable. The new `read_span_field` tool fetches the complete, untruncated value of a single field (input, output, reasoning, content, errors, or context) for one span, so your coding agent can pull the full text only when it actually needs it. `read_traces` now points you to it whenever a field is truncated. Available across the Claude, Cursor, and Codex plugins.
## Experiments show when a replay is awaiting labels
In the experiments view, a replay that has finished running but hasn't been scored yet is now marked "awaiting agent labels" instead of an indefinite loading spinner. The state appears on the run's pass-rate pill, its trace rows, and the progress bar, and stays out of the pass rate until a verdict lands, so a run that is done replaying reads as settled rather than stuck.
## Studio windows close more reliably at the end of a flow
Studio windows now close themselves when a flow finishes, with a backup that helps close the window if the browser does not. You will see fewer stray Studio windows left open after setup, assistant, and other flows wrap up.
## Reliable sign-in from the Bitfab CLI
Setting up Bitfab with `npx bitfab-cli init` (or signing in with `bitfab login`) now completes reliably. A recent CLI version could fail to open the sign-in window and report that you were not authenticated; the CLI now opens Studio and finishes login as expected.
## Faster trace loading in the assistant
When the Bitfab assistant builds or labels a dataset, it now loads the dataset's traces in parallel batches instead of one group at a time. Large datasets come into context faster and more reliably, with no change to how you run the assistant. This applies across the Claude, Cursor, and Codex plugins.
## Studio session daemon for persistent browser management
Bitfab plugins now include a session daemon that manages Studio browser windows as a persistent background process. The daemon keeps Studio alive across coding agent restarts: if a browser window crashes, it re-spawns automatically; if you navigate to a new page, it reuses the existing window instead of opening a second one; and if a page refresh fires a transient close event, the daemon waits briefly before treating it as a real close.
## Built-in help on every command
Every Bitfab plugin and CLI command now responds to `-h` / `--help` with a usage line, a short description, and per-argument details, then exits without running. You (and your coding agent) can discover how to call a command without leaving the terminal, and `bitfab init -h` now prints help instead of starting the full onboarding flow.
## Studio sessions reconnect after expiring
Returning to Bitfab Studio after a session has been idle now reconnects automatically instead of showing a "Could not connect to this session" error. Previously an expired session left you stuck until you signed in again; the session is now re-established transparently when you come back.
## No duplicate Studio window after signing in
Signing in to Bitfab from your coding agent no longer leaves a leftover Studio window or opens a duplicate. The next action reuses the window you signed in with instead of spawning a second one.
## More reliable tracing for Python agent frameworks
The Python SDK no longer drops a span or trace when a value that can't be JSON-serialized (such as a Pydantic model) appears outside a span's input or response, like a tool's output or a trace's metadata. Spans and traces from the OpenAI Agents, LangGraph, and Claude Agent SDK integrations now always reach Bitfab, and any value that couldn't be captured faithfully is flagged as non-replayable instead of silently lost.
## Compact trace reads by default
The `read_traces` tool now returns bounded, truncated span details by default (`scope: "summary"`), so the Bitfab assistant can scan many traces in one pass without overflowing its context. Each span keeps the head and tail of its largest fields, so big inputs and outputs stay legible at a glance; pass `scope: "full"` when you need complete, untruncated detail on a handful of traces.
## Trace streaming functions without a refactor
You can now instrument a streaming function (one that returns a live response stream) by adding a single `finalize` option, with no need to restructure your code. Your function still streams to users exactly as before, while Bitfab records a clean, replayable summary of the turn (text, token usage, tool calls) as the trace output.
In the TypeScript SDK, pass `finalize` to `withSpan` and use the built-in `finalizers.aiSdk` helper for the Vercel AI SDK:
```typescript theme={null}
import { finalizers } from "@bitfab/sdk"
const runChatTurn = bitfab.withSpan(
"chat-turn",
{ type: "agent", finalize: finalizers.aiSdk },
() => streamText({ model, messages }),
)
```
In the Python SDK, decorate an async generator that yields its chunks and pass `finalizers.openai_chunks` or `finalizers.anthropic_events`. And when you run setup on a streaming endpoint, Bitfab now guides you straight to this instead of asking you to refactor.
## Experiments stay linked to their dataset
When you benchmark or run an experiment against a dataset, the run is now durably attached to that dataset, so it always appears on the dataset's experiments page, even when the underlying traces can't be matched back by lineage. Before, the link was inferred from shared traces and could be missed.
## `datasetId` option when replaying
The TypeScript, Python, and Ruby SDKs' `replay()` now takes a `datasetId` / `dataset_id` option that attributes the run to a dataset. Passed on its own, it replays exactly that dataset's traces, so you no longer need to look up and pass the trace IDs yourself.
```ts theme={null}
await client.replay("my-function", processInput, { datasetId: "" })
```
The Claude, Cursor, and Codex plugins use this automatically when you benchmark a dataset.
## Point Bitfab setup at a specific workflow
When you set up Bitfab tracing, the plugin now asks how you want to find what to instrument: let it scan your codebase for AI workflows, or point it straight at a specific file, function, or directory. Choose the targeted option when you already know what you want traced and want to skip the full scan. Available in the Claude, Cursor, and Codex plugins.
## Faster Bitfab assistant runs
The Bitfab assistant no longer pauses between steps while it reports progress to the Studio sidebar. Those activity updates are now sent in the background, so the assistant keeps moving instead of waiting on the network. The difference is most noticeable during longer flows like dataset building and experiments.
## Studio follows your work across organizations
Opening a Studio link for a dataset, trace plan, trace, test run, or experiment that lives in a different organization than your session started in now switches you into the right organization and loads the page, instead of showing a not-found error. If you belong to only one organization, nothing changes. When you are viewing a page in an organization other than the one your coding agent is connected to, a small indicator in the Studio header makes that clear.
## Sign-in returns you to where you were headed
Following a link to a specific trace, trace plan, or other page while signed out now takes you to that exact page after you sign in or sign up, instead of dropping you on your default traces view. This works for both email and Google sign-in.
## Uncached token basis in the experiments cost view
The experiments cost lens can now show "uncached" tokens (input minus cached reads, plus output), so you can see the tokens you actually pay full price for instead of the cheap cache reads. With the token lens on, switch between All and Uncached from the "Token count" control in the experiments header, and each trace's cost trend recolors so the two views read apart at a glance. This surfaces cost regressions that prompt caching would otherwise hide behind a flat total.
## Claude Agent SDK reports the full prompt size in inputTokens
The Claude Agent SDK handler (TypeScript and Python) now folds cache reads and cache creation into `inputTokens`, so it reports the full prompt size, consistent with the LangGraph integration. The cached portion is still reported separately as `cacheReadTokens`. If you read `inputTokens` directly for cache-heavy calls, expect a larger value than before.
## Spans never silently drop on non-serializable inputs
All four SDKs now keep a span even when an input or output can't be JSON-encoded: the offending value is replaced with a placeholder and a warning flags that the trace may not be replayable, instead of the whole span being dropped. The LangGraph, OpenAI Agents, and Claude Agent SDK handlers, along with the span decorator, also capture nested values more completely, so handler-instrumented traces stay replayable.
## Cleared the phantom "Grading" status on trace rows
Trace rows no longer show a spinning "Grading..." status when your project has no graders configured. Previously every incoming trace briefly displayed a grading indicator that never resolved. Traces that do have graders are unaffected and still show their pass and fail scores.
## Cached tokens in the per-span token breakdown
When the experiments token view is on, each span's token breakdown now shows cached input tokens alongside input and output, so you can see how much of a span's prompt was read from cache rather than processed fresh.
## Token usage in experiment trace results
The `get_experiment_traces` tool now reports token usage (input, output, cached, and total) for each replay trace and the original it is compared against. Your coding agent can use this to reason about cost and cache-read changes between runs, not just the pass or fail verdict.
## Before/after comparison for replays with an unlabeled original
In the experiments view, a replay whose original trace was never labeled now shows a neutral "Unlabeled" badge alongside its own Pass or Fail result and opens the same before/after comparison as a scored trace, instead of hiding the comparison entirely. Open the replay to toggle between the original and updated runs and see the change in context, even while the replay is still being evaluated.
## Replay tracks whether the database snapshot was actually used
When you replay traces against a historical database branch, each replayed trace now records whether a branch was provisioned for it and whether your code actually read the branch connection URL. A replay that checks `env.active` but never reads `env.databaseUrl` (`env.database_url` in Python and Ruby) is recorded as not having used the branch, which catches the common silent failure where a connection pool created at module import sends the replay to your live database instead. Tracking is automatic in the TypeScript (0.18.2), Python (0.18.2), and Ruby (0.17.1) SDKs with no code changes; the Database Branching docs describe the three states a replayed trace can record.
## See your organization's usage
A new Usage page (in the user menu) shows how many traces your organization has ingested and how much data they carry over time. Switch between daily, weekly, and monthly views, pick a date range, and read totals with trend indicators at the top; a usage-by-period table breaks it down further and exports to CSV. Dates and time buckets follow your local time zone.
## Ask Bitfab to cut token usage
The Bitfab assistant now responds to token and cost reduction requests. Tell your coding agent something like "use Bitfab to reduce token usage on one of my datasets" and it enters the experiment flow: it changes prompts or code, replays your labeled dataset, and checks that token usage drops without hurting the pass rate. The experiments view shows original vs. replay token totals next to each verdict, so you can see the savings as results stream in.
## Skipped replay traces are marked done, not left spinning
When you skip a replay trace during review, the experiments view now shows it with a "Skipped" badge instead of leaving the row spinning as if it were still being evaluated. The run summary reads "9/11 passing · 1 skipped" so a skipped trace is clearly counted as intentionally unscored, rather than dropped from the totals or mistaken for a result that is still loading.
## Replay environments branch your organization's own database copy
When replaying with a `ReplayEnvironment`, each replayed item now branches your organization's own managed database copy (the one provisioned when you connect a database in the dashboard) at the trace's capture time, so replays read your data as it was, never anyone else's. Replay branches are cleaned up automatically after each run, with a background sweep catching anything an interrupted run leaves behind.
## Token cost in experiments and the trace viewer
When an experiment is about reducing token usage, you can now see per-trace and per-span token counts alongside pass/fail, so you can tell whether a change actually cut cost as the results stream in. It shows up in the experiments comparison and the trace viewer, with the token figures styled distinctly from the green/red verdict so cost reads as a measurement, not another pass/fail signal. The Bitfab assistant turns it on automatically when an iteration is framed around cutting tokens or cost.
## Guided setup for database branching in replay
A new `/bitfab:setup db-branching` flow walks you through replaying traces against your database as it was at trace time, instead of today's data (TypeScript, Python, and Ruby). Connect a Postgres database once in the dashboard, and the flow polls until the branchable copy is ready, then wires a `ReplayEnvironment` into your replay scripts so each replayed item reads from a per-trace branch. A new connection-status check lets the flow tell you exactly when your database is connected and provisioned.
## Reliable database connection setup
Connecting a source database for trace replay now completes reliably regardless of database size. Setup runs in the background in resumable stages and the dashboard's connection status tracks it as it progresses, instead of timing out partway through on large databases. Failures (an unreachable database, an invalid connection string) now surface promptly as a failed connection with the underlying reason, rather than leaving the status stuck on checking.
## Full token-usage capture for LangChain and LangGraph
LLM spans traced through the LangChain/LangGraph callback handler now record token usage from modern LangChain releases, which put counts on each message's `usage_metadata` rather than the legacy `llm_output.token_usage` location. Streaming agents are covered too: usage is read from the final aggregated chunk. Provider-native shapes from OpenAI, Anthropic, and Google are recognized as fallbacks, and the legacy location keeps working.
Cached prompt tokens are now reported as `cachedInputTokens` on the span, and Anthropic input counts include cache reads so they reflect the true prompt size (heavily cached agents will see input counts go up; that is the previous under-count being fixed). Only provider-reported numbers are recorded, nothing is estimated. For OpenAI streaming, enable `stream_usage=True` (Python) or `stream_options: {"include_usage": true}` (TypeScript) so usage arrives on the stream.
## Replay for handler-instrumented workflows
Workflows traced through a framework handler (LangGraph, LangChain, Claude Agent SDK, OpenAI Agents) can now be replayed, even though they have no decorated root function in your code. Pass the handler's trace function key plus any plain callable to `replay()`, and the SDK wraps it internally so every replayed run records a trace tied to the experiment:
```python theme={null}
handler = bitfab.get_langgraph_callback_handler("my-agent")
def replay_my_agent(state):
return graph.invoke(state, config={"callbacks": [handler]})
result = bitfab.replay("my-agent", replay_my_agent, limit=10)
```
The same works in TypeScript with `bitfab.replay("my-agent", fn, options)`, where plain callables are wrapped automatically. The setup and assistant plugin skills now offer handler instrumentation as a first-class option for workflows whose entry points take live objects (database handles, billing callbacks), and write the matching replay script for you. See "Replaying handler-instrumented functions" in the Python and TypeScript SDK docs.
## Fixed database connector provisioning in production
Fixed a bug where connecting a database for trace snapshots failed in production before provisioning could start. Database connector setup from the dashboard now completes as expected.
## More reliable Studio sessions
Studio now keeps a single, durable connection open for the whole assistant session. Moving between pages (dataset review, experiments, trace plans) reuses the open Studio window instead of opening extra background connections, so signals like marking a dataset done or ending a session are no longer missed. The live Studio updates in the assistant flow are dependable from start to finish, even when a session is reused from an earlier run.
## Replay a trace against its historical database state
The Python and Ruby SDKs can now replay a recorded trace against the database as it was when the trace ran. Pass a `ReplayEnvironment` to `replay()` and read its database URL inside your function; Bitfab resolves a per-trace database branch for each item and releases it when the item finishes. This brings Python and Ruby to parity with the TypeScript SDK.
```python theme={null}
env = ReplayEnvironment()
@client.span("lookup-user")
def lookup_user(user_id):
db_url = env.database_url if env.active else LIVE_DATABASE_URL
...
client.replay(lookup_user, environment=env)
```
Every trace is now automatically pinned to its capture-time snapshot, so any trace can be replayed this way later, with no extra configuration.
## Trace LangChain by its own name
The Bitfab callback handler has always traced plain LangChain chains as well as LangGraph graphs (both share the same callback system), but every entry point was named after LangGraph. The SDKs now expose LangChain-named aliases that return the identical handler: `getLangChainCallbackHandler()` in TypeScript and `get_langchain_callback_handler()` in Python, with the class also exported as `BitfabLangChainCallbackHandler`.
```python theme={null}
handler = bitfab.get_langchain_callback_handler("summarize-doc")
result = chain.invoke({"document": doc_text}, config={"callbacks": [handler]})
```
The plugin's setup flow now recommends the LangChain-named methods when instrumenting a project that uses LangChain without LangGraph, and the framework docs include a plain-LangChain quick start.
## Keep your app's API key in sync when you switch organizations
When you switch organizations with the Bitfab plugin, it now offers to update your project's local `BITFAB_API_KEY` as well. Previously the switch only repointed the plugin, so traces your own code sent kept landing in the old organization until you updated the key by hand. After you run `/bitfab:setup switch-org` (or `/bitfab-setup switch-org` in Cursor, `$bitfab:setup switch-org` in Codex), the agent finds every `.env` file that defines the key and, with your go-ahead, updates them in place.
## Connection fields on the Integrations page no longer clear while you type
Entering a database connection string or integration secret on the Integrations page now stays put as you type. Browser password managers were treating these masked fields as login passwords and overwriting them mid-entry; they're now opted out, so your input holds.
## Switch organizations from your coding agent
You can now switch which Bitfab organization a plugin reads and writes without leaving your editor. Run `/bitfab:setup switch-org` (or `/bitfab-setup switch-org` in Cursor, `$bitfab:setup switch-org` in Codex): the agent lists the organizations you belong to, switches to the one you pick, and swaps the plugin's API key to the new org. The agent can also call the new `list_organizations` tool on its own to check which org it is currently pointed at. Your already-open browser tabs keep showing their current org until the next time the plugin opens Studio.
## Faster traces list
The traces list now loads significantly faster for functions with a large number of traces. Opening a function's traces no longer waits on a full count of every matching trace before showing results, so the page appears as soon as the traces are ready.
## Sign in without a duplicate Studio window
When you run a Bitfab plugin command while signed out and a Studio window is already open, the plugin now reuses that window to sign you in instead of opening a second one. There's only ever one Studio window now, where before a fresh window could appear and leave the original orphaned. Running login while you're already signed in is also a no-op, and you can re-authenticate on demand by running login with `--force`.
## See a dataset's experiment history in Studio
You can now open the experiments page scoped to a single dataset. Ask the Bitfab assistant to "show experiments for a dataset" and Studio lists every experiment that replayed one of the dataset's traces, giving you that dataset's full run history in one place. Previously the experiments page opened only for specific test runs or an experiment group.
## More reliable Studio session cleanup
Studio sessions now close themselves cleanly when an assistant or setup run finishes: the browser tab closes, background processes stop, and nothing is left running even if the window refuses to close. Recovering from an unreachable Studio window ("Open a new Studio") also cleans up the old window and its background process instead of leaving them behind.
## Accurate error indicators in the trace viewer
Spans with no recorded errors no longer show a false "Error detected" badge or an "Execution Error" section in the trace viewer. Error detection now correctly ignores empty error data, and a related fix ensures a real execution error can no longer be hidden by an empty error list.
## Live updates arrive reliably on the hosted dashboard
Pages that update in real time, such as dataset review, trace lists, and experiment results, could miss updates on the hosted dashboard and only show new data after a manual refresh. Event delivery now completes reliably, so new traces, labels, template changes, and experiment results appear the moment they happen.
## Replay a single trace to check your fix
The assistant has a new `replay` mode for the quickest version of the improvement loop: you already made a fix and just want to know whether one specific trace passes now. Run `/bitfab:assistant replay ` (or simply ask "did my fix work on ``?") and the agent finds your replay script, re-runs that one trace through your current code, and reports a pass/fail verdict in chat. It skips everything heavier: no browser, no dataset, no labeling, and nothing is persisted, so it's safe to run as often as you like while iterating.
## Replay accepts limit and trace IDs together
Passing both `limit` and `traceIds` (`trace_ids` in Python and Ruby) to `replay()` no longer throws. The SDK now logs a warning and ignores `limit`, since an explicit trace ID list already determines how many traces replay. This applies to the TypeScript, Python, and Ruby SDKs, so replay scripts that forward both flags keep working instead of crashing.
## Replay traces whose function signature changed
When you rename, reorder, or restructure a traced function's arguments, replay can no longer feed it the inputs recorded against the old shape. `replay()` in the TypeScript, Python, and Ruby SDKs now takes an `adaptInputs` (`adapt_inputs` in Python and Ruby) hook that reshapes each trace's recorded inputs onto the current signature, so older traces keep running.
```typescript theme={null}
await bitfab.replay("my-function", updatedFn, {
adaptInputs: (inputs, ctx) => [{ userId: inputs[0], limit: inputs[1] }],
})
```
The hook runs per trace and is isolated per item: if one trace can't be reshaped, that item alone reports the error and the rest of the run continues.
## The assistant recovers replays broken by signature changes
When you iterate on a traced function in your coding agent and a replay fails because the signature drifted since the traces were captured, the Bitfab assistant now recognizes the mismatch instead of treating it as an environment error, and helps you write a small committed input adapter so those traces rejoin the run.
## Spot traces that can't be replayed
The trace viewer now flags traces that won't replay against your current code, either because they captured no top-level span or because their recorded inputs no longer fit the function's current signature. A "Can't replay" badge appears on dataset rows and in the trace detail view, so you can see at a glance which traces a replay will actually cover before you run it.
When your coding agent opens a dataset, the Bitfab plugins pass along your function's current input shape, and the check runs live against it. Nothing is stored, so the badge can't go stale.
## Replay results now persist reliably
`replay()` in the TypeScript, Python, and Ruby SDKs now waits for each replayed item's trace to be fully persisted before completing the test run. `item.traceId` (`trace_id` in Python and Ruby) is a real server trace ID you can use immediately. Previously, a race could leave every trace ID null and the experiments page empty, even though the replay appeared to succeed.
Failures are no longer silent. If none of the replayed items' traces reached the server (for example, the replayed function isn't instrumented), `replay()` raises an error explaining why. If only some items fail to persist, those items return a null trace ID with a logged error and the rest of the run comes back intact, so one bad trace costs you one data point instead of the whole run.
## Clearer skill routing in plugin flows
The setup, assistant, and update skills now state exactly where each choice leads ("Update all → step 7", "Skip → stop"), so coding agents follow multi-step flows more reliably instead of inferring the wiring from prose. The update skill on Claude Code also runs as chained sub-skills: each phase hands off directly to the next with the invocation mode attached, removing a class of lost-context routing mistakes in long sessions.
## Live dataset review in Studio
Dataset review now always happens on the dataset's own page in Studio, which updates in real time as your coding agent adds traces and applies labels. Previously the agent could leave you on a function-level review page that only showed new activity after a manual refresh.
That older function-level page now redirects to the function's most recent dataset, so existing links and older plugin versions keep working.
## Replay by trace IDs no longer truncates
Replaying specific traces by ID previously capped the list at the default `limit`, silently dropping the rest of your selection: 12 IDs in could mean only 5 replayed, skewing experiment results without warning. An explicit ID list now always replays every trace in it (up to 100).
`limit` and `traceIds` are now mutually exclusive: `limit` means "replay my last N traces", and an ID list speaks for itself. Passing both raises a clear error instead of guessing.
```typescript theme={null}
// Replay your last 10 traces
await bitfab.replay("my-function", fn, { limit: 10 })
// Replay exactly these traces (up to 100)
await bitfab.replay("my-function", fn, { traceIds: ["id-1", "id-2"] })
```
Available in the TypeScript, Python, and Ruby SDKs v0.14.0, with matching `trace_ids` semantics in Python and Ruby. Older SDK versions get the core fix server-side: explicit ID lists are no longer truncated by a default limit.
## No more duplicate Studio windows
Your coding agent now keeps exact track of its Studio window. Ending a session no longer makes the next Studio command open a second window while the old one lingers: the existing window is reused, and the agent only forgets a window once the browser confirms it actually closed. Refreshing the Studio page mid-session is also safe; the connection re-establishes itself instead of being mistaken for a close.
Commands that reconnect to an already-open Studio window now react only to what happens after they connect, so a previously ended session or an earlier run's activity can no longer end a new command prematurely. If a Studio window disappears without a trace (for example the browser quit entirely), the agent detects that it is unreachable and offers to reopen instead of guessing.
## Watch benchmark runs live in Studio
The assistant skill's `benchmark` mode is terminal-only by default, but you can now add the `studio` keyword (for example, `benchmark studio`, or just ask in natural language to "open studio") to open Studio's experiments page and watch each trace's pass/fail verdict stream in as the replay runs. The default stays terminal-only, so existing benchmark runs are unchanged unless you opt in.
## Traces are flagged errored only when your code fails
A trace is now marked as errored only when your traced code throws, not when the Bitfab SDK hits a serialization or ingestion error while recording the trace. The error indicator in the trace list now reflects failures in your own functions, so SDK-side noise no longer surfaces as a failed trace.
## Errored traces open on the failing span
When you open a trace that recorded an error, the trace viewer now jumps straight to the first span that failed instead of starting on the trace root. You land on the error and its message right away, without scanning the span tree for the red marker. Traces without errors open exactly as before, and any span you deep-link to still takes precedence.
## Errored spans highlighted in the trace viewer
Spans that recorded an error are now flagged directly in the trace viewer. The span tree marks failed spans in red, and the span header shows an Error tag with the error message on hover, so you can spot failures in a trace without opening each span.
## Replay keeps going when an individual trace fails
The Python and Ruby SDKs now isolate per-trace errors during `replay()`: if one historical trace fails to load or its function raises, that result is marked with an error and the rest of the run still completes, instead of the whole replay aborting. This matches the TypeScript SDK's behavior, so a single bad trace no longer costs you the entire run.
## Copy buttons in custom span templates
Custom span templates can now drop in a clipboard icon button next to any field value with `{{ value | copyButton | safe }}`. Pass a string for the tooltip and accessible label, e.g. `{{ span.id | copyButton("Copy span id") | safe }}`. The button copies the value (objects and arrays are pretty-printed as JSON), flashes "Copied", and works inside the template's isolated shadow DOM without any extra JavaScript or CSS in the template.
## More reliable Studio sign-in
When your coding agent opens Bitfab Studio and you sign in, the Studio tab now connects reliably on the first try. Previously a timing issue could leave a freshly opened session showing "Could not connect to this session" until you re-ran the command; signing in now hands off cleanly to the live session every time.
## Error source classification on spans
Span errors now carry an explicit source tag so you can tell whether an error came from your code or from the SDK itself. When your traced function throws, the error is recorded with `source: "code"`. SDK-internal failures (like serialization errors) are tagged `source: "sdk"`. Both types appear in the unified `errors` field on the span, replacing the previous split between `span_data.error` and the errors column.
All four SDKs support this: set `error_source: "code"` automatically when a traced function fails.
```typescript theme={null}
const result = await bitfab.span("my-function", async () => {
throw new Error("something went wrong")
// error_source: "code" is set automatically
})
```
When the same span is sent multiple times (for example, during retries), errors from all payloads are merged and deduplicated rather than replaced.
## Connect a database for per-trace snapshots
You can now connect your Postgres database from the new Database page so Bitfab can take an isolated snapshot per trace at replay time. Paste a connection string and activate: replays run against a fresh branch of your database, so they never touch or slow down production. The page shows live status (activating, active, or failed, with a support contact if it can't reach your database), and you can deactivate at any time from a confirmation dialog.
## More reliable trace reads under heavy use
Reading several traces at once through the Bitfab plugin (the `read_traces` tool) no longer returns intermittent errors when many requests are in flight at the same time. Trace reads and agent-label updates are now more efficient, so larger reads stay fast and reliable.
## Quickly add a trace to a dataset
The `/bitfab:assistant` command has a new lightweight `add-trace` mode that attaches one or more existing traces to a dataset and stops, without the full label-and-iterate flow. Run `/bitfab:assistant add-trace ` (the function key is inferred from the trace) or just ask your coding agent to "add this trace to a dataset". It picks or creates the right dataset for you, and if you point it at several traces it makes sure they all belong to the same function before attaching.
## Diagnose your tracing setup with `/bitfab:setup inspect`
A new `inspect` mode checks whether your Bitfab tracing is healthy: whether you're authenticated, what's instrumented in this repo, whether the plugin and SDK are up to date, whether your replay scripts cover every trace function, and whether traces are actually arriving. It then walks the available fixes one at a time, asking before each change. Run `/bitfab:setup inspect`, or just ask your coding agent something like "why aren't my traces showing up?".
## Get oriented with `/bitfab:setup explain`
A new read-only `explain` mode prints a quick overview of what Bitfab is and what each setup mode does, without authenticating or scanning your code. Run `/bitfab:setup explain` or ask "what is Bitfab?".
## Smarter Studio window reuse
The Bitfab plugin now reuses your existing Studio window whenever it's still open, instead of risking a duplicate window or a stale "not responding" prompt. Close the Studio tab and the next action opens a fresh window right away; only a window that's genuinely unreachable, like after a crash or your machine sleeping, will ask whether to retry or open a new one.
## Clearer Studio connection status
Studio's connection indicator now tells you exactly what's happening: "Studio connected" when your agent is live, "Awaiting agent" while it's away, and "Studio disconnected" if the browser loses its live connection to the session (it reconnects automatically). Reconnecting to a session you already have open now reuses that Studio tab instead of opening a second window.
## Benchmark a dataset against your current code
Run `/bitfab:assistant benchmark ` to replay a labeled dataset against your current code without changing anything, then read a pass/fail scorecard that shows which traces still pass, still fail, regressed, or were fixed. Use it to measure where your function stands right now, as a regression baseline or a quick check after unrelated work, instead of starting an experiment loop. You can also just say "benchmark my dataset" in plain language and the assistant routes there.
## More reliable Studio sessions
Studio no longer shows a stray "agent disconnected" popup on the session-complete page. When your agent disconnects mid-session, the reconnect prompt now rejoins your existing Studio session instead of starting a new one. And if you run more than one coding agent in the same project, each keeps its own Studio session instead of overwriting the other's.
## Sign-in URL shown when the browser doesn't open
When you sign in to Bitfab from your coding agent and the browser doesn't open automatically, the login flow now prints the sign-in URL so you can open it manually. Previously it told you to visit the URL without showing one.
## Simplified plugin login
The setup login flow now uses a single authentication method that works everywhere, including SSH sessions, containers, and cloud IDEs. The separate `login headless` mode has been removed since the standard login already handles these environments automatically via its server-polled channel.
## Seamless login in the assistant flow
Running `/bitfab:assistant` without being authenticated now logs you in inline through Studio instead of stopping the flow. Previously, unauthenticated users were told to run a separate login command first, breaking the workflow. The assistant now opens Studio's sign-in page directly and continues automatically once you've signed in.
## Plugin login no longer hangs for CLI users
The `bitfab init` login flow now completes reliably instead of hanging after sign-in. Previously, the published CLI used a query parameter the close page didn't recognize, so the authentication callback never fired and the CLI waited indefinitely. The close page also now displays "Login complete" instead of the generic "Session Complete" message.
## Experiment results show label annotations
The experiments page now displays the label annotation for each trace instead of the raw function output. This matches how the dataset page already renders traces and makes it easier to scan experiment results for what passed, what failed, and why. Annotations from both human reviewers and the agent's automated labeling are shown.
## Replay script upgrades no longer interrupt the assistant flow
When the assistant detects that your replay script needs an upgrade (missing code-change or experiment-group support), it now edits the script directly instead of launching a separate setup flow. Previously, this would break the assistant's continuity and drop you to an empty prompt. The upgrade happens inline and the experiment flow continues automatically.
## Faster disconnect detection in Studio
When an agent closes its Studio session, the browser now shows "disconnected" within milliseconds instead of up to 60 seconds. Previously, the connection status indicator could display "Agent connected" long after the agent process had exited because the server-side heartbeat lingered in cache.
## Studio redirects to sign-in instead of showing a session error
Opening a Studio page without being signed in now redirects you to the sign-in page instead of showing a "Could not connect to this session" error. After signing in, you're returned to the page you originally requested with the session intact.
## Studio commands now open login instead of erroring
Plugin commands that open Studio pages (experiments, trace plans, datasets, template previews) no longer fail with "Not authenticated" when you haven't logged in yet. Instead, they open Studio directly and redirect you to the sign-in page. After you sign in, you land on the page the command originally requested. For interactive commands like trace plan confirmation and dataset review, the plugin saves your credentials automatically so the bidirectional event channel works normally after login.
## Automatic replay script capability detection
The assistant now checks whether your replay script supports the latest experiment features before running experiments. If your script is missing support for code diffs, experiment groups, or trace ID tracking, the assistant offers to upgrade your SDK and regenerate the script in place. You can also choose to continue without the missing features. This replaces the previous behavior where outdated scripts would silently skip features or produce incomplete experiment results.
## Dataset mode now continues through failure diagnosis and experiments
Fixed a bug where the assistant's dataset mode could stop after building the dataset instead of continuing to diagnose failures and run experiments. The flow's internal instructions contradicted its routing in three places, which could cause the agent to exit early. Dataset mode now reliably progresses through the full pipeline: build dataset, diagnose failures, iterate with experiments, and wrap up.
## Clear error messages when Studio can't connect to a session
When Studio fails to connect to an agent session or switch to the correct organization, it now shows a clear error message instead of silently loading in the wrong context. This prevents the confusing state where traces appear missing because Studio was looking in a different organization than the one your plugin authenticated against.
## Live experiment streaming
The experiments page now streams results in real time as replays complete. When the assistant runs experiments, it opens the experiments viewer before the first replay starts and new results appear automatically via server-sent events as each test run finishes. Previously, the experiments page only opened after all replays completed.
All three SDKs now accept an `experimentGroupId` parameter on `replay()` that groups multiple test runs into a single experiment batch:
```typescript theme={null}
// TypeScript
await client.replay("my-function", traceIds, {
experimentGroupId: "550e8400-e29b-41d4-a716-446655440000",
})
```
```python theme={null}
# Python
await client.replay("my-function", trace_ids,
experiment_group_id="550e8400-e29b-41d4-a716-446655440000")
```
```ruby theme={null}
# Ruby
client.replay("my-function", trace_ids,
experiment_group_id: "550e8400-e29b-41d4-a716-446655440000")
```
The experiments page accepts `?experimentGroupId=` as a query parameter, and falls back to `?testRunIds=` for replay scripts that haven't been updated yet.
## Inline template editing during labeling
You can now edit trace view templates in chat while labeling traces, without leaving the dataset review page. When you ask the assistant to change how a span type renders (e.g. "edit the LLM template"), it reads and updates the template inline using MCP tools, and the dataset page re-renders automatically. Previously, template editing required invoking the setup flow, which navigated Studio away from the dataset and broke the labeling session.
## Auto-update trace plan in Studio
When you modify your span capture setup and create a new trace plan, Studio now automatically navigates to the updated plan. Previously, you had to manually refresh or re-navigate to see the latest version after making changes to the capture configuration.
## Investigate mode continues through the full pipeline
When you run the assistant in investigate mode, the flow now continues through diagnosis and experiments after building a dataset, matching how dataset mode already works. Previously, investigate mode stopped after dataset building, requiring you to restart in experiment mode to iterate on fixes. All assistant modes now serve as entry points into the same pipeline, converging at wrap-up regardless of where they start.
## Graceful Studio close on flow exit
When the assistant finishes a flow (wrap-up, early stop, or sub-mode completion like dataset-only), the Studio tab now closes gracefully instead of lingering in a disconnected state. The agent navigates to a close route that lets the Studio clean up before the background process is terminated.
## Better error messages when agents pass wrong CLI arguments
Plugin commands now validate arguments against a declarative schema before executing, catching common agent mistakes like invented `--flag` syntax, missing arguments, or invalid UUIDs. When a command receives bad input, it prints a usage string showing the expected arguments and a clear error message, so the agent can self-correct on the next attempt.
## Reconnect guidance when Studio loses agent connection
When your coding agent disconnects from Studio, a popup now appears after 30 seconds with a copiable prompt you can paste into your agent to reconnect. The popup includes your session ID so the agent can rejoin the same session. If you dismiss it, the popup reappears with increasing intervals, and it auto-dismisses if the agent reconnects on its own.
## Dataset mode continues to diagnosis and experiments
When you run `/bitfab:assistant dataset`, the flow now continues past labeling into Phase 4 (diagnosis and experiments), matching the behavior of the full `/bitfab:assistant` flow. Previously, dataset mode stopped after labeling, requiring you to restart in the default mode to run experiments on the same dataset.
## Live experiment verdicts in Studio
When your replay script returns trace IDs (requires SDK v0.13.5+), the assistant now opens the experiments page in Studio before running evaluations, so you can watch pass/fail verdicts populate in real time. If your SDK predates trace ID support, the assistant prompts you to update and falls back to showing evaluation results as text in the agent.
## Code-change diffs in the experiment viewer
When the assistant runs experiments, it now captures before-and-after file snapshots for every edit and attaches them to the replay. The experiment viewer can then display the literal code change alongside pass/fail results, so you can see exactly what was tried in each iteration. Existing replay scripts that predate this feature continue to work; the assistant detects whether the script supports the new `--code-change` flag and gracefully skips the metadata if it doesn't.
## Replay results include trace IDs
Each replay result item now includes a `traceId` (or `trace_id` in Python/Ruby) that links directly to the server-side trace created during replay. Previously, matching a replay result back to its trace in the dashboard required heuristics based on input similarity. Now you can navigate straight to the trace.
```typescript theme={null}
const result = await bitfab.replay("my-function-key", myFunction, {
limit: 5,
});
for (const item of result.items) {
console.log(`Trace: ${item.traceId}`);
}
```
Available in TypeScript SDK v0.13.5, Python SDK v0.13.2, and Ruby SDK v0.12.2.
## Experiments distinguish errored traces from pending
When a replay trace fails during execution (before it can be graded), the experiments page now shows it as "errored" instead of lumping it in with pending traces. Errored traces appear with a red "Error" badge, a red row tint, and a dedicated segment in the progress bar. The pass-rate pill also now shows context-appropriate states: "N errored" when all traces errored, "N pending" with a spinner when grading is in progress, and "X/Y so far" for partially graded runs.
## Assistant reviews existing dataset traces before searching for new ones
When you pick a dataset that already has traces, the assistant now goes straight to the review page instead of asking how to source new candidates. Previously, datasets with unlabeled (but present) traces were treated like empty datasets, which skipped past the traces you already had. Empty datasets still get the "what kind of traces should I find?" prompt as before.
## Experiments page updates after replay completion
The experiments page now refreshes automatically when a replay finishes and as labels are applied. Previously, completing a replay didn't trigger an update, so the page could appear empty until manually refreshed. Summary counts and individual trace labels now appear as soon as they're available.
## Studio session recovery after agent interruptions
The plugin assistant now automatically reconnects to an existing Studio browser session when the background polling process is interrupted (for example, by a long conversation triggering context compaction). Previously, losing the background process meant the assistant would open a duplicate Studio window. Now it resumes the existing session seamlessly, keeping the same browser tab and session state.
## Experiment and dataset tools for the assistant flow
The plugin assistant can now work with experiments and datasets directly. Two new MCP tools, `list_experiments` and `get_experiment_traces`, let the assistant list recent experiments for a function and drill into individual trace verdicts (fixed, regressed, still-passing, still-failing). The `search_traces` tool also now accepts `testRunId` and `datasetId` parameters, so you can scope trace searches to a specific experiment run or dataset without manual filtering.
## Fixed investigate mode opening a broken Studio page
Running `/bitfab:assistant investigate ` no longer opens Studio to a non-existent page. The investigate mode now lands on the Studio root, which loads correctly. The investigation itself (trace reading, code exploration, findings summary) was unaffected since it runs via tool calls, not Studio navigation.
## Trace plan links now open correctly in Studio
Clicking a trace plan link from the plugin assistant now opens the trace plan inside the Studio session. Previously, the link pointed to a standalone route outside of Studio, which bypassed the authenticated Studio context.
## Experiment results stream in real time
Running experiments on a dataset no longer blocks until every test finishes. The experiments page opens immediately and results stream in trace by trace: the progress bar fills, pass/fail counts update, and trace rows appear as each test completes. If background execution fails, the run is marked as failed and the page updates accordingly instead of getting stuck on "pending."
## Trace list loads reliably for high-volume functions
The traces page now loads correctly on the first visit for functions with very high trace creation rates. Previously, real-time update events could interfere with the initial page load, causing the trace list to appear empty until you navigated away and back.
## Reliable Studio navigation during assistant sessions
Studio navigations that include query parameters (such as opening the experiments page with specific test run IDs) no longer time out with "not responding." The same fix also ensures that navigating to the same page with different parameters is recognized correctly, so the assistant flow proceeds without interruption.
## Dataset traces now stream in real-time in Studio
Traces added to a dataset while viewing it in Studio now appear immediately without a page refresh. A recent migration to Studio's route tree accidentally dropped the real-time event connection, so newly added traces were invisible until you reloaded.
## Fixed experiment mode navigation
The plugin assistant's experiment mode now correctly opens the experiments page. Previously, it attempted to navigate to a non-existent per-function experiments route.
## Studio reconnects automatically after sleep
Studio connections now recover automatically when your laptop wakes from sleep. Previously, closing your lid and reopening could leave the session unresponsive until you manually refreshed the page.
## Dataset page shows errors instead of misleading empty state
The dataset review page now displays a clear error message when traces fail to load, instead of incorrectly showing "No traces yet." This helps you quickly identify loading failures, such as viewing a dataset while signed into the wrong organization.
## Per-trace DB branching for replay (alpha)
Replay can now run against the database state at the moment a trace was recorded, not your current production database. This raises fidelity for agents whose behavior depends on stored state, like a refund decision that read a since-cancelled order or a retrieval agent that saw last week's index. Available in the TypeScript SDK; backed by Neon preview branches on the Bitfab service.
Wire it up:
```ts theme={null}
import { Bitfab } from "bitfab"
const bitfab = new Bitfab({
apiKey: process.env.BITFAB_API_KEY,
dbSnapshot: { provider: "neon" },
})
const env = new Bitfab.ReplayEnvironment()
await bitfab.replay("refund-agent", async (orderId) => {
const url = env.active ? env.databaseUrl : process.env.DATABASE_URL
const pool = new Pool({ connectionString: url })
return decideRefund(orderId, pool)
}, { environment: env })
```
**Alpha caveats.** Single-tenant in this release; per-org Neon project configuration and the Ardent customer-LSN to replica-LSN mapping land in a follow-up. Reach out if you want it turned on for your account.
## Studio sessions survive page refreshes
Refreshing the Studio tab no longer kills your coding agent's session. Previously, a browser refresh was indistinguishable from closing the tab, so the plugin would immediately tear down the connection. Now the plugin waits up to 10 seconds for the page to reload before ending the session, so you can refresh freely without interrupting your workflow.
## Plugin login respects project-local credential isolation
When your project has a `.bitfab/credentials.local.json` file (used to isolate credentials per project), logging in now writes the new API key to that file instead of the global credentials store. Previously, login always wrote to `~/.config/bitfab/credentials.json`, which meant the project-local file stayed empty and the plugin fell back to the global key, defeating the isolation.
## Studio auto-switches org to match the plugin session
When your browser's active org differs from the org bound to the plugin's API key, Studio now detects the mismatch and automatically switches to the correct org on load. This fixes the "Awaiting agent" stuck state that could occur when you belong to multiple organizations and your browser happened to be on a different one than your plugin.
If you're not a member of the session's org, the plugin now cleanly aborts and tells you why, instead of hanging indefinitely.
## Studio agent connection recovers after laptop sleep
Studio now correctly restores the agent connection indicator after your laptop sleeps and wakes. Previously, the "agent disconnected" banner could get stuck even though the agent had successfully reconnected. The fix ensures heartbeat tracking refreshes on every poll and that the browser verifies the actual connection state when resuming after a gap.
## TypeScript SDK is now `@bitfab/sdk`
The TypeScript SDK package has moved from `bitfab` to `@bitfab/sdk`. New installs should use the scoped name:
```bash theme={null}
npm install @bitfab/sdk
```
```typescript theme={null}
import { Bitfab } from "@bitfab/sdk";
```
The old `bitfab` package continues to work but now prints a deprecation warning on import. Running the plugin's update command (`/bitfab:update`) detects the legacy package and warns you to switch, even if the version number is current.
## Studio is now the single browser surface for all plugin flows
Every plugin CLI flow (login, trace plan confirmation, dataset review, template preview) now opens inside Studio instead of launching a separate browser window. If Studio is already open, the plugin navigates it in place rather than opening a new window. This means fewer browser tabs, a consistent UI, and the ability to stay in one window while working with the assistant.
Headless login is now available for environments where a browser can't reach your terminal (SSH, cloud IDEs, CI). Visit `/studio/auth/claude` in any browser, sign in, copy the token, and paste it back into your coding agent.
Closing Studio during a trace plan confirmation now cleanly cancels the operation instead of leaving the CLI in an error state.
## Automatic package rename in update flow
Running `/bitfab:update` now detects the legacy `bitfab` npm package and offers to switch it to `@bitfab/sdk`. The update flow removes the old package, installs the new one, and rewrites imports in your source files. If you're already on `@bitfab/sdk`, nothing changes; the flow works as before.
## Studio URL guard and auth verification
Studio now always opens at the correct `/studio` path. Previously, certain launch conditions could cause the Studio window to open at the site root instead of the Studio interface. A path guard now normalizes the URL before the browser window opens.
The plugin's auth status check now verifies your API key against the current server. If you switch between servers (e.g., local development to production), the status command correctly reports that re-authentication is needed instead of showing a stale "authenticated" state.
## Live-streaming dataset pages in Studio
When the Studio assistant creates or picks a dataset, the dataset review page now opens immediately instead of waiting for all traces to be labeled and attached first. Traces appear on the page in real time as the agent finds, labels, and attaches them. Label changes on traces already in a dataset also update live, so you can watch rows move between the "Agent labeled," "Labeled," and "Unlabeled" sections without refreshing.
If the page is empty while the agent is still working, a "Building your dataset" indicator shows that traces are on the way.
## Trace plan review page scrolls
The trace plan review page now scrolls when a plan has more captured nodes than fit on screen. Before, long plans clipped the Advanced selection toggle and any inline error messages below the fold; opening a plan with around 40 captured nodes now scrolls cleanly through the full call tree.
## TypeScript SDK available as @bitfab/sdk
The TypeScript SDK is now published under the scoped package name `@bitfab/sdk` in addition to the existing `bitfab` package. Both names resolve to the same code and will stay in sync on every release. If you prefer scoped package names for clarity in your `package.json`, you can switch your import at any time:
```bash theme={null}
npm install @bitfab/sdk
```
No code changes are required beyond updating the package name in your imports. Both `import { Bitfab } from "bitfab"` and `import { Bitfab } from "@bitfab/sdk"` work identically.
## SDK serialization hardening
Trace spans now ship reliably even when function inputs or outputs are difficult to serialize. Objects with circular references, oversized payloads (over 512 KB), or classes that throw during serialization no longer cause lost spans. Instead, the SDK replaces the problematic value with a descriptive `` stub so the span still appears in your traces with full timing and metadata.
This fix applies to the TypeScript, Python, and Ruby SDKs. No code changes are needed on your side; update to the latest SDK version to get the improvement automatically.
## Plugin can query Studio browser state
Plugins can now check whether the Studio browser tab is connected and which page is currently active via the new `getStudioState` function. This lets the plugin make smarter decisions before navigating, for example skipping a navigation command when Studio is already on the target page, or surfacing a connection warning when the browser tab has been closed.
## Studio crash recovery and agent navigation guardrails
Studio now shows a recoverable error screen when a runtime error or unexpected crash occurs during a session. Instead of a blank page or a full 404, you see a "Try again" button that retries without losing your session context.
Agent-initiated navigation is now validated against a known route whitelist. When an agent tries to navigate to an invalid or out-of-scope path, it receives an immediate `navigation-blocked` event with a reason string instead of waiting for a 12-second timeout. This helps agents self-correct faster when a requested page doesn't exist.
## Studio stays connected through long conversations
The Bitfab plugin now persists the link between your coding agent conversation and your Studio session. Previously, when a long conversation triggered context compaction, the agent lost track of which Studio window it had opened, requiring you to reopen Studio manually. Now the mapping is written to disk and recovered automatically after compaction, so Studio commands continue working seamlessly in extended sessions.
## Code change diffs in experiments
When an experiment replays traces against a code change, you can now view the exact diff that was tested. Click the file stats on any experiment card or the code-change pill in the trace detail header to open a side-by-side diff modal. The modal also shows how the dataset reacted overall (fixed, regressed, still passing, still failing) or, when opened from a single trace, whether that specific trace flipped.
## Fixed plugin login falling through to manual paste flow
Signing in to a Bitfab plugin (Claude Code, Cursor, or Codex) via the browser now reliably completes the automatic handoff back to your terminal. Previously, the login page could lose the callback parameters during a redirect, causing every login to fall through to the manual "copy and paste this token" flow even when the browser and terminal were on the same machine.
## Investigate a trace function with /bitfab:assistant
Run `/bitfab:assistant investigate []` to characterize an issue in a trace function without going through the full assistant flow. The agent reads recent traces and your code based on what you describe, then offers three follow-ups: stop with an in-chat summary, save a written report under `.bitfab/analysis/`, or hand off to dataset building when the findings include reproducible failures worth labeling. The function key is optional; when omitted, the agent picks it from your description or asks.
## Agent-initiated Studio session close
Agents can now programmatically end a Studio session when their work is complete. The `closeStudio()` helper sends a completion event with an optional message, and the browser automatically closes or shows a "Session Complete" screen with the agent's message. This replaces the need for users to manually click "End session" when the agent is done.
## Trace plan confirmation lands inside Studio
The Bitfab plugin's trace-plan confirmation page (where you review which spans your function will capture) now renders inside your existing Studio tab during `/bitfab:setup` instead of spawning a second browser window. Studio's header and agent indicator stay visible while you decide; Confirm or Cancel keeps the tab open for the rest of the flow, no more orphan windows.
If no Studio is running (you invoked `/bitfab:setup` outside an `/bitfab:assistant` session), the confirmation falls back to the standalone chromeless window as before.
## Annotate a closed trace from any process
The TypeScript and Python SDKs now expose a detached `client.getTrace(id)` handle that lets you add context, merge metadata, or set the session id on a trace **after** its root span has closed. The handle works from any process, thread, or agent that knows the trace id, with no shared in-memory state. Useful when a downstream worker or a forked AI agent needs to attach information to the original conversation's trace.
```typescript theme={null}
// From any process, by trace id:
const trace = client.getTrace(traceId);
await trace.addContext({ refund_status: "approved" });
await trace.setMetadata({ region: "us-west" });
await trace.setSessionId("session_xyz");
```
Available in `bitfab` v0.13.0 for TypeScript and Python.
## Smarter Studio session management
The assistant flow now reuses an existing Studio session instead of opening a new browser window each time. If the Studio becomes unresponsive (tab closed, page crashed), the agent detects this within 12 seconds and offers options to refresh the tab or open a fresh session.
## Live activity progress in Studio
Studio now shows which phase the assistant is working on in real time. As the skill progresses through steps like identifying the trace function, building a dataset, or running experiments, the header displays the active phase name with a live elapsed timer. When one phase completes and the next begins, you see the previous phase's duration before it transitions.
If the agent disconnects or crashes mid-phase, the activity indicator automatically resets within 30 seconds instead of showing stale state indefinitely.
## Open a trace plan from inside `/bitfab:assistant`
Ask the assistant to "open the trace plan for X" (or "show me what's captured") and it now routes your open Studio tab to that function's most recent trace plan in place. The Studio shell stays mounted around the plan, so your agent session, header, and connection indicator persist across the navigation, and no new browser tab pops up. The canonical `/trace-plan/[id]` URL still works as a standalone shareable link outside Studio.
## Click any trace while reviewing a dataset
Fixed a bug that blocked clicks while a trace detail was open. You can now switch traces or press Done without closing the open one first.
## Redesigned trace planner
The trace planner now leads with what you actually need to know: a validation summary at the top that calls out anything blocking replay (live writes inside captured spans, missing samples, disconnected roots), then a flow diagram of the captured spans and a sample-trace preview of how the recorded trace will look in the viewer. The legacy two-pane tree picker is still there, tucked behind an Advanced selection toggle for power-users. Confirm and Cancel still flow through the same Cmd+Enter / Esc handoff, so muscle memory carries over.
## Live agent connection indicator in Studio
Studio now shows a real-time connection status in the header. A green dot with "Agent connected" appears when the coding agent is actively polling, and transitions to a gray dot with "Awaiting agent" if the agent disconnects. The indicator updates instantly when the agent reconnects, with no page refresh needed.
## Mutual presence detection for plugins
The agent plugin now receives `browserConnected` in its poll response, indicating whether a user has Studio open in the browser. This enables plugins to adapt their behavior based on whether someone is actively watching the session.
## Studio is now the default assistant mode
The `/assistant` skill now opens Studio automatically on every invocation. You no longer need to pass a `studio` argument to get the companion browser surface. Studio is always there, from start to finish.
## Studio opens directly at the relevant page
When you start in dataset or experiment mode (`/assistant dataset ` or `/assistant experiment `), Studio now opens directly at that function's datasets or experiments page instead of opening at the root and navigating after. This shaves a few seconds off each focused session and puts you in context immediately.
## Logout redirects to sign-in page
Signing out no longer lands on a blank page. You're now redirected to the sign-in page, where you can immediately log back in or close the tab.
## Session log capture fix and standalone opt-in
Session log capture now works correctly after opting in during setup. A configuration mismatch previously caused the plugin to silently skip session capture even when you'd consented, so no session data was being collected. You can also now toggle session log capture on or off by running `/bitfab:setup session-logs`, a standalone mode that doesn't require authentication.
## Trace viewer skips empty spans on open
Opening any trace now lands on the first span that has data instead of a blank trace root or an empty span. This applies across the dashboard: trace detail pages, the labeling panel, the experiments comparison view, the dataset detail panel, and the template preview studio. The hard template filter still hides non-matching spans; you just no longer have to scroll past empty ones to see meaningful content.
## Studio navigation events for coding agents
When you navigate between pages in Studio, the coding agent now receives real-time navigation events with the current path. This gives the agent immediate awareness of where you are in Studio, so it can tailor its responses and actions to the page you're viewing without needing to ask.
## Studio sign-in stays within the Studio shell
When your coding agent opens Studio and you're not signed in, you now see a branded sign-in page inside the Studio window instead of being redirected to the main Bitfab login. The session context persists across the sign-in flow, so the agent picks up exactly where it left off once you authenticate. The CLI receives real-time `auth-required` and `authenticated` events, letting it wait for sign-in without polling.
## Accurate offline SDK update checks
The plugin's session-start update check now always reports the correct latest SDK versions. Previously the baked version snapshot could lag behind by one release, causing the plugin to miss update notifications or report you were up to date when a newer SDK was available.
## Hill-climb from existing labels in /bitfab:assistant
When you start a new dataset for a function that already has validated labels, the assistant now offers a Reuse option that seeds the dataset with those labels instead of starting from scratch. Pick Reuse when you're spinning up a different cut for experimentation but want to keep the labeling work you already trust. Define and Open are still there for the from-scratch and broad-sample cases.
## Replay verdicts persist with a coverage gate
After a replay in Phase 5, the assistant writes its pass/fail verdicts on the replay traces through a bundled script that verifies every replay trace got a verdict before moving on. Previously a verdict could die mid-session if the agent forgot to persist it; now the script enforces full coverage before continuing. If a trace is genuinely ambiguous, you can record it as an explicit skip rather than leaving it silently unverdicted.
## Plugins surface which Bitfab org they're writing to
The plugin MCP now flags which Bitfab org it reads and writes from, so you'll catch mismatches between your project's `BITFAB_API_KEY` and the org open in your Studio tab before traces land somewhere unexpected. Coding agents now call `get_api_key_context` at the start of a plugin MCP session, and again whenever you mention data you just wrote isn't visible in Studio. The same tightening applies to the remote MCP server in the Dashboard for direct (non-plugin) callers.
## Ruby SDK: skip child spans during replay
When you replay historical traces through `client.replay(...)`, you can now have child spans return their recorded outputs instead of running real code. Three strategies control which children get short-circuited:
* `mock: "none"` (default) reruns every child span as before.
* `mock: "all"` returns historical output for every child.
* `mock: "marked"` returns historical output only for spans declared with `mock_on_replay: true`, and runs everything else real.
```ruby theme={null}
class Pipeline
include Bitfab::Traceable
bitfab_function "pipeline"
bitfab_span :classify, type: "llm", mock_on_replay: true
def classify(text)
# paid LLM call
end
bitfab_span :process, type: "agent"
def process(text)
classify(text)
end
end
client.replay(Pipeline.new, :process,
trace_function_key: "pipeline",
mock: "marked")
```
Use `mock: "marked"` to iterate on agent logic without paying for the marked child calls on each replay. Use `mock: "all"` for the cheapest possible replay (only the root function runs real code). Brings the Ruby SDK to parity with the existing `mock` option in the Python and TypeScript SDKs.
## Ruby SDK: fluent wrapper for shared trace function keys
`client.get_function(key)` returns a wrapper bound to that trace function key, so you can wrap multiple methods or classes without repeating the key on every call.
```ruby theme={null}
fn = Bitfab.client.get_function("openai")
fn.wrap(OpenAI::Client, :chat, name: "Chat", type: "llm")
fn.wrap(OpenAI::Client, :embeddings, name: "Embed", type: "llm")
```
Matches `client.get_function` in the Python SDK and `client.getFunction` in TypeScript.
## Accurate experiment counts with multi-label traces
Experiment pass/fail counts now correctly deduplicate traces that have labels from multiple sources (human review, approved agent, unapproved agent). Previously, a trace with both a human and an agent label could be double-counted in experiment totals. The viewer now picks the highest-priority label per trace: human labels take precedence over approved agent labels, which take precedence over unapproved ones.
## Experiments auto-label replayed traces
When you run an experiment through the assistant, replayed traces now receive agent labels automatically. The experiment viewer shows pass/fail results immediately after a replay completes, without requiring a manual labeling step first.
## Replay with mocks: shared-key spans return the correct output
Fixed an off-by-one in the replay mockTree when the function under test and one of its children share one `traceFunctionKey` (the canonical `getFunction(key).withSpan(...)` pattern). The marked child was returning the root's historical output instead of its own. The mockTree is now keyed by `(traceFunctionKey, spanName, callIndex)`, which also unblocks recursive same-key replays.
```ts theme={null}
const summarize = bitfab.getFunction("summarize-thread")
const buildTranscript = summarize.withSpan(
{ name: "buildTranscript", mockOnReplay: true },
async (id) => db.getTranscript(id),
)
const processSummarize = summarize.withSpan(
{ name: "processSummarize" },
async (id) => generate(await buildTranscript(id)),
)
// `mock: "marked"` now returns buildTranscript's own historical
// transcript string, not processSummarize's full result object.
```
## Mocked non-async Promise-returning functions stay Promises
If you wrap a `function fetchX() { return fetch(...) }` (no `async`, but returns a Promise) and mock it during replay, the mocked return is now a `Promise`, not a raw value. Downstream `.then(...)` callers no longer crash. Detected at wrap time.
## `/bitfab:assistant` experiments auto-pick parallel or serial
Phase 5 of the assistant skill now checks whether subagent worktrees inherit bypass permissions before forking parallel experiments. If `permissions.defaultMode: "bypassPermissions"` is set in committed `.claude/settings.json` or `~/.claude/settings.json`, experiments fork to worktree-isolated subagents; otherwise they run serially in the main agent. Cursor and Codex always run serial since they don't support worktree-isolated subagent calls.
## Organization switcher fix
Fixed the organization switcher dropdown not appearing in the header. After upgrading to Clerk v7, the switcher silently returned no memberships, making it impossible to switch between teams. The switcher now reliably shows all your organizations, with your personal workspace listed first and the rest sorted alphabetically.
## Live agent activity in Studio
The Studio home page now shows what your coding agent is doing in real time. While the assistant is working, the agent card highlights green and displays the current tool action (e.g., "Reading traces...", "Creating grader..."). When the agent finishes or goes idle, the card fades back to its neutral state. Activity persists across page navigation within Studio, so you won't lose track of the agent's progress.
## Studio detects which coding agent opened it
When Studio is launched from Cursor or Codex, the UI now shows that agent's logo and name instead of defaulting to Claude Code. The welcome page, header, and "Return to" button all reflect the agent that started the session.
## Replay failure handling in the assistant skill
`/bitfab:assistant` now separates infrastructure failures (missing DB rows, rejected writes) from real regressions during replay, and keeps unreplayable traces out of the pass-rate. When a child span fails environmentally, it suggests either flipping the span to `mockOnReplay` or pointing replay at the trace's source environment.
## Replay mocks return the correct child span's output
Fixed ordering bugs in `mock: "marked"` that caused a marked child span to return a sibling's historical output instead of its own. Upgrade to TypeScript SDK 0.12.1 if you're using `mock: "marked"` on 0.12.0.
## Mock child spans during replay
When you replay a recorded trace against new code, child spans sometimes fail locally for reasons unrelated to what you're iterating on, like a paid API key you don't have set, a flaky external service, or a production database row that isn't seeded in your local environment. Replay now supports skipping those children and returning their recorded outputs instead, so the root function can still run.
Pass a `mock` strategy to `replay()` to control it. `"none"` (default) runs every child for real. `"all"` returns the historical output for every descendant. `"marked"` only short-circuits descendants you've tagged at definition time, leaving everything else to run real, which is the iteration-friendly mode.
Tag a span with `mockOnReplay: true` in TypeScript or `mock_on_replay=True` in Python:
```ts theme={null}
bitfab.withSpan("fetch-article-from-db", { mockOnReplay: true }, async (id) => db.find(id))
```
```python theme={null}
@bitfab.span("fetch-article-from-db", mock_on_replay=True)
def fetch_article_from_db(article_id): ...
```
Then replay with `mock: "marked"` (TS) or `mock="marked"` (Python). The flagged child returns its recorded output and downstream spans run real code, so you can iterate on the analysis or formatting steps without standing up the upstream dependency.
When the assistant skill is replaying a function and a child span fails environmentally, it'll now suggest this fix directly. Full docs: TypeScript SDK and Python SDK reference under "Mocking child spans during replay".
## Reliable focus restoration for macOS terminals
When clicking "Return to coding agent" in Studio, focus now reliably returns to the correct terminal app. The previous approach could target the wrong window if the terminal's environment was modified (common inside Claude Code). The plugin now identifies your terminal by walking the process tree to find the parent application. For iTerm2 users with multiple windows, focus targets the exact session pane.
## Persistent Studio session for the assistant flow
Add `studio` to any `/bitfab:assistant` invocation (e.g., `/bitfab:assistant studio`) to keep a single Studio window open for the entire flow. Dataset review and experiment results open inside the same window instead of launching separate ones, so you stay in one place while iterating. Without the `studio` argument, the flow works exactly as before.
## See which template renders each span at a glance
Iterating on the right template is faster when you can tell which one runs for the span you're looking at. In the template preview, click or arrow-key through any span in the trace viewer and the matching card in the left rail lights up in that span's color. That's the template to edit.
## Know when your coding agent is mid-edit
Stay out of the agent's way and watch its work land in context. When your agent saves a template, the studio names who is editing (Claude Code, Cursor, or Codex), pulses the affected card in the rail, and outlines the exact region inside the rendered span, even on instant saves, so you don't miss it.
## Chat session capture
Bitfab plugins can now capture your coding-agent chat sessions and send them to the dashboard. Session capture is **opt-in**: enable it by setting `BITFAB_CAPTURE_SESSIONS=true` or adding `"captureSessions": true` to `~/.config/bitfab/config.json`. Nothing is captured until you explicitly turn it on.
Once enabled, sessions are only recorded after you invoke a Bitfab tool or slash command in the same conversation, so ordinary non-Bitfab conversations are never captured. Works across Claude Code, Cursor, and Codex.
## Cross-platform focus restoration
When a plugin opens a browser window (OAuth login, Studio preview), focus now returns to your terminal or editor automatically on Linux and Windows. Previously this only worked on macOS. If platform tools aren't available (e.g., Wayland on Linux), the handoff completes normally without focus restoration.
## Studio connection errors surface immediately
When your coding agent opens the Studio preview, connection problems (expired API key, network timeout) are now caught before the browser window opens. Previously, errors could surface mid-session after you'd already started editing.
## Click-to-target template editing
In the template preview studio, you can now click directly on a rendered span to tell your coding agent exactly which region you want changed. No more "make the user message smaller" guesswork: point at the element and describe the change.
## Live preview auto-refresh
Templates saved in the studio now re-render in the preview automatically. Previously, you had to reload the page to see your changes.
## Template reference for coding agents
The new `get_template_reference` MCP tool returns a catalog of every editable region in the standard template, so coding agents can discover what's available without you having to describe it.
## Template preview is faster on large functions
The template preview page loads significantly faster for functions with many spans. Pages that previously made dozens of parallel requests now resolve in a single batched call.
## Template rendering page: template-first layout
The template rendering page now starts from the templates instead of starting from a trace. You pick a template and see exactly which spans it affects.
The new three-column layout shows all templates for a function on the left, the current trace in the center (with non-matching spans dimmed), and affected spans across recent traces on the right. If the current trace has no spans for the selected template, the viewer auto-navigates to one that does.
## API key context
Coding agents can now call `get_api_key_context` to find out which organization and environment their API key belongs to before sending traces. No more guesswork.
```
Organization: Acme, Inc. (5 members)
Your role: admin
Environment: production
API key: "Claude Plugin" (bf_c659...bfae)
User: Ankur Toshniwal
```
Available in all three plugins (Claude Code, Cursor, Codex) as of v0.4.15.
## API key descriptions
You can now add a description when creating API keys in the dashboard. Descriptions show up in the key list and are returned by `get_api_key_context`, so your coding agent can tell you which key it's using without you having to check.
# Claude Plugin
Source: https://docs.bitfab.ai/claude-plugin
Install the Bitfab plugin for Claude Code to get tracing, diagnostics, replay, and improvement tools directly in your editor
The Bitfab Claude Code plugin brings the full evaluation workflow into Claude Code. It provides MCP tools for trace inspection, datasets, labeling, experiments, and setup, slash commands for authentication, and automatic notifications -- so you never have to leave your editor.
## Installation
### CLI (recommended)
One command installs the plugin, opens your browser to log in, and launches `/bitfab:setup`:
```bash theme={null}
npx bitfab-cli init --editor claude
```
Pass an initial setup request with `--prompt` (or `-p`) to send it straight to the agent:
```bash theme={null}
npx bitfab-cli init --editor claude --prompt "instrument the chat workflow"
```
The CLI checks that Claude Code is signed in before it launches the Bitfab agent. If needed, run `claude auth login` first.
For the experimental terminal-native flow, run `npx bitfab-cli init --v2` instead. It keeps plan review, setup decisions, and edit approvals in the CLI and uses the Claude Agent SDK directly. Set `ANTHROPIC_API_KEY` or configure a supported Agent SDK cloud provider first. Use `npx bitfab-cli setup --v2 instrument` to run one setup mode, or add `--diagram` to print its declarative state graph without starting the agent.
### Plugin installation
To do the same three things yourself:
A [plugin marketplace](https://code.claude.com/docs/en/plugin-marketplaces) is a catalog of plugins you can browse and install into Claude Code. Add Bitfab's so the `bitfab` plugin is available to install.
```bash theme={null}
claude plugin marketplace add Project-White-Rabbit/bitfab-claude-plugin
```
```
/plugin marketplace add Project-White-Rabbit/bitfab-claude-plugin
```
Install `bitfab` from the marketplace you just added.
```bash theme={null}
claude plugin install bitfab@bitfab
```
```
/plugin install bitfab@bitfab
```
Start Claude Code and run setup to log in and instrument your codebase. If you installed from inside Claude Code and the summary asks for it, run `/reload-plugins` first.
```
/bitfab:setup
```
## What the Plugin Does
### Automatic Setup
The `/bitfab:setup` command runs a multi-phase workflow:
1. **Login** -- Opens your browser for OAuth authentication, saves credentials securely
2. **Explain** -- Walks through the two primitives you instrument with, `withSpan` and `replay`, including the five ways replay can change a method's execution. Everything after this asks you to make per-method decisions, so it comes first
3. **Approach** -- Asks whether the agent should walk you through instrumenting or hand you the docs so you can do it yourself
4. **Instrument + Replay** (in parallel, per workflow) -- Reads your codebase, finds all AI workflows (LLM calls, agents, AI-driven decisions), and presents them as a numbered list. You choose which to instrument, or name a file, function, or directory yourself and it reads only that instead of scanning. Either way it adds tracing with minimal diffs and creates a registry module for the replay command shipped with the SDK
You can run individual phases:
```
> /bitfab:setup explain # Explain withSpan + replay and list the modes (read-only, no login)
> /bitfab:setup login # Auth only
> /bitfab:setup instrument # Trace instrumentation only
> /bitfab:setup inspect # Diagnose (and offer to fix) your tracing setup
> /bitfab:setup replay # Replay registry creation only
> /bitfab:setup analyze-repo # Scan the repo and upload draft trace plans without prompts
```
The setup is interactive -- it presents 2-5 concrete options per decision point with a recommended choice, so you stay in control throughout.
### Assistant
The `/bitfab:assistant` command turns production traces into code improvements, whether the goal is correctness (improving pass rates) or efficiency (cutting token usage and cost). Your agent will do the mechanical work and collaborate with you on three steps:
1. **Build a dataset** from production traces -- search for failures, label them with expected outcomes
2. **Experiment** against that dataset -- make isolated code changes, replay, compare results
3. **Hill climb** -- repeat until the best change is found, then present results
Run it with an optional trace function key:
```
> /bitfab:assistant
> /bitfab:assistant order-processing
```
#### Building the Dataset
Your coding agent does the data wrangling -- it searches production traces for failures, reads full inputs and outputs, and identifies edge cases. It then presents edge cases for your judgment: is this a failure (and what should the output be), correct, or irrelevant? This labeled dataset becomes the benchmark for all experiments.
The plugin opens a rich UI for navigating and labeling the dataset, then brings you back to your coding agent so you stay in flow. You can label every trace yourself, or label a few and let the agent classify the rest based on the patterns you've established.
#### Running Experiments
The command reads your code, diagnoses failure patterns, and categorizes proposed changes:
* **Code fixes** -- deterministic bugs, bundled into one experiment as a foundation
* **Judgment-based fixes** -- prompt changes, search tuning, output formatting -- each gets its own experiment
* **Infrastructure proposals** -- larger changes noted for future work, not experimented on
Independent experiments run in parallel -- each in its own isolated subagent on a separate git worktree. Each subagent edits the code, runs the SDK's replay command with your registry module and labeled dataset, and compares new outputs to expected outcomes.
#### Results
After each round, you see which traces now match expected outcomes, which still diverge, and whether any regressions occurred. The assistant works through the planned experiments in turn without pausing to ask whether to keep going, then wraps up once the plan is complete. The final summary shows pass rate improvement and all files changed, uncommitted in your working tree for review.
### MCP Tools
The plugin registers MCP tools that Claude Code can call during conversations. These let you inspect traces, manage datasets and labels, run experiments, and improve your code without leaving the editor.
#### Core
##### `get_bitfab_api_key`
Retrieve your API key for SDK initialization and environment variable configuration.
##### `get_api_key_context`
Returns which Bitfab org the plugin reads/writes to (it can differ from the project's `BITFAB_API_KEY` and from the org open in Studio). Call before the first plugin write of a session, or when data you wrote isn't visible in Studio.
```
"Which org is this plugin connected to?"
```
##### `list_organizations`
List the Bitfab organizations available to the signed-in user, marking the current plugin org.
##### `get_database_connection_status`
Reports whether the org has a database connected for per-trace replay branching (`none`, `checking`, `connected`, or `failed`) and identifies it as direct Neon or a managed Postgres mirror. For direct Neon connections, the response also includes the pinned project name and ID. Used by `/bitfab:setup db-snapshot` to tell when the branchable copy is ready.
```
"Is my database connected for replay branching?"
```
#### Trace Inspection
##### `list_trace_functions`
List all traced functions in your organization.
```
"Show me all my traced functions"
```
##### `search_traces`
Search and filter traces with full-text search, date ranges, status filters, regex matching, environment, and label filters. Supports drill-down to narrow results progressively. Replay traces are excluded by default; set `includeReplays` to include them, or filter by `testRunId` to scope to a specific experiment's replay traces and enable them automatically. Filter by `datasetId` to scope to a specific dataset, or by `hasDbSnapshot` to scope to traces that captured (or lack) a database snapshot reference.
```
"Find all failed traces for order-processing from the last week"
```
##### `get_traces`
Read one or more traces by ID with the trace environment plus summary (truncated) or full span details (input, output, reasoning, context, errors, per-span duration, tokens, and model).
```
"Show me the full details of trace abc-123"
```
##### `get_trace_labels`
Read just the labels for one or more traces by ID: each trace's pass/fail verdict, its annotation, and whether the label is human-validated, followed by one indented line per scored assertion carrying that assertion's own verdict, annotation, confidence, and author. A verdict written per assertion reads back per assertion, not as a tally. No span content is loaded, so one call accepts up to 100 IDs, ideal for loading a whole dataset's verdicts at once.
```
"Load the labels for every trace in this dataset"
```
##### `get_grader_labels`
Read the individual verdicts each automated grader recorded, one row per grader per trace, with the grader's reason, its failure diagnostic, the confidence, and whether the verdict came from a human or a grader run. Pass trace IDs to see every grader's verdict on those traces, or a grader ID to see that grader's most recent verdicts across traces.
```
"Which grader failed trace abc-123, and what did it say?"
```
##### `get_span_field`
Fetch the complete, untruncated value of a single span field (input, output, reasoning, content, errors, or contexts) when `get_traces` truncated it. Pass the trace ID, the span's `[ID: ...]`, and the field name.
```
"Get the full input for span def-456 in trace abc-123"
```
#### Labeling and Datasets
##### `save_agent_labels`
Set, skip, or archive the agent's pass/fail verdict on one or more traces. Supports confidence levels (`VeryLow` through `VeryHigh`) and annotations shown to human reviewers. Pass an `assertionId` to score one of the trace's assertions, one verdict per assertion, and omit it for the trace's single whole-trace verdict. New verdicts start unapproved; once a human approves one in the UI, the label joins the validated dataset.
```
"Label these traces as passing with high confidence"
```
##### `save_human_labels`
Write a validated human pass/fail verdict on one or more traces, with an optional annotation. Pass an `assertionId` to score one of the trace's assertions, one verdict per assertion, and omit it for the trace's single whole-trace verdict. Unlike `save_agent_labels`, these labels are validated the instant they're written (no UI approval step). Used by `assistant fix` before adding the failing trace to a dataset once the fix is verified.
```
"Save this failing trace as a validated test"
```
##### `save_trace_assertions`
Record what a trace SHOULD do the next time it is replayed. Each assertion is one statement, with optional pass/fail criteria, an optional target scoping the check to the trace's output or to a single span by name, and an optional people-only `humanNote`. The note can be created or edited through this tool but is never assessment evidence. Distinct from a verdict, which judges a run that already happened: a trace holds at most one verdict per author and any number of assertions.
```
"This booking should have picked the 6am flight, not the 9am"
```
##### `get_trace_assertions`
Read the assertions on up to 100 traces by ID, with no span content. Reads include each people-only human note so an agent can preserve or edit it, but the note is never assessment evidence. Call it before judging a replay so the verdict is measured against what the user actually asked for. A replay carrying none of its own reads its original trace's assertions and is told which trace they came from.
##### `archive_trace_assertions`
Retire one or more assertions on a trace so later replays stop checking them. Archiving is non-destructive: the row is hidden from reads and kept for audit, and verdicts already recorded against it are untouched. The call is all-or-nothing, so one unknown, already-archived, or wrong-trace id fails it and archives nothing.
```
"That 9am constraint was wrong, drop that assertion"
```
##### `save_dataset`
Create a named dataset for a traced function. Datasets are buckets of traces that humans review and that experiments replay against.
##### `list_datasets`
List all datasets for a traced function with IDs, names, descriptions, trace counts, and assigned graders.
##### `add_traces_to_dataset` / `remove_traces_from_dataset`
Add or remove traces from a dataset (idempotent, up to 100 per call). Removing a trace from a dataset does not delete the trace itself.
##### `add_graders_to_dataset` / `remove_graders_from_dataset`
Assign or remove graders from a dataset (idempotent, up to 100 per call). Assignments are restricted to graders in the same organization and trace function as the dataset. Removing an assignment does not delete the grader.
##### `save_grader` / `list_graders`
Create or edit an automated grader (an LLM-as-judge pass/fail check) for a traced function, or list the graders defined for one. `save_grader` upserts by grader id or by name, supports renaming, clearing pass/fail criteria, archive/restore, and selecting the judge model (Gemini 2.5 Flash by default, or GPT-5.6 Sol/Terra, Claude Opus 4.8, or Gemini 3.1 Pro). `list_graders` returns each grader's name, status, evaluation focus, and criteria; supports case-insensitive name search; and paginates newest-first with 20 results by default, up to 50 per page. Pass the returned cursor to retrieve the next page. Archived graders are hidden by default.
#### Experiments
##### `save_experiment_group`
Create or reuse a group across selected experiments (test runs), optionally giving it a human-readable name and notes. To update an existing group, pass its id and a new name and/or notes; editing never changes which experiments belong to the group.
##### `list_experiment_groups`
List recent experiment groups with their names, notes, creation times, and member experiment ids.
##### `get_experiment_group`
Get one experiment group by id with its name, notes, creation and update times, plus each member experiment's id, name, status, and creation time.
##### `save_experiment`
Update an existing experiment's human-readable name, notes, or experiment group. Omitted fields keep their current values; an empty notes string clears the notes, an experiment group id moves the experiment into that existing group, and `null` removes it from its group.
##### `rerun_graders_on_dataset` / `rerun_graders_on_experiment`
Re-score a dataset's traces, or an experiment's completed replays, with the graders already attached to them, overwriting the previous verdicts. This is how traces added before a grader existed get scored, and how everything gets re-scored after a grader's criteria change: attaching a grader does not itself grade anything. Defaults to every attached grader; name a subset to narrow it, and ids that are not attached are rejected rather than skipped. Waits up to 90 seconds and reports how many traces were graded; a repeat call reports the running job instead of starting a second one.
##### `add_graders_to_experiment_group`
Assign one or more graders directly to every experiment currently in a group. Completed experiments queue any missing evaluations immediately; pending experiments use the graders when they complete. Because assignments live on experiments rather than the group, experiments added later do not inherit them automatically.
##### `remove_graders_from_experiment_group`
Detach one or more graders from every experiment currently in a group, undoing `add_graders_to_experiment_group`. Graders that are not assigned are ignored, the graders themselves are not deleted, and archived graders can still be detached.
##### `list_experiments`
List experiments (replay test runs) for a traced function with name, notes, status, pass/fail totals, and delta (fixed, regressed, still passing, still failing) for each.
```
"Show me past experiments for order-processing"
```
##### `get_experiment`
Get a single experiment (replay test run) by id with its name, notes, status, pass/fail totals, delta, experiment group, and grader results: the aggregate passing/failing checks and pass rate, plus a per-grader breakdown of each assigned grader's own passing/failing on the run. Use it when you already have an experiment id, instead of paging the whole function's history with `list_experiments`.
```
"Show me the grader results for experiment abc-123"
```
##### `list_experiment_traces`
Get individual trace results for an experiment, including each replay trace's verdict by comparing against the original trace's label, plus token usage (input, output, cached, total) for the replay and the paired original.
```
"Show me the trace-level results for experiment abc-123"
```
##### `add_graders_to_experiment` / `remove_graders_from_experiment`
Attach or detach graders directly on an experiment (test run) so they run against its replay traces (idempotent, up to 100 per call). Attachments are restricted to active graders in the same organization and trace function as the experiment. The effective grader set at completion is the union of these direct attachments and the dataset's current runnable graders. Detaching does not delete the grader. Its effect depends on run state: for an in-progress experiment a detached grader that is also a dataset grader is re-added from the dataset at completion (so it still runs), while for an already-completed experiment the grader set is a finalized snapshot with no dataset re-union, so detaching permanently drops that grader from the run's results. Attaching to an already-completed experiment persists the assignment but does not immediately grade its existing replay traces; those traces are graded only the next time the experiment runs completion again (e.g. an SDK replay retry) or a fresh replay runs.
```
"Attach the no-fabricated-order-ids grader to experiment abc-123"
```
##### `get_replay_status`
Read the current replay test run status, including the mapping from local replay trace IDs to server trace IDs while a replay is still running.
#### Templates
##### `get_template_reference`
Read the agent-facing reference for Bitfab span templates: the Nunjucks engine, render-context schema, registered filters, and common patterns. Call once per session before editing templates.
##### `get_template`
Read the rendering template for a span type (llm, agent, function, guardrail, handoff, custom), scoped to a trace function key or org-global.
##### `save_template`
Upsert a rendering template for a span type. Controls how span input/output renders in the Bitfab UI.
#### Instrumentation
##### `save_trace_plan`
Create a tracing instrumentation plan (call tree with recommended captured nodes) and get a URL the user opens to review or adjust selections. To change an existing plan in place, pass its `planId` and either adjust which nodes are captured (`capture` pulls in every node above the one you name, `uncapture` drops every node below it) and mocked (`mockOnReplayByNodeId`), or replace the whole node tree (`tree` + `capturedNodeIds`) when the code changed. A structural replacement can change the root when the trace boundary moved or was replaced. Replacing the tree reopens a confirmed plan for re-confirmation; a targeted edit leaves its status alone.
##### `confirm_trace_plan`
Confirm a trace plan without the browser, for the continue path where the user accepted the ASCII plan in chat, or left the Studio plan page without saving. Persists it as the confirmed plan for its trace function key so `setup view` and `setup modify` can find it later, and returns the final captured set and per-node replay decisions.
##### `get_trace_plan`
Read a trace plan by ID (after user confirmation) or by trace function key. During a Modify cycle, the prior plan preserves earlier decisions while the agent rereads current instrumentation and reconciles added, removed, moved, renamed, or re-rooted calls before proposing an update.
##### `list_trace_plans`
List the organization's trace plans, newest first (filterable by source, status, and trace function key). Used before instrumenting to reuse the unconfirmed drafts an earlier `analyze-repo` run uploaded instead of re-scanning the codebase, and before saving a plan to find the one that already exists for a key so it is updated rather than duplicated.
##### `cancel_trace_plan`
Retire an unconfirmed trace plan nobody will act on, so it stops coming back as a reusable draft: its workflow is already instrumented, or the workflow no longer exists. Only unconfirmed plans can be cancelled, and cancelling is final.
##### `get_sim_plan`
Read a trace function's sim plan (alpha): its span nodes overlaid from recent traces, each with its span type, call count, average payload, estimated monthly cost, share of traces, and whether its content (inputs and outputs) is captured. Use it to learn the exact node names before `save_sim_plan`, or to answer what a workflow records today. A node is its span name inside the trace function key.
##### `save_sim_plan`
Turn content capture off or on for span nodes of one trace function (alpha). The span itself, its name, type, timing, and errors are always recorded; content off strips only inputs and outputs. A node recorded by a framework integration (OpenAI Agents, LangGraph, Claude Agent SDK, Vercel AI SDK), a node that is the root of its traces, a node imported from another platform, or a node named by Bitfab rather than the SDK keeps its content, and turning it off is refused with the reason. Turning a node off also turns content off for every node beneath it in the sim plan, the way untracing a span in a trace plan drops the spans under it; a node beneath it that keeps its content stays on, and a node named in the same call keeps the value given to it. Turning a node back on leaves the nodes beneath it as they are. Takes the nodes to change by span name and returns the updated sim plan, so a decision made here shows up on the Sim plan page as set by agent.
### Slash Commands
| Command | Description |
| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `/bitfab:setup` | Full setup workflow -- authenticate, instrument, create replay registries |
| `/bitfab:setup login` | Auth only |
| `/bitfab:assistant` | Build a dataset from traces, experiment with code changes, improve pass rates or cut token costs |
| `/bitfab:assistant ` | Iterate on a specific trace function |
| `/bitfab:assistant fix ` | Fix one failing trace: diagnose it, make the focused code change, and replay just that trace; once it passes, add it to a dataset, then choose to inspect the before/after in Studio or re-run the full dataset |
| `/bitfab:logout` | Remove saved credentials |
| `/bitfab:status` | Check auth status, plugin version, and available updates |
| `/bitfab:update` | Update the plugin to the latest version |
### Session Notifications
The plugin runs a hook on every session start and resume that checks:
* **Authentication**: If you're not logged in, it reminds you to run `/bitfab:setup`
* **Updates**: If a new plugin version is available, it tells you how to update (or auto-updates if you've enabled it)
## Example Workflows
### Instrument a new project
```
> /bitfab:setup
```
The agent detects your project language, finds AI workflows, presents options, and instruments your chosen workflows -- all interactively.
### Diagnose and fix a failing function
Ask Claude Code naturally:
```
"My order-processing traces are failing. What's going wrong and can you fix it?"
```
The plugin calls `search_traces` and `get_traces` to inspect failing traces, then suggests code fixes directly.
For a specific failing trace, run `/bitfab:assistant fix `. The agent diagnoses the failure, confirms why the original trace is wrong before editing when the trace or conversation does not already make that clear, makes the focused code fix, and replays only that trace first. Once the fix passes, it adds that trace to a dataset with a validated failing label, then branches: inspect the before/after in Studio, re-run the full dataset (in Studio or terminal-only), keep iterating, or stop. If that full-dataset re-run reveals real regressions (previously-passing traces the fix broke), it reports them and keeps the target trace saved as a red test to revisit. If the replay still fails, it offers to keep iterating or save the trace as a failing test instead.
### Iterate on a trace function
```
> /bitfab:assistant memory-search
```
The agent finds failing traces, walks you through labeling them with expected outcomes, diagnoses the failure patterns in your code, then runs experiments -- editing prompts or code, replaying against your labeled dataset, and reporting what improved. You stay in control at every decision point.
### Replay after a code change
After updating a function, pass your registry module to the replay command installed by the SDK:
```bash theme={null}
pnpm exec bitfab-replay --registry scripts/replayRegistry.ts extraction --limit 20
```
Or ask Claude Code to do it for you: it can run the script and interpret the results.
While the replay runs, Claude runs it in the background and reports progress to you as it goes: one line per trace as it finishes (a pass/fail mark, the running count, and how long that trace took), with any error reason inline, plus a periodic "still running" heartbeat when a slow trace takes a while so the run never looks stuck, then a summary with the total and average time. Full per-item outputs are written under that replay run's `.bitfab/replays//items/` folder and referenced from `.bitfab/replays//events.jsonl`. If the replay command succeeds but its local result cannot be captured, the plugin reports the outcome as unverified instead of failed, then checks the server test run to recover the final result.
## Configuration
### Credentials
Credentials are stored in `.bitfab/credentials.local.json` when that project-local file exists, otherwise in `~/.config/bitfab/credentials.json` (created by `/bitfab:setup login` with owner-readable permissions).
### Environment Variables
| Variable | Description |
| ---------------- | --------------------------- |
| `BITFAB_API_KEY` | Override the stored API key |
## Troubleshooting
### Not authenticated
If you see "Not authenticated" on session start:
1. Run `/bitfab:setup login` to authenticate via browser
2. Check that `~/.config/bitfab/credentials.json` exists and contains your API key
3. If using an environment variable, verify `BITFAB_API_KEY` is set
### MCP tools not available
If Claude Code can't access the Bitfab tools:
1. Run `/bitfab:status` to check connection status
2. Run `/mcp` and check whether **Bitfab** is disabled. Turning an MCP server off is
remembered per project, so it survives reinstalling the plugin and restarting Claude
Code. Enable it there and the tools load into the running session.
3. Verify the plugin is installed: check `/plugin list`
`claude mcp list` is not a reliable check here: it health-checks the server in a separate
process and reports `Connected` even when the current session has it disabled.
### Stale session
The plugin automatically detects and recovers from stale MCP sessions. If tools stop working mid-conversation, they'll reconnect on the next call.
### Plugin updates
Run `/bitfab:status` to check for updates, then `/bitfab:update` to install the latest version. Restart Claude Code after updating.
# CLI
Source: https://docs.bitfab.ai/cli
Install and configure the Bitfab plugin from your terminal
The Bitfab CLI installs the plugin, authenticates, and launches setup for Claude Code, Codex, Cursor, or Amp in one command.
## Quick Start
```bash theme={null}
npx bitfab-cli init
```
This runs the full onboarding flow:
1. **Detects** which editors you have installed
2. **Installs** the Bitfab plugin in your chosen editor
3. **Authenticates** via browser-based OAuth
4. **Launches** the setup command (`/bitfab:setup`) in your editor
If multiple editors are detected, you'll be prompted to choose one. To skip the prompt, pass `--editor`:
```bash theme={null}
npx bitfab-cli init --editor claude
npx bitfab-cli init --editor codex
npx bitfab-cli init --editor cursor
npx bitfab-cli init --editor amp
```
With `--editor amp`, step 2 clones the [Amp plugin](/amp-plugin) into `~/.config/amp/plugins/bitfab` and step 4 starts Amp with the setup skill to invoke. See [Amp](#amp).
## Commands
Every command supports `-h` and `--help`. You can also use `help `:
```bash theme={null}
npx bitfab-cli --help
npx bitfab-cli analyze-repo --help
npx bitfab-cli help analyze-repo
```
### `init`
Full onboarding: install the plugin, authenticate, and launch setup.
```bash theme={null}
npx bitfab-cli init [--editor ]
```
### `plugin-install`
Install the Bitfab plugin without authenticating or launching setup. Useful if you want to handle login separately.
```bash theme={null}
npx bitfab-cli plugin-install [--editor ]
```
### `login`
Authenticate with Bitfab. Opens your browser for OAuth, then saves credentials to `.bitfab/credentials.local.json` when that project-local file exists, otherwise to `~/.config/bitfab/credentials.json`.
```bash theme={null}
npx bitfab-cli login
```
Pass `--force` to re-authenticate when you're already signed in. `--force` also opens a fresh Studio window even if one is already open, so it's how you recover when login reports that a Studio window is already open but isn't responding.
```bash theme={null}
npx bitfab-cli login --force
```
### `logout`
Remove stored credentials. If project-local credentials exist, removes `.bitfab/credentials.local.json`; otherwise removes `~/.config/bitfab/credentials.json`.
```bash theme={null}
npx bitfab-cli logout
```
### `session-logs`
Read or update the saved session-log collection preference. This is the same shared setting used by setup and `analyze-repo` when no per-run `--upload-logs` or `--no-upload-logs` flag is passed.
```bash theme={null}
npx bitfab-cli session-logs status
npx bitfab-cli session-logs enable
npx bitfab-cli session-logs disable
```
### `setup`
Launch the editor's setup command (`/bitfab:setup` for Claude Code, `$bitfab:setup` for Codex) without reinstalling the plugin. For Cursor, prints the setup command and next steps.
```bash theme={null}
npx bitfab-cli setup [--editor ]
```
### `analyze-repo`
Headlessly scan the repository for AI workflows and upload a **draft trace plan** for each of the top candidates, with no prompts and no code changes. Runs the editor's analyze-repo command non-interactively through Claude Code, Codex, or Cursor Agent, or Amp headless (`amp -x`) with `--editor amp`. After upload, it prints a compact terminal summary of the suggested workflows, instrumentation effort, capture methods, and replay mocks. If you're not signed in and the CLI is attached to a terminal, it opens the browser login flow first; in non-interactive environments, sign in ahead of time with `npx bitfab-cli login` or set `BITFAB_API_KEY`. Use `--limit` to cap how many plans it uploads (default 5), and `--prompt` (or a trailing quoted argument) to steer what the scan focuses on.
```bash theme={null}
npx bitfab-cli analyze-repo [--editor ] [--limit ] [--prompt ] [--upload-logs | --no-upload-logs]
npx bitfab-cli analyze-repo --prompt "focus on the billing and checkout flows"
```
### `assistant`
Launch the Bitfab assistant (`/bitfab:assistant` for Claude Code, `$bitfab:assistant` for Codex) to iterate on traced functions. Any extra arguments are passed through to the editor command. To forward an argument that looks like a bitfab flag, place it after `--`: everything after `--` is passed through verbatim.
With `--editor amp`, starts Amp and prints the `bitfab:assistant` invocation to type, since Amp takes no initial message on launch.
```bash theme={null}
npx bitfab-cli assistant [--editor ] [args...]
npx bitfab-cli assistant -- --some-editor-flag value
```
### `update`
Update the Bitfab plugin and SDKs. The CLI refreshes the plugin directly (marketplace update + plugin update), then launches the editor's update command for SDK updates. Pass `plugin`, `sdk`, or `all` to control scope.
With `--editor amp`, `plugin` pulls the latest plugin into `~/.config/amp/plugins/bitfab` (run `plugins: reload` in Amp afterward), and `sdk` starts Amp with `bitfab:update sdk` to invoke.
```bash theme={null}
npx bitfab-cli update [--editor ] [plugin | sdk | all]
```
## Flags
### `--editor`, `-e`
Target a specific editor (`claude`, `codex`, `cursor`, or `amp`). If omitted, the CLI detects installed editors and prompts you to choose.
### `--skip-permissions`
Run the launched agent without permission prompts (translates to `--dangerously-skip-permissions` for Claude Code, `--dangerously-bypass-approvals-and-sandbox` for Codex). If you omit this flag, the CLI will ask interactively whether to skip permissions. Pass `--no-skip-permissions` to keep permission prompts without being asked.
Applies to: `init`, `setup`, `assistant`, `update`. Cursor and Amp have no launch flag for this, so the CLI never asks there; passing the flag explicitly prints where the equivalent lives instead (`amp.dangerouslyAllowAll` in Amp's settings).
### `--limit`
Cap how many draft trace plans `analyze-repo` uploads (default 5). Passed through to the `analyze-repo` skill as its plan cap, on every editor including `--editor amp`.
Applies to: `analyze-repo`. Note that `setup --v2 analyze-repo` has no `--limit` flag of its own and ignores one silently, so cap the run through `analyze-repo --editor amp --limit `.
### `--prompt`, `-p`
Free-text guidance steering what `analyze-repo` focuses on (for example, `--prompt "focus on the billing and checkout flows"`). You can also pass it as a trailing quoted argument without the flag. Passed through to the `analyze-repo` skill, which biases its scan and ranking toward the areas you name while still filling any remaining plan slots from the rest of the repo.
Applies to: `analyze-repo`.
### `--upload-logs`
Upload the `analyze-repo` run's session logs to Bitfab to help diagnose issues (translates to the plugin's session-log capture for that run). Pass `--no-upload-logs` to keep them local. If you omit both, the CLI uses your saved session-log preference, asking once if it isn't set yet.
Applies to: `analyze-repo`. Session log capture on Amp ships in a follow-up release, so with `--editor amp` the flag is accepted and currently has no effect.
## What It Does Per Editor
### Claude Code
* Adds the `bitfab` marketplace (from `Project-White-Rabbit/bitfab-claude-plugin`)
* Enables auto-updates for the marketplace
* Installs the plugin at user scope
* Launches `claude /bitfab:setup`
### Codex
* Adds the `bitfab` marketplace (from `Project-White-Rabbit/bitfab-codex-plugin`)
* Enables the plugin in `~/.codex/config.toml`
* Launches `codex $bitfab:setup`
### Cursor
* Copies the `/add-plugin` command to your clipboard
* Opens Cursor so you can paste and run the command
* Prints next steps for completing setup in Cursor
### Amp
* Clones the [Bitfab plugin](/amp-plugin) into `~/.config/amp/plugins/bitfab` and confirms Amp loaded it
* Starts Amp and prints the skill invocation to type (`bitfab:setup`, `bitfab:assistant`, `bitfab:update sdk`)
* Runs `analyze-repo` headless through `amp -x`
```bash theme={null}
npx bitfab-cli init --editor amp
```
Every command works with `--editor amp`. What differs from the other hosts is listed in the [Amp plugin docs](/amp-plugin).
## Requirements
* Node.js 18+
* At least one supported editor installed: `claude`, `codex`, `cursor`, or `amp` must be on your PATH
* `--editor amp` also needs `git`, which installs the plugin
# Codex Plugin
Source: https://docs.bitfab.ai/codex-plugin
Bitfab plugin for OpenAI Codex: trace, diagnose, and iterate on AI workflows directly in your terminal
The Bitfab Codex plugin brings the full evaluation workflow into Codex. It provides MCP tools for trace inspection and diagnostics, skills for authentication and setup, and automatic notifications so you never have to leave your terminal.
## Installation
Run the CLI from your project directory:
```bash theme={null}
npx bitfab-cli init --editor codex
```
This installs the Bitfab plugin, opens your browser to log in, and launches `$bitfab:setup`.
Pass an initial setup request with `--prompt` (or `-p`) to send it straight to the agent:
```bash theme={null}
npx bitfab-cli init --editor codex --prompt "instrument the chat workflow"
```
The CLI checks that Codex is signed in before it launches the Bitfab agent. If needed, run `codex login` first.
For the experimental terminal-native flow, run `npx bitfab-cli init --v2` instead. It does not launch Codex: plan review, setup decisions, and edit approvals stay in the CLI while the Claude Agent SDK performs repository analysis and approved edits. Set `ANTHROPIC_API_KEY` or configure a supported Agent SDK cloud provider first. Use `npx bitfab-cli setup --v2 instrument` to run one setup mode, or add `--diagram` to print its declarative state graph without starting the agent.
Set your API key as an environment variable:
```bash theme={null}
export BITFAB_API_KEY="YOUR_API_KEY"
```
Then add this to `~/.codex/config.toml` (global) or `.codex/config.toml` (project-level):
```toml theme={null}
[mcp_servers.bitfab]
url = "https://bitfab.ai/mcp"
bearer_token_env_var = "BITFAB_API_KEY"
```
Visit the [setup page](https://bitfab.ai/setup) to get your API key and see the full configuration steps.
## What the Plugin Does
### Automatic Setup
The `$bitfab:setup` skill runs a multi-phase workflow:
1. **Login** -- Opens your browser for OAuth authentication, saves credentials securely
2. **Explain** -- Walks through the two primitives you instrument with, `withSpan` and `replay`, including the five ways replay can change a method's execution. Everything after this asks you to make per-method decisions, so it comes first
3. **Approach** -- Asks whether the agent should walk you through instrumenting or hand you the docs so you can do it yourself
4. **Instrument + Replay** (in parallel, per workflow) -- Reads your codebase, finds all AI workflows (LLM calls, agents, AI-driven decisions), and presents them as a numbered list. You choose which to instrument, or name a file, function, or directory yourself and it reads only that instead of scanning. Either way it adds tracing with minimal diffs and creates a registry module for the replay command shipped with the SDK
You can run individual phases:
```
$bitfab:setup explain # Explain withSpan + replay and list the modes (read-only, no login)
$bitfab:setup login # Auth only
$bitfab:setup instrument # Trace instrumentation only
$bitfab:setup inspect # Diagnose (and offer to fix) your tracing setup
$bitfab:setup replay # Replay registry creation only
$bitfab:setup analyze-repo # Scan the repo and upload draft trace plans without prompts
```
The setup is interactive: it presents 2-5 concrete options per decision point with a recommended choice, so you stay in control throughout.
### Assistant
The `$bitfab:assistant` skill turns production traces into code improvements, whether the goal is correctness (improving pass rates) or efficiency (cutting token usage and cost). Your agent will do the mechanical work and collaborate with you on three steps:
1. **Build a dataset** from production traces: search for failures, label them with expected outcomes
2. **Experiment** against that dataset: make isolated code changes, replay, compare results
3. **Hill climb**: repeat until the best change is found, then present results
Run it with an optional trace function key:
```
$bitfab:assistant
$bitfab:assistant order-processing
```
#### Building the Dataset
Your coding agent does the data wrangling: it searches production traces for failures, reads full inputs and outputs, and identifies edge cases. It then presents edge cases for your judgment: is this a failure (and what should the output be), correct, or irrelevant? This labeled dataset becomes the benchmark for all experiments.
The plugin opens a rich UI for navigating and labeling the dataset, then brings you back to your coding agent so you stay in flow. You can label every trace yourself, or label a few and let the agent classify the rest based on the patterns you've established.
#### Running Experiments
The skill reads your code, diagnoses failure patterns, and categorizes proposed changes:
* **Code fixes**: deterministic bugs, bundled into one experiment as a foundation
* **Judgment-based fixes**: prompt changes, search tuning, output formatting, each gets its own experiment
* **Infrastructure proposals**: larger changes noted for future work, not experimented on
Independent experiments run in parallel, each in its own isolated subagent on a separate git worktree. Each subagent edits the code, runs the SDK's replay command with your registry module and labeled dataset, and compares new outputs to expected outcomes.
#### Results
After each round, you see which traces now match expected outcomes, which still diverge, and whether any regressions occurred. The assistant works through the planned experiments in turn without pausing to ask whether to keep going, then wraps up once the plan is complete. The final summary shows pass rate improvement and all files changed, uncommitted in your working tree for review.
### MCP Tools
The plugin registers MCP tools that Codex can call during conversations. These let you inspect traces, diagnose failures, and improve your code without leaving the terminal.
#### Core
| Tool | Description |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `get_bitfab_api_key` | Retrieve your API key for SDK initialization and environment variable configuration |
| `get_api_key_context` | Returns which Bitfab org the plugin reads/writes to. Call before the first plugin write, or when data you wrote isn't visible in Studio |
| `list_organizations` | List the Bitfab organizations available to the signed-in user, marking the current plugin org |
| `get_database_connection_status` | Report whether the org has connected a database for per-trace replay branching (`none`, `checking`, `connected`, or `failed`), identify it as direct Neon or a managed Postgres mirror, and include the pinned project name and ID for direct Neon connections |
#### Trace Inspection
| Tool | Description |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `list_trace_functions` | List all traced functions in your organization |
| `search_traces` | Search and filter traces with keyword search, date ranges, status filters, regex, environment, label filters, dataset, test run, and db-snapshot scoping; replay traces are excluded by default unless `includeReplays` is set or a `testRunId` is provided |
| `get_traces` | Read one or more traces by ID with the trace environment plus summary (truncated) or full span details (input, output, reasoning, context, errors, per-span duration and tokens) |
| `get_trace_labels` | Read just the labels for up to 100 traces by ID in one call, no span content: each trace's verdict, annotation, and approved flag, plus one line per scored assertion carrying that assertion's own verdict, annotation, confidence, and author |
| `get_grader_labels` | Read the individual verdicts each automated grader recorded (reason, failure diagnostic, confidence, human or grader run), by trace IDs, by grader ID, or both |
| `get_span_field` | Fetch the complete, untruncated value of a single span field (input, output, reasoning, content, errors, or contexts) when `get_traces` truncated it |
#### Labeling and Datasets
| Tool | Description |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `save_agent_labels` | Set, skip, or archive agent pass/fail verdicts on traces. Supports confidence levels and annotations for human review. Pass an `assertionId` to score one assertion, omit it for the whole-trace verdict |
| `save_human_labels` | Write validated human pass/fail verdicts (with annotations) on traces. Validated immediately with no UI approval step; used by `assistant fix` before adding a trace to a dataset |
| `save_trace_assertions` | Record what a trace SHOULD do when replayed: one assertion per statement, with optional pass/fail criteria, target, and people-only human note. Agents can edit the note but must never use it as assessment evidence |
| `get_trace_assertions` | Read the assertions and people-only human notes on up to 100 traces by ID, no span content. Notes are never assessment evidence; a replay with no assertions of its own reads its original's |
| `archive_trace_assertions` | Retire assertions on a trace so later replays stop checking them. Non-destructive and all-or-nothing: one unknown, already-archived, or wrong-trace id fails the call and archives nothing |
| `save_dataset` | Create a labeled dataset for a traced function (named buckets of traces for review and replay) |
| `list_datasets` | List all datasets for a traced function with trace counts and assigned graders |
| `add_traces_to_dataset` | Add traces to a dataset (idempotent, 1-100 per call) |
| `remove_traces_from_dataset` | Remove traces from a dataset without deleting the traces themselves |
| `add_graders_to_dataset` | Assign graders to a dataset (idempotent, organization- and function-scoped, 1-100 per call) |
| `remove_graders_from_dataset` | Remove grader assignments from a dataset without deleting the graders |
| `save_grader` | Create or edit an automated grader (LLM-as-judge pass/fail check) for a traced function: upsert by id or name, rename, clear pass/fail criteria, archive/restore, select the judge model |
| `list_graders` | List automated graders for a traced function with optional name search and cursor pagination (20 by default, 50 maximum; archived hidden by default) |
#### Experiments
| Tool | Description |
| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `save_experiment_group` | Create a group from selected experiments with optional name and notes, or update an existing group's metadata without changing membership |
| `list_experiment_groups` | List recent experiment groups with their names, notes, and member experiment ids |
| `get_experiment_group` | Get one experiment group with its metadata and detailed member experiment summaries |
| `save_experiment` | Update an existing experiment's name or notes, move it into an existing experiment group, or remove it from its group |
| `add_graders_to_experiment_group` | Assign one or more graders directly to every current experiment in a group and queue missing evaluations for completed runs; later experiments do not inherit them automatically |
| `remove_graders_from_experiment_group` | Detach one or more graders from every current experiment in a group; unassigned ids are ignored and the graders themselves are not deleted |
| `list_experiments` | List experiments (replay test runs) for a traced function with name, notes, status, totals, and delta (fixed/regressed/still passing/still failing) |
| `get_experiment` | Get a single experiment by id with name, notes, status, totals, delta, experiment group, and grader results, including a per-grader passing/failing breakdown |
| `list_experiment_traces` | Get individual trace results for an experiment with each replay trace's verdict compared to the original, plus token usage (input, output, cached, total) for the replay and paired original |
| `add_graders_to_experiment` / `remove_graders_from_experiment` | Attach or detach active graders on an experiment so they run against its replay traces (idempotent, organization- and function-scoped, 1-100 per call); the effective set at completion is the union with the dataset's runnable graders; detaching a dataset-overlapping grader from an in-progress run is re-added at completion (still runs) but from a completed run permanently drops it from the finalized snapshot, and attaching to a completed experiment grades existing traces only on the next completion/replay |
| `rerun_graders_on_dataset` / `rerun_graders_on_experiment` | Re-score a dataset's traces or an experiment's completed replays with their attached graders, overwriting the previous verdicts (defaults to every attached grader; a subset may be named, and ids that are not attached are rejected); waits up to 90s and reports traces graded, and a repeat call reports the running job instead of starting a second one |
| `get_replay_status` | Read a replay test run's current status and local replay trace ID to server trace ID mapping while replay is still running |
#### Templates
| Tool | Description |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `get_template_reference` | Read the Nunjucks template engine reference, render-context schema, and available filters. Call once per session before editing templates |
| `get_template` | Read the rendering template for a span type, scoped to a trace function key or org-global |
| `save_template` | Upsert a rendering template for a span type. Controls how span data renders in the Bitfab UI |
#### Instrumentation
| Tool | Description |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `save_trace_plan` | Create a tracing plan, or structurally update an existing plan (its root included) in place by passing its plan ID |
| `confirm_trace_plan` | Confirm a plan without the browser (the continue path, or a plan page left without saving) so `setup view`/`setup modify` can find it by key later |
| `get_trace_plan` | Read a trace plan by ID (after confirmation) or by trace function key; Modify preserves prior decisions while reconciling the plan and its root with current instrumentation |
| `list_trace_plans` | List the org's trace plans, newest first, filterable by source, status, and trace function key; used to reuse unconfirmed `analyze-repo` drafts instead of re-scanning, and to find the plan a key already has so it is updated rather than duplicated |
| `cancel_trace_plan` | Retire an unconfirmed plan nobody will act on (already instrumented, or the workflow is gone) so it stops coming back as a reusable draft |
| `get_sim_plan` | Read a trace function's sim plan (alpha): its span nodes from recent traces with type, call count, average payload, estimated monthly cost, share of traces, and whether content is captured. A node is its span name inside the trace function key |
| `save_sim_plan` | Turn content capture off or on per span node (alpha). The span itself, its name, type, timing, and errors are always recorded; content off strips only inputs and outputs. A node recorded by a framework integration, a node that is the root of its traces, a node imported from another platform, or a node named by Bitfab rather than the SDK keeps its content, and turning it off is refused with the reason. Turning a node off also turns content off for every node beneath it in the sim plan, the way untracing a span in a trace plan drops the spans under it; a node beneath it that keeps its content stays on, and a node named in the same call keeps the value given to it. Turning a node back on leaves the nodes beneath it as they are. Returns the updated sim plan |
### Skills
| Skill | Description |
| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `$bitfab:setup` | Full setup workflow: authenticate, instrument, create replay registries |
| `$bitfab:setup login` | Auth only |
| `$bitfab:setup analyze-repo` | Non-interactively scan the repo, pick traceable workflows, and upload draft trace plans |
| `$bitfab:assistant` | Build a dataset from traces, experiment with code changes, improve pass rates or cut token costs |
| `$bitfab:assistant ` | Iterate on a specific trace function |
| `$bitfab:assistant fix ` | Fix one failing trace: diagnose it, make the focused code change, and replay just that trace; once it passes, add it to a dataset, then choose to inspect the before/after in Studio or re-run the full dataset |
| `$bitfab:update` | Update the plugin to the latest version |
## Example Workflows
### Instrument a new project
```
$bitfab:setup
```
The agent detects your project language, finds AI workflows, presents options, and instruments your chosen workflows, all interactively.
### Diagnose and fix a failing function
Ask Codex naturally:
```
"My order-processing traces are failing. What's going wrong and can you fix it?"
```
The plugin calls `search_traces` and `get_traces` to inspect failing traces and suggests code fixes directly.
For a specific failing trace, run `$bitfab:assistant fix `. The agent diagnoses the failure, confirms why the original trace is wrong before editing when the trace or conversation does not already make that clear, makes the focused code fix, and replays only that trace first. Once the fix passes, it adds that trace to a dataset with a validated failing label, then branches: inspect the before/after in Studio, re-run the full dataset (in Studio or terminal-only), keep iterating, or stop. If that full-dataset re-run reveals real regressions (previously-passing traces the fix broke), it reports them and keeps the target trace saved as a red test to revisit. If the replay still fails, it offers to keep iterating or save the trace as a failing test instead.
### Iterate on a trace function
```
$bitfab:assistant memory-search
```
The agent finds failing traces, walks you through labeling them with expected outcomes, diagnoses the failure patterns in your code, then runs experiments: editing prompts or code, replaying against your labeled dataset, and reporting what improved. You stay in control at every decision point.
### Replay after a code change
After updating a function, pass your registry module to the replay command installed by the SDK:
```bash theme={null}
pnpm exec bitfab-replay --registry scripts/replayRegistry.ts extraction --limit 20
```
Or ask Codex to do it for you.
While the replay runs, Codex runs it in the background and reports progress to you as it goes: one line per trace as it finishes (a pass/fail mark, the running count, and how long that trace took), with any error reason inline, plus a periodic "still running" heartbeat when a slow trace takes a while so the run never looks stuck, then a summary with the total and average time. Full per-item outputs are written under that replay run's `.bitfab/replays//items/` folder and referenced from `.bitfab/replays//events.jsonl`. If the replay command succeeds but its local result cannot be captured, the plugin reports the outcome as unverified instead of failed, then checks the server test run to recover the final result.
## Configuration
### Credentials
Credentials are stored in `.bitfab/credentials.local.json` when that project-local file exists, otherwise in `~/.config/bitfab/credentials.json` (created by `$bitfab:setup login` with owner-readable permissions).
### Environment Variables
| Variable | Description |
| ---------------- | --------------------------- |
| `BITFAB_API_KEY` | Override the stored API key |
## Troubleshooting
### Not authenticated
If you see "Not authenticated" on session start:
1. Run `$bitfab:setup login` to authenticate via browser
2. Check that `~/.config/bitfab/credentials.json` exists and contains your API key
3. If using an environment variable, verify `BITFAB_API_KEY` is set
### MCP tools not available
If Codex can't access the Bitfab tools:
1. Verify the MCP server configuration in `~/.codex/config.toml` or `.codex/config.toml`
2. Check that `BITFAB_API_KEY` is set in your environment
3. Try restarting Codex
### Plugin updates
Run `$bitfab:update` to install the latest version.
# Cursor Plugin
Source: https://docs.bitfab.ai/cursor-plugin
Bitfab plugin for Cursor: trace, diagnose, and iterate on AI workflows directly in your editor
The Bitfab Cursor plugin brings the full evaluation workflow into Cursor. It provides MCP tools for trace inspection and diagnostics, slash commands for setup and iteration, and automatic notifications so you never have to leave your editor.
## Installation
Run the CLI from your project directory:
```bash theme={null}
npx bitfab-cli init --editor cursor
```
This installs the Bitfab plugin, opens your browser to log in, and launches `/bitfab-setup`.
Pass an initial setup request with `--prompt` (or `-p`) to carry it into the Cursor handoff:
```bash theme={null}
npx bitfab-cli init --editor cursor --prompt "instrument the chat workflow"
```
The CLI checks that Cursor Agent is signed in before it launches the Bitfab agent. If needed, run `cursor agent login` first.
For the experimental terminal-native flow, run `npx bitfab-cli init --v2` instead. It does not launch Cursor: plan review, setup decisions, and edit approvals stay in the CLI while the Claude Agent SDK performs repository analysis and approved edits. Set `ANTHROPIC_API_KEY` or configure a supported Agent SDK cloud provider first. Use `npx bitfab-cli setup --v2 instrument` to run one setup mode, or add `--diagram` to print its declarative state graph without starting the agent.
Add this to your `.cursor/mcp.json` (project-level) or `~/.cursor/mcp.json` (global):
```json theme={null}
{
"mcpServers": {
"bitfab": {
"type": "streamable-http",
"url": "https://bitfab.ai/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
Visit the [setup page](https://bitfab.ai/setup) to get a pre-configured snippet with your API key already embedded, or use the one-click deep link installation.
## What the Plugin Does
### Automatic Setup
The `/bitfab-setup` command runs a multi-phase workflow:
1. **Login** -- Opens your browser for OAuth authentication, saves credentials securely
2. **Explain** -- Walks through the two primitives you instrument with, `withSpan` and `replay`, including the five ways replay can change a method's execution. Everything after this asks you to make per-method decisions, so it comes first
3. **Approach** -- Asks whether the agent should walk you through instrumenting or hand you the docs so you can do it yourself
4. **Instrument + Replay** (in parallel, per workflow) -- Reads your codebase, finds all AI workflows (LLM calls, agents, AI-driven decisions), and presents them as a numbered list. You choose which to instrument, or name a file, function, or directory yourself and it reads only that instead of scanning. Either way it adds tracing with minimal diffs and creates a registry module for the replay command shipped with the SDK
You can run individual phases:
```
/bitfab-setup explain # Explain withSpan + replay and list the modes (read-only, no login)
/bitfab-setup login # Auth only
/bitfab-setup instrument # Trace instrumentation only
/bitfab-setup inspect # Diagnose (and offer to fix) your tracing setup
/bitfab-setup replay # Replay registry creation only
/bitfab-setup analyze-repo # Scan the repo and upload draft trace plans without prompts
```
The setup is interactive: it presents 2-5 concrete options per decision point with a recommended choice, so you stay in control throughout.
### Assistant
The `/bitfab-assistant` command turns production traces into code improvements, whether the goal is correctness (improving pass rates) or efficiency (cutting token usage and cost). Your agent will do the mechanical work and collaborate with you on three steps:
1. **Build a dataset** from production traces: search for failures, label them with expected outcomes
2. **Experiment** against that dataset: make isolated code changes, replay, compare results
3. **Hill climb**: repeat until the best change is found, then present results
Run it with an optional trace function key:
```
/bitfab-assistant
/bitfab-assistant order-processing
```
#### Building the Dataset
Your coding agent does the data wrangling: it searches production traces for failures, reads full inputs and outputs, and identifies edge cases. It then presents edge cases for your judgment: is this a failure (and what should the output be), correct, or irrelevant? This labeled dataset becomes the benchmark for all experiments.
The plugin opens a rich UI for navigating and labeling the dataset, then brings you back to your coding agent so you stay in flow. You can label every trace yourself, or label a few and let the agent classify the rest based on the patterns you've established.
#### Running Experiments
The command reads your code, diagnoses failure patterns, and categorizes proposed changes:
* **Code fixes**: deterministic bugs, bundled into one experiment as a foundation
* **Judgment-based fixes**: prompt changes, search tuning, output formatting, each gets its own experiment
* **Infrastructure proposals**: larger changes noted for future work, not experimented on
Independent experiments run in parallel, each in its own isolated subagent on a separate git worktree. Each subagent edits the code, runs the SDK's replay command with your registry module and labeled dataset, and compares new outputs to expected outcomes.
#### Results
After each round, you see which traces now match expected outcomes, which still diverge, and whether any regressions occurred. The assistant works through the planned experiments in turn without pausing to ask whether to keep going, then wraps up once the plan is complete. The final summary shows pass rate improvement and all files changed, uncommitted in your working tree for review.
### MCP Tools
The plugin registers MCP tools that Cursor can call during conversations. These let you inspect traces, diagnose failures, and improve your code without leaving the editor.
#### Core
| Tool | Description |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `get_bitfab_api_key` | Retrieve your API key for SDK initialization and environment variable configuration |
| `get_api_key_context` | Returns which Bitfab org the plugin reads/writes to. Call before the first plugin write, or when data you wrote isn't visible in Studio |
| `list_organizations` | List the Bitfab organizations available to the signed-in user, marking the current plugin org |
| `get_database_connection_status` | Report whether the org has connected a database for per-trace replay branching (`none`, `checking`, `connected`, or `failed`), identify it as direct Neon or a managed Postgres mirror, and include the pinned project name and ID for direct Neon connections |
#### Trace Inspection
| Tool | Description |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `list_trace_functions` | List all traced functions in your organization |
| `search_traces` | Search and filter traces with keyword search, date ranges, status filters, regex, environment, label filters, dataset, test run, and db-snapshot scoping; replay traces are excluded by default unless `includeReplays` is set or a `testRunId` is provided |
| `get_traces` | Read one or more traces by ID with the trace environment plus summary (truncated) or full span details (input, output, reasoning, context, errors, per-span duration and tokens) |
| `get_trace_labels` | Read just the labels for up to 100 traces by ID in one call, no span content: each trace's verdict, annotation, and approved flag, plus one line per scored assertion carrying that assertion's own verdict, annotation, confidence, and author |
| `get_grader_labels` | Read the individual verdicts each automated grader recorded (reason, failure diagnostic, confidence, human or grader run), by trace IDs, by grader ID, or both |
| `get_span_field` | Fetch the complete, untruncated value of a single span field (input, output, reasoning, content, errors, or contexts) when `get_traces` truncated it |
#### Labeling and Datasets
| Tool | Description |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `save_agent_labels` | Set, skip, or archive agent pass/fail verdicts on traces. Supports confidence levels and annotations for human review. Pass an `assertionId` to score one assertion, omit it for the whole-trace verdict |
| `save_human_labels` | Write validated human pass/fail verdicts (with annotations) on traces. Validated immediately with no UI approval step; used by `assistant fix` before adding a trace to a dataset |
| `save_trace_assertions` | Record what a trace SHOULD do when replayed: one assertion per statement, with optional pass/fail criteria, target, and people-only human note. Agents can edit the note but must never use it as assessment evidence |
| `get_trace_assertions` | Read the assertions and people-only human notes on up to 100 traces by ID, no span content. Notes are never assessment evidence; a replay with no assertions of its own reads its original's |
| `archive_trace_assertions` | Retire assertions on a trace so later replays stop checking them. Non-destructive and all-or-nothing: one unknown, already-archived, or wrong-trace id fails the call and archives nothing |
| `save_dataset` | Create a labeled dataset for a traced function (named buckets of traces for review and replay) |
| `list_datasets` | List all datasets for a traced function with trace counts and assigned graders |
| `add_traces_to_dataset` | Add traces to a dataset (idempotent, 1-100 per call) |
| `remove_traces_from_dataset` | Remove traces from a dataset without deleting the traces themselves |
| `add_graders_to_dataset` | Assign graders to a dataset (idempotent, organization- and function-scoped, 1-100 per call) |
| `remove_graders_from_dataset` | Remove grader assignments from a dataset without deleting the graders |
| `save_grader` | Create or edit an automated grader (LLM-as-judge pass/fail check) for a traced function: upsert by id or name, rename, clear pass/fail criteria, archive/restore, select the judge model |
| `list_graders` | List automated graders for a traced function with optional name search and cursor pagination (20 by default, 50 maximum; archived hidden by default) |
#### Experiments
| Tool | Description |
| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `save_experiment_group` | Create a group from selected experiments with optional name and notes, or update an existing group's metadata without changing membership |
| `list_experiment_groups` | List recent experiment groups with their names, notes, and member experiment ids |
| `get_experiment_group` | Get one experiment group with its metadata and detailed member experiment summaries |
| `save_experiment` | Update an existing experiment's name or notes, move it into an existing experiment group, or remove it from its group |
| `add_graders_to_experiment_group` | Assign one or more graders directly to every current experiment in a group and queue missing evaluations for completed runs; later experiments do not inherit them automatically |
| `remove_graders_from_experiment_group` | Detach one or more graders from every current experiment in a group; unassigned ids are ignored and the graders themselves are not deleted |
| `list_experiments` | List experiments (replay test runs) for a traced function with name, notes, status, totals, and delta (fixed/regressed/still passing/still failing) |
| `get_experiment` | Get a single experiment by id with name, notes, status, totals, delta, experiment group, and grader results, including a per-grader passing/failing breakdown |
| `list_experiment_traces` | Get individual trace results for an experiment with each replay trace's verdict compared to the original, plus token usage (input, output, cached, total) for the replay and paired original |
| `add_graders_to_experiment` / `remove_graders_from_experiment` | Attach or detach active graders on an experiment so they run against its replay traces (idempotent, organization- and function-scoped, 1-100 per call); the effective set at completion is the union with the dataset's runnable graders; detaching a dataset-overlapping grader from an in-progress run is re-added at completion (still runs) but from a completed run permanently drops it from the finalized snapshot, and attaching to a completed experiment grades existing traces only on the next completion/replay |
| `rerun_graders_on_dataset` / `rerun_graders_on_experiment` | Re-score a dataset's traces or an experiment's completed replays with their attached graders, overwriting the previous verdicts (defaults to every attached grader; a subset may be named, and ids that are not attached are rejected); waits up to 90s and reports traces graded, and a repeat call reports the running job instead of starting a second one |
| `get_replay_status` | Read a replay test run's current status and local replay trace ID to server trace ID mapping while replay is still running |
#### Templates
| Tool | Description |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `get_template_reference` | Read the Nunjucks template engine reference, render-context schema, and available filters. Call once per session before editing templates |
| `get_template` | Read the rendering template for a span type, scoped to a trace function key or org-global |
| `save_template` | Upsert a rendering template for a span type. Controls how span data renders in the Bitfab UI |
#### Instrumentation
| Tool | Description |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `save_trace_plan` | Create a tracing plan, or structurally update an existing plan (its root included) in place by passing its plan ID |
| `confirm_trace_plan` | Confirm a plan without the browser (the continue path, or a plan page left without saving) so `setup view`/`setup modify` can find it by key later |
| `get_trace_plan` | Read a trace plan by ID (after confirmation) or by trace function key; Modify preserves prior decisions while reconciling the plan and its root with current instrumentation |
| `list_trace_plans` | List the org's trace plans, newest first, filterable by source, status, and trace function key; used to reuse unconfirmed `analyze-repo` drafts instead of re-scanning, and to find the plan a key already has so it is updated rather than duplicated |
| `cancel_trace_plan` | Retire an unconfirmed plan nobody will act on (already instrumented, or the workflow is gone) so it stops coming back as a reusable draft |
| `get_sim_plan` | Read a trace function's sim plan (alpha): its span nodes from recent traces with type, call count, average payload, estimated monthly cost, share of traces, and whether content is captured. A node is its span name inside the trace function key |
| `save_sim_plan` | Turn content capture off or on per span node (alpha). The span itself, its name, type, timing, and errors are always recorded; content off strips only inputs and outputs. A node recorded by a framework integration, a node that is the root of its traces, a node imported from another platform, or a node named by Bitfab rather than the SDK keeps its content, and turning it off is refused with the reason. Turning a node off also turns content off for every node beneath it in the sim plan, the way untracing a span in a trace plan drops the spans under it; a node beneath it that keeps its content stays on, and a node named in the same call keeps the value given to it. Turning a node back on leaves the nodes beneath it as they are. Returns the updated sim plan |
### Slash Commands
| Command | Description |
| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `/bitfab-setup` | Full setup workflow: authenticate, instrument, create replay registries |
| `/bitfab-setup login` | Auth only |
| `/bitfab-setup analyze-repo` | Non-interactively scan the repo, pick traceable workflows, and upload draft trace plans |
| `/bitfab-assistant` | Build a dataset from traces, experiment with code changes, improve pass rates or cut token costs |
| `/bitfab-assistant ` | Iterate on a specific trace function |
| `/bitfab-assistant fix ` | Fix one failing trace: diagnose it, make the focused code change, and replay just that trace; once it passes, add it to a dataset, then choose to inspect the before/after in Studio or re-run the full dataset |
| `/bitfab-update` | Update the plugin to the latest version |
## Example Workflows
### Instrument a new project
```
/bitfab-setup
```
The agent detects your project language, finds AI workflows, presents options, and instruments your chosen workflows, all interactively.
### Diagnose and fix a failing function
Ask Cursor naturally:
```
"My order-processing traces are failing. What's going wrong and can you fix it?"
```
The plugin calls `search_traces` and `get_traces` to inspect failing traces and suggests code fixes directly.
For a specific failing trace, run `/bitfab-assistant fix `. The agent diagnoses the failure, confirms why the original trace is wrong before editing when the trace or conversation does not already make that clear, makes the focused code fix, and replays only that trace first. Once the fix passes, it adds that trace to a dataset with a validated failing label, then branches: inspect the before/after in Studio, re-run the full dataset (in Studio or terminal-only), keep iterating, or stop. If that full-dataset re-run reveals real regressions (previously-passing traces the fix broke), it reports them and keeps the target trace saved as a red test to revisit. If the replay still fails, it offers to keep iterating or save the trace as a failing test instead.
### Iterate on a trace function
```
/bitfab-assistant memory-search
```
The agent finds failing traces, walks you through labeling them with expected outcomes, diagnoses the failure patterns in your code, then runs experiments: editing prompts or code, replaying against your labeled dataset, and reporting what improved. You stay in control at every decision point.
### Replay after a code change
After updating a function, pass your registry module to the replay command installed by the SDK:
```bash theme={null}
pnpm exec bitfab-replay --registry scripts/replayRegistry.ts extraction --limit 20
```
Or ask Cursor to do it for you.
While the replay runs, Cursor runs it in the background and reports progress to you as it goes: one line per trace as it finishes (a pass/fail mark, the running count, and how long that trace took), with any error reason inline, plus a periodic "still running" heartbeat when a slow trace takes a while so the run never looks stuck, then a summary with the total and average time. Full per-item outputs are written under that replay run's `.bitfab/replays//items/` folder and referenced from `.bitfab/replays//events.jsonl`. If the replay command succeeds but its local result cannot be captured, the plugin reports the outcome as unverified instead of failed, then checks the server test run to recover the final result.
## Configuration
### Credentials
Credentials are stored in `.bitfab/credentials.local.json` when that project-local file exists, otherwise in `~/.config/bitfab/credentials.json` (created by `/bitfab-setup login` with owner-readable permissions).
### Environment Variables
| Variable | Description |
| ---------------- | --------------------------- |
| `BITFAB_API_KEY` | Override the stored API key |
## Troubleshooting
### Not authenticated
If you see "Not authenticated" on session start:
1. Run `/bitfab-setup login` to authenticate via browser
2. Check that `~/.config/bitfab/credentials.json` exists and contains your API key
3. If using an environment variable, verify `BITFAB_API_KEY` is set
### MCP tools not available
If Cursor can't access the Bitfab tools:
1. Verify the MCP configuration in `.cursor/mcp.json` or `~/.cursor/mcp.json`
2. Try restarting Cursor after adding or modifying the MCP configuration
3. Check the [MCP Setup guide](/mcp-setup) for troubleshooting steps
### Plugin updates
Run `/bitfab-update` to install the latest version.
# Database Branching for Replay
Source: https://docs.bitfab.ai/db-branching
Replay traces against the database state that existed when they were captured
By default, replay re-runs a historical trace's inputs through your current code, but your function still talks to your **current** database. That breaks fidelity for anything that reads stored state: a refund decision over a since-cancelled order, a retrieval step over last week's rows, a tool call gated by a row that no longer exists.
**Database branching** pins each eligible trace to the database state at its captured instant, subject to replication lag. On replay, Bitfab provisions an ephemeral branch at that pinned instant and hands your replayed function a connection URL pointing at it. Traces captured by older SDKs without a snapshot reference use the normal database path.
Available in the **TypeScript, Python, Ruby, and Go** SDKs.
## How it works
Capture is **automatic in current SDKs**: eligible root traces record the wall-clock instant they ran and a snapshot reference, with no SDK configuration. So there's nothing to turn on for new traces; setup is just two steps:
1. **Connect your database once** in the Bitfab dashboard. Your source database can be any Postgres; Bitfab provisions a managed, branchable copy from it.
2. **Wire replay** by passing `dbBranch` to the replay call and reading the per-trace branch URL inside the replayed function.
At replay time, for each eligible item Bitfab branches the managed copy to the trace's captured instant and returns a short-lived connection URL. Your replayed function connects to that branch instead of the normal database. The branch is deleted after the run. Items without a snapshot reference use the normal database path and are not branched.
## 1. Connect your database
In the Bitfab dashboard, open **Integrations → Database** and paste your Postgres connection string. Bitfab provisions a branchable managed copy from it; discovery and engine setup take a few minutes, after which the Database section shows **Connected**.
Your source database can be any Postgres. You do **not** set any `NEON_*` environment variables (those are Bitfab-side server configuration). You only provide your source connection string in the dashboard. The connection string is encrypted at rest. When a replay item has a captured snapshot and uses its provisioned branch, writes stay on that isolated, ephemeral copy and never touch your source database. Items without a snapshot use the normal database path, so keep unsafe writes behind replay-mocked spans.
## 2. Wire replay to the branch
Pass `dbBranch` to the replay call and connect through the resolved branch's URL inside the replayed function. `true` requests a branch per replay item with the mirror's own sizing; pass an object instead to tune it, and see [Sizing and warming the branch](#sizing-and-warming-the-branch) for the fields. The accessor that reads the branch differs slightly per SDK.
```typescript TypeScript theme={null}
import { Bitfab, getCurrentReplayBranch } from "@bitfab/sdk"
const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
const pipeline = bitfab.getFunction("checkout-agent")
// The replayed function connects to the branch URL when a per-trace branch is
// available, and to the normal env var otherwise.
const runCheckout = pipeline.withSpan(
{ name: "CheckoutAgent", type: "agent" },
async (orderId: string) => {
const branch = getCurrentReplayBranch()
const url = branch?.databaseUrl ?? process.env.DATABASE_URL
const db = makeDbClient(url) // your normal client factory
const order = await db.orders.find(orderId)
return decideRefund(order)
},
)
const result = await bitfab.replay("checkout-agent", runCheckout, {
limit: 10,
dbBranch: true,
})
console.log(result.testRunUrl)
```
```python Python theme={null}
import os
from bitfab import Bitfab, get_current_replay_branch
bitfab = Bitfab(api_key=os.environ["BITFAB_API_KEY"])
pipeline = bitfab.get_function("checkout-agent")
@pipeline.span(name="CheckoutAgent", type="agent")
def run_checkout(order_id: str):
branch = get_current_replay_branch()
url = branch.database_url if branch else os.environ["DATABASE_URL"]
db = make_db_client(url) # your normal client factory
order = db.orders.find(order_id)
return decide_refund(order)
result = bitfab.replay(run_checkout, limit=10, db_branch=True)
print(result["test_run_url"])
```
```ruby Ruby theme={null}
require "bitfab"
Bitfab.configure(api_key: ENV["BITFAB_API_KEY"])
client = Bitfab.client
class CheckoutAgent
include Bitfab::Traceable
bitfab_function "checkout-agent"
bitfab_span :run_checkout, name: "CheckoutAgent", type: "agent"
def run_checkout(order_id)
branch = Bitfab.current_replay_branch
url = branch ? branch.database_url : ENV["DATABASE_URL"]
db = make_db_client(url) # your normal client factory
decide_refund(db.orders.find(order_id))
end
end
result = client.replay(
CheckoutAgent.new, :run_checkout,
trace_function_key: "checkout-agent",
limit: 10,
db_branch: true,
)
puts result[:test_run_url]
```
```go Go theme={null}
func runCheckout(ctx context.Context, orderID string) (Decision, error) {
databaseURL := os.Getenv("DATABASE_URL")
if branch := bitfab.GetCurrentReplayBranch(ctx); branch != nil {
databaseURL = branch.DatabaseURL()
}
db := makeDBClient(databaseURL)
order, err := db.Orders.Find(ctx, orderID)
if err != nil {
return Decision{}, err
}
return decideRefund(order), nil
}
result, err := client.Replay(
context.Background(),
"checkout-agent",
runCheckout,
&bitfab.ReplayOptions{DBBranch: &bitfab.DBBranchOptions{}},
)
```
### Sizing and warming the branch
A replay branch is created fresh, so it starts with an empty cache and whatever compute size the mirror was provisioned with. If your replay's query latency looks nothing like the original trace's, that gap is usually the branch, not your code. Two optional settings close it, both passed in `dbBranch`:
```typescript TypeScript theme={null}
await bitfab.replay("checkout-agent", runCheckout, {
dbBranch: {
minCu: 2,
maxCu: 2,
warmupSql: "SELECT count(*) FROM orders; SELECT count(*) FROM line_items;",
},
})
```
```python Python theme={null}
bitfab.replay(
run_checkout,
db_branch={
"min_cu": 2,
"max_cu": 2,
"warmup_sql": "SELECT count(*) FROM orders; SELECT count(*) FROM line_items;",
},
)
```
```ruby Ruby theme={null}
client.replay(
CheckoutAgent.new, :run_checkout,
trace_function_key: "checkout-agent",
db_branch: {
min_cu: 2,
max_cu: 2,
warmup_sql: "SELECT count(*) FROM orders; SELECT count(*) FROM line_items;"
}
)
```
```go Go theme={null}
result, err := client.Replay(ctx, "checkout-agent", runCheckout, &bitfab.ReplayOptions{
DBBranch: &bitfab.DBBranchOptions{
MinCU: 2,
MaxCU: 2,
WarmupSQL: "SELECT count(*) FROM orders; SELECT count(*) FROM line_items;",
},
})
```
* **`minCu` / `maxCu`** set the branch compute's autoscaling floor and ceiling, in Neon Compute Units. Setting them equal pins the size, which is what you want for an experiment: with a range, an item that runs later can hit an endpoint that has already scaled up and post a better number for identical code. Equal values are a fixed-size compute, allowed up to 56 CU. A range is autoscaling, which Neon caps at an 8 CU span and a 16 CU ceiling, so `minCu: 2, maxCu: 16` is rejected. Sizes are a discrete set, not a continuum: `0.25`, `0.5`, every integer to `16`, then even numbers to `56`. Anything invalid fails the replay immediately rather than silently falling back to your live database. Note that a pinned size above 16 CU stays always-active, since Neon does not scale those to zero, so it keeps billing until the branch is released.
* **`warmupSql`** runs as part of the branch's readiness check, before your function ever sees the lease. Warm-up time is therefore not charged to the replayed call. Passing it raises the readiness check's budget from 10 seconds to **240 seconds**, since warming a real working set is not a `SELECT 1`. Invalid SQL, or a warm-up that outruns that budget, fails the lease rather than quietly handing back a cold branch.
Both are optional. Pass `dbBranch: true` instead and the branch keeps the mirror's own defaults, while branching stays on.
### Handling branch failures
Bitfab never falls back to the live database after a requested branch fails. The replay function is not invoked, and that attempt still returns an item with no replay trace ID and a structured database branch replay error in `replayError` (TypeScript and Go) or `replay_error` (Python and Ruby).
The error is `DbBranchReplayError` (`Bitfab::DbBranchReplayError` in Ruby, `DBBranchReplayError` in Go). Its code and message are the values produced by the server resolver, and its original trace ID identifies the affected attempt. Codes such as `branch_create_failed`, `branch_warmup_invalid`, `branch_warmup_failed`, and `snapshot_from_replaced_origin` can be handled without parsing the compatible item error string. A malformed captured ref reports `invalid_snapshot_ref`; a seeded source, which pinned no database instant, reports `seeded_trace_has_no_snapshot`; an unexpected server resolver failure reports `internal_error`; and an HTTP, timeout, or network failure while asking for the branch reports `lease_request_failed` with the original client exception retained as `cause`. If the entire replay later raises, the same typed errors remain on the items attached to the top-level `ReplayError`.
Only a trace with no captured snapshot reference may use the normal live-database fallback. Once a snapshot reference is present, malformed or unreadable snapshot data fails that item rather than running it against current data.
### Reading the branch
The accessor returns a branch only inside a replay item that has a resolved one, and a nullish result is the normal fallback path:
* On the **live request path** it is null: your function keeps using its normal `DATABASE_URL`. The same code works in production and in replay.
* For **traces captured before the SDK version that added always-on snapshot capture**, it is null (no snapshot ref), so the item replays against your normal database.
* A [**seeded trace**](/typescript-sdk#seeding-traces) records no snapshot ref, so there is no state to restore. With branching requested, the item fails with the replay-error code `seeded_trace_has_no_snapshot` rather than falling back to live data, because a silent fallback would return a result that looks like it used the historical data you asked for. Replay a seeded trace without `dbBranch`, or replay a captured trace.
Database snapshots provide historical fidelity; they are not the general replay safety boundary. Mark unsafe database writes and other side effects for replay mocking. If database behavior itself must run real, replay only traces whose accessor returns a branch and treat a missing snapshot as unreplayable in your workflow.
| | TypeScript | Python | Ruby | Go |
| -------------- | -------------------------- | ----------------------------- | ------------------------------ | ----------------------------- |
| Accessor | `getCurrentReplayBranch()` | `get_current_replay_branch()` | `Bitfab.current_replay_branch` | `GetCurrentReplayBranch(ctx)` |
| Branch URL | `branch.databaseUrl` | `branch.database_url` | `branch.database_url` | `branch.DatabaseURL()` |
| Expiry | `branch.expiresAt` | `branch.expires_at` | `branch.expires_at` | `branch.ExpiresAt` |
| Read-only | `branch.readOnly` | `branch.read_only` | `branch.read_only` | `branch.ReadOnly` |
| Source trace | `branch.traceId` | `branch.trace_id` | `branch.trace_id` | `branch.TraceID` |
| Branch region | `branch.region` | `branch.region` | `branch.region` | `branch.Region` |
| Pinned instant | `branch.snapshotTimestamp` | `branch.snapshot_timestamp` | `branch.snapshot_timestamp` | `branch.SnapshotTimestamp` |
| Branch id | `branch.neonBranchId` | `branch.neon_branch_id` | `branch.neon_branch_id` | `branch.NeonBranchID` |
| Env var name | `branch.envKey` | `branch.env_key` | `branch.env_key` | `branch.EnvKey` |
Every field the service puts on the lease is exposed, so a field added server-side reaches your code without an SDK upgrade. Only the connection string is special: reading it is what marks the branch as used.
### Resolve the connection per call, not at import
The most common reason a wired replay still hits production is a database client created **once at module import**: a module-level pool/engine bound to `DATABASE_URL` captures that value before any replay context exists. Refactor so the connection string is resolved **per call** (or per replay item) and your replayed function can build, or be handed, a client from the branch URL. Each replay item receives its own branch URL at runtime, so an import-time pool cannot be made safe by changing the process-wide `DATABASE_URL`.
## Verifying it works
Capture is automatic, but a trace only carries a snapshot ref if it was recorded by an SDK version with always-on capture, so smoke-test with a **recently captured** trace:
1. Run the instrumented function once so a new trace lands.
2. Replay that trace and confirm, inside the function, that the accessor returned a branch and that `snapshotTimestamp` matches the moment the source trace ran. A differing URL host only proves some branch was handed over; the pinned instant proves it is the right point in history.
3. Open the test run URL from the replay result to inspect the experiment.
### Bitfab tracks whether the branch was actually used
Each replayed trace records whether your code actually obtained the branch URL during that item (reading `databaseUrl` / `database_url` counts; calling the accessor or reading any other field does not). The experiment data for the trace then shows one of three states:
* **Used**: a branch was provisioned and your function took its URL.
* **Provisioned but never read**: a branch was ready, but the function never asked for the URL. This usually means the replay silently hit your live database; the most common cause is a connection pool created at module import (see "Resolve the connection per call, not at import" above).
* **No branch**: the item had no snapshot to branch from, so the function ran against its normal database. A provisioning failure does not produce a replay trace because the function is not invoked; it returns an item with a structured replay error instead.
### Where the provisioning time went
Every replay item reports how long its branch took to provision, broken down by
phase, so a slow replay can be attributed rather than guessed at:
```typescript theme={null}
const result = await bitfab.replay("my-key", myFn, { dbBranch: {} })
for (const item of result.items) {
console.log(item.dbBranchTimings)
// {
// startedAt: "2026-05-22T14:29:55.000Z",
// projectResolveMs: 12, // resolving your project, its retention and region
// branchCreateMs: 1800, // creating the branch, until it exists
// connectionUriMs: 0, // resolving the connection URI
// computeConnectMs: 340, // the compute accepting a connection
// baseProbeMs: 8, // the branch answering a readiness query
// warmupMs: 2500, // your warm-up SQL
// totalMs: 4660,
// }
}
```
Python exposes the same object as `item["db_branch_timings"]`, Ruby as
`item[:db_branch_timings]`, and Go as `item.DBBranchTimings`. It is also
recorded on the replayed trace itself, so the breakdown outlives the run that
produced it.
Three things worth knowing when you read these numbers:
* **They are measured server-side**, from Bitfab to your database provider. Your
own runner will observe the total plus its round trip to the branch's region,
which the lease reports as `region`.
* **They are per item**, not per replay. Each item gets its own branch and its
own compute, so a cold one and a warm one are not comparable.
* **A failed resolve still reports them**, carrying the phases it reached and a
`totalMs` that is time-to-failure. Phases after the failure are absent rather
than zero, so "it spent four minutes and then failed" is answerable.
Durations, not timestamps: any instant is `startedAt` plus the running sum of the
phases before it. Reporting a wall-clock stamp per phase would make clock skew
between our servers and your runner read as latency.
## Optional: pin the provider (TypeScript)
Capture works with no configuration. In TypeScript you *may* pass a provider to pin it at capture time; it is not required, and the provider is otherwise resolved at replay time:
```typescript theme={null}
const bitfab = new Bitfab({
apiKey: process.env.BITFAB_API_KEY,
dbSnapshot: { provider: "neon" },
})
```
Python, Ruby, and Go have no equivalent option; capture is fully automatic.
## Limitations
* **TypeScript, Python, Ruby, and Go** support historical database branches.
* **Postgres only.**
* Branch leases are **short-lived** (a few minutes) and created fresh per replay item. Replay completes well within that window; the lease is released and the branch deleted afterward.
* The branch reflects the source database's state at the captured instant, **bounded by replication lag** (typically sub-second to a few seconds).
* When a branch is provisioned and used, replay writes land only on that ephemeral branch and are discarded; they never propagate to your source database. A trace with no snapshot uses the normal database path, so unsafe writes on that path must still be mocked.
* Only traces captured by an SDK version with always-on snapshot capture can be branched. Older traces replay against your normal database.
* **Seeded traces cannot be branched.** They were written from cases rather than captured, so no database instant was ever pinned. Requesting a branch for one fails the item instead of falling back to live data.
# BAML
Source: https://docs.bitfab.ai/frameworks/baml
Auto-capture prompts and LLM metadata from BAML function calls with Bitfab
Bitfab integrates with [BAML](https://docs.boundaryml.com/) to automatically capture rendered prompts and LLM metadata on the current span - no manual `setPrompt` or `addContext` calls needed. Wrap a BAML method with `wrapBAML` / `wrap_baml` and Bitfab extracts everything automatically.
**Canonical signatures:** [TypeScript `wrapBAML`](/reference/typescript#wrapbaml) · [Python `wrap_baml`](/reference/python#wrap-baml)
## Supported Languages
| Language | Method | Status |
| ---------- | ------------- | ----------------- |
| TypeScript | `wrapBAML()` | ✅ Supported |
| Python | `wrap_baml()` | ✅ Supported |
| Ruby | - | Not yet supported |
| Go | - | Not yet supported |
## Quick Start
```typescript TypeScript theme={null}
import { Bitfab } from "@bitfab/sdk"
import { b } from "./baml_client"
const bitfab = new Bitfab({
apiKey: process.env.BITFAB_API_KEY,
bamlClient: b,
})
const tracedClassify = bitfab.withSpan(
"classify",
{ type: "llm" },
bitfab.wrapBAML(b.ClassifyText),
)
const result = await tracedClassify("Hello world")
```
```python Python theme={null}
import os
from bitfab import Bitfab
from baml_client import b
bitfab = Bitfab(api_key=os.environ["BITFAB_API_KEY"], baml_client=b)
@bitfab.span("classify", type="llm")
async def classify(text: str):
return await bitfab.wrap_baml(b.ClassifyText)(text=text)
result = await classify("Hello world")
```
## What Gets Captured
`wrapBAML` / `wrap_baml` creates a BAML `Collector`, runs the method through a tracked client, then extracts:
| Data | Span Field | Source |
| ---------------------------------------- | ----------------------------------- | --------------------------------------- |
| Rendered prompt (system + user messages) | `span_data.prompt` | BAML Collector HTTP request body |
| Model name | `span_data.contexts[].model` | BAML Collector HTTP request body or URL |
| Provider | `span_data.contexts[].provider` | BAML Collector call metadata |
| Input tokens | `span_data.contexts[].inputTokens` | BAML Collector usage |
| Output tokens | `span_data.contexts[].outputTokens` | BAML Collector usage |
| Duration | `span_data.contexts[].durationMs` | BAML Collector timing |
If `@boundaryml/baml` (TypeScript) or `baml-py` (Python) is not installed, the BAML method is called directly without instrumentation.
### Limitations
* **Streaming functions are not auto-instrumented.** `wrapBAML` / `wrap_baml` awaits a single result, so streaming calls (`b.stream.ClassifyText`, which return a stream rather than a final value) are not captured. Instrument the non-streaming form (`b.ClassifyText`), or capture the call manually with `setPrompt` / `addContext` inside a `withSpan` / `@span` root after draining the stream.
* **Functions that make more than one LLM call capture only one of them.** When a BAML function issues multiple HTTP calls (retry policies, `fallback` clients, round-robin clients), the captured prompt, model, and provider come from the selected call (or the first one if none is marked selected). Token usage is taken from the Collector's aggregate, so on a fallback chain the recorded `model` / `provider` may not correspond to the call that produced the tokens.
## TypeScript
### Installation
```bash theme={null}
npm install @bitfab/sdk @boundaryml/baml
```
### Method Signature
```typescript theme={null}
// Form 1: bamlClient passed in constructor
bitfab.wrapBAML(method: Function): WrappedBamlFn
bitfab.wrapBAML(method: Function, options: WrapBAMLOptions): WrappedBamlFn
// Form 2: bamlClient passed at call site
bitfab.wrapBAML(bamlClient: unknown, method: Function): WrappedBamlFn
bitfab.wrapBAML(bamlClient: unknown, method: Function, options: WrapBAMLOptions): WrappedBamlFn
```
**Parameters:**
* `method` (Function, required) - The BAML method to wrap (e.g., `b.ClassifyText`)
* `bamlClient` (unknown, optional) - The BAML client instance. Required if not passed in the `Bitfab` constructor
* `options` (WrapBAMLOptions, optional) - Configuration options
**WrapBAMLOptions:**
* `onCollector?: (collector: unknown) => void` - Callback fired after each invocation with the BAML Collector instance
**Returns:** A `WrappedBamlFn` - an async function with the same signature as the original, plus a `.collector` property.
### Usage
#### Constructor-Based (Recommended)
```typescript theme={null}
import { Bitfab } from "@bitfab/sdk"
import { b } from "./baml_client"
const bitfab = new Bitfab({
apiKey: process.env.BITFAB_API_KEY,
bamlClient: b,
})
const tracedClassify = bitfab.withSpan(
"classify",
{ type: "llm" },
bitfab.wrapBAML(b.ClassifyText),
)
const result = await tracedClassify("Hello world")
```
#### Explicit Client
```typescript theme={null}
const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
const tracedClassify = bitfab.withSpan(
"classify",
{ type: "llm" },
bitfab.wrapBAML(b, b.ClassifyText),
)
```
#### Accessing the BAML Collector
The wrapped function exposes a `.collector` property containing the BAML `Collector` from the most recent call:
```typescript theme={null}
const tracedClassify = bitfab.withSpan(
"classify",
{ type: "llm" },
bitfab.wrapBAML(b.ClassifyText),
)
await tracedClassify("Hello world")
const collector = tracedClassify.collector // BAML Collector instance
```
The `.collector` is `null` before the first call or if `@boundaryml/baml` is not installed.
#### `onCollector` Callback
For more control, pass an `onCollector` callback:
```typescript theme={null}
const tracedClassify = bitfab.withSpan(
"classify",
{ type: "llm" },
bitfab.wrapBAML(b.ClassifyText, {
onCollector: (collector) => {
console.log("BAML collector:", collector)
},
}),
)
```
With the explicit client form, options go in the third argument:
```typescript theme={null}
bitfab.wrapBAML(b, b.ClassifyText, {
onCollector: (collector) => { /* ... */ },
})
```
If the callback throws, the error is silently caught and never crashes your application.
### Error Handling
If `@boundaryml/baml` is not installed, `wrapBAML` falls back to calling the method directly on the BAML client without instrumentation. Metadata extraction errors are silently caught.
## Python
### Installation
```bash theme={null}
pip install bitfab-py baml-py
```
### Method Signature
```python theme={null}
# Form 1: baml_client passed in constructor
bitfab.wrap_baml(method: Callable, *, on_collector: Callable | None = None) -> Callable
# Form 2: baml_client passed at call site
bitfab.wrap_baml(baml_client: Any, method: Callable, *, on_collector: Callable | None = None) -> Callable
```
**Parameters:**
* `method` (Callable, required) - The BAML method to wrap (e.g., `b.ClassifyText`)
* `baml_client` (Any, optional) - The BAML client instance. Required if not passed in the `Bitfab` constructor
* `on_collector` (Callable, optional, keyword-only): Callback fired after each invocation with the BAML `Collector` instance
**Returns:** An async wrapper with the same signature as the original method, plus a `.collector` attribute holding the most recent call's `Collector` (`None` before the first call or if `baml-py` is not installed).
### Usage
#### Constructor-Based (Recommended)
```python theme={null}
import os
from bitfab import Bitfab
from baml_client import b
bitfab = Bitfab(api_key=os.environ["BITFAB_API_KEY"], baml_client=b)
@bitfab.span("classify", type="llm")
async def classify(text: str):
return await bitfab.wrap_baml(b.ClassifyText)(text=text)
result = await classify("Hello world")
```
#### Explicit Client
```python theme={null}
bitfab = Bitfab(api_key=os.environ["BITFAB_API_KEY"])
@bitfab.span("classify", type="llm")
async def classify(text: str):
return await bitfab.wrap_baml(b, b.ClassifyText)(text=text)
```
#### Accessing the BAML Collector
The wrapper exposes a `.collector` attribute with the BAML `Collector` from the most recent call, and accepts an `on_collector` callback fired after each invocation:
```python theme={null}
classify = bitfab.wrap_baml(
b.ClassifyText,
on_collector=lambda collector: print("BAML collector:", collector),
)
@bitfab.span("classify", type="llm")
async def run(text: str):
return await classify(text=text)
await run("Hello world")
collector = classify.collector # BAML Collector instance (None before first call)
```
If the callback throws, the error is silently caught and never crashes your application.
### Error Handling
If `baml-py` is not installed, `wrap_baml` falls back to calling the method directly on the BAML client without instrumentation. Metadata extraction errors are silently caught.
# Claude Agent SDK
Source: https://docs.bitfab.ai/frameworks/claude-agent-sdk
Automatic tracing for Claude Agent SDK with Bitfab
Bitfab integrates with the [Claude Agent SDK](https://docs.anthropic.com/en/docs/agents-and-tools/claude-agent-sdk) via a handler that automatically captures LLM turns, tool invocations, and subagent execution as traced spans. The handler instruments the SDK's hook system and wraps the response stream to capture all execution data.
**Canonical signatures:** [TypeScript reference](/reference/typescript#framework-integrations) · [Python reference](/reference/python#framework-integrations)
## Supported Languages
| Language | Method | Status |
| ---------- | ---------------------------- | ----------------- |
| TypeScript | `getClaudeAgentHandler()` | ✅ Supported |
| Python | `get_claude_agent_handler()` | ✅ Supported |
| Ruby | - | Not yet supported |
| Go | - | Not yet supported |
## Quick Start
```typescript TypeScript theme={null}
import { Bitfab } from "@bitfab/sdk"
import { query } from "@anthropic-ai/claude-agent-sdk"
const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
const handler = bitfab.getClaudeAgentHandler("my-agent")
// Instrument the SDK options with Bitfab hooks (tool + subagent spans)
const options = handler.instrumentOptions({
model: "claude-sonnet-4-6",
})
// Wrap the query() stream to capture LLM turns. Pass `{ input }` (the prompt)
// so the run records a replayable root span - see Replay below.
const prompt = "What's the weather?"
for await (const message of handler.wrapQuery(
query({ prompt, options }),
{ input: prompt },
)) {
// Process messages as normal
}
```
```python Python theme={null}
import os
from bitfab import Bitfab
bitfab = Bitfab(api_key=os.environ["BITFAB_API_KEY"])
handler = bitfab.get_claude_agent_handler("my-agent")
# Instrument the SDK options with Bitfab hooks
options = handler.instrument_options(
ClaudeAgentOptions(model="claude-sonnet-4-6")
)
prompt = "What's the weather?"
async with ClaudeSDKClient(options=options) as client:
await client.query(prompt)
# Wrap the response stream to capture LLM turns. Pass `input` (the prompt)
# so the run records a replayable root span - see Replay below.
async for message in handler.wrap_response(
client.receive_response(), input=prompt
):
# Process messages as normal
pass
```
## What Gets Captured
The handler captures three types of spans:
| Event | Span Type | Captured Data |
| ------------------ | ---------- | -------------------------------------------------------------------------------------------------------------------------- |
| LLM turns | `llm` | Full conversation history, assistant response content, model name, token usage (input, output, cache read, cache creation) |
| Tool invocations | `function` | Tool input, tool response or error |
| Subagent execution | `agent` | Agent type, start/stop lifecycle |
### Token Usage
For LLM spans, token usage is extracted from the message stream:
* `inputTokens` - Input tokens
* `outputTokens` - Output tokens
* `cacheReadTokens` - Cache read tokens (if applicable)
* `cacheCreationTokens` - Cache creation tokens (if applicable)
* `model` - Model name
## TypeScript
### Installation
```bash theme={null}
npm install @bitfab/sdk
```
### Method Signature
```typescript theme={null}
bitfab.getClaudeAgentHandler(traceFunctionKey: string): BitfabClaudeAgentHandler
```
**Parameters:**
* `traceFunctionKey` (string, required) - Groups all traces from this handler under one key in Bitfab
**Returns:** A `BitfabClaudeAgentHandler` instance with methods for instrumenting options and wrapping streams.
### Handler Methods
#### `instrumentOptions(options)`
Injects Bitfab hooks into the SDK options object. Mutates the options in-place and returns them.
```typescript theme={null}
const options = handler.instrumentOptions({
model: "claude-sonnet-4-6",
// your other options...
})
```
The injected hooks capture:
* **PreToolUse** → Creates a `function` span with the tool input
* **PostToolUse** → Completes the span with the tool response
* **PostToolUseFailure** → Completes the span with an error
* **SubagentStart** → Creates an `agent` span
* **SubagentStop** → Completes the agent span
#### `wrapQuery(stream, opts?)`
Wraps the `query()` async iterator to capture LLM turns from the message stream. Messages are yielded unchanged. Tool and subagent spans come from the hooks injected by `instrumentOptions`.
Pass `{ input }` (the prompt) to record a replayable root `agent` span - see [Replay](#replay).
```typescript theme={null}
import { query } from "@anthropic-ai/claude-agent-sdk"
const prompt = "Hello"
for await (const message of handler.wrapQuery(
query({ prompt, options }),
{ input: prompt },
)) {
// Messages pass through unchanged
console.log(message)
}
```
Each LLM turn creates an `llm` span containing:
* **Input**: Full conversation history snapshot up to this turn
* **Output**: Assistant message content blocks
* **Context**: Model name, token usage
#### `wrapResponse(stream)`
Identical to `wrapQuery` - wraps any Claude Agent SDK message stream. Provided for naming symmetry with the Python SDK (whose `ClaudeSDKClient.receiveResponse()` it wraps). In TypeScript, prefer `wrapQuery` around `query()`.
The TypeScript Claude Agent SDK exposes a single `query()` entry point. There is no `ClaudeSDKClient` class - that exists only in the Python SDK.
### Usage
```typescript theme={null}
import { Bitfab } from "@bitfab/sdk"
import { query } from "@anthropic-ai/claude-agent-sdk"
const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
const handler = bitfab.getClaudeAgentHandler("my-agent")
const options = handler.instrumentOptions({
model: "claude-sonnet-4-6",
// mcpServers, allowedTools, systemPrompt, etc.
})
for await (const message of handler.wrapQuery(
query({ prompt: "Analyze the latest sales data", options })
)) {
// Tool calls, subagent execution, and LLM turns
// are all automatically traced
}
```
### Nesting with Core Tracing
Wrapping the agent call in `withSpan` with the **same key** records the call's
arguments as a replayable root and nests every handler span underneath it (see
[Replay](#replay)).
```typescript theme={null}
const pipeline = bitfab.getFunction("my-agent")
const tracedRun = pipeline.withSpan(
{ name: "RunClaudeAgent", type: "agent" },
async (prompt: string) => {
const handler = pipeline.getClaudeAgentHandler() // same key as the root
const options = handler.instrumentOptions({ model: "claude-sonnet-4-6" })
const messages = []
for await (const msg of handler.wrapQuery(query({ prompt, options }))) {
messages.push(msg)
}
return messages
},
)
await tracedRun("What's the weather?")
// Claude Agent spans appear nested under the "RunClaudeAgent" span
```
### Error Handling
* All hook callbacks are wrapped in try/catch - errors are silently ignored and return an empty object.
* Stream processing continues even if individual message capture fails.
* The handler never throws or affects the SDK's execution.
## Python
### Installation
```bash theme={null}
pip install bitfab-py
```
### Method Signature
```python theme={null}
bitfab.get_claude_agent_handler(trace_function_key: str) -> BitfabClaudeAgentHandler
```
**Parameters:**
* `trace_function_key` (str, required) - Groups all traces from this handler under one key in Bitfab
**Returns:** A `BitfabClaudeAgentHandler` instance with methods for instrumenting options and wrapping streams.
### Handler Methods
#### `instrument_options(options)`
Injects Bitfab hooks into the SDK options object. Returns the modified options.
```python theme={null}
options = handler.instrument_options(
ClaudeAgentOptions(model="claude-sonnet-4-6")
)
```
#### `wrap_response(stream, input=...)`
Wraps the `receive_response()` async iterator to capture LLM turns. Pass `input` (the prompt) to record a replayable root `agent` span - see [Replay](#replay):
```python theme={null}
async for message in handler.wrap_response(
client.receive_response(), input=prompt
):
# Messages pass through unchanged
print(message)
```
#### `wrap_query(stream, input=...)`
Same, for the `query()` API:
```python theme={null}
async for message in handler.wrap_query(
query(prompt=prompt, options=options), input=prompt
):
print(message)
```
### Usage
```python theme={null}
import os
from bitfab import Bitfab
bitfab = Bitfab(api_key=os.environ["BITFAB_API_KEY"])
handler = bitfab.get_claude_agent_handler("my-agent")
options = handler.instrument_options(
ClaudeAgentOptions(
model="claude-sonnet-4-6",
tools=[...],
)
)
async with ClaudeSDKClient(options=options) as client:
await client.query("Analyze the latest sales data")
async for message in handler.wrap_response(client.receive_response()):
# Tool calls, subagent execution, and LLM turns
# are all automatically traced
pass
```
### Nesting with Core Tracing
Bind the key once with `get_function` so the root and the handler share it (no repeated string to keep in sync):
```python theme={null}
pipeline = bitfab.get_function("my-pipeline")
@pipeline.span(type="agent")
async def run_claude_agent(query: str):
handler = pipeline.get_claude_agent_handler() # same key as the root
options = handler.instrument_options(
ClaudeAgentOptions(model="claude-sonnet-4-6")
)
async with ClaudeSDKClient(options=options) as client:
await client.query(query)
async for message in handler.wrap_response(client.receive_response()):
pass
await run_claude_agent("What's the weather?")
# Claude Agent spans appear nested under the "my-pipeline" span
```
The plain `@bitfab.span("my-pipeline")` / `bitfab.get_claude_agent_handler("my-pipeline")` forms work too; `get_function` just keys both from one place.
### Error Handling
* All hook callbacks are wrapped in try/except - errors are logged at DEBUG level and return an empty dict.
* Stream processing continues even if individual message capture fails.
* The handler never raises or affects the SDK's execution.
## Replay
Pass the prompt as `input` to the wrap call and the handler records a root
`agent` span carrying it, with every LLM / tool / subagent span nested
underneath. That root is the replayable unit: `replay(key, fn)` re-feeds each
historical prompt to a callable that re-issues the query. No `@bitfab.span` /
`withSpan` wrapper is required.
The prompt is **not** present anywhere in the message stream, so the handler
cannot recover it on its own - you must pass it as `input`. Without `input`
the run still traces, but it has no replayable root.
```python theme={null}
async def run_my_agent(prompt: str) -> str:
handler = bitfab.get_claude_agent_handler("my-agent")
options = handler.instrument_options(
ClaudeAgentOptions(model="claude-sonnet-4-6")
)
final = ""
async with ClaudeSDKClient(options=options) as client:
await client.query(prompt)
# `input=prompt` records the replayable root span.
async for message in handler.wrap_response(
client.receive_response(), input=prompt
):
... # collect output
return final
# Replays re-run run_my_agent with each historical prompt.
result = bitfab.replay("my-agent", run_my_agent, limit=10)
```
```typescript theme={null}
async function runMyAgent(prompt: string) {
const handler = bitfab.getClaudeAgentHandler("my-agent")
const options = handler.instrumentOptions({ model: "claude-sonnet-4-6" })
// `{ input: prompt }` records the replayable root span.
for await (const msg of handler.wrapQuery(query({ prompt, options }), {
input: prompt,
})) {
// collect output
}
}
await bitfab.replay("my-agent", runMyAgent, { limit: 10 })
```
Keep `input` serializable (a prompt string, a small params object) - it is the
recorded root input replay re-feeds. Rebuild runtime wiring inside the function
(agent options, tools, API keys). Put unsafe calls behind replay-mockable marked
spans; use a no-op only for a replay-only callback slot with no recorded call to
mock.
If you already wrap the agent call in a `@bitfab.span` / `withSpan` with the
**same key** (for example, to also capture surrounding work), that outer span
is the replayable root and the handler nests under it automatically - passing
`input` is then unnecessary. Full details: **Replaying functions** in the
[Python SDK](/python-sdk#replay) and [TypeScript SDK](/typescript-sdk#replay) pages.
## Trace metadata
This integration exports its own trace under the same canonical trace id as the traced function, and that export carries metadata of its own. Metadata the caller recorded, through `seedTrace` / `seed_trace` or `setMetadata` / `set_metadata`, is merged onto that export and takes precedence per key, so a seeded case's provenance survives and reaches a later replay's `adaptInputs` / `adapt_inputs` hook as `ctx.metadata` / `ctx["metadata"]`.
Keys the integration set that the caller did not are kept, so `ctx.metadata` can carry keys the caller never wrote. A key set on both sides to different values keeps the caller's value and warns once per trace.
The merge needs both sides on the same canonical trace id, which is what running the integration nested inside a traced function gives them. A processor-only setup, with no enclosing traced function, resolves its own trace id and has no caller metadata to merge. An export that lands after the traced function returned, as a streamed run's does, still merges. Requires the TypeScript or Python SDK v0.52.3 or later.
# LangGraph / LangChain
Source: https://docs.bitfab.ai/frameworks/langgraph
Automatic tracing for LangGraph and LangChain agents with Bitfab
Bitfab integrates with [LangGraph](https://langchain-ai.github.io/langgraph/) and [LangChain](https://python.langchain.com/) via a callback handler that automatically captures graph node execution, LLM calls, tool invocations, and retriever queries as traced spans. For LangGraph tools that need replay mocking, the first-class integration intercepts `ToolNode` execution so an output mock can be returned without running the tool against the real world.
**Canonical signatures:** [TypeScript reference](/reference/typescript#framework-integrations) · [Python reference](/reference/python#framework-integrations)
## Supported Languages
| Language | Method | Status |
| ---------- | ------------------------------------------------------------- | ----------------------- |
| TypeScript | `getLangGraphCallbackHandler()` | ✅ Supported |
| Python | `get_langgraph_callback_handler()` | ✅ Supported |
| TypeScript | `getLangGraphIntegration()` for replayable `ToolNode` tools | 🧪 Experimental (alpha) |
| Python | `get_langgraph_integration()` for replayable `ToolNode` tools | 🧪 Experimental (alpha) |
| Ruby | - | Not yet supported |
| Go | - | Not yet supported |
Working with plain LangChain (no LangGraph)? The same handler serves both, and the SDK exposes it under a LangChain name too: `getLangChainCallbackHandler()` / `get_langchain_callback_handler()` and the `BitfabLangChainCallbackHandler` class are aliases of their LangGraph counterparts. Use whichever name matches your stack; the behavior is identical.
## Quick Start
```typescript TypeScript theme={null}
import { Bitfab } from "@bitfab/sdk"
const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
const handler = bitfab.getLangGraphCallbackHandler("my-agent")
const result = await agent.invoke(
{ messages: [{ role: "user", content: "What's the weather?" }] },
{ callbacks: [handler] },
)
```
```python Python theme={null}
import os
from bitfab import Bitfab
bitfab = Bitfab(api_key=os.environ["BITFAB_API_KEY"])
handler = bitfab.get_langgraph_callback_handler("my-agent")
result = agent.invoke(
{"messages": [{"role": "user", "content": "What's the weather?"}]},
config={"callbacks": [handler]},
)
```
## Mock LangGraph tool calls on replay
**Experimental (alpha):** The tool replay integration uses public LangGraph APIs, but its API and matching behavior may change before it is stable. The callback-only tracing APIs above remain fully supported.
Callbacks observe tool execution but cannot prevent it. Use the first-class LangGraph integration when tools need output mocks during replay. It combines:
* `wrapTools()` in TypeScript, or LangGraph's native `ToolNode` hooks in Python, at the tool execution boundary
* `createInvoker()` / `create_invoker()` for the normal graph entry point; it adds the callback handler and replayable root together
* `callbackHandler` / `callback_handler` and `wrapInvoke()` / `wrap_invoke()` as low-level primitives for workflows with meaningful application work around the graph invocation
```typescript TypeScript theme={null}
import { Bitfab } from "@bitfab/sdk"
import { END, MessagesAnnotation, START, StateGraph } from "@langchain/langgraph"
import { ToolNode, toolsCondition } from "@langchain/langgraph/prebuilt"
const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
const integration = bitfab.getLangGraphIntegration("support-agent")
const tools = integration.wrapTools([lookupCustomer, searchDocs])
const modelWithTools = model.bindTools(tools)
const graph = new StateGraph(MessagesAnnotation)
.addNode("agent", async (state: typeof MessagesAnnotation.State) => ({
messages: [await modelWithTools.invoke(state.messages)],
}))
.addNode("tools", new ToolNode(tools))
.addEdge(START, "agent")
.addConditionalEdges("agent", toolsCondition, ["tools", END])
.addEdge("tools", "agent")
.compile()
const runSupportAgent = integration.createInvoker(graph)
await runSupportAgent(input)
await bitfab.replay("support-agent", runSupportAgent, { limit: 10 })
```
```python Python theme={null}
import os
from bitfab import Bitfab
from langgraph.graph import START, MessagesState, StateGraph
from langgraph.prebuilt import ToolNode, tools_condition
bitfab = Bitfab(api_key=os.environ["BITFAB_API_KEY"])
integration = bitfab.get_langgraph_integration("support-agent")
tools = [lookup_customer, search_docs]
model_with_tools = model.bind_tools(tools)
tool_node = ToolNode(
tools,
wrap_tool_call=integration.wrap_tool_call,
awrap_tool_call=integration.awrap_tool_call,
)
def call_model(state: MessagesState):
return {"messages": [model_with_tools.invoke(state["messages"])]}
builder = StateGraph(MessagesState)
builder.add_node("agent", call_model)
builder.add_node("tools", tool_node)
builder.add_edge(START, "agent")
builder.add_conditional_edges("agent", tools_condition)
builder.add_edge("tools", "agent")
graph = builder.compile()
run_support_agent = integration.create_invoker(graph)
run_support_agent(input)
bitfab.replay("support-agent", run_support_agent, limit=10)
```
Python uses LangGraph's native interception API directly: Bitfab supplies the two hook callables and the application remains responsible for constructing and configuring `ToolNode`. LangGraph JS does not currently expose equivalent tool-call hooks, so the TypeScript integration wraps each tool's public Runnable `invoke()` method and returns a normal tools array for `bindTools()` and `ToolNode`. It does not subclass `ToolNode` or depend on protected internals.
The invoker uses LangGraph's public `withConfig()` / `with_config()` API, so invocation-time configuration and existing callbacks are preserved. Only the graph input is recorded as the replay root input; callback managers and runtime configuration are not serialized. For an async-only Python graph, use `create_async_invoker()` and replay the returned async callable.
When input preparation or post-processing should be inside the same trace, keep using the low-level form: pass `callbackHandler` / `callback_handler` in the graph invocation config and wrap the full application function with `wrapInvoke()` / `wrap_invoke()`.
By default, every integration-managed tool is marked for replay mocking. Pass `mockToolsOnReplay: ["searchDocs"]` or `mock_tools_on_replay=["search_docs"]` to mark only selected tools; pass `false` to leave them unmarked. Replay's `mock: "all"` strategy and matching mock overrides can still substitute any integration-managed tool.
The integration records a serializable view of LangGraph's result and reconstructs a real `ToolMessage` or `Command` during replay. Every reconstructed tool message uses the current invocation's tool-call ID, so a recorded result remains valid when the model generates a different ID.
Recorded tool calls are matched by tool name and occurrence order within one replay item. If a tool is marked for replay mocking and that occurrence has no recorded output or matching override, Bitfab refuses to execute the live tool. LangGraph then applies the `ToolNode`'s configured error behavior. Use replay with `mock: "none"` only when running real tools is intentional.
### Current alpha limitations
* **Repeated tools are order-sensitive.** Calls are matched by tool name and occurrence order. If the same tool runs concurrently, or current code changes its call order, replay can select the wrong recorded occurrence. Use this integration first on graphs whose same-name tool calls have deterministic ordering.
* **Every tool call must cross the integration boundary.** In TypeScript, use the array returned by `wrapTools()` everywhere the graph references those tools. In Python, construct each replayable `ToolNode` with the integration's hooks. Calling the original tool directly is not intercepted and cannot be mocked.
* **TypeScript proxies the public `invoke()` method.** Standard LangChain tools work through this boundary. Custom tool objects that depend on exact object identity, custom proxy behavior, or a path that bypasses `invoke()` may be incompatible.
* **Tool inputs and outputs must serialize.** Native `ToolMessage` and `Command` results are supported. Streaming tool results and unusual custom return objects are not yet guaranteed to round-trip.
* **Fail-closed errors follow `ToolNode` error handling.** When tool-error handling is enabled, LangGraph may turn a missing-recording error into an error `ToolMessage` and let the graph continue. Configure the node to propagate tool errors if a missing recording should abort the replay.
* **The convenience invoker covers `invoke()` / `ainvoke()`.** For streaming, batch execution, or custom graph entry points, use `callbackHandler` / `callback_handler` and `wrapInvoke()` / `wrap_invoke()` directly.
Test the replay path with non-production dependencies before relying on it for a workflow with unsafe side-effects. Callback-only tracing does not provide this protection because callbacks run after tool execution has already started.
### Plain LangChain Chains
The handler works the same way on LangChain chains and runnables; pass it in `callbacks` when invoking:
```typescript TypeScript theme={null}
const handler = bitfab.getLangChainCallbackHandler("summarize-doc")
const result = await chain.invoke(
{ document: docText },
{ callbacks: [handler] },
)
```
```python Python theme={null}
handler = bitfab.get_langchain_callback_handler("summarize-doc")
result = chain.invoke(
{"document": doc_text},
config={"callbacks": [handler]},
)
```
## What Gets Captured
The callback handler hooks into LangChain's callback system and creates spans automatically:
| Event | Span Type | Captured Data |
| -------------------- | ---------- | ------------------------------------------------------------------------------------------- |
| Graph nodes (chains) | `agent` | Configured run name or node name, inputs, outputs, LangGraph metadata |
| Chat model calls | `llm` | Configured run name or model name, messages (role/content), token usage, LangGraph metadata |
| LLM calls | `llm` | Configured run name or model name, prompts, token usage, LangGraph metadata |
| Tool invocations | `function` | Configured run name or tool name, input, output |
| Retriever queries | `function` | Configured run name or retriever name, query, documents |
### Trace Lifecycle
The first callback in a framework invocation becomes the trace root. For full LangGraph runs this is usually a chain/graph node; for plain LangChain usage it can also be a direct chat model, LLM, tool, or retriever call. The handler registers that root immediately as a pending external trace, then completes it when the root callback ends, so long-running invocations can appear as in progress before their final output is available.
If the handler runs inside an active Bitfab `withSpan` / `@span` root, its spans attach to that outer trace instead. In that case the handler still records the framework span tree, but the outer Bitfab root owns final trace completion.
You do **not** need an outer `withSpan` / `@span` root when the workflow is just the LangGraph or LangChain invocation. Add one only when there is meaningful application work around `invoke()` that should be part of the same trace, such as input preparation, retrieval outside LangChain, post-processing, persistence, or downstream service calls. When you do add an outer root, use the same trace function key for the outer span and the callback handler.
### LangGraph Metadata
LangGraph-specific metadata is automatically extracted and stored as span context:
* `langgraph_step`: Current step number
* `langgraph_node`: Current node name
* `langgraph_triggers`: What triggered this node
* `langgraph_path`: Execution path
* `langgraph_checkpoint_ns`: Checkpoint namespace
### Token Usage
For LLM spans, token usage is captured from the LLM result. The handler prefers LangChain's standardized `usage_metadata` on each generation's message (which is also how usage arrives on the final aggregated chunk of streaming runs), then falls back to provider-native `response_metadata` shapes (OpenAI, Anthropic, Google Gemini / Vertex), and finally the legacy `llm_output.token_usage` location:
* `inputTokens`: Prompt tokens. For Anthropic, cache reads and cache creation are added back so the value reflects the true prompt size.
* `outputTokens`: Completion tokens
* `totalTokens`: Total tokens
* `cachedInputTokens`: Cached prompt tokens, from `usage_metadata.input_token_details.cache_read`, OpenAI `prompt_tokens_details.cached_tokens`, or Anthropic `cache_read_input_tokens`
* `model`: Model name (extracted from serialized config or metadata)
When a result has multiple generations, usage is summed across them. Only provider-reported numbers are recorded: if the provider reports nothing (for example OpenAI streaming without `stream_usage: true` / `stream_options: {"include_usage": true}`), the fields are left unset rather than estimated.
## TypeScript
### Installation
```bash theme={null}
npm install @bitfab/sdk @langchain/core @langchain/langgraph @langchain/openai zod
```
### Method Signature
```typescript theme={null}
bitfab.getLangGraphCallbackHandler(traceFunctionKey: string): BitfabLangGraphCallbackHandler
```
**Parameters:**
* `traceFunctionKey` (string, required): Groups all traces from this handler under one key in Bitfab
**Returns:** A `BitfabLangGraphCallbackHandler` that implements the LangChain callback handler interface (duck-typed, no `@langchain/core` dependency required).
### Usage
```typescript theme={null}
import { Bitfab } from "@bitfab/sdk"
const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
const handler = bitfab.getLangGraphCallbackHandler("weather-agent")
const result = await graph.invoke(
{ messages: [{ role: "user", content: "What's the weather in SF?" }] },
{ callbacks: [handler] },
)
```
### Callback Hooks
LangChain invokes these callback hooks on the handler:
| Method | Creates Span | Type |
| --------------------------- | -------------------------- | ---------- |
| `handleChainStart(...)` | Yes | `agent` |
| `handleChainEnd(...)` | Completes span | - |
| `handleChainError(...)` | Completes span with error | - |
| `handleChatModelStart(...)` | Yes | `llm` |
| `handleLLMStart(...)` | Yes | `llm` |
| `handleLLMEnd(...)` | Completes span with tokens | - |
| `handleLLMError(...)` | Completes span with error | - |
| `handleToolStart(...)` | Yes | `function` |
| `handleToolEnd(...)` | Completes span | - |
| `handleToolError(...)` | Completes span with error | - |
| `handleRetrieverStart(...)` | Yes | `function` |
| `handleRetrieverEnd(...)` | Completes span | - |
| `handleRetrieverError(...)` | Completes span with error | - |
### Nesting with Core Tracing
The handler integrates with Bitfab's span stack. If the application has meaningful work around the graph or chain invocation, create a same-key `withSpan` wrapper around that outer workflow and the LangGraph spans nest as children. Handler-only instrumentation is enough for a plain LangGraph / LangChain call:
```typescript theme={null}
const pipeline = bitfab.getFunction("my-pipeline")
const tracedRun = pipeline.withSpan(
{ name: "RunAgent", type: "agent" },
async (query: string) => {
const handler = pipeline.getLangGraphCallbackHandler() // same key as the root
return agent.invoke(
{ messages: [{ role: "user", content: query }] },
{ callbacks: [handler] },
)
},
)
await tracedRun("What's the weather?")
// LangGraph spans appear nested under the "RunAgent" span
```
**Use the same trace function key in both places.** Both `bitfab.getFunction(...)` and `bitfab.getLangGraphCallbackHandler(...)` take a key; pass the **same key** (`"my-pipeline"` above) to both. If you use two different keys here, the same flow will register as two separate overlapping trace functions in the dashboard, an anti-pattern to avoid.
### Error Handling
* **GraphBubbleUp**: LangGraph's internal interrupt mechanism. Detected automatically and completed silently (no error recorded).
* **All other errors**: Error message captured in the span. The handler never throws; all callbacks are wrapped in try/catch.
* **Missing serialized callback data**: Some framework versions can pass nullish serialized payloads. The handler still records the span using fallback names such as `chain`, `llm`, `tool`, or `retriever`.
* **Reusability**: The handler resets after each root span completes and can be reused across multiple invocations.
## Python
### Installation
```bash theme={null}
pip install "bitfab-py[langgraph]" langchain-openai
```
### Method Signature
```python theme={null}
bitfab.get_langgraph_callback_handler(trace_function_key: str) -> BitfabLangGraphCallbackHandler
```
**Parameters:**
* `trace_function_key` (str, required): Groups all traces from this handler under one key in Bitfab
**Returns:** A `BitfabLangGraphCallbackHandler` instance that extends `BaseCallbackHandler` from `langchain-core`.
### Usage
```python theme={null}
import os
from bitfab import Bitfab
bitfab = Bitfab(api_key=os.environ["BITFAB_API_KEY"])
handler = bitfab.get_langgraph_callback_handler("weather-agent")
result = graph.invoke(
{"messages": [{"role": "user", "content": "What's the weather in SF?"}]},
config={"callbacks": [handler]},
)
```
### Callback Methods
The handler implements these LangChain callback methods:
| Method | Creates Span | Type |
| -------------------------------------------------------------------- | -------------------------- | ---------- |
| `on_chain_start(serialized, inputs, *, run_id, parent_run_id?, ...)` | Yes | `agent` |
| `on_chain_end(outputs, *, run_id, ...)` | Completes span | - |
| `on_chain_error(error, *, run_id, ...)` | Completes span with error | - |
| `on_chat_model_start(serialized, messages, *, run_id, ...)` | Yes | `llm` |
| `on_llm_start(serialized, prompts, *, run_id, ...)` | Yes | `llm` |
| `on_llm_end(response, *, run_id, ...)` | Completes span with tokens | - |
| `on_llm_error(error, *, run_id, ...)` | Completes span with error | - |
| `on_tool_start(serialized, input_str, *, run_id, ...)` | Yes | `function` |
| `on_tool_end(output, *, run_id, ...)` | Completes span | - |
| `on_tool_error(error, *, run_id, ...)` | Completes span with error | - |
| `on_retriever_start(serialized, query, *, run_id, ...)` | Yes | `function` |
| `on_retriever_end(documents, *, run_id, ...)` | Completes span | - |
| `on_retriever_error(error, *, run_id, ...)` | Completes span with error | - |
### Nesting with Core Tracing
Bind the key once with `get_function` so the root and the handler share it (no repeated string to keep in sync). Use this only when the application has meaningful work around the graph or chain invocation; handler-only instrumentation is enough for a plain LangGraph / LangChain call:
```python theme={null}
pipeline = bitfab.get_function("my-pipeline")
@pipeline.span(type="agent")
def run_agent(query: str):
handler = pipeline.get_langgraph_callback_handler() # same key as the root
return agent.invoke(
{"messages": [{"role": "user", "content": query}]},
config={"callbacks": [handler]},
)
run_agent("What's the weather?")
# LangGraph spans appear nested under the "my-pipeline" span
```
**Use the same trace function key for the root and the handler.** Binding via `get_function` does this for you. The plain `@bitfab.span("my-pipeline")` / `bitfab.get_langgraph_callback_handler("my-pipeline")` forms work too, but you must pass the **same key** to both, otherwise the same flow registers as two separate overlapping trace functions in the dashboard, an anti-pattern to avoid.
### Error Handling
* **GraphBubbleUp**: LangGraph's internal interrupt mechanism. Detected automatically and completed silently (no error recorded).
* **All other errors**: `repr(error)` captured in the span. The handler never raises; all callbacks are wrapped in try/except.
* **Missing serialized callback data**: Some framework versions can pass nullish serialized payloads. The handler still records the span using fallback names such as `chain`, `llm`, `tool`, or `retriever`.
* **Reusability**: The handler resets after each root span completes and can be reused across multiple invocations.
## Replay
Callback-only graphs are replayable, even though no `@span` / `withSpan` root exists in the application code. The handler records the graph invocation as the root span, with the initial graph state as the recorded input. Their observed framework calls run live during replay. To replay, pass the handler's key plus a plain callable that re-invokes the graph (the SDK wraps it internally):
```python Python theme={null}
handler = bitfab.get_langgraph_callback_handler("weather-agent") # same key
def replay_weather_agent(state): # recorded state arrives as one argument
return agent.invoke(state, config={"callbacks": [handler]})
result = bitfab.replay("weather-agent", replay_weather_agent, limit=10)
```
```typescript TypeScript theme={null}
const handler = bitfab.getLangGraphCallbackHandler("weather-agent")
const replayWeatherAgent = async (
state: AgentState, // recorded state arrives as a single argument
) => agent.invoke(state, { callbacks: [handler] })
const result = await bitfab.replay("weather-agent", replayWeatherAgent, {
limit: 10,
})
```
The key is the only link between the handler-recorded production traces and the replay callable. Rebuild runtime wiring inside the callable (`config["configurable"]` values, dependency objects, API keys); the trace records only the graph state. Put unsafe calls behind replay-mockable marked spans; use a no-op only for a replay-only callback slot with no recorded call to mock. For LangGraph tools that must not run, use `getLangGraphIntegration()` / `get_langgraph_integration()` as shown above. On SDKs that predate explicit-key replay, wrap the callable under the same key yourself (Python: `@bitfab.span("weather-agent")` with a `(**state)` signature; TypeScript: `getFunction("weather-agent").withSpan(...)`). Full details: **Replaying handler-instrumented functions** in the [Python SDK](/python-sdk#replay) and [TypeScript SDK](/typescript-sdk#replay) pages.
# OpenAI Agents SDK
Source: https://docs.bitfab.ai/frameworks/openai-agents
Automatic tracing for OpenAI Agents SDK with Bitfab
Bitfab integrates with the [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/) via a tracing processor that automatically captures agent runs, tool calls, handoffs, and guardrails as traced spans - no manual `withSpan` or `@span` decorators needed. To make those runs replayable, a thin run wrapper (`wrapRun` / `wrap_run`) records a root span carrying the run input; see [Replayable runs with the run wrapper](#replayable-runs-with-the-run-wrapper).
**Canonical signatures:** [TypeScript reference](/reference/typescript#framework-integrations) · [Python reference](/reference/python#framework-integrations)
## Supported Languages
| Language | Method | Status |
| ---------- | --------------------------------------------------------------- | ----------------- |
| TypeScript | `getOpenAiTracingProcessor()` + `getOpenAiAgentHandler()` | ✅ Supported |
| Python | `get_openai_tracing_processor()` + `get_openai_agent_handler()` | ✅ Supported |
| Ruby | - | Not yet supported |
| Go | - | Not yet supported |
## Quick Start
```typescript TypeScript theme={null}
import { Bitfab } from "@bitfab/sdk"
import { addTraceProcessor, Agent } from "@openai/agents"
const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
// Register the processor once: it captures agent internals (LLM/tool/handoff spans).
addTraceProcessor(bitfab.getOpenAiTracingProcessor())
const agent = new Agent({ name: "my-agent", instructions: "..." })
const handler = bitfab.getOpenAiAgentHandler("my-agent")
// Use wrapRun (a drop-in for run) so the trace gets a replayable root.
const result = await handler.wrapRun(agent, "user input here")
```
```python Python theme={null}
import os
from bitfab import Bitfab
from agents import Agent, add_trace_processor
bitfab = Bitfab(api_key=os.environ["BITFAB_API_KEY"])
# Register the processor once: it captures agent internals (LLM/tool/handoff spans).
add_trace_processor(bitfab.get_openai_tracing_processor())
agent = Agent(name="my-agent", instructions="...")
handler = bitfab.get_openai_agent_handler("my-agent")
# Use wrap_run (a drop-in for Runner.run) so the trace gets a replayable root.
result = await handler.wrap_run(agent, "user input here")
```
Use `addTraceProcessor` / `add_trace_processor` (shown above) to **add** the Bitfab processor alongside any existing ones. The OpenAI Agents SDK also exposes `setTraceProcessors` / `set_trace_processors`, which **replaces** the entire processor list, including the SDK's default exporter to the OpenAI platform dashboard. Only use the `set` form if you want Bitfab to be the sole destination and intend to disable OpenAI's own tracing.
## TypeScript
### Installation
```bash theme={null}
npm install @bitfab/sdk @openai/agents
```
### Method Signature
```typescript theme={null}
bitfab.getOpenAiTracingProcessor(): BitfabOpenAITracingProcessor
```
**Parameters:** None.
**Returns:** A `BitfabOpenAITracingProcessor` instance that implements the OpenAI Agents SDK `TracingProcessor` interface.
### Usage
```typescript theme={null}
import { Bitfab } from "@bitfab/sdk"
import { addTraceProcessor, Agent } from "@openai/agents"
const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
// Processor captures agent internals; register once.
addTraceProcessor(bitfab.getOpenAiTracingProcessor())
const agent = new Agent({
name: "my-agent",
instructions: "You are a helpful assistant.",
model: "gpt-4o",
})
// wrapRun is a drop-in for run() that records a replayable root.
const handler = bitfab.getOpenAiAgentHandler("my-agent")
const result = await handler.wrapRun(agent, "What's the weather?")
```
### What Gets Captured
The processor implements the `TracingProcessor` interface and captures:
| Event | What's Captured |
| -------------- | ---------------------------------------------------------------------------------------------- |
| `onTraceStart` | Trace ID, workflow name, group ID |
| `onTraceEnd` | Trace completion with timing |
| `onSpanStart` | No payload is sent; the processor waits for the completed span |
| `onSpanEnd` | The authoritative completed snapshot: IDs, span type, name, output, error (if any), and timing |
Span types from the OpenAI Agents SDK (agent, function, generation, guardrail, handoff, etc.) are mapped to Bitfab span data automatically.
### Nesting with Core Tracing
If you wrap an agent invocation with `withSpan`, the OpenAI Agents spans nest as children:
```typescript theme={null}
const pipeline = bitfab.getFunction("my-pipeline")
const tracedRun = pipeline.withSpan(
{ name: "RunAgent", type: "agent" },
async (query: string) => {
return run(agent, query)
},
)
await tracedRun("What's the weather?")
// OpenAI Agents spans appear nested under the "RunAgent" span
```
### Error Handling
All processor callbacks are wrapped in try/catch - errors are logged but never thrown. Your agent execution is never affected by tracing failures.
## Python
### Installation
```bash theme={null}
pip install bitfab-py[openai-tracing]
```
The `openai-tracing` extra installs `openai-agents` as a dependency.
### Method Signature
```python theme={null}
bitfab.get_openai_tracing_processor() -> BitfabOpenAITracingProcessor
```
**Parameters:** None.
**Returns:** A `BitfabOpenAITracingProcessor` instance that implements the OpenAI Agents SDK `TracingProcessor` interface.
### Usage
```python theme={null}
import os
from bitfab import Bitfab
from agents import Agent, add_trace_processor
bitfab = Bitfab(api_key=os.environ["BITFAB_API_KEY"])
# Processor captures agent internals; register once.
add_trace_processor(bitfab.get_openai_tracing_processor())
agent = Agent(
name="my-agent",
instructions="You are a helpful assistant.",
model="gpt-4o",
)
# wrap_run is a drop-in for Runner.run that records a replayable root.
handler = bitfab.get_openai_agent_handler("my-agent")
result = await handler.wrap_run(agent, "What's the weather?")
```
### What Gets Captured
Same as TypeScript - the processor implements the `TracingProcessor` interface:
| Event | What's Captured |
| ---------------- | ---------------------------------------------------------------------------------------------- |
| `on_trace_start` | Trace ID, workflow name, group ID |
| `on_trace_end` | Trace completion with timing |
| `on_span_start` | No payload is sent; the processor waits for the completed span |
| `on_span_end` | The authoritative completed snapshot: IDs, span type, name, output, error (if any), and timing |
### Nesting with Core Tracing
```python theme={null}
@bitfab.span("my-pipeline", type="agent")
async def run_agent(query: str):
return await Runner.run(agent, query)
await run_agent("What's the weather?")
# OpenAI Agents spans appear nested under the "my-pipeline" span
```
### Error Handling
All processor callbacks are wrapped in try/except - errors are logged but never raised. Your agent execution is never affected by tracing failures. Traces flush automatically via `atexit` hook.
## Replayable runs with the run wrapper
The tracing processor captures the agent run for observability, but **the processor alone records a root span with no input**. The OpenAI Agents *agent* span is the trace root, and the run input never lands on it (only a response *child* span carries the model-request input). Replay re-runs the trace's root span against its recorded input, so a processor-only trace would replay with nothing to feed it.
**The run wrapper makes a run replayable with no hand-written root.** `getOpenAiAgentHandler(key).wrapRun` (TS) / `get_openai_agent_handler(key).wrap_run` (Python) is a drop-in for the run call: it opens a keyed root span carrying the run input and final output, and the processor's auto-captured spans nest underneath. Keep the processor registered for the internals; swap the run call for the wrapper.
```typescript TypeScript theme={null}
import { Bitfab } from "@bitfab/sdk"
import { addTraceProcessor, Agent } from "@openai/agents"
const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
addTraceProcessor(bitfab.getOpenAiTracingProcessor()) // captures internals
const agent = new Agent({ name: "my-agent", instructions: "..." })
const handler = bitfab.getOpenAiAgentHandler("my-agent-workflow")
// Swap run(agent, input) -> handler.wrapRun(agent, input)
const result = await handler.wrapRun(agent, "What is 21 + 21?")
// Replay re-runs each recorded input through the same wrapper, by key.
await bitfab.replay("my-agent-workflow", (input: string) =>
handler.wrapRun(agent, input),
)
```
```python Python theme={null}
import os
from bitfab import Bitfab
from agents import Agent, add_trace_processor
bitfab = Bitfab(api_key=os.environ["BITFAB_API_KEY"])
add_trace_processor(bitfab.get_openai_tracing_processor()) # captures internals
agent = Agent(name="my-agent", instructions="...")
handler = bitfab.get_openai_agent_handler("my-agent-workflow")
# Swap Runner.run(agent, input) -> handler.wrap_run(agent, input)
result = await handler.wrap_run(agent, "What is 21 + 21?")
# Replay re-runs each recorded input through the same wrapper, by key.
await bitfab.replay("my-agent-workflow", lambda input: handler.wrap_run(agent, input))
```
The wrapper's root carries the serializable run input, so replay re-runs each historical input through your code by key - no `@span`/`withSpan`-decorated function needs to exist. Rebuild runtime wiring inside the callable (agent construction, tools, API keys). Put unsafe calls behind replay-mockable marked spans; use a no-op only for a replay-only callback slot with no recorded call to mock.
**Alternative: a hand-written root.** When there is meaningful work *around* the run (input prep, orchestration, post-processing), wrap the whole workflow in a `withSpan` / `@span` root that takes the workflow input (the same wrap shown in [Nesting with Core Tracing](#nesting-with-core-tracing)). The processor's spans nest underneath it, and replay re-runs the root against its recorded input. Full details in the [Python SDK](/python-sdk#replay) and [TypeScript SDK](/typescript-sdk#replay) Replay sections.
## Streaming runs
Streamed runs are also traced: the run input lands on the root span and the final output is captured once the stream drains, so first-byte latency is untouched.
```typescript TypeScript theme={null}
// wrapRun is a drop-in for run(agent, input, { stream: true }).
const handler = bitfab.getOpenAiAgentHandler("my-agent-workflow")
const stream = await handler.wrapRun(agent, "Find X", { stream: true })
for await (const event of stream) {
// handle each streamed event
}
// The root's output is recorded once the stream completes.
```
```python Python theme={null}
# wrap_run_streamed is an async-generator drop-in for Runner.run_streamed:
# iterate it to consume the same stream events while the run is traced.
handler = bitfab.get_openai_agent_handler("my-agent-workflow")
async for event in handler.wrap_run_streamed(agent, "Find X"):
... # handle each streamed event
# The root's output (the run's final_output) is recorded once the stream drains.
```
In TypeScript, `wrapRun` returns the streamed run result (drained by the caller) and records the root in the background once it completes. In Python, `wrap_run_streamed` is itself an async generator that yields each event from `Runner.run_streamed(...).stream_events()`; the span stays open for the whole iteration so the processor's spans nest beneath the root.
To replay a recorded run as a regression with `bitfab.replay()`, use the non-streaming wrapper (`wrapRun` / `wrap_run`), which replay re-runs directly. `bitfab.replay()` does not drive the Python `wrap_run_streamed` async generator.
## Trace metadata
This integration exports its own trace under the same canonical trace id as the traced function, and that export carries metadata of its own. Metadata the caller recorded, through `seedTrace` / `seed_trace` or `setMetadata` / `set_metadata`, is merged onto that export and takes precedence per key, so a seeded case's provenance survives and reaches a later replay's `adaptInputs` / `adapt_inputs` hook as `ctx.metadata` / `ctx["metadata"]`.
Keys the integration set that the caller did not are kept, so `ctx.metadata` can carry keys the caller never wrote. A key set on both sides to different values keeps the caller's value and warns once per trace.
The merge needs both sides on the same canonical trace id, which is what running the integration nested inside a traced function gives them. A processor-only setup, with no enclosing traced function, resolves its own trace id and has no caller metadata to merge. An export that lands after the traced function returned, as a streamed run's does, still merges. Requires the TypeScript or Python SDK v0.52.3 or later.
# Framework Integrations
Source: https://docs.bitfab.ai/frameworks/overview
Automatic tracing for popular AI frameworks with Bitfab
Bitfab provides automatic tracing for popular AI frameworks. Instead of manually wrapping every function with `withSpan` or `@span`, use a framework-specific handler that hooks into the framework's execution lifecycle and captures spans automatically.
## Framework Support
| Framework | TypeScript | Python | Ruby | Go |
| ------------------------------------------------ | :-----------------: | :----: | :--: | :-: |
| [LangGraph / LangChain](/frameworks/langgraph) | ✅ | ✅ | - | - |
| [OpenAI Agents SDK](/frameworks/openai-agents) | ✅ | ✅ | - | - |
| [BAML](/frameworks/baml) | ✅ | ✅ | - | - |
| [Claude Agent SDK](/frameworks/claude-agent-sdk) | ✅ | ✅ | - | - |
| [Vercel AI SDK](/frameworks/vercel-ai-sdk) | ✅ | - | - | - |
| [Rivet step-body tracing](/frameworks/rivet) | Experimental recipe | - | - | - |
## Choose a Framework
Callback tracing for framework calls, plus experimental replayable LangGraph `ToolNode` execution in TypeScript and Python.
Trace processor that captures agent runs, tool calls, and handoffs from the OpenAI Agents SDK, plus a run wrapper that records a replayable root.
Auto-capture rendered prompts and LLM metadata (model, tokens, duration) from BAML function calls.
Handler that captures LLM turns, tool invocations, and subagent execution from the Claude Agent SDK.
Language model middleware that captures every generateText / streamText / generateObject / streamObject call, with full streaming support.
[Rivet tracing](/frameworks/rivet) uses a subtree root inside an existing workflow step. It records the business function's executed attempts. It does not use a framework callback handler or add replay mocking.
## How It Works
Each framework handler integrates with Bitfab's tracing pipeline:
1. **Create a handler** via the Bitfab client (e.g., `bitfab.getLangGraphCallbackHandler("my-agent")`)
2. **Pass the handler** to the framework (as a callback, processor, or hook)
3. **Spans are created automatically** for each execution step (LLM calls, tool invocations, etc.)
4. **Add core tracing only around real surrounding work** - callback handlers such as LangGraph / LangChain already create a replayable root for the framework invocation. Use a same-key `withSpan` or `@span` outer root only when input prep, non-framework retrieval, post-processing, persistence, or downstream calls should be part of the same trace.
Tracing-only handlers are fire-and-forget and do not affect application execution. The experimental LangGraph replay-aware execution hooks can substitute selected calls; if an expected tool recording is missing, the tool does not run and LangGraph applies the `ToolNode`'s configured error behavior.
# Rivet
Source: https://docs.bitfab.ai/frameworks/rivet
Trace business functions inside Rivet workflow steps
Trace the business function inside an existing Rivet step with `withTrace()`. Bitfab records its input, output, errors, and first-party calls. Rivet continues to own step scheduling and recovery.
This initial integration is a tracing recipe using the existing TypeScript SDK.
Subtree tracing is experimental. The actor fixture is verified with RivetKit
2.3.15 on Node.js using the TypeScript transform. Tool mocking and full actor
replay are separate work.
## Configure capture
Install `@bitfab/sdk` and configure a [TypeScript transform adapter](/typescript-sdk#experimental-subtree-tracing) for your server build. The transform records first-party functions called beneath `withTrace()`.
Without a transform, the root still records its input and output. Its descendants are absent. Use `withNode()` to name or type a discovered call. A node does not create a span by itself.
## Trace an existing step
Keep the step name and callback in place. Create the traced function inside the callback so the live step context stays in a closure. Pass the message as the recorded input.
```typescript theme={null}
import { Bitfab, getCurrentTrace } from "@bitfab/sdk"
const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
// Inside your existing workflow callback, after receiving a message:
await ctx.step("process-turn", async (step) => {
const runTurn = bitfab.withTrace(
"chat-turn",
{ name: "Process turn", type: "agent" },
async (input: { turnId: string; message: string }) => {
getCurrentTrace().setSessionId(step.actorId)
getCurrentTrace().setMetadata({
framework: "rivet",
actorId: step.actorId,
actorName: step.name,
turnId: input.turnId,
})
return processTurn(input)
},
)
return runTurn(message.body)
})
```
`processTurn` is your existing application function. It may close over the step context when it needs actor state or database access. Await all work that uses that context before the step callback returns. Rivet restricts actor capabilities to an active step callback. See [Rivet's step context rules](https://rivet.dev/workflows/docs/steps/).
Record message data as input. Store actor and turn identifiers in trace metadata. Passing a live context as input would record runtime internals without reconstructing actor state.
## Trace shape and retries
The root represents one execution attempt of the business function. Calls such as input preparation, model invocation, and persistence appear underneath it when their first-party code is transformed.
* Queue waits and workflow sleeps stay outside the root. Waiting for a message does not create a failed turn trace.
* A failed business function records its error and rethrows it. Rivet applies its existing retry policy.
* A retry creates a new trace for the new attempt. The actor session and turn metadata let you correlate attempts.
* A completed step restored from Rivet's history does not invoke the callback again. It produces no new business-function trace.
This boundary does not produce one continuous trace across actor eviction or process restart. It does not import Rivet's journal or display recovered model outputs as new model calls. A turn spanning several durable steps can trace each existing step body with the same actor session and turn metadata.
Do not merge or reorder durable steps to obtain a larger trace. Keep tracing inside the existing execution boundary. Leave workflow-level retry and error hooks unchanged.
## Verified scope
The real actor tests exercise queue delivery, local SQLite, actor state, concurrent actors, workflow sleep, and a failed model attempt followed by a successful retry. They assert root input/output, child parenting, actor metadata, session identity, and error capture. The same successful workload is tested with capture disabled.
The tests intercept Bitfab transport and use a model test double. They do not verify delivery to a hosted Bitfab service, forced eviction, process restart, streaming, or every Rivet version. Whole-workflow suspension status handling remains outside this step-body recipe.
Tracing does not make external operations safe to replay. This recipe adds no tool mocking. See [Instrumentation](/instrumentation) for capture configuration.
# Vercel AI SDK
Source: https://docs.bitfab.ai/frameworks/vercel-ai-sdk
Automatic tracing for the Vercel AI SDK with Bitfab
Bitfab integrates with the [Vercel AI SDK](https://ai-sdk.dev) (`ai`) via a [language model middleware](https://ai-sdk.dev/docs/ai-sdk-core/middleware). Wrap any model with `wrapLanguageModel` and Bitfab captures every `generateText`, `streamText`, `generateObject`, and `streamObject` call as a keyed `llm` span, no hand-written `withSpan` required. Streaming is captured without disturbing the live stream.
**Canonical signatures:** [TypeScript reference](/reference/typescript#framework-integrations)
## Supported Languages
| Language | Method | Status |
| ---------- | ------------------------- | ----------------------------------------------- |
| TypeScript | `getVercelAiMiddleware()` | ✅ Supported |
| Python | - | The Vercel AI SDK is JavaScript/TypeScript only |
| Ruby | - | Not yet supported |
| Go | - | Not yet supported |
## Quick Start
```typescript theme={null}
import { Bitfab } from "@bitfab/sdk"
import { openai } from "@ai-sdk/openai"
import { streamText, wrapLanguageModel } from "ai"
const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
// Wrap the model once; reuse it everywhere you call the AI SDK.
const model = wrapLanguageModel({
model: openai("gpt-4o"),
middleware: bitfab.getVercelAiMiddleware("chat-turn"),
})
const result = streamText({ model, messages })
return result.toUIMessageStreamResponse() // live stream untouched
```
Every call through `model` records a `chat-turn` span. The same middleware works for non-streaming calls:
```typescript theme={null}
const { text } = await generateText({ model, prompt: "Summarize this." })
```
## What Gets Captured
Each model call creates one `llm` span:
| Field | Captured Data |
| ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Input | The call parameters (the `prompt`/messages and settings), recorded as a single positional argument so the call replays by key |
| Output | A serializable summary: `{ text, toolCalls, usage, finishReason, model }`, where `model` is `{ provider, modelId }` - the provider/model that actually served the call |
| Type | `llm` |
Because the middleware hooks the model, it captures the resolved provider and model id on every call. That makes a multi-provider setup (one primary model, a fallback) fully observable: each span shows which provider answered.
For streaming calls, the assembled text, tool calls, and final usage are accumulated from the model's stream as the caller consumes it. Parts pass through as they are consumed. The span finalizes on normal completion, upstream error, or reader cancellation, retaining partial output and recording the error or cancellation. The original error and cancellation reason propagate unchanged. Once the stream ends or is cancelled, `flushTraces()` can finish delivering the model span; flushing does not consume an active stream.
The middleware hooks the **model**, so it captures one span per model call. A multi-step `streamText` run (tool calls that trigger follow-up model calls) records one span per step. To group those under a single root, wrap the whole call (see [Nesting with core tracing](#nesting-with-core-tracing)).
## TypeScript
### Installation
```bash theme={null}
npm install @bitfab/sdk
```
Requires the Vercel AI SDK (`ai` v5 or v6) as a peer dependency.
### Method Signature
```typescript theme={null}
bitfab.getVercelAiMiddleware(traceFunctionKey: string): BitfabLanguageModelMiddleware
```
**Parameters:**
* `traceFunctionKey` (string, required) - Groups all traces from this middleware under one key in Bitfab
**Returns:** A language model middleware object you pass to the AI SDK's `wrapLanguageModel`. It implements `wrapGenerate` and `wrapStream`; the AI SDK reads only those, so it drops straight in.
### Usage
```typescript theme={null}
import { Bitfab } from "@bitfab/sdk"
import { anthropic } from "@ai-sdk/anthropic"
import { generateText, tool, wrapLanguageModel } from "ai"
import { z } from "zod"
const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
const model = wrapLanguageModel({
model: anthropic("claude-sonnet-4-6"),
middleware: bitfab.getVercelAiMiddleware("weather-agent"),
})
const { text } = await generateText({
model,
prompt: "What's the weather in San Francisco?",
tools: {
getWeather: tool({
description: "Get the weather for a city",
inputSchema: z.object({ city: z.string() }),
execute: async ({ city }) => `Foggy in ${city}`,
}),
},
})
// Each model call (initial + post-tool) is traced under "weather-agent"
```
### Multiple providers and fallback
Because the middleware wraps the model, it is provider-agnostic: wrap each model you use (Anthropic, OpenAI, anything that implements the AI SDK model interface) with the same middleware. The provider/model that served each call is recorded on the span, so a Claude-primary, GPT-4o-fallback setup shows exactly who answered.
```typescript theme={null}
import { anthropic } from "@ai-sdk/anthropic"
import { openai } from "@ai-sdk/openai"
import { generateText, wrapLanguageModel } from "ai"
const middleware = bitfab.getVercelAiMiddleware("deal-brief")
const primary = wrapLanguageModel({
model: anthropic("claude-sonnet-4-6"),
middleware,
})
const fallback = wrapLanguageModel({
model: openai("gpt-4o"),
middleware,
})
async function generateBrief(prompt: string) {
try {
return await generateText({ model: primary, prompt })
} catch {
// The fallback call is traced too, with model = { provider: "openai", ... }.
return await generateText({ model: fallback, prompt })
}
}
```
Every call (primary or fallback) lands under the `deal-brief` key; the `model` field on each span tells the two apart.
### Next.js App Router
In a route handler (or server action), wrap the model once and use it as normal. For streaming, return the AI SDK response directly - the middleware's span finalizes as the stream drains, so it does not delay first byte:
```typescript theme={null}
// app/api/chat/route.ts
import { streamText } from "ai"
import { model } from "@/lib/ai" // wrapLanguageModel(...) with the middleware
export async function POST(req: Request) {
const { messages } = await req.json()
const result = streamText({ model, messages })
return result.toUIMessageStreamResponse()
}
```
For **non-streaming** calls in serverless, the span upload is fire-and-forget; if the function returns immediately the upload may be cut off. Keep it alive with `after()` so the span lands:
```typescript theme={null}
import { after } from "next/server"
import { flushTraces } from "@bitfab/sdk"
export async function POST(req: Request) {
// ... generateText(...) ...
after(() => flushTraces())
return Response.json(result)
}
```
(Streaming responses keep the function alive while the stream is consumed, so this is only needed for non-streaming calls.)
### Vercel Workflow SDK
The middleware works inside [Vercel Workflow SDK](https://useworkflow.dev) durable steps (`"use step"`). Each `"use step"` runs in its own invocation, so two things need care for durable, multi-step pipelines (extraction, research, deal-brief/memo generation):
1. **Flush before the step returns.** Span uploads are fire-and-forget. A non-streaming model call inside a step that returns immediately can have its upload cut off when the function freezes, so `await flushTraces()` at the end of the step.
2. **Stitch the steps into one run.** Steps run in separate invocations with no shared context, so each step records its own trace. Set a shared **session id** (the workflow run id) on every step, and Bitfab groups them as one run. Wrapping each step body in `withSpan` gives it a replayable root with the model-call `llm` spans nested underneath.
Keep the AI logic in `"use step"` functions (they have full Node.js access); the `"use workflow"` function only orchestrates.
```typescript theme={null}
import { MockLanguageModelV3 } from "ai/test"
import { generateText, wrapLanguageModel } from "ai"
import { anthropic } from "@ai-sdk/anthropic"
import { Bitfab, flushTraces, getCurrentTrace } from "@bitfab/sdk"
const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
async function extract(runId: string, doc: string): Promise {
"use step"
const model = wrapLanguageModel({
model: anthropic("claude-sonnet-4-6"),
middleware: bitfab.getVercelAiMiddleware("extract"),
})
const run = bitfab.withSpan("extract", { type: "agent" }, async (input: string) => {
getCurrentTrace().setSessionId(runId) // group every step under the run
const { text } = await generateText({ model, prompt: input })
return text
})
const result = await run(doc)
await flushTraces() // the step invocation ends here - make the span land
return result
}
// ...research(runId, ...) and writeMemo(runId, ...) follow the same shape...
export async function dealBriefPipeline(doc: string) {
"use workflow"
const runId = crypto.randomUUID()
const facts = await extract(runId, doc)
const notes = await research(runId, facts)
return await writeMemo(runId, notes)
}
```
Each step's model calls are captured; all steps share the `runId` session, so the whole pipeline reads as one run in Bitfab. (This pattern is verified end to end against the real Workflow runtime in the SDK's test suite.)
For agents built with `DurableAgent` from `@workflow/ai`, the model still flows through `wrapLanguageModel`, so wrap the model the same way and pass it to the agent.
### Nesting with Core Tracing
To group a multi-step run under one replayable root, wrap the AI SDK call in `withSpan`. The middleware spans nest underneath it, and the `finalizers.aiSdk` helper records a clean root output from the streaming result without consuming the live stream.
Bind the key once with `getFunction` so the root and the middleware share it (no repeated string to keep in sync):
```typescript theme={null}
import { finalizers } from "@bitfab/sdk"
const chatTurn = bitfab.getFunction("chat-turn")
const model = wrapLanguageModel({
model: openai("gpt-4o"),
middleware: chatTurn.getVercelAiMiddleware(), // same key as the root
})
const runChatTurn = chatTurn.withSpan(
{ type: "agent", finalize: finalizers.aiSdk },
(messages) => streamText({ model, messages }),
)
const result = runChatTurn(messages) // caller still gets the live stream
return result.toUIMessageStreamResponse()
```
Here the root `chat-turn` span carries the messages as input and `{ text, usage, finishReason, toolCalls }` as output, with each model call nested beneath it. The plain `bitfab.getVercelAiMiddleware("chat-turn")` / `bitfab.withSpan("chat-turn", …)` forms work too; `getFunction` just keys both from one place.
### Streaming
Streaming is handled automatically. The middleware passes the model's stream through a transform that accumulates the assembled output as the AI SDK reads it, so:
* The caller's stream is returned unchanged - every part is enqueued in order.
* First-byte latency is unaffected.
* The span is finalized once the stream emits its `finish` part.
You do not need `finalizers.aiSdk` for the per-call middleware spans; it is only for an optional outer `withSpan` root around the AI SDK call (see [Nesting with core tracing](#nesting-with-core-tracing)).
### Error Handling
* The middleware never throws; span capture is wrapped so a tracing failure cannot break the model call or the stream.
* If the client is disabled, the middleware is a transparent pass-through.
## Replay
Each model call records a keyed `llm` span carrying its parameters as input, so the call replays by key. With no outer `withSpan`, the span is the trace root and `replay(key, fn)` re-feeds each historical call's parameters to a callable that re-issues the model call.
If you wrap the AI SDK call in a `withSpan` with the **same key** (see [Nesting with core tracing](#nesting-with-core-tracing)), that outer span is the replayable root and the middleware spans nest under it. Full details: **Replaying functions** in the [TypeScript SDK](/typescript-sdk#replay) page.
# Go SDK
Source: https://docs.bitfab.ai/go-sdk
Instrument Go AI workflows as a span tree with Start/End and context propagation
The Bitfab Go SDK captures your AI function calls to automatically generate evaluations. Re-run your prompts with different models, parameters, and inputs to iterate faster.
Framework-native adapters (LangGraph, OpenAI Agents, BAML, Claude Agent SDK) are not yet available for Go. See [Frameworks overview](/frameworks/overview) for current coverage. Instrument Go code manually via `client.Span` or `client.Start`.
## Installation
```bash theme={null}
go get github.com/Project-White-Rabbit/bitfab-go
```
## Quick Start
```go theme={null}
package main
import (
"context"
"os"
"time"
bitfab "github.com/Project-White-Rabbit/bitfab-go"
)
func main() {
client := bitfab.NewClient(os.Getenv("BITFAB_API_KEY"))
ctx := context.Background()
client.Span(ctx, "my-service", func(ctx context.Context) (any, error) {
return map[string]any{"result": "..."}, nil
})
client.FlushTraces(5 * time.Second)
}
```
Need an API key? Get one from the [Bitfab dashboard](https://bitfab.ai/setup) or see the [API Keys guide](/api-keys) for detailed setup instructions.
Copy this prompt into your coding agent (tested with Cursor and Claude Code using Sonnet 4.5):
```text theme={null}
Modify existing Go code to add Bitfab tracing.
Do NOT browse or web search. Use ONLY the API described below.
Bitfab Go SDK (authoritative excerpt):
- Install: `go get github.com/Project-White-Rabbit/bitfab-go`
- Init:
import bitfab "github.com/Project-White-Rabbit/bitfab-go"
client := bitfab.NewClient(os.Getenv("BITFAB_API_KEY"))
- Start/End style (PREFERRED for existing functions):
func myFunc(ctx context.Context, arg1 string) (Result, error) {
ctx, span := client.Start(ctx, "", "", bitfab.WithType("function"))
defer span.End()
span.SetInput(arg1)
result, err := doWork(ctx, arg1)
if err != nil {
span.SetError(err)
return Result{}, err
}
span.SetOutput(result)
return result, nil
}
- Closure style (for inline code):
result, err := client.Span(ctx, "", func(ctx context.Context) (any, error) {
return doWork(ctx), nil
}, bitfab.WithName(""), bitfab.WithType("function"), bitfab.WithInput(args...))
- Fluent API:
fn := client.GetFunction("")
ctx, span := fn.Start(ctx, "", bitfab.WithType("function"))
defer span.End()
- Span types: "llm", "agent", "function", "guardrail", "handoff", "custom"
- Always pass ctx from Start or Span callback into nested calls for parent-child linking.
- Always call client.FlushTraces(5 * time.Second) before program exit.
Task:
1) Ensure the bitfab-go module is added (`go get github.com/Project-White-Rabbit/bitfab-go`).
2) Ensure a client is initialized with the API key.
3) Read the codebase and identify ALL AI workflows (LLM calls, agent runs, AI-driven decisions).
4) Present me with a numbered list of workflows you found. For each, describe:
- What it does
- Why it's worth instrumenting - what visibility tracing gives you into each step
5) After I choose which workflow(s) to instrument:
- Add Start/End at the top of each function (preferred) or wrap in Span closure
- Use SetInput/SetOutput/SetError to capture data
- Produce a SPAN TREE, not one span. The workflow root gets a span, and so does every step inside it:
each model call, each read of external state (DB query, HTTP GET, storage, vector search, cache),
each transform of the model output (parsing, validation, ranking, formatting), each retry or loop
iteration, and each external write. A single span around the outer function records one input and
one output for the whole workflow, which leaves per-step diagnosis with nothing to work on. Do not wrap trivial in-memory helpers or per-item work inside a large loop.
- Pass `ctx` through for nested span support
6) Do not change function signature, behavior, or return value. Minimal diff.
Output:
- First: your numbered list of workflows with why each is worth instrumenting
- After my selection: minimal diffs for go.mod and the instrumented functions
```
## Basic Configuration
```go theme={null}
// Default (production URL)
client := bitfab.NewClient(apiKey)
// Omit the key: NewClient reads BITFAB_API_KEY from the environment
client := bitfab.NewClient("")
// Set the key via an option instead of the positional argument
client := bitfab.NewClient("", bitfab.WithAPIKey(apiKey))
// Custom service URL
client := bitfab.NewClient(apiKey, bitfab.WithServiceURL("http://localhost:4000"))
// Disable tracing (functions still execute, but no spans are sent)
client := bitfab.NewClient(apiKey, bitfab.WithEnabled(false))
```
**Missing API key doesn't crash.** If the API key is missing, empty, or whitespace-only, the SDK automatically disables tracing and logs a warning. All instrumented functions still execute normally - no spans are sent, no errors are thrown. You don't need any conditional logic around the API key.
When no key is passed (empty argument and no `WithAPIKey`), the SDK reads `BITFAB_API_KEY` from the environment. Unlike the JS and Python SDKs, Go resolves the key eagerly in `NewClient`: clients are constructed explicitly (normally in `main`, after env has loaded), so there is no import-time construction-before-env trap to defer around. For standalone programs where a run that emits no traces should be a hard failure, use `WithStrict`:
```go theme={null}
// Panics in NewClient if no key resolves, instead of disabling quietly
client := bitfab.NewClient(apiKey, bitfab.WithStrict(true))
```
## Tracing
**Trace the whole workflow, not just its entrypoint.** A single `Start`/`End` around the outer function records one input and one output for everything inside it, which leaves per-step diagnosis and prompt iteration with nothing to work on. Spans exist only where you create them: nesting is automatic, but only between spans that exist.
Give a step its own span when any of these is true:
* **It calls a model.** Always. This is the span you iterate on, compare across experiments, and attach graders to.
* **It reads external mutable state** (DB query, HTTP `GET`, object storage, vector search, cache). These are the spans you will want to mock on replay.
* **It writes external state** (DB write, queue publish, email, charge, file write). Mark these to mock on replay so a replayed trace does not repeat the side effect.
* **It transforms the model output** (parsing, validation, ranking, formatting), so a quality regression points at the model or at your post-processing.
* **It retries or loops**, one span per attempt or iteration, so a trace shows how many attempts it really took.
Skip trivial in-memory helpers, per-item work inside a large loop (wrap the loop or the batch), and internals a [framework integration](/frameworks/overview) already captures.
Go replay defaults to `MockMarked`: closure-style child spans tagged with `WithMockOnReplay(true)` reuse their recorded output. Untagged child code still runs, so mark or override unsafe side effects before replaying a workflow. Manual `Start`/`End` spans cannot skip caller-owned code; use closure-style `Client.Span` for mockable boundaries.
Worked examples, replay-mocking decisions, and common pitfalls: [Instrumentation](/instrumentation).
## Replay Historical Traces
`Client.Replay` fetches historical root inputs, decodes them into your current Go function's parameter types, runs the items concurrently, and creates an experiment in Bitfab. Pass the same top-level function production uses. Replay accepts typed functions with an optional leading `context.Context` and an optional final `error` return.
```go theme={null}
result, err := client.Replay(
context.Background(),
"document-pipeline",
processDocument,
&bitfab.ReplayOptions{
Limit: 10,
MaxConcurrency: 4,
OnItemStart: func(progress bitfab.ReplayItemStartProgress) {
bitfab.ReportReplayProgress(progress)
},
OnItemFinish: func(progress bitfab.ReplayItemFinishProgress) {
bitfab.ReportReplayProgress(progress)
},
},
)
if err != nil {
var replayErr *bitfab.ReplayError
if errors.As(err, &replayErr) {
log.Printf("replay failed after %d items: %v", len(replayErr.Items), replayErr.Cause)
}
log.Fatal(err)
}
encoded, err := bitfab.SerializeReplayResult(result)
if err != nil {
log.Fatal(err)
}
fmt.Println(encoded)
```
Replay wraps each invocation in a root span under the supplied trace function key. If the production function also uses `Start`/`End`, those spans nest beneath the replay root and retain the same key. The replay context is passed as the function's leading `context.Context`, so continue passing the returned context through nested work.
`ReplayOptions` supports:
* `Limit` - recent traces to replay (default `5`, maximum `5000`). Omitted from the request when `TraceIDs` is set.
* `TraceIDs` - explicit historical trace IDs (maximum `100`). Passed alongside `DatasetID` or `DatasetIDs` they pin which members of that selection replay, and the server rejects any ID none of those datasets contains.
* `Name`, `ExperimentGroupID`, and `DatasetID` / `DatasetIDs` - experiment organization and attribution. `DatasetIDs` runs against several datasets at once, replaying the union of their traces.
* `GraderIDs` - graders attached directly to this experiment.
* `MaxConcurrency` - bounded worker count (default `10`).
* `CodeChangeDescription` and `CodeChangeFiles` - explicit code-change context. When files are omitted, Go checks `BITFAB_CODE_CHANGE_PATH` and then captures the rename-aware working-tree diff against trunk. `DisableCodeChangeCapture` opts one replay out.
* `Mock` and `MockOverrides` - recorded-output strategy and selective substitutions for closure-style child spans.
* `DBBranch` - non-nil enables a trace-time database branch; `MinCU`, `MaxCU`, and `WarmupSQL` tune it.
* `AdaptInputs` - reshape historical positional inputs before type decoding.
* `OnItemStart` and `OnItemFinish` - lifecycle callbacks with running totals. The finish callback normally includes the new server trace ID after a per-item flush; the final result is the fallback when that delivery cannot be confirmed. Callback panics are isolated from the replay.
When a signature changes, adapt the recorded values into the new positional shape:
```go theme={null}
type DocumentRequest struct {
Text string
Limit int
}
options := &bitfab.ReplayOptions{
AdaptInputs: func(inputs []any, ctx bitfab.AdaptContext) ([]any, error) {
return []any{DocumentRequest{Text: inputs[0].(string), Limit: 20}}, nil
},
}
result, err := client.Replay(ctx, "document-pipeline", processDocumentV2, options)
```
### Mock child spans
Mark an expensive or unsafe closure-style child span once in production code:
```go theme={null}
value, err := client.Span(
ctx,
"call-model",
func(ctx context.Context) (any, error) {
return callModel(ctx, prompt)
},
bitfab.WithName("CallModel"),
bitfab.WithInput(prompt),
bitfab.WithMockOnReplay(true),
bitfab.WithMockOutputType[ModelResponse](),
)
```
The default `MockMarked` strategy reuses that occurrence's historical output during replay. Repeated same-name calls are matched in call order; the SDK uses strictly increasing microsecond timestamps so rapid sibling calls remain distinguishable. `WithMockOutputType[T]()` decodes recorded JSON into the concrete Go type the caller expects; it is unnecessary for primitive or deliberately dynamic `any`/map outputs. `MockNone` runs real child code unless an override matches it; `MockAll` substitutes every recorded child occurrence. A selected occurrence that is missing fails the item closed.
Overrides run before the base strategy, with per-call overrides before client-registered overrides:
```go theme={null}
options := &bitfab.ReplayOptions{
Mock: bitfab.MockNone,
MockOverrides: []bitfab.MockOverride{{
Match: func(node bitfab.SpanNodeMeta) bool {
return node.TraceFunctionKey == "call-model"
},
Resolve: func(ctx bitfab.MockOverrideContext) (any, error) {
original, err := ctx.GetOriginalOutput()
if err != nil {
return nil, err
}
return improveRecordedAnswer(original), nil
},
}},
}
```
`GetOriginalOutput` fetches lazily and is memoized per item. Mocked uploads record whether the output came from `recorded` history or an `override`.
### Replay against trace-time database state
Every Go root trace now carries a no-I/O wall-clock snapshot reference. To request a historical branch, pass a non-nil `DBBranch` and read it inside the replayed function:
```go theme={null}
func processDocument(ctx context.Context, input Document) (Result, error) {
databaseURL := os.Getenv("DATABASE_URL")
if branch := bitfab.GetCurrentReplayBranch(ctx); branch != nil {
databaseURL = branch.DatabaseURL()
}
return runWithDatabase(ctx, databaseURL, input)
}
result, err := client.Replay(ctx, "document-pipeline", processDocument, &bitfab.ReplayOptions{
DBBranch: &bitfab.DBBranchOptions{
MinCU: 1,
MaxCU: 1,
WarmupSQL: "SELECT count(*) FROM documents",
},
})
```
Branch resolution runs inside the bounded replay workers, so `MaxConcurrency` also bounds live branches. A requested branch that cannot be resolved fails that item instead of silently using the live database. The SDK reports whether `DatabaseURL()` was obtained, includes provisioning timings on the trace and replay item, and releases every branch after the item; the connection string is excluded from branch JSON and formatting.
### Bound replay keys
Plain Go function values carry no tracing metadata, so explicit-key replay remains valid. When code wants a declared-key guard, bind the callable:
```go theme={null}
target := bitfab.BindReplayFunction("document-pipeline", processDocument)
result, err := client.Replay(ctx, "document-pipeline", target, options)
// Or bind once through the fluent API:
result, err = client.GetFunction("document-pipeline").Replay(ctx, processDocument, options)
```
Passing a bound target to `Client.Replay` under a different key fails before Bitfab selects any historical traces.
An item failure does not stop the batch. `ReplayItem.TraceError` retains an error or panic from the replayed function, `ReplayItem.ReplayError` retains setup/adaptation failures, and `ReplayItem.Error` is the compatible message. A persistence or finalization failure returns `*bitfab.ReplayError`, whose `Items`, `TestRunID`, `TestRunURL`, and `Cause` preserve the partial run.
Before finalizing, replay flushes the normal OpenTelemetry pipeline. Every carrier keeps a private delivery reference that never goes on the wire; successful ingestion acknowledges those references and returns each server `TraceID`. Replay finishes immediately when all carriers are acknowledged, and polls final trace status plus expected persisted span counts only when delivery is ambiguous. The final items include both the original trace lineage (`OriginalTraceID`, `OriginalSpanID`) and the new server `TraceID`, plus original duration/token/model measurements, replay duration, and replay token usage.
When `BITFAB_REPLAY_RESULT_PATH` is set, the SDK writes the same structured result JSON there automatically. This is how the Bitfab plugin captures results without parsing stdout.
### Custom (Recommended)
#### Using `Start`/`End` to Instrument Existing Functions
The recommended way to add tracing to existing functions without restructuring them:
```go theme={null}
func processOrder(ctx context.Context, orderID string, amount float64) (Order, error) {
ctx, span := client.Start(ctx, "order-processing", "ProcessOrder",
bitfab.WithType("agent"))
defer span.End()
span.SetInput(orderID, amount)
// Pass the returned ctx down so each step records as a child of this span
order, err := classifyOrder(ctx, orderID, amount)
if err != nil {
span.SetError(err)
return Order{}, err
}
span.SetOutput(order)
return order, nil
}
func classifyOrder(ctx context.Context, orderID string, amount float64) (Order, error) {
ctx, span := client.Start(ctx, "order-processing", "ClassifyOrder",
bitfab.WithType("llm"))
defer span.End()
span.SetInput(orderID, amount)
order, err := callModel(ctx, orderID, amount)
if err != nil {
span.SetError(err)
return Order{}, err
}
span.SetOutput(order)
return order, nil
}
```
Calling `processOrder` records one trace with two spans:
```
ProcessOrder
└── ClassifyOrder
```
Instrumenting only `processOrder` would record the same work as a single node. The nesting comes from passing the `ctx` that `Start` returns into the next step: hand a step the original `ctx` and its span becomes a separate root.
* `Start` returns an updated `context.Context` (for nested span propagation) and an `ActiveSpan`
* `defer span.End()` ensures the span is always completed and sent
* `SetInput` / `SetOutput` / `SetError` record data on the span
* `End` is idempotent - calling it multiple times is safe
#### Multi-File Projects
For projects with instrumented functions spread across multiple files, create a dedicated package that initializes the client and exposes a function handle. Import it wherever you need to instrument.
```go theme={null}
// pkg/tracing/tracing.go - single source of truth
package tracing
import (
"os"
bitfab "github.com/Project-White-Rabbit/bitfab-go"
)
var Client = bitfab.NewClient(os.Getenv("BITFAB_API_KEY"))
var OrderService = Client.GetFunction("order-processing")
```
```go theme={null}
// services/process_order.go
package services
import (
"context"
"myapp/pkg/tracing"
bitfab "github.com/Project-White-Rabbit/bitfab-go"
)
func ProcessOrder(ctx context.Context, orderID string) (Order, error) {
ctx, span := tracing.OrderService.Start(ctx, "ProcessOrder", bitfab.WithType("agent"))
defer span.End()
span.SetInput(orderID)
// Pass the ctx from Start so ValidateOrder nests under this span
order, err := ValidateOrder(ctx, orderID)
if err != nil {
span.SetError(err)
return Order{}, err
}
span.SetOutput(order)
return order, nil
}
```
```go theme={null}
// services/validate_order.go
package services
import (
"context"
"myapp/pkg/tracing"
bitfab "github.com/Project-White-Rabbit/bitfab-go"
)
func ValidateOrder(ctx context.Context, orderID string) (Order, error) {
ctx, span := tracing.OrderService.Start(ctx, "ValidateOrder", bitfab.WithType("guardrail"))
defer span.End()
span.SetInput(orderID)
order, err := doWork(ctx, orderID)
if err != nil {
span.SetError(err)
return Order{}, err
}
span.SetOutput(order)
return order, nil
}
```
```go theme={null}
// main.go
package main
import (
"context"
"myapp/pkg/tracing"
"myapp/services"
"time"
)
func main() {
defer tracing.Client.FlushTraces(5 * time.Second)
ctx := context.Background()
services.ProcessOrder(ctx, "order-123")
}
```
Spans from different files are automatically linked as parent-child when you pass `ctx` between instrumented functions.
#### Using `client.Span` (Closure Style)
Wrap inline code in a closure. Output is captured automatically from the return value. Use `WithInput` to record inputs:
```go theme={null}
result, err := client.Span(ctx, "order-processing", func(ctx context.Context) (any, error) {
return map[string]any{"order_id": "123", "total": 100}, nil
}, bitfab.WithName("ProcessOrder"), bitfab.WithType("function"),
bitfab.WithInput("order-123", 100))
```
#### Using `GetFunction` for a Static Trace Key
Bind a trace function key once, then create multiple spans without repeating it:
```go theme={null}
orderService := client.GetFunction("order-processing")
result, err := orderService.Span(ctx, func(ctx context.Context) (any, error) {
return map[string]any{"order_id": "123"}, nil
}, bitfab.WithName("ProcessOrder"), bitfab.WithType("function"))
result, err = orderService.Span(ctx, func(ctx context.Context) (any, error) {
return map[string]any{"valid": true}, nil
}, bitfab.WithName("ValidateOrder"), bitfab.WithType("guardrail"))
```
#### Automatic Nesting
Spans nest automatically when you pass `ctx` from the outer span callback:
```go theme={null}
client.Span(ctx, "pipeline", func(ctx context.Context) (any, error) {
// This span becomes a child of the outer span
client.Span(ctx, "pipeline", func(ctx context.Context) (any, error) {
// This span becomes a grandchild
return client.Span(ctx, "pipeline", func(ctx context.Context) (any, error) {
return map[string]any{"safe": true}, nil
}, bitfab.WithName("CheckFraud"), bitfab.WithType("guardrail"))
}, bitfab.WithName("Validate"), bitfab.WithType("guardrail"))
return map[string]any{"status": "done"}, nil
}, bitfab.WithName("Process"), bitfab.WithType("agent"))
```
For reusable helpers that should appear only inside an existing trace, pass `bitfab.WithCaptureWhen(bitfab.CaptureWhenNested)`:
```go theme={null}
helper := func(ctx context.Context) (any, error) {
return transform(ctx), nil
}
result, err := client.Span(
ctx,
"pipeline",
helper,
bitfab.WithName("Helper"),
bitfab.WithCaptureWhen(bitfab.CaptureWhenNested),
)
```
With no parent span in `ctx`, `helper` runs normally without creating a trace. Pass the child context from `Span` or `Start` to capture it as a nested span.
#### Span Options
**Parameters:**
* `traceFunctionKey` (required): Groups spans under a function key in Bitfab
* `WithName(name)` (optional): Display name. Defaults to the trace function key
* `WithType(spanType)` (optional): Span type. Defaults to `"custom"`. A label only, used to organize and filter spans in the dashboard; it does not change how the span is traced, replayed, or evaluated
* `WithFunctionName(name)` (optional): Override the function name in span data
* `WithInput(args...)` (optional, closure style only): Record input data. A single arg is stored directly; multiple args as a slice
* `WithCaptureWhen(CaptureWhenNested)` (optional): Capture only with an active parent span; otherwise run untraced. Defaults to `CaptureWhenAlways`; unknown values warn once and use that default
**Span Types:**
```go theme={null}
// Valid span types
"llm" // LLM calls
"agent" // Agent workflows
"function" // Function calls
"guardrail" // Safety checks
"handoff" // Human handoffs
"custom" // Default
```
**Examples:**
```go theme={null}
// LLM call
client.Span(ctx, "chat-service", func(ctx context.Context) (any, error) {
return callOpenAI(prompt), nil
}, bitfab.WithName("ChatCompletion"), bitfab.WithType("llm"))
// Safety check
client.Span(ctx, "safety-service", func(ctx context.Context) (any, error) {
return map[string]any{"safe": true}, nil
}, bitfab.WithName("ContentFilter"), bitfab.WithType("guardrail"))
```
#### Span Context
Use `span.AddContext()` on an `ActiveSpan` (Start/End style) to attach contextual key-value pairs at runtime - useful when context depends on computed values:
```go theme={null}
ctx, span := client.Start(ctx, "order-processing", "ProcessOrder",
bitfab.WithType("function"))
defer span.End()
requestID := generateRequestID()
span.AddContext(map[string]any{"request_id": requestID, "user_id": "u-123"})
```
Each `AddContext` call pushes the entire map as one entry. Multiple calls accumulate entries:
```go theme={null}
span.AddContext(map[string]any{"user_id": "u-123"})
span.AddContext(map[string]any{"request_id": "req-789"})
// Result: contexts: [{"user_id": "u-123"}, {"request_id": "req-789"}]
```
#### Span Prompt
Use `span.SetPrompt()` on an `ActiveSpan` (Start/End style) to set the prompt string on the current span. This is stored in `span_data.prompt` and is useful for capturing the exact prompt text sent to an LLM:
```go theme={null}
func classifyText(ctx context.Context, text string) (string, error) {
ctx, span := client.Start(ctx, "classification", "ClassifyText",
bitfab.WithType("llm"))
defer span.End()
span.SetInput(text)
prompt := fmt.Sprintf("Classify the following text: %s", text)
span.SetPrompt(prompt)
result, err := llm.Complete(prompt)
if err != nil {
span.SetError(err)
return "", err
}
span.SetOutput(result)
return result, nil
}
```
The prompt is metadata only. It records the prompt text for display and reference in the dashboard; it does not send the prompt to any model or change what the span executes. The last `SetPrompt` call wins -- it overwrites any previously set prompt on the span. Calling `SetPrompt` outside a span context is a no-op (it never crashes).
#### Trace Context
Use `bitfab.GetCurrentTrace(ctx)` to set context that applies to the entire trace (all spans within a single execution). This is useful for grouping traces by session or attaching trace-level metadata:
```go theme={null}
result, err := client.Span(ctx, "order-processing", func(ctx context.Context) (any, error) {
trace := bitfab.GetCurrentTrace(ctx)
// Set session ID (stored as database column, filterable in dashboard)
trace.SetSessionID("session-123")
// Name the trace (its title in Bitfab, searchable and filterable)
trace.SetName("Order 8f21")
// Set trace metadata (stored in raw trace data)
trace.SetMetadata(map[string]any{"region": "us-west-2", "environment": "production"})
// Add context entries (stored as key-value pairs, accumulates across calls)
trace.AddContext(map[string]any{"workflow": "checkout-flow", "batch_id": "batch-2024-01"})
return map[string]any{"status": "completed"}, nil
}, bitfab.WithName("ProcessOrder"), bitfab.WithType("function"))
```
* `SetSessionID(id)` - Groups traces by user session. Stored as a database column for efficient filtering.
* `SetName(name)` - The trace's title in Bitfab, and a field you can search and filter on. Use it for the case, ticket, or record the run is about. Unset, the trace is titled by its trace function key.
* `SetMetadata(map)` - Arbitrary key-value metadata on the trace. Merges with existing metadata.
* `AddContext(map)` - Key-value context entries. Accumulates across multiple calls.
* `TraceID()` - Returns the canonical Bitfab trace ID used for persisted lookups.
#### Read One Persisted Span
Fetch one span without loading the full trace. Repeated name matches default to the last span.
```go theme={null}
span, err := client.GetTraceSpan(ctx, traceID, bitfab.SpanLookup{
Name: "GenerateAnswer",
})
first, err := client.GetTraceSpan(ctx, traceID, bitfab.SpanLookup{
Name: "GenerateAnswer",
Occurrence: bitfab.FirstSpanOccurrence,
})
exact, err := client.GetTraceSpan(ctx, traceID, bitfab.SpanLookup{ID: spanID})
```
Both IDs are canonical Bitfab IDs; ingestion source IDs are not accepted. Use `bitfab.SpanOccurrenceAt(index)` for a zero-based occurrence. A missing trace or span returns `nil, nil`.
#### Dropping a Trace
Use `bitfab.GetCurrentTrace(ctx).Drop()` to discard the in-flight trace. Once flagged, spans that complete afterward are not uploaded at all, and the flag rides out on the completion payload, so when the trace completes the server scrubs any payloads that already raced out (the trace, its external trace, and sibling spans), deletes the archived S3 objects, and marks it `dropped` instead of `completed`, keeping only a skeleton audit row. Use it to discard runs you never want stored (health checks, test traffic) or a run you know carries sensitive data.
```go theme={null}
result, err := client.Span(ctx, "order-processing", func(ctx context.Context) (any, error) {
if isHealthCheck(ctx) {
bitfab.GetCurrentTrace(ctx).Drop()
}
return map[string]any{"status": "completed"}, nil
}, bitfab.WithName("ProcessOrder"), bitfab.WithType("function"))
```
* Safe to call outside a span (`GetCurrentTrace` returns `nil`, and `Drop()` is a no-op on a `nil` receiver), and never panics into your application.
#### Error Handling
Errors are captured in the span and returned to the caller:
```go theme={null}
result, err := client.Span(ctx, "risky-service", func(ctx context.Context) (any, error) {
return nil, errors.New("something went wrong")
}, bitfab.WithName("RiskyOperation"), bitfab.WithType("function"))
// err contains "something went wrong"
// The span records the error message and timing
```
Each error is classified by source. Errors returned by your function are recorded with `error_source: "code"`. SDK-internal errors are recorded with `source: "sdk"`. Both appear in the span's `errors` array in the Bitfab dashboard.
#### Flushing Traces
```go theme={null}
if !client.FlushTraces(5 * time.Second) { // Wait up to 5s for pending spans
log.Println("some spans were not delivered")
}
```
`FlushTraces` returns `false` when an export failed or the deadline expired.
Go does not have an automatic `atexit` hook. You must call `FlushTraces` before your program exits to ensure all pending spans are sent.
#### Closing a Client
```go theme={null}
client := bitfab.NewClient(os.Getenv("BITFAB_API_KEY"))
defer client.Close(5 * time.Second)
```
`Close` flushes and permanently shuts down the client's background delivery worker. It is idempotent, and returns `false` when an export failed or the deadline expired. A closed client no longer records spans, so prefer it over `FlushTraces` for the final flush in `main`.
## OpenTelemetry Transport
Span delivery runs on OpenTelemetry. This is an internal transport detail: `Span`, `Start`/`End`, and `GetFunction` still produce Bitfab spans with Bitfab trace IDs, and no OTel type appears in the Bitfab API. The SDK owns a private tracer provider per client and never touches your application's global OTel provider.
Each client lazily starts one bounded batching worker on its first span, so a client that never traces starts no worker.
Replay carriers use this same worker. A private carrier reference follows each
queued OTel span through batching and trimming but is excluded from the request
body. A successful ingestion response returns the server trace IDs and
acknowledges the references in that request. Replay polls the status endpoint
only as a fallback when the flush leaves delivery unconfirmed.
| Variable | Default | Purpose |
| -------------------------------- | --------- | --------------------------------------------- |
| `BITFAB_OTEL_EXPORT_CONCURRENCY` | `32` | Concurrent direct requests, `1` through `64`. |
| `BITFAB_OTEL_MAX_REQUEST_BYTES` | `3000000` | Request-size target. Can only be lowered. |
A single span may use up to 7,800,000 carrier bytes when its dedicated request
gzips below the 3,000,000-byte wire target and remains below the 8,000,000-byte
decompressed ingress limit. The carrier is the payload re-escaped into the OTLP
attribute. If it does not compress enough, compression is unavailable, or it
exceeds the raw ceiling, the SDK replaces its largest fields with
`` placeholders until it fits the
2,800,000-byte fallback budget. The trim is recorded on the span's `errors` so
the trace is flagged as incomplete.
Adopting OpenTelemetry raises the SDK's minimum Go version to **1.25**. The SDK links only the OpenTelemetry trace SDK, so protobuf and gRPC stay out of your build.
See [OpenTelemetry Transport Architecture](/otel-architecture) for the full design.
## Datasets
`client.Datasets` creates, reads, and modifies datasets programmatically, with the same operations your coding agent reaches through the Bitfab MCP tools. A dataset is a named bucket of traces under one trace function. Experiments replay against it and its graders score its members.
```go theme={null}
saved, err := client.Datasets.Save(ctx, bitfab.SaveDatasetParams{
TraceFunctionKey: "checkout-agent",
Name: "Refund failures",
Description: "Checkout runs where the refund was declined",
})
if err != nil {
return err
}
datasetID := saved.Dataset.ID
added, err := client.Datasets.AddTraces(ctx, datasetID, []string{traceID})
if err != nil {
return err
}
if len(added.SkippedTraceIDs) > 0 {
log.Println("not in this trace function:", added.SkippedTraceIDs)
}
membership, _ := client.Datasets.ListTraces(ctx, datasetID)
client.Datasets.AddGraders(ctx, datasetID, []string{graderID})
rerun, err := client.Datasets.RerunGraders(ctx, datasetID, bitfab.RerunGradersOptions{})
fmt.Println(rerun.Run.Status, rerun.Run.Result)
```
`Save` is an upsert on the dataset name within its trace function, so re-running a program does not accumulate duplicates. Membership and grader calls accept up to 100 ids and report ids they skipped rather than failing the whole call. `RemoveTraces` only drops membership. Traces are never deleted. `RerunGraders` waits for the run by default (90 seconds, configurable through `Timeout`) and returns whatever state it last saw. Set `NoWait` to return immediately and poll with `GetGraderRerun`. See the [reference](/reference/go#datasets) for every method and result type.
# Instrumentation
Source: https://docs.bitfab.ai/instrumentation
Choosing your spans: capture enough to see and evaluate what happened, and decide what gets mocked on replay
Start by choosing how a trace decides what to record.
| | Opt-out **(recommended)** | Opt-in |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- |
| The idea | Capture everything, then tune it down to what you want | Capture exactly what you want |
| What you write | One root, once: `trace()` / `withTrace()` | One per function: the `span()` decorator, or `withSpan()` to wrap |
| What it records | The root and every first-party call beneath it, unless excluded | Only the functions you mark |
| Tuning one step | `node()` / `withNode()` names, types, or mocks a discovered call without making it a boundary | Options on that function's own span |
| Replay boundaries | The root | Every span |
| Status | **Experimental.** TypeScript needs a [`@bitfab/transform` adapter](/typescript-sdk#experimental-subtree-tracing), Python needs 3.12+ | Stable. TypeScript, Python, Ruby, and Go |
**Start with opt-out.** One root records the whole subtree and you narrow from there, which beats deciding every boundary up front. Use opt-in for a small exact set of spans, on Ruby or Go, or on a runtime without subtree capture, where the root still records and its descendants are skipped.
Do not mix the two in one call stack. Both SDKs raise `MixedTracingError` where they meet, naming which surface was entered inside which.
Either way you end up with spans, and each carries two decisions: whether you can **see and evaluate** that step, and whether it **runs or serves its recording** on replay. One span around a whole workflow makes neither. One input, one output, one duration, and nothing inside it readable, gradable, or mockable.
## Goal 1: capture enough to see and evaluate
Opt-out captures these for you, so the list is what to check for in the trace and what deserves a `node`. Opt-in is the list of spans to write.
Give a step its own span when any of these is true:
* **It calls a model.** Always. This is the span you iterate on, compare across experiments, and attach graders to.
* **It reads external mutable state** (DB query, HTTP `GET`, object storage, vector search, cache). These are the spans you will want to mock on replay.
* **It writes external state** (DB write, queue publish, email, charge, file write). Mark these to mock on replay so a replayed trace does not repeat the side effect.
* **It transforms the model output** (parsing, validation, ranking, formatting), so a quality regression points at the model or at your post-processing.
* **It retries or loops**, one span per attempt or iteration, so a trace shows how many attempts it really took.
Skip trivial in-memory helpers, per-item work inside a large loop (wrap the loop or the batch), and internals a [framework integration](/frameworks/overview) already captures.
### What one span costs you
| You lose | Why |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------ |
| Failure localization | You see that the workflow failed, not which step failed or what it was handed. |
| Latency and token attribution | Model time and tokens are not separable from storage, parsing, and persistence. |
| Prompt iteration and grading | With no `llm` span there is no recorded model input and output to compare across runs or attach a grader to. |
| Replay mocking | Mocking applies only to descendants, so a trace with none can never mock anything. |
### Example
One `trace` on the root records everything it calls. Add `node` only where a step needs what the default did not give it, here typing the model call as `llm`:
```typescript TypeScript theme={null}
export class DocSummarizer {
@bitfab.trace("summarize-doc", { type: "agent" })
async summarizeDoc(id: string): Promise {
const doc = await this.readDoc(id)
const summary = await this.summarize(doc)
await this.persist(id, summary)
return summary
}
async readDoc(id: string): Promise {
return storage.read(id)
}
@bitfab.node({ type: "llm" })
async summarize(doc: Doc): Promise {
return generateObject({ model, schema, prompt: buildPrompt(doc) })
}
async persist(id: string, summary: Summary): Promise {
await db.summaries.insert(id, summary)
}
}
```
```python Python theme={null}
@bitfab.trace("summarize-doc", type="agent")
def summarize_doc(doc_id: str) -> Summary:
doc = read_doc(doc_id)
summary = summarize(doc)
persist(doc_id, summary)
return summary
def read_doc(doc_id: str) -> Doc:
return storage.read(doc_id)
@bitfab.node(type="llm")
def summarize(doc: Doc) -> Summary:
return client.responses.parse(model=MODEL, input=build_prompt(doc), text_format=Summary)
def persist(doc_id: str, summary: Summary) -> None:
db.summaries.insert(doc_id, summary)
```
```
summarize_doc
├── read_doc
├── summarize
└── persist
```
`read_doc` and `persist` are recorded without a decorator. Spans take their function's name, so TypeScript records `readDoc` where Python records `read_doc`. Pass `name` to `node` to rename one.
Spans nest by call stack in every SDK, so you never wire parents and children by hand.
`trace` and `node` are experimental. TypeScript needs a [`@bitfab/transform` adapter](/typescript-sdk#experimental-subtree-tracing) wired into your build (one config wrapper for Next.js, a plugin for Vite, webpack, esbuild and friends, or `--import @bitfab/transform/register` for direct Node). Python needs 3.12+. Without them the root still records and descendants are skipped, so a workflow degrades to a single-node trace rather than breaking.
The decorator forms need TypeScript 5+ with standard ECMAScript decorators and apply to class methods. Use `withTrace` and `withNode` for TypeScript 4.x, standalone functions, or functions from another library; same options.
## Goal 2: decide what gets mocked on replay
[Replay](/replay-mocking) re-runs recorded inputs through your current code. The root always runs for real; each descendant either runs for real or serves its recorded output. Steps that serve recordings are **mocks**, and you choose them at definition time with `mockOnReplay` / `mock_on_replay`, on a `node` under opt-out or on the span itself under opt-in. Only descendants can be mocks: a root always executes.
**Mock the world, run your code.** The point of a replay is to test a change against production scenarios, so anything that is not the change should be held fixed and cheap:
| Step | Replay | Why |
| ------------------------------------------------------- | ---------------- | --------------------------------------------------------------------------------------------------------- |
| External reads (DB, HTTP `GET`, storage, vector search) | **Mark to mock** | The recorded rows are the scenario. Re-reading gives you today's data, or fails without prod credentials. |
| External writes (DB write, queue, email, charge) | **Mark to mock** | Replaying a trace must not re-send the email or re-charge the card. |
| The model call | **Leave live** | Improving it is usually the point. Mock it and you are testing nothing. |
| Your own logic (parsing, validation, orchestration) | **Leave live** | This is the code under test. |
Mark a step when you define it and the default `"marked"` strategy does the rest. The two IO steps needed no decorator to be captured; they need one now, because marking is what the default did not give them:
```typescript TypeScript theme={null}
@bitfab.node({ mockOnReplay: true })
async readDoc(id: string): Promise {
return storage.read(id)
}
@bitfab.node({ mockOnReplay: true })
async persist(id: string, summary: Summary): Promise {
await db.summaries.insert(id, summary)
}
```
```python Python theme={null}
@bitfab.node(mock_on_replay=True)
def read_doc(doc_id: str) -> Doc:
return storage.read(doc_id)
@bitfab.node(mock_on_replay=True)
def persist(doc_id: str, summary: Summary) -> None:
db.summaries.insert(doc_id, summary)
```
Now a replay of `summarize-doc` feeds each historical document straight from its recording, runs your new prompt for real, and writes nothing.
To mock by default instead, set `mockOnReplayDefault` / `mock_on_replay_default` on the `trace`. Every configured node then mocks under `"marked"`, and the step under test opts back out with `mockOnReplay: false` / `mock_on_replay=False`.
Either way, **goal 2 only covers steps you captured in goal 1.** An excluded step, or one set to `capture: false`, has no recording to serve.
Replay also needs a root whose own arguments are JSON-serializable, since that is what it re-runs the trace against. Prefer an id, a request object, or a message list over a live connection, a stream, or a class instance.
## Pitfalls
### One span around the whole workflow
An opt-in mistake, and the most common one by a wide margin.
```typescript theme={null}
// ❌ Every step happens inside this span, so none of them is a span
@bitfab.span("summarize-doc", { type: "agent" })
async summarizeDoc(id: string): Promise {
const doc = await storage.read(id)
const summary = await generateObject({ model, schema, prompt: buildPrompt(doc) })
await db.summaries.insert(id, summary)
return summary
}
```
One node. No prompt recorded, no per-step timing, and replay re-reads storage and re-runs the insert every time. Swapping `span` for `trace` fixes it outright, which is the main reason to start with opt-out: the shape you get by default is the one you wanted.
### Calling the Vercel AI SDK without wrapping the model
```typescript theme={null}
// ❌ The model call is inside the span, but it is not a span
const chatTurn = bitfab.withSpan("chat-turn", () => generateText({ model, messages }))
// ✅ Wrap the model too, under the same key
const model = wrapLanguageModel({
model: openai("gpt-4o"),
middleware: bitfab.getVercelAiMiddleware("chat-turn"),
})
```
A `withSpan` around a function that calls a model does not create a span for the model call. See the [Vercel AI SDK integration](/frameworks/vercel-ai-sdk).
### Wrapping in an anonymous function
Applies to `withSpan`, which you reach for when the boundary is not a class method.
```typescript theme={null}
// ❌ The span records no input: the arrow function takes no arguments
const result = await bitfab.withSpan("summarize-doc", { name: "Summarize" }, async () => summarize(doc))()
// ✅ Pass the function directly so its arguments are captured
const result = await bitfab.withSpan("summarize-doc", { name: "Summarize" }, summarize)(doc)
```
An input-less root is not replayable, because replay has nothing to re-run the trace against.
### An unserializable root
```typescript theme={null}
// ❌ A live Request object cannot be recorded, so the trace cannot be replayed
export const handler = bitfab.withSpan("summarize-doc", { name: "handler" }, async (req: Request) => { ... })
// ✅ Move the boundary inward to the values the workflow actually needs
export const handler = async (req: Request) => summarizeDoc(await req.json())
```
The trace is still observable, but replay needs recorded inputs that round-trip.
### Mocking the thing you are changing
Mocking the model call, or running with `mock: "all"`, makes a replay return its recorded outputs and prove nothing about your change. Mock the setup around the step you are iterating on, never the step itself.
### Marking nothing, then replaying against production
With no span marked, the default strategy mocks nothing and every descendant runs for real. If the workflow writes, a replay repeats the write for every trace in the run. Mark writes before the first replay, not after.
### Hand-wrapping framework internals
If you use a [framework integration](/frameworks/overview), the handler, processor, or middleware already records the graph nodes, tools, and model calls. Wrapping them yourself produces duplicate spans. Add manual spans only for work above, alongside, or below the framework call.
## Self-check
Open a trace in the [web portal](/web-portal/overview):
* Is there more than one node? Under opt-out, a single-node trace usually means the transform or Python version is missing and descendants were skipped.
* Is there a span for the model call, carrying the prompt and the output, and typed `llm`?
* Is every external read and write its own span, marked to mock on replay?
# Integrations
Source: https://docs.bitfab.ai/integrations
Connect external services for tracing and observability
Bitfab integrates with popular LLM observability platforms to automatically trace your function calls. When configured, every SDK call is automatically logged to your connected services.
## Available Integrations
Enterprise-grade LLM evaluation and monitoring
Open-source LLM observability and tracing
## Braintrust
[Braintrust](https://braintrust.dev) is an enterprise-grade platform for LLM evaluation, monitoring, and experimentation.
### Setup
1. Go to **Settings > Integrations** in the Bitfab portal
2. Click **Connect** on the Braintrust card
3. Enter your **API Key** (starts with `sk-`)
4. Click **Test Connection** to verify
5. Click **Save**
### What Gets Traced
When Braintrust is connected, each SDK call creates a span with:
* **Input**: The function inputs you provided
* **Output**: The parsed result from the LLM
* **Model and provider**: Which LLM was used
* **Token usage**: Input, output, and cached tokens
* **Metadata**: Function ID, version ID, and Bitfab trace ID
* **Tags**: `bitfab` and `sdk-call`
### Viewing Traces
After making SDK calls, you can view spans in your Braintrust dashboard. The Bitfab trace ID is included in the metadata for cross-referencing.
## Langfuse
[Langfuse](https://langfuse.com) is an open-source LLM observability platform that provides tracing, analytics, and prompt management.
### Setup
1. Go to **Settings > Integrations** in the Bitfab portal
2. Click **Connect** on the Langfuse card
3. Enter your credentials:
* **Host**: Your Langfuse host (e.g., `https://cloud.langfuse.com` or your self-hosted URL)
* **Public Key**: Your Langfuse public key (starts with `pk-lf-`)
* **Secret Key**: Your Langfuse secret key (starts with `sk-lf-`)
4. Click **Test Connection** to verify
5. Click **Save**
### What Gets Traced
When Langfuse is connected, each SDK call creates a trace with:
* **Input**: The function inputs you provided
* **Output**: The parsed result from the LLM
* **Generation details**: Model, token usage, and timing
* **Metadata**: Function ID, version ID, and Bitfab trace ID
### Viewing Traces
After making SDK calls, you can view traces in your Langfuse dashboard. Each trace includes a direct link back to the corresponding trace in Bitfab.
## Managing Integrations
### Updating Credentials
To update an integration's credentials:
1. Go to **Settings > Integrations**
2. Click **Update** on the integration card
3. Enter new credentials
4. Click **Save**
### Disconnecting
To disconnect an integration:
1. Go to **Settings > Integrations**
2. Click **Disconnect** on the integration card
3. Confirm the disconnection
Disconnecting an integration stops future traces from being sent. Existing traces in the external service are not affected.
## Organization Scope
Integrations are configured per organization. Each organization can have its own Langfuse and Braintrust credentials, allowing different teams to use their own observability accounts.
# Introduction
Source: https://docs.bitfab.ai/introduction
Test code changes and run experiments on your AI features by replaying traces with mocks for unsafe side-effects and snapshots for database fidelity
## What is Bitfab?
Bitfab helps you test code changes and run experiments on your AI features by replaying traces while substituting unsafe side-effects. Put email sends, payment charges, database writes, and other unsafe calls behind replay-mockable spans; marked replay returns their recorded outputs instead of executing them. You capture traces with our SDK, then emulate and verify changes directly from your coding agent - with the [web portal](https://bitfab.ai) available to dive deeper when you need it.
Add the SDK to capture inputs, outputs, errors, and context for every AI call. 1-shot setup with the Claude plugin or MCP - your coding agent reads your codebase, finds AI workflows, instruments them, and creates registries for the replay command included with the SDK.
Replay production traces through your changed code. Bitfab's emulation engine reproduces cases that process messy real-world data through a mix of regular code and LLM calls, and handles unsafe side-effects: [replay mocking](/replay-mocking) returns recorded outputs for the calls you don't want to re-run, and [database snapshots](/db-branching) restore the database state at capture time.
Score every replay with [graders](/primitives/graders) and run [experiments](/primitives/experiments) across whole datasets. Each replayed trace comes back fixed, regressed, still passing, or still failing, so you know a change works before you ship.
## Quick Start
Install the CLI and run init:
```bash theme={null}
npx bitfab-cli init
```
This detects your editor (Claude Code, Codex, Cursor, or Amp), installs the Bitfab plugin, authenticates you, and launches the setup workflow. See the [Claude Code plugin](/claude-plugin) docs for full details.
For agents without a dedicated plugin, connect via [MCP](/mcp-setup) or visit the [setup page](https://bitfab.ai/setup).
## How It Works
### Capture
Your coding agent handles the entire setup:
1. **Instrument + create replay registries** - Reads your codebase, finds AI workflows, and in parallel adds tracing with minimal diffs and generates registry modules so you can regression-test against production data with the SDK's standard replay command
2. **Get traces** - Run your app and traces flow into Bitfab automatically. No traffic yet? [Seed traces](/typescript-sdk#seeding-traces) from cases you already hold, and replay selects them like captured ones
Instrumenting by hand instead? Read [Instrumentation](/instrumentation) first. It covers the two decisions you make by choosing spans: capturing enough to see and evaluate each step, and deciding what gets mocked on replay.
### Emulate
Once traces are flowing, test changes directly from your coding agent:
1. **Build a dataset** - Tell your agent which traces to test against, in plain language:
```
Build a dataset of checkout-agent failures from the last week
Random-sample 100 traces into a baseline dataset
Build a dataset of the agent turns above p90 latency
```
2. **Fix and replay** - Your agent suggests fixes and replays the dataset through your updated code. The emulation engine reproduces each case as it happened, even when the workflow mixes regular code and LLM calls over messy real-world data.
3. **Contain side-effects** - [Mock the calls you're not testing](/replay-mocking) so LLM calls, API calls, and database lookups return their recorded outputs instead of re-running for real, and [snapshot your database](/db-branching) so every replay sees the database state at capture time.
### Verify
Every replay gets scored, so verification is automatic:
1. **Grade replays** - [Graders](/primitives/graders) mark each replay pass or fail on the properties that matter
2. **Run experiments** - Replay a whole [dataset](/primitives/datasets) against your changed code and see each trace come back fixed, regressed, still passing, or still failing
3. **Dive deeper** - Use the [web portal](https://bitfab.ai) to inspect individual traces, compare outputs side-by-side, and explore patterns across your data
## Core Workflows
Traces, datasets, graders, and experiments: the four objects Bitfab is built from and how they chain into an iteration loop.
Choose your spans: capture enough to see and evaluate each step, and decide what gets mocked on replay.
Replace selected child calls with recorded outputs so you can iterate on the root behavior without paying for every dependency again.
Replay each trace against the database state at capture time.
# MCP Setup
Source: https://docs.bitfab.ai/mcp-setup
Connect your coding agent to Bitfab via Model Context Protocol
Bitfab provides an [MCP (Model Context Protocol)](https://modelcontextprotocol.io/) server that lets coding agents like Cursor, Claude Code, Codex, and Windsurf set up Bitfab tracing in your codebase automatically.
The MCP server provides tools for setup, trace inspection, datasets, labels, experiments, templates, and database-backed replay. Core setup tools include:
* **`get_bitfab_api_key`** -- Retrieves your API key for the agent to configure
* **`get_api_key_context`** -- Returns which Bitfab org the connection reads/writes to. Useful when data written by the agent isn't visible in Studio (the org may differ from the project's `BITFAB_API_KEY` or the org open in the browser)
* **`list_organizations`** -- Lists the organizations available to the signed-in user and marks the current plugin org
* **`get_database_connection_status`** -- Reports whether the org has connected a database for per-trace replay branching, identifies it as direct Neon or a managed Postgres mirror, and includes the pinned project name and ID for direct Neon connections
SDK setup content (install commands, initialization, instrumentation patterns, and replay) lives in the language-specific SDK reference pages on this site (TypeScript, Python, Ruby, Go). Agents fetch those pages directly rather than through an MCP tool.
## API Key Requirements
The MCP server requires authentication to access your organization's API keys and provide personalized setup guidance. You have two options:
1. **Automatic Authentication** (Recommended): Use the setup page which automatically configures authentication
2. **Manual Configuration**: Add authentication headers to your MCP client configuration
## Supported Languages
`bitfab` -- withSpan wrapper
`bitfab-py` -- @span decorator
`bitfab` -- bitfab\_span macro
`bitfab-go` -- Start/End spans
## Quick Setup
### CLI
The fastest way to get connected:
```bash theme={null}
npx bitfab-cli init
```
Detects your editor, installs the plugin, authenticates, and launches setup. Works with Claude Code, Codex, Cursor, and Amp.
Amp is the one host that does not use MCP for this. The [Amp plugin](/amp-plugin) registers the same tools directly with Amp under their bare names (`search_traces`, `get_traces`, ...), so you can ask Amp to query your traces the same way, without an MCP server.
The CLI checks that your editor's own agent is signed in before launching Bitfab setup (`claude auth login`, `codex login`, or `cursor agent login`).
The experimental terminal-native alternative keeps the complete setup workflow in the CLI:
```bash theme={null}
npx bitfab-cli init --v2
npx bitfab-cli setup --v2 instrument
npx bitfab-cli setup --v2 --diagram
```
This path uses the Claude Agent SDK directly instead of launching an editor agent. Set `ANTHROPIC_API_KEY` or configure a supported Agent SDK cloud provider first. Repository reads are automatic; setup decisions, edits, shell commands, API-key retrieval, and saved plan or template changes require terminal approval.
### Setup Page
For agents not supported by the CLI, or if you prefer manual configuration, use the [Bitfab setup page](https://bitfab.ai/setup):
1. **Choose your agent**: The page shows setup instructions for Claude Code, Cursor, Codex, and other MCP clients
2. **Follow the steps**: Claude Code uses a plugin install flow; other agents use MCP configuration with your API key embedded
3. **Copy and paste**: Each configuration is ready to use -- no manual key copying needed
The setup page provides:
* **Claude Code**: Plugin install commands that handle MCP setup and authentication automatically
* **Cursor**: One-click deep link installation
* **Other agents**: Ready-to-use MCP configuration with your API key embedded
## Manual Configuration
If you prefer to set up the MCP connection manually, you'll need to include your API key in the configuration.
### Getting Your API Key
1. **From the setup page**: Visit [bitfab.ai/setup](https://bitfab.ai/setup) and copy your API key
2. **From the dashboard**: Click your profile avatar → API Keys → Create/view your key
See the [API Keys guide](/api-keys) for detailed key management instructions.
### Cursor
Add this to your `.cursor/mcp.json` (project-level) or `~/.cursor/mcp.json` (global):
```json theme={null}
{
"mcpServers": {
"bitfab": {
"type": "streamable-http",
"url": "https://bitfab.ai/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
Replace `YOUR_API_KEY` with your actual API key, or use the pre-configured setup from the [setup page](https://bitfab.ai/setup).
### Claude Code
Install the Bitfab plugin, which automatically configures the MCP connection and provides slash commands for setup, labeling, and diagnostics.
Recommended: the CLI does all three steps in one command.
```bash theme={null}
npx bitfab-cli init
```
To do them yourself:
```bash theme={null}
claude plugin marketplace add Project-White-Rabbit/bitfab-claude-plugin
```
```
/plugin marketplace add Project-White-Rabbit/bitfab-claude-plugin
```
```bash theme={null}
claude plugin install bitfab@bitfab
```
```
/plugin install bitfab@bitfab
```
Start Claude Code and run `/bitfab:setup` to log in and instrument your codebase.
### Codex
Set your API key as an environment variable:
```bash theme={null}
export BITFAB_API_KEY="YOUR_API_KEY"
```
Then add this to `~/.codex/config.toml` (global) or `.codex/config.toml` (project-level):
```toml theme={null}
[mcp_servers.bitfab]
url = "https://bitfab.ai/mcp"
bearer_token_env_var = "BITFAB_API_KEY"
```
### Windsurf
Add this to your MCP configuration:
```json theme={null}
{
"mcpServers": {
"bitfab": {
"serverUrl": "https://bitfab.ai/mcp",
"headers": {
"Authorization": "Bearer YOUR_API_KEY"
}
}
}
}
```
## How It Works
Once connected, your coding agent can:
1. **Call `get_bitfab_api_key`** to retrieve your API key (requires your Bitfab session)
2. **Call `get_api_key_context`** to confirm which org the connection targets (helpful when data isn't showing up in the expected place)
3. **Fetch the language-specific SDK reference** from this site (e.g. `https://docs.bitfab.ai/typescript-sdk`) to get the full API surface, install commands, instrumentation patterns, and replay setup
4. **Follow the reference** to install the SDK, initialize the client, and instrument your functions
## Example Workflow
Ask your coding agent:
```
Set up Bitfab tracing for my project
```
The agent will:
1. Detect your project language
2. Call `get_bitfab_api_key` to get the API key and `get_api_key_context` to confirm the target org, then fetch the language-specific SDK reference from docs.bitfab.ai
3. Install the SDK
4. Read your codebase and identify all AI workflows
5. Present you with a list of workflows it found, and why each is worth instrumenting
6. Instrument whichever workflows you choose, one, several, or all
The agent presents options and lets you choose which workflows to instrument, so you stay in control. Each option explains what visibility tracing gives you into that workflow.
## Troubleshooting
### Authentication Issues
If your agent can't access the MCP tools:
1. **Check your API key**: Ensure you're using the correct API key from your dashboard
2. **Verify the header format**: Use `Authorization: Bearer YOUR_API_KEY` (note the space after "Bearer")
3. **Organization access**: Make sure you're logged into the correct organization in Bitfab
4. **Try the setup page**: Use the [automatic configuration](https://bitfab.ai/setup) to avoid manual errors
### Connection Problems
Common issues and solutions:
* **MCP server not found**: Verify the URL is `https://bitfab.ai/mcp`. If using Claude Code with the Bitfab plugin, run `/bitfab:setup` to reconfigure.
* **Tools not available**: Restart your coding agent after adding the MCP configuration
* **API key errors**: Generate a new API key from the [API Keys page](/api-keys)
* **Network issues**: Ensure your network allows HTTPS connections to bitfab.ai
### Getting Help
If you're still having trouble:
1. Check the [API Keys documentation](/api-keys) for key management
2. Visit the [setup page](https://bitfab.ai/setup) for automatic configuration
3. Contact support if the issue persists
# Organizations
Source: https://docs.bitfab.ai/organizations
Understand how organizations work and how resources are scoped in Bitfab
## Overview
Organizations are the top-level container for all your Bitfab resources. Everything in Bitfab is scoped to an organization:
* Functions
* Traces
* API Keys
* Tags
* Integrations
* Team members
## How Organizations Work
When you sign up for Bitfab, a personal organization is automatically created for you. You can also create additional organizations or be invited to join existing ones.
### Personal vs Team Organizations
| Type | Description |
| ------------ | -------------------------------------------------------------------- |
| **Personal** | Created automatically for each user. Ideal for individual projects. |
| **Team** | Created manually for collaboration. Multiple members can be invited. |
## Resource Scoping
All resources in Bitfab are scoped to the currently selected organization:
### Functions
Functions belong to a single organization. When you create a function, it's created in your currently selected organization.
```typescript theme={null}
// This call uses the API key's organization
const result = await client.call("ExtractName", { text: "John Doe" })
```
### API Keys
API keys are scoped to an organization. When you use an API key:
* You can only call functions in that organization
* Traces are recorded to that organization
* The key only has access to that organization's resources
### Traces
Traces are automatically associated with the organization of the API key used to make the call.
### Tags
Tags are organization-specific. Each organization has its own set of tags for organizing traces.
## Switching Organizations
If you belong to multiple organizations, you can switch between them:
1. Click the organization switcher in the header (next to your profile)
2. Select the organization you want to work in
3. The page will refresh with that organization's resources
Your current organization is shown in the header. Make sure you're in the correct organization before creating resources.
## Creating an Organization
To create a new organization:
1. Click the organization switcher in the header
2. Click **Create Organization**
3. Enter a name for your organization
4. Click **Create**
## Inviting Team Members
To invite members to your organization:
1. Click your profile avatar and select **Manage Organization**
2. Go to the **Members** tab
3. Click **Invite Member**
4. Enter their email address
5. Select a role
6. Click **Send Invite**
### Roles
| Role | Permissions |
| ---------- | ------------------------------------------------ |
| **Admin** | Full access to all resources, can manage members |
| **Member** | Can create and edit functions, view traces |
## Best Practices
* **Separate environments**: Create separate organizations for development, staging, and production
* **Use descriptive names**: Name organizations clearly (e.g., "Acme Corp - Production")
* **Manage access carefully**: Only invite members who need access
* **Check your organization**: Always verify you're in the correct organization before creating resources
## SDK Usage
When using the SDK, the organization is determined by the API key:
```typescript theme={null}
// This API key determines which organization is used
const client = new Bitfab({
apiKey: process.env.BITFAB_API_KEY, // Scoped to a specific organization
})
// All calls use that organization
const result = await client.call("MyFunction", { input: "value" })
```
To work with multiple organizations, use different API keys:
```typescript theme={null}
const prodClient = new Bitfab({
apiKey: process.env.BITFAB_PROD_API_KEY,
})
const devClient = new Bitfab({
apiKey: process.env.BITFAB_DEV_API_KEY,
})
```
# OpenTelemetry Transport Architecture
Source: https://docs.bitfab.ai/otel-architecture
How the TypeScript, Python, Ruby, and Go SDKs use OpenTelemetry for batching and delivery while preserving Bitfab trace identity and replay correctness
The TypeScript, Python, Ruby, and Go SDKs use OpenTelemetry as their internal
queueing, batching, and delivery engine. The public Bitfab API does not change:
`withSpan`, decorators, traced methods, manual spans, `Start`/`End`, and
framework handlers still create Bitfab spans with Bitfab trace IDs.
The design below is shared by all four SDKs. Where a name differs:
| Concept | TypeScript | Python | Ruby | Go |
| --------------------- | ---------------------------------------- | ------------------------------------------------- | -------------------------------------------- | ---------------------------------------- |
| Transport seam | `TraceTransport` | `bitfab.transport` | `Bitfab::Transport` | `traceTransport` |
| Per-client owner | `HttpClient` | `HttpClient` | `Bitfab::HttpClient` | `httpClient` |
| Submit entry points | `sendExternalSpan` / `sendExternalTrace` | `send_external_span` / `send_external_trace` | `send_external_span` / `send_external_trace` | `sendExternalSpan` / `sendExternalTrace` |
| Global flush | `flushTraces(timeoutMs)` | `flush_traces(timeout)` | `Bitfab.flush_traces(timeout:)` | `FlushTraces(timeout)` |
| Per-client close | `client.close(timeoutMs)` | `client.close()` or `with Bitfab(...) as client:` | `client.close(timeout:)` | `Close(timeout)` |
| Private provider type | `BasicTracerProvider` | `TracerProvider` | `TracerProvider` | `TracerProvider` |
OpenTelemetry is a transport dependency here, not the source of Bitfab's trace
model. This boundary lets the SDK use standard OTel infrastructure without
merging an application's OTel traces into Bitfab traces or exposing OTel types
through the Bitfab API.
## Design invariants
* Each `Bitfab` client's `HttpClient` lazily owns exactly one private OTel
provider and one bounded `BatchSpanProcessor`.
* Framework integrations created by a `Bitfab` client reuse that same
`HttpClient`; creating handlers does not create additional OTel workers.
* Live and replay traffic use the same pipeline.
* The private provider does not replace the application's global OTel provider
or export application spans.
* OTel owns queueing, count-based batching, scheduling, flushing, shutdown,
and its batch worker.
* Bitfab owns logical trace identity, payload validation, persistence,
idempotency, and replay readiness.
* Each carrier is encoded exactly once; a request body is assembled from those
encodings rather than re-encoded to measure its size.
## Components and ownership
| Component | Owns | Does not own |
| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | --------------------------------------------- |
| `withSpan`, decorators, traced methods, and framework handlers | Logical spans, trace IDs, parent relationships, replay metadata, serialized inputs and outputs | Network queues or exporter workers |
| `TraceTransport` | The small boundary used by every instrumentation path: submit, flush, and shutdown | Bitfab trace semantics |
| Private provider + `BatchSpanProcessor` | Bounded queue, batch worker, count threshold, schedule delay, force-flush, shutdown | Application OTel spans or the global provider |
| Bitfab OTLP/JSON exporter | OTLP/JSON encoding, count-and-byte-bounded request packing, retries, and bounded concurrent requests | Replay finalization |
| Bitfab OTLP ingress | Carrier extraction, schema validation, idempotent span/trace service dispatch, OTLP `partialSuccess` | SDK queueing |
| Replay status service | Final trace status and expected persisted-span-count barrier | Transport scheduling |
## Carrier spans versus Bitfab spans
Every Bitfab external-span, trace-completion, or `call()` BAML-trace payload is
placed inside an internal OTel carrier span with two attributes:
* `bitfab.operation`: `external_span`, `external_trace`, or `internal_trace`
* `bitfab.payload`: the JSON-encoded Bitfab payload
Detached trace patches (`client.getTrace(id)` / `client.get_trace(id)`) are not
carried here. They are trace-API calls, not telemetry, and go straight to
`PATCH /api/sdk/traces/{id}` like `getTraceSpan` / `get_trace_span`.
The carrier's OTel trace and span IDs are transport details. A carrier may
inherit the process's ambient OTel context because OTel context propagation is
process-wide, but Bitfab ingress does not use that topology to construct the
stored trace tree. It extracts `bitfab.payload` and uses the Bitfab
`sourceTraceId`, span ID, and parent ID inside that payload.
This is why an application can have its own OTel trace while recording one or
more Bitfab traces safely: the exporter and processor are private, and Bitfab
trace grouping comes from the payload rather than the carrier topology. Two
Bitfab trace IDs carried inside one ambient OTel trace remain two stored Bitfab
traces.
## Delivery
The flow is:
```text theme={null}
withSpan / decorator / traced method / framework handler
→ HttpClient.sendExternalSpan / sendExternalTrace / sendInternalTrace
→ private OTel BatchSpanProcessor
→ Bitfab OTLP/JSON exporter
→ POST /api/sdk/otel/v1/traces
→ existing external-span / external-trace / internal-trace services
→ Postgres
```
The exporter treats each OTel export batch as a candidate window, packs its
carriers into final HTTP requests bounded by both a carrier-count limit
(128 in TypeScript; eight in Python, Ruby, and Go) and the exact encoded request
size, and sends up to 32 complete requests
concurrently by default. Set
`BITFAB_OTEL_EXPORT_CONCURRENCY` to an integer from `1` through `64` to tune
that request concurrency. Invalid values warn once and fall back to
`32`.
Each carrier is encoded once, up front, and its byte count is carried alongside
it. Packing sums those counts against the request limit and the body is built by
concatenating the encodings, so the largest thing in a request, the
`bitfab.payload` string, is escaped exactly once no matter how many carriers a
window holds.
This preserves OTel's bounded queue, batch scheduling, force-flush, shutdown,
and single batch worker while restoring the small independent requests that
Bitfab's serverless ingress is designed to scale.
Each request retries transient network errors and the responses that mean the
server is busy or unreachable rather than unhappy with the payload: `429`,
`502`, `503`, `504`, and `500`. (`500` is not in OTLP's retryable set. Bitfab
ingestion answers every unhandled error with it, so a connection blip is
indistinguishable from a permanent fault and is worth a second attempt.) Every
other status is a verdict that will not change, and is not retried. Malformed
carriers can be rejected through the standard OTLP `partialSuccess` response
without affecting sibling requests.
A server that sends `Retry-After` is asking for a specific wait, and that wait
is honored exactly rather than shortened. It is refused only when it cannot be
served inside the export budget the batch processor enforces, since a wait
outliving that budget is killed mid-wait and loses the batch anyway. Half of the
remaining budget is reserved for the request itself, so a wait is only taken
when what follows it can still carry the send. Absent that instruction,
attempts back off exponentially with jitter, so a struggling server is not hit
on a fixed cadence and a fleet of clients does not return in lockstep. While a
throttle is active it applies to the whole transport, not only the request that
was refused.
Retrying is safe for spans and trace completions, which ingestion keys
idempotently on their source span and trace IDs. A `call()` BAML trace carries
no such key, so a retried batch holding one can create a duplicate trace, and a
request that times out client-side may still be persisted server-side. This is
a known limitation in every SDK that executes BAML; resolving it needs a
client-supplied idempotency key that ingestion dedupes on.
Bitfab ingress validates the complete request before processing accepted
carriers through a flat bounded worker pool. The pool defaults to eight
concurrent carriers and can be changed with
`BITFAB_OTLP_INGEST_CONCURRENCY`, up to a safety ceiling of 32. Carrier
processing does not rely on arrival order, matching the existing ingestion
contract for independently delivered SDK requests. The endpoint acknowledges
only after every accepted carrier has persisted.
## Batching and payload size
OTel's processor batches by count and time. The current SDK defaults are a
bounded queue of 8,192 carriers and a five-second scheduled export delay. OTel hands up to 512 carriers to one exporter call; the exporter immediately
packs that window into count-and-byte-bounded HTTP requests and sends those
complete requests concurrently. It does not own a second queue, timer, or
retained batch state.
Count alone is not safe for Bitfab because captured inputs and outputs vary
widely in size. Requests contain at most 128 carriers in TypeScript or eight in
Python, Ruby, and Go, and are packed using the exact encoded OTLP/JSON request
size under a target of approximately 3 MB. This
leaves headroom beneath Bitfab's fixed ingress request limit.
Set `BITFAB_OTEL_MAX_REQUEST_BYTES` to a positive integer no greater than
`3000000` to use a smaller request target for a proxy with a stricter limit. The value is read when the client's lazy OTel transport is
created. Invalid, non-positive, or larger values warn once and fall back to
`3000000`; the setting cannot raise the Bitfab-safe ceiling.
A carrier above the raw packing target gets its own request. If that request
compresses beneath the configured byte target and stays below the 8 MB
decompressed ingress ceiling, it is sent intact. Otherwise the carrier takes
the trimming fallback below. A `413 Payload Too Large` response from ingress
still fails the request. Tracing failures never interrupt the host application.
## The per-span payload budget
The SDK first preserves up to 7,800,000 bytes of a span's whole payload—its
input, output, contexts, prompt, and metadata together—when the resulting
single-span request compresses beneath the wire target. The ceiling leaves
room under ingress's 8 MB decompressed request limit. Both ceilings are
measured on the *carrier*, the payload as it is re-escaped into the OTLP
attribute, because that is the size the request limit applies to. Measuring the
payload itself would not bound anything: the second
escaping only has to escape `"` and `\\`, but it doubles each one, so
backslash-dense content (escaped JSON, Windows paths, regexes) grows up to 2x on
the way out. A payload sized to a 2.4 MB raw cap produced a 2.4 MB carrier as
prose and a 4.8 MB carrier as backslashes, and the exporter dropped the latter.
Measuring the carrier costs nothing on ordinary spans. Escaping can never shrink
a body and can at most double it, so a payload under half the budget always fits
and one past the budget never does; both are integer comparisons. Only a payload
between those bounds is scanned exactly, and that scan counts bytes without
allocating a second copy.
The cap is on the payload as a whole, not on each value. Capping values
independently cannot deliver this guarantee: two values that each fit can still
add up to a span that has to be dropped. It also means a single legitimately
large value, a document, a long message history, an agent state, may use the
entire budget when the rest of the span is small.
A payload past 7,800,000 carrier bytes is trimmed immediately. A smaller
oversized carrier is tried intact once; if gzip is unavailable or its request
still exceeds the wire target, the SDK trims it to the 2,800,000-byte fallback
budget and prepares the request again. Trimming replaces the largest fields
with `` placeholders, largest first, until
the encoded payload fits, and never trims fields that identify the span. Each
trim is recorded in the payload's own `errors`
under a `payload_budget` step, so the trace is flagged as incomplete and not
replayed as though it were faithful, and the SDK warns once in the host
application's logs.
## Compression
A packed request is gzipped before it is sent once its body reaches 8,192
bytes and gzip makes it smaller. Smaller or incompressible requests go
uncompressed. Ordinary batches are packed under the raw byte target and
compressed exactly once. Only a single carrier above that raw target checks the
prepared wire size; it is compressed a second time only when the first result
does not fit and the existing trimming fallback must rebuild it. A request
already above the 8,000,000-byte decompressed ceiling is trimmed before gzip,
because compression cannot make it admissible under that raw limit.
The SDKs use their runtimes' standard compression level; Python is explicitly
pinned to level 6 instead of `gzip.compress`'s CPU-heavier level-9 default.
Compression never runs on a thread the host application is waiting on: the
TypeScript SDK uses Node's asynchronous gzip, which runs on the libuv
threadpool, and Python, Ruby, and Go release the interpreter lock or compress
on their own goroutine.
Compression is best-effort: if it fails, the SDK sends the original body rather
than dropping the span. Set `BITFAB_DISABLE_COMPRESSION` to any value to send
everything uncompressed. In the browser, the TypeScript SDK uses
`CompressionStream` and falls back to uncompressed where it is unavailable.
## Burst limits
The queue holds 8,192 carriers, and a span and its trace completion are two
carriers. A burst that queues faster than the exporter drains loses the excess:
OTel drops the oldest carriers to keep the queue bounded, which is what keeps a
tracing backlog from growing into the host application's memory. A measured
example, against a sink that answers immediately: 20,000 spans submitted in
1.4 seconds queued 40,000 carriers, of which 8,704 were delivered. Staying
inside the queue's capacity delivers everything, exactly once.
Those drops happen before delivery is attempted, so the global flush still
reports success: its verdict covers what reached the exporter, not what the
queue had to discard. Treat it as "everything queued was delivered," and keep
bursts under the queue size when every span matters.
A second limit is server-side rather than client-side. A carrier that opens a
new trace costs substantially more to ingest than one appending a span to an
existing trace, so a burst of many independent traces can exceed the export
deadline even when the same span count spread across one trace tree would not.
Nested workloads are unaffected.
## Behavior against a slow endpoint
Delivery runs up to 32 concurrent requests, each with a 30-second
deadline, and retries a request that times out or returns a retryable status.
Against an endpoint slower than that concurrency (a laptop dev server, a
constrained self-hosted proxy), requests can exceed the deadline and be re-sent
while the first attempt is still being processed. Bitfab ingestion is
idempotent per span ID, so the retries cost bandwidth rather than duplicate
rows: one measured run re-sent 46% of its carriers and still produced exactly
one row per span. Lower `BITFAB_OTEL_EXPORT_CONCURRENCY` to match what the
endpoint can absorb.
## Replay persistence barrier
Replay is implemented in all four SDKs and uses the same live OpenTelemetry
pipeline as ordinary trace traffic.
The SDK tracks each OTel exporter's result so the global flush reports failure
when a batch fails or the deadline expires, instead of reporting only that the
processor queue drained. A flush by itself does not prove that Bitfab committed
the trace and all of its spans; the ingestion response acknowledges persistence.
Replay therefore adds a server-authoritative barrier without adding another
transport pipeline:
```text theme={null}
Run replay items concurrently
→ submit every span and trace completion through normal OTel pipelines
→ track a private reference for every span and trace-completion carrier
→ globally force-flush live Bitfab OTel transports
→ finish immediately when local delivery acknowledgments cover every carrier
→ otherwise poll POST /api/sdk/replay/status with expectedSpanCounts
→ wait until every trace is final and its persisted span count is sufficient
→ POST /api/sdk/replay/complete
```
For the fallback status request, the submission tracker counts unique source
span IDs rather than delivery attempts. This matches the server's idempotent
span key, so a duplicate submission cannot make replay wait for a duplicate
database row that should never exist.
Direct exporters keep each carrier's delivery reference with that carrier's
OTel span until encoding. A successful ingestion response acknowledges those
references locally, which proves the rows are durable without another server
request. A dropped span drops its reference with it; references are never held
or evicted independently of their spans. If delivery was not confirmed locally,
the authenticated status endpoint remains the authoritative fallback.
The existing status endpoint remains backward-compatible:
* Without `expectedSpanCounts`, it returns every trace mapping currently
associated with the test run.
* With `expectedSpanCounts`, it returns a mapping only when that trace has a
final status and at least the expected number of persisted spans.
If the barrier times out, replay raises and does not finalize an incomplete test
run. This is stronger than relying on OTel force-flush alone, while avoiding a
second server round trip when the ingestion acknowledgment is unambiguous.
## Framework integrations
`withSpan`, decorators, OpenAI Agents, LangGraph/LangChain, the Claude Agent
SDK, and (in TypeScript) the Vercel AI SDK middleware all submit through the
owning `Bitfab` client's `HttpClient` and the same `TraceTransport`
abstraction. Closing the client therefore flushes and shuts down the worker
used by its instrumented functions and framework integrations together.
Handlers instantiated directly, without a `Bitfab` client, own their
`HttpClient`; call the handler's close method (or, in Python, use it as a
context manager) to release that worker. Replay still flushes all live Bitfab
OTel transports in the process so multiple intentional `Bitfab` clients are
covered before server readiness is checked.
The LangGraph/LangChain handler keeps `langsmith:hidden` scheduler callbacks
only as local parenting state. It reparents visible children to the nearest
visible ancestor and does not submit hidden callbacks through OTel. The server
retains its hidden-span filter for older SDK versions.
Framework integrations do not create a separate replay exporter or special
single-item batches.
Ruby and Go ship no framework integrations; every traced method submits through
its client's `HttpClient`.
## Lifecycle and failure behavior
* Construction is lazy: a client that never sends a trace starts no OTel
worker.
* The global flush asks every active Bitfab processor in the process to flush
within one deadline and returns false on exporter failure or timeout.
* The per-client close flushes and permanently shuts down that client's
processor. In Python, `with Bitfab(...) as client:` closes it on context
exit.
* An exit handler gives remaining transports a bounded shutdown attempt.
* After `fork()`, OTel reinitializes its batch worker, queue, and export lock;
Bitfab rebuilds its own pipeline in the child, including its exporter.
* Submissions after shutdown are rejected without rebuilding an unowned
pipeline.
## TypeScript specifics
TypeScript shares the design above, with differences that follow from the
runtime:
* **The private provider is constructed with every option supplied
explicitly**, including its sampler and span limits. Left to its defaults it
would read `OTEL_*` environment variables, so a host application's global
sampler or attribute-length limit could silently drop or truncate Bitfab
payloads.
* **Every OTel package the SDK depends on is isomorphic**, so the SDK still
builds and runs in browsers.
* **A failed flush is a hint, not a verdict.** The export deadline can fire
while requests are still in flight and the server can still persist every one
of them, so replay does not fail on the flush result alone: it polls, and
fails only if the server itself never confirms. When both fail, the error
names the flush failure as the likely cause. Python raises on the flush
result instead.
* **No fork handling.** Node.js has no `fork()` that duplicates the event loop,
so there is no post-fork worker to rebuild.
* **Deferred spans are tracked per client.** A span recorded through `finalize`
reaches the transport only after its finalize chain settles, because the
caller gets the live stream back untouched. That deferred work is tracked per
client, so `client.close()` waits for its own spans without being failed by
another client's slow finalize; `flushTraces()` remains process-wide and
waits for all of it. Python's finalize runs inline, so it has nothing to
track.
* **A `beforeExit` handler** gives remaining transports a bounded shutdown
attempt, which is what lets a plain script exit with its spans delivered.
## Go specifics
Go shares the design above, with differences that follow from the runtime:
* **Adopting OTel raises the SDK's minimum Go version to 1.25**, the floor the
OpenTelemetry Go modules declare. Only the OpenTelemetry trace SDK is linked,
so protobuf and gRPC stay out of consumers' builds.
* **The request envelope is derived from an encoded empty request.** Go orders
map keys when marshalling, so the head and tail are cut from a marshalled
request with an empty span list rather than written by hand, keeping an
assembled body byte-identical to marshalling the whole request.
* **No fork handling.** A Go program that forks does not carry its goroutines
into the child, so there is no post-fork worker to rebuild.
* **No automatic last-chance flush.** Go has no `atexit` equivalent, so call
`defer client.Close(5 * time.Second)` in `main`, or `FlushTraces` before exit.
* **Replay uses the live pipeline and the shared acknowledgment mechanics.** A
wrapping span processor keeps each private carrier reference attached to its
queued OTel span without putting it on the wire. The HTTP response records
server trace IDs, and the exporter acknowledges the original references only
after successful ingestion. Replay polls with expected span counts only when
delivery remains ambiguous after flush.
* **Carriers start from a background context** rather than the ambient one, and
a submission is a non-blocking enqueue: spans and trace completions are queued
in the order they occur, and ingress processes carriers idempotently without
relying on arrival order.
## Deployment order
The replay status request is backward-compatible, but the expected-span
readiness behavior lives on the server. Deploy the Bitfab web/server change
before publishing an SDK version that relies on the stronger barrier. That
change shipped with the Python SDK's adoption, so the SDKs that adopted the
transport afterwards needed no further server deploy.
## Deliberate non-goals
* The SDK does not install a Bitfab processor or exporter on the application's
global OTel provider.
* The SDK does not infer Bitfab trace grouping from OTel carrier trace IDs.
* The SDK does not expose transport acknowledgments as public API or treat them
as the sole persistence authority.
See [TypeScript SDK](/typescript-sdk), [Python SDK](/python-sdk),
[Ruby SDK](/ruby-sdk), and [Go SDK](/go-sdk) for public usage, and
[HTTP endpoints](/reference/http) for the ingestion and replay-status
contracts.
# Datasets
Source: https://docs.bitfab.ai/primitives/datasets
Curate traces into named sets that graders score and experiments replay against
## Overview
A **dataset** is a named bucket of traces for one trace function: the traces your team reviews, that graders score, and that experiments replay against. It is the unit of evaluation in Bitfab. Without a dataset, a grader has nothing to run on and an experiment has nothing to replay.
Datasets usually start as a failure mode you want to fix, for example "Hallucinated order numbers" or "Long multi-turn threads", and grow as you find more traces that belong in them.
## Creating a Dataset
Ask your coding agent:
```
Build a dataset of failing checkout-agent traces from last week
```
The agent calls `save_dataset` with the trace function key, a short name, and an optional description, then populates it with `add_traces_to_dataset`. The portal's **New dataset** button shows this prompt rather than opening a form.
## Membership
* Adding traces is idempotent. Re-adding a trace already in the dataset does nothing.
* Traces are validated against the dataset's trace function key. Ids that belong to another organization or another trace function are skipped, and the response reports how many were added versus skipped.
* Up to 100 traces per call.
* Removing a trace from a dataset does not delete the trace.
Your agent can also add the trace you are currently working on, which is the normal way a dataset grows during a debugging session: fix the trace, confirm the fix by replaying it, then add it to the dataset so a future change cannot silently break it again.
### Filling a dataset before production has traces
A dataset holds traces, so a function that has not run in production yet has nothing to collect. If you already hold the cases somewhere else (a spreadsheet, a fixtures file, an export from another tool), seed them into Bitfab as traces and add those trace IDs to the dataset like any other:
```typescript TypeScript theme={null}
const traceId = await bitfab.seedTrace("checkout-agent", runCheckout, {
args: [caseRow.input],
metadata: { caseUid: caseRow.id },
name: caseRow.id,
})
await bitfab.datasets.addTraces(dataset.id, [traceId])
```
```python Python theme={null}
trace_id = bitfab.seed_trace(
"checkout-agent",
run_checkout,
args=[case_row["input"]],
metadata={"case_uid": case_row["id"]},
name=case_row["id"],
)
bitfab.datasets.add_traces(dataset["id"], [trace_id])
```
Seeded traces behave like captured ones from here on. Experiments replay them, graders score them, and the dataset's pass rate counts them. Seeding is available in the TypeScript and Python SDKs. See [Seeding traces](/typescript-sdk#seeding-traces) for TypeScript and [Seeding traces](/python-sdk#seeding-traces) for Python.
## Reading a Dataset
The datasets list for a trace function shows each dataset's name, trace count, and one verdict signal:
| Signal | When it shows |
| -------------------------- | ------------------------------------------------------------------------------------------- |
| **Grader pass-rate pills** | The dataset has graders attached. One pill per grader, filled green to red at its pass rate |
| **Human pass/fail** | No graders attached, but reviewers have labeled traces by hand |
| **Nothing** | The dataset is unlabeled and ungraded |
A dataset shows exactly one of these. Grader results and human labels are never mixed into a single verdict.
A grader that is attached but has not run yet reads as 0/0 rather than a wall of failures, so an empty pill means "not scored", not "everything failed".
Open a dataset to see its traces, the graders attached to it, and the experiments that have replayed it.
## Attaching Graders
The dataset detail view has a **Graders** section. Use **Manage** to attach or detach graders, and **Re-run** to grade the dataset's traces with the graders you select. Only one re-run runs at a time per dataset.
Graders must belong to the same organization and trace function as the dataset. Detaching a grader from a dataset never deletes the grader. See [Graders](/primitives/graders) for what the graders themselves do.
## Running Experiments Against a Dataset
An [experiment](/primitives/experiments) replays a dataset's traces against your current code and scores the results. The experiment inherits the dataset's runnable graders at completion, so attaching a grader to the dataset is usually all the setup an experiment needs.
## From the SDK
Every operation above is also available programmatically through `client.datasets` in the TypeScript, Python, Ruby, and Go SDKs, for scripts and CI jobs that build or maintain datasets without an agent in the loop.
```typescript TypeScript theme={null}
const { dataset } = await bitfab.datasets.save({
traceFunctionKey: "checkout-agent",
name: "Refund failures",
})
await bitfab.datasets.addTraces(dataset.id, traceIds)
await bitfab.datasets.addGraders(dataset.id, [graderId])
const { run } = await bitfab.datasets.rerunGraders(dataset.id)
```
```python Python theme={null}
dataset = bitfab.datasets.save("checkout-agent", "Refund failures")["dataset"]
bitfab.datasets.add_traces(dataset["id"], trace_ids)
bitfab.datasets.add_graders(dataset["id"], [grader_id])
run = bitfab.datasets.rerun_graders(dataset["id"])["run"]
```
```ruby Ruby theme={null}
dataset = client.datasets.save(trace_function_key: "checkout-agent", name: "Refund failures")["dataset"]
client.datasets.add_traces(dataset["id"], trace_ids)
client.datasets.add_graders(dataset["id"], [grader_id])
run = client.datasets.rerun_graders(dataset["id"])["run"]
```
```go Go theme={null}
saved, _ := client.Datasets.Save(ctx, bitfab.SaveDatasetParams{TraceFunctionKey: "checkout-agent", Name: "Refund failures"})
client.Datasets.AddTraces(ctx, saved.Dataset.ID, traceIDs)
client.Datasets.AddGraders(ctx, saved.Dataset.ID, []string{graderID})
rerun, _ := client.Datasets.RerunGraders(ctx, saved.Dataset.ID, bitfab.RerunGradersOptions{})
```
Dataset listing and trace-ID listing automatically fetch every page in all four SDKs and return the complete result. You can keep using `list` / `List` and `listTraces` / `list_traces` / `ListTraces` without managing cursors.
The methods are save, list, get, list traces, add and remove traces, add and remove graders, re-run graders, and read a re-run's status. They follow the membership rules on this page: adds are idempotent, foreign or wrong-function ids are reported as skipped rather than rejected, and removing a trace never deletes it. See the [TypeScript reference](/reference/typescript#datasets), the [Python reference](/reference/python#datasets), the [Ruby reference](/reference/ruby#datasets), the [Go reference](/reference/go#datasets), and the [HTTP API](/reference/http#datasets).
## Best Practices
* **Include both directions.** A dataset of only failures cannot tell you when a change makes things worse. Add the passing traces you refuse to regress.
* **Keep a dataset about one thing.** "Hallucinated order numbers" tells you what broke. "Bad outputs" does not.
* **Grow it from real failures.** Every production failure worth fixing is worth adding, so the fix is locked in.
* **Size it for the loop you want.** Small enough to replay quickly while iterating, large enough that the pass rate means something.
# Experiments
Source: https://docs.bitfab.ai/primitives/experiments
Replay a dataset against your changed code and score every trace as fixed, regressed, or unchanged
## Overview
An **experiment** is a replay of a dataset against your current code, scored against what the original traces did. It answers the only question that matters after a change: did this fix what I meant to fix, and did it break anything else?
Each experiment replays historical inputs through your code now, pairs every replay with the original trace it came from, and classifies the pair.
## Running an Experiment
Ask your coding agent:
```
Run an experiment on checkout-agent to improve pass rate
```
You can also drive replays directly from the SDK with `replay()`, which returns a `testRunId` and a `testRunUrl` for the resulting experiment. See the [TypeScript](/reference/typescript), [Python](/reference/python), and [Ruby](/reference/ruby) references for the call signature, and [Replay Mocking](/replay-mocking) for controlling which child spans re-run for real.
A trace replays only when its root span has serializable inputs, or it was captured through a framework handler. Traces whose inputs were stubbed as non-serializable at capture time cannot be replayed.
A dataset does not have to be built from production traffic. [Seeded traces](/typescript-sdk#seeding-traces), written from cases you already hold, replay in an experiment exactly as captured ones do and are graded the same way. They carry no database snapshot pin, so an experiment that needs [database branching](/db-branching) has to run against captured traces.
## Status
| Status | Meaning |
| ------------- | ----------------------------------------------- |
| **Pending** | Replays are still running |
| **Completed** | Every replay settled. Graders run at completion |
| **Failed** | The run itself failed |
## Verdicts
Each replayed trace is classified by comparing the original trace's label with the replay's:
| Verdict | Original | Replay | Read it as |
| ---------------------- | -------------------------------- | ------ | ------------------------------------------------------------- |
| **Fixed** | Fail | Pass | The change did what you wanted |
| **Regressed** | Pass | Fail | The change broke something that worked |
| **Still passing** | Pass | Pass | Protected, no change |
| **Still failing** | Fail | Fail | Not fixed yet |
| **Original unlabeled** | Not scored | Any | Nothing to compare against, but you can still A/B the outputs |
| **Unpaired** | No partner, or replay unresolved | | No comparison available |
| **Skipped** | | | The labeler deliberately declined to score it |
Fixed and regressed are the two numbers to read first. A run with 8 fixed and 3 regressed is not a win.
### Totals and Pass Rate
Alongside verdicts, an experiment tracks how each replay ended: succeeded, failed, errored, still pending, awaiting labels (the replay finished but no verdict has been written yet), and skipped. The pass rate counts only resolved traces, so awaiting-labels and skipped traces are excluded from the denominator rather than counted as failures.
## Graders on an Experiment
When an experiment completes, its grader set is the union of the graders attached directly to the experiment and the runnable graders on its dataset at that moment. That set is then frozen onto the experiment, so its results stay reproducible even if the dataset's attachments change later.
Attaching a grader to an already completed experiment records the assignment but does not regrade the traces it already produced. Use **Re-run** on the experiment, or launch a fresh replay. See [Graders](/primitives/graders).
## Experiment Groups
Every experiment launched in a single iteration shares an **experiment group**, so the runs you kicked off together stay together in the list instead of interleaving with older work. Groups are ordered by their most recent run.
Runs launched ad hoc, without a group, collect in a single **Ungrouped** bucket, which is ordered by recency like any other group.
Your agent can also group existing experiments after the fact, as long as they are not already split across different groups.
## Who Ran It, and From Where
Every experiment records who launched it and exactly which code ran, so a pass rate three weeks old still says whose change it measured.
Six fields are stored on the run:
| Field | What it is |
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Bitfab user id | The owner of the API key the replay authenticated with, resolved on the server rather than reported by the SDK, so it cannot be forged. A team sharing one key in CI attributes every run to that key's owner. |
| GitHub email | `user.email` from the launching machine's git config. |
| Branch | The checked-out branch, or empty on a detached head. |
| Commit sha | `HEAD` on the launching machine. |
| Base sha | The tree at `HEAD`: the committed state. |
| Experiment sha | The tree that actually ran, uncommitted and untracked files included. |
The last two are the useful pair. `HEAD` alone lies whenever the working tree is dirty, so the experiment sha is written from a throwaway git index and covers what was really on disk. That gives you two properties:
* `git diff ` is the change the experiment tested, computed by git with no tooling from us.
* Two runs with the same experiment sha ran byte-identical code. Two runs where base and experiment match ran a clean tree.
Writing that tree never touches your index or your stash: it uses a temporary `GIT_INDEX_FILE`, so a replay is safe to run mid-edit.
Everything is read locally from git, with no network call. `BITFAB_DISABLE_GIT_STATE` opts a process out, and older SDKs simply report nothing.
Both appear on the experiment row and in its code change view.
## Comparing Runs
An experiment always shows you the before and after: the original trace's input and output next to the replay's. Per-trace results also carry token usage for both sides, so you can see the cost and cache-read delta of a change rather than just its pass rate. That makes an experiment as useful for a cost optimization pass as it is for a correctness one.
## Best Practices
* **Change one thing per experiment.** Two changes in one run give you a pass rate you cannot attribute.
* **Read regressions before fixes.** The traces that used to pass are the ones your users already rely on.
* **Mock what you are not changing.** Keep the code under test real and mock the expensive setup around it. See [Replay Mocking](/replay-mocking).
* **Attach graders to the dataset, not the run.** Dataset graders flow into every experiment automatically, so each run is scored the same way.
# Graders
Source: https://docs.bitfab.ai/primitives/graders
Define automated pass/fail judges for a traced function and run them against datasets and experiments
## Overview
A **grader** is an automated judge that scores a trace as pass or fail against one specific property of a traced function, for example "the reply never invents an order number". Graders turn a quality bar you would otherwise check by hand into a check that runs across every trace in a dataset and every replay in an experiment.
Each grader belongs to one organization and one trace function. A trace function usually has several graders, one per property worth protecting.
Graders only run where they are attached. A grader with no dataset or experiment attached is a definition that never executes.
Graders are LLM-as-judge: a judge model reads the trace and returns pass or fail with a reason. You define what it should look for in plain language, not in code.
## Viewing Graders
Open a trace function and select **Graders**. The list shows one row per grader with:
| Column | Description |
| --------------- | ----------------------------------------------------- |
| **Grader** | Name, status badge, and the grader's evaluation focus |
| **Datasets** | How many datasets the grader is attached to |
| **Experiments** | How many experiments have run it |
Sort by status, most evaluated, recently updated, or name, and switch between the **Active** and **Archived** views. Click a row to open the grader.
### Grader Detail
The detail page shows the grader's criteria, its judge model, and every dataset and experiment it is attached to. From here you can change the judge model, archive the grader, or restore it.
### Status
| Status | Meaning |
| ------------ | ---------------------------------------------------------------- |
| **Active** | Usable. Runs when you grade an attached dataset or experiment |
| **Archived** | Retired. Hidden from lists and runs nowhere. Restore at any time |
## Creating and Editing Graders
Graders are authored from your coding agent through the Bitfab plugin, not from a form in the web portal. Ask your agent in plain language:
```
Add a grader to checkout-agent: the reply must never invent an order number
```
The agent calls the `save_grader` tool, which upserts. Pass a grader id to edit a specific grader, or reuse the name of an existing grader on the same trace function to update that one instead of creating a duplicate. The same tool renames graders, clears pass and fail criteria, selects the judge model, and archives or restores. `list_graders` enumerates what is already defined for a function.
See the [Claude Code](/claude-plugin), [Cursor](/cursor-plugin), or [Codex](/codex-plugin) plugin reference for the full tool signatures.
## Writing Good Criteria
A grader is defined by a small set of fields:
| Field | Purpose |
| ------------------------- | --------------------------------------------------------------------------------------- |
| **Evaluation focus** | The property a trace must satisfy, written as a concrete, checkable statement. Required |
| **Pass criteria** | What specifically counts as a pass |
| **Fail criteria** | What specifically counts as a fail |
| **Evaluation steps** | The order in which the judge should check things |
| **Acceptable variations** | Differences that must not be penalized, such as wording or formatting |
| **Span selection** | Which part of the trace the judge should look at |
One property per grader. A grader that checks tone, factuality, and formatting at once produces a fail you cannot act on. Three graders tell you which one broke.
Keep the focus checkable from the trace alone, and spell out acceptable variations so the judge does not fail a correct answer for phrasing it differently.
## Judge Model
Every grader runs on a selectable judge model:
| Model | Notes |
| -------------------- | ----------------------------------------------------- |
| **Gemini 2.5 Flash** | Default. Fast and cost-effective |
| **GPT-5.6 Sol** | OpenAI flagship. Strongest reasoning for hard grading |
| **GPT-5.6 Terra** | OpenAI balanced. Faster and cheaper than Sol |
| **Claude Opus 4.8** | Anthropic flagship reasoning model |
| **Gemini 3.1 Pro** | Google advanced reasoning model |
Change it from the dropdown on the grader detail page, or have your agent pass a model to `save_grader`. Start on the default and move up only for criteria the default judge gets wrong.
## Running Graders
### On a Dataset
Open a [dataset](/primitives/datasets) and use **Manage** in its Graders section to attach or detach graders. Your agent can do the same with `add_graders_to_dataset` and `remove_graders_from_dataset`. Only graders belonging to the same organization and trace function as the dataset can be attached, and detaching an assignment never deletes the grader itself.
Once graders are attached, **Re-run** grades the dataset's traces with the graders you select. Your agent can do the same with `rerun_graders_on_dataset`, which re-runs every attached grader unless you name a subset. Only one re-run can be in flight per dataset at a time; a second request joins the running one instead of starting another.
### On an Experiment
When an [experiment](/primitives/experiments) finishes replaying a dataset, its grader set is the union of the graders attached directly to that experiment and the dataset's runnable graders at that moment. That set is then frozen onto the experiment, so its results stay reproducible even if the dataset's attachments change afterwards.
Attaching a grader to an experiment that has already completed records the assignment but does not regrade the traces it already produced. Use **Re-run** on the experiment (or `rerun_graders_on_experiment` from your agent), or run a fresh replay. Only the experiment's completed replays are graded, never the originals they were compared against.
### On New Traces
Graders do not currently run automatically as new production traces arrive. Add traces to a dataset, attach the graders you want, and use **Re-run** (or `rerun_graders_on_dataset`) to grade them.
## Reading Results
* **Trace lists** show grader outcomes in the Grader Results column, and you can filter traces by pass or fail on a specific grader.
* **Dataset rows** show a pass-rate pill once graders are attached, splitting green to red at the trace's pass rate across those graders.
* **Trace detail** shows each grader's verdict with the reason it gave. A failing grader also produces a failure diagnostic explaining what went wrong.
* **Corrections** appear when a reviewer labels a trace as a pass that a grader failed. The trace shows who corrected it, which is the clearest signal that the grader's criteria need tightening.
Human labeling and automated grading are complementary: see the [Labeling guide](/web-portal/labeling) for how reviewers mark traces by hand.
## Archiving
Archive a grader from its row in the list or from its detail page. Archived graders are hidden from the default view, stop running everywhere, and keep their history. Switch the list to **Archived** to restore one.
## Best Practices
* **One property per grader**: a focused grader produces an actionable failure.
* **Attach before expecting results**: a grader runs only on the datasets and experiments it is attached to.
* **Re-run after editing criteria**: existing evaluations reflect the criteria in place when they ran.
* **Grade both directions**: a dataset of only failures cannot tell you whether a grader is too strict.
* **Escalate the judge model deliberately**: reach for a stronger model when the default judge is measurably wrong, not by default.
# Primitives Overview
Source: https://docs.bitfab.ai/primitives/overview
The four objects Bitfab is built from: traces, datasets, graders, and experiments
## Overview
Bitfab has four primitives. Everything in the product is one of them, or a view over them.
| Primitive | What it is | Created by |
| ----------------------------------------- | --------------------------------------------------------------------------------- | -------------------------------------- |
| **[Trace](/web-portal/overview)** | A recording of one workflow run: inputs, outputs, and every step inside | Your instrumented code, automatically |
| **[Dataset](/primitives/datasets)** | A named bucket of traces your team reviews and that experiments replay against | Your coding agent |
| **[Grader](/primitives/graders)** | An automated pass/fail check on a trace, attached to the datasets it should score | Your coding agent |
| **[Experiment](/primitives/experiments)** | A replay of a dataset against your changed code, scored against assertions | Your coding agent, or the SDK directly |
## How They Fit Together
The primitives chain into one loop:
1. **[Instrument](/instrumentation)** your code with the SDK. Every run of the instrumented function produces a **trace**.
2. **Collect** the interesting traces into a **dataset**, usually the failures you want to fix and the successes you refuse to break.
3. **Define graders** for the properties that matter, and attach them to the dataset. The dataset now has a pass rate instead of an opinion.
4. **Run an experiment**: replay the dataset against your changed code, and the graders score the replays. Each replayed trace comes back as fixed, regressed, still passing, or still failing.
5. Repeat, with the dataset growing every time you find a failure mode it did not cover.
Each primitive belongs to one organization and one **trace function key**, the name you passed to `getFunction()` or `get_function()`. A grader can only be attached to a dataset with the same key, and a dataset can only hold traces from that key.
## Creating Primitives
Datasets, graders, and experiments are created from your coding agent, in plain language, through the Bitfab plugin:
```
Build a dataset of failing checkout-agent traces from last week
Add a grader on checkout-agent that fails when the reply invents an order number
Run an experiment on checkout-agent to improve pass rate
```
The web portal deliberately has no create forms for these. Its **New** buttons explain the agent prompt to use instead. The portal is where you read, review, and manage what already exists.
Traces are the exception: they are recorded by your instrumented code, so you create them by instrumenting a workflow and running it.
Curate the traces you evaluate against
Score traces pass or fail automatically
Replay a dataset against changed code
## Where to Go Next
* Setting up tracing for the first time: [Introduction](/introduction) and your SDK reference.
* Driving the UI: [Web Portal Overview](/web-portal/overview).
* Reviewing traces by hand: [Labeling](/web-portal/labeling).
# Python SDK
Source: https://docs.bitfab.ai/python-sdk
Instrument Python AI workflows as a span tree with the @span decorator, then replay production traces against your current code
The Bitfab Python SDK captures your AI function calls to automatically generate evaluations. Re-run your prompts with different models, parameters, and inputs to iterate faster.
## Installation
Python 3.10 or newer is required.
```bash theme={null}
# pip
pip install bitfab-py
# Poetry
poetry add bitfab-py
# uv
uv add bitfab-py
```
## Quick Start
```python theme={null}
import os
from bitfab import Bitfab
bitfab = Bitfab(api_key=os.environ["BITFAB_API_KEY"])
```
Need an API key? Get one from the [Bitfab dashboard](https://bitfab.ai/setup) or see the [API Keys guide](/api-keys) for detailed setup instructions.
Copy this prompt into your coding agent (tested with Cursor and Claude Code using Sonnet 4.5):
```text theme={null}
Modify existing Python code to add Bitfab tracing.
Do NOT browse or web search. Use ONLY the API described below.
Bitfab Python SDK (authoritative excerpt):
- Install: `pip install bitfab-py` or `poetry add bitfab-py` or `uv add bitfab-py`
- Init:
import os
from bitfab import Bitfab
bitfab = Bitfab(api_key=os.environ["BITFAB_API_KEY"])
- Framework integrations:
If the codebase uses LangGraph or LangChain (`langgraph`, `langchain`, `langchain_core`, or other `langchain_*` packages), use the callback handler instead of manually decorating graph nodes, tools, retrievers, or model calls:
handler = bitfab.get_langgraph_callback_handler("")
graph.invoke(input, config={"callbacks": [handler]})
For plain LangChain chains, `get_langchain_callback_handler("")` is an identical alias. The handler records a replayable root from the framework input, so no outer `@span` root is needed when the workflow is just the graph/chain invocation. Add a same-key outer root only for meaningful surrounding application work. Callbacks tagged `langsmith:hidden` remain local for parent resolution and are not submitted through the OTel transport as Bitfab spans.
If replay must return recorded `ToolNode` results instead of executing live tools, use the Experimental (alpha) first-class integration:
integration = bitfab.get_langgraph_integration("")
tool_node = ToolNode(tools, wrap_tool_call=integration.wrap_tool_call, awrap_tool_call=integration.awrap_tool_call)
run_graph = integration.create_invoker(graph)
Every integration-managed tool is marked for recorded-output mocking by default. Use `mock_tools_on_replay=["tool_name"]` to select tools. Callback-only tool spans remain observable but cannot be short-circuited.
- Manual instrumentation (when no framework handler applies, or for meaningful work around a framework call):
# Declare trace function key once
my_service = bitfab.get_function("")
# Decorate methods with span
@my_service.span()
def method_name(): ...
# Or with options:
@my_service.span(name="DisplayName", type="function")
def method_name(): ...
# Span types: "llm", "agent", "function", "guardrail", "handoff", "custom"
- Decorator form ONLY; must be placed immediately ABOVE the `def` it instruments.
- DO NOT use context managers or manual span creation.
- DO NOT extract helper methods.
Task:
1) Ensure bitfab-py is installed and initialization exists.
2) Read the codebase and identify ALL AI workflows (LLM calls, agent runs, AI-driven decisions). Check for LangGraph/LangChain before planning manual instrumentation.
3) Present me with a numbered list of workflows you found. For each, describe:
- What it does
- Why it's worth instrumenting -- what visibility tracing gives you into each step
4) After I choose which workflow(s) to instrument:
- If it uses LangGraph/LangChain, add the Bitfab callback handler to the framework invoke config instead of decorating framework-managed internals. Use `get_function("").get_langgraph_callback_handler()` only when a same-key outer `@span` root is needed for surrounding application work.
- For non-framework workflows, create a function wrapper with `bitfab.get_function("")`
- Add `@my_service.span()` directly ABOVE each non-framework method's `def`
- Produce a SPAN TREE, not one span. The workflow root gets a span, and so does every step inside it:
each model call, each read of external state (DB query, HTTP GET, storage, vector search, cache),
each transform of the model output (parsing, validation, ranking, formatting), each retry or loop
iteration, and each external write. A single span around the outer function records one input and
one output for the whole workflow, which leaves replay mocking and per-step diagnosis with nothing
to work on. Do not wrap trivial in-memory helpers or per-item work inside a large loop.
- Mark reads and writes to mock on replay so a replayed trace does not repeat side effects
- Ensure the bitfab client is initialized and accessible
5) Do not change method signature, behavior, or return value. Minimal diff.
Output:
- First: your numbered list of workflows with why each is worth instrumenting
- After my selection: minimal diffs for dependencies, initialization, and the method changes
```
## Basic Configuration
```python theme={null}
Bitfab(api_key="...")
# Omit api_key entirely: the SDK reads BITFAB_API_KEY from the environment
Bitfab()
# Capture off: decorated functions still execute, but no spans are sent.
# Replay still records inside each item, and seed_trace records its one call.
Bitfab(api_key="...", capture_enabled=False)
```
**Missing API key doesn't crash.** If the API key is missing, empty, or whitespace-only, the SDK automatically disables tracing and logs a one-time warning at first use. All decorated functions still execute normally -- no spans are sent, no errors are thrown. You don't need any conditional logic around the API key.
### API key resolution
The key is resolved **lazily, the first time a span runs**, not when the client is constructed. This matters in scripts: a module that builds the client at import time can run before the entrypoint calls `load_dotenv()`, so a key read at construction would be empty even though it is set moments later. Resolving at first use reads the key after env loading has happened.
```python theme={null}
# Pass a callable to defer resolution explicitly (resolved at first use):
Bitfab(api_key=lambda: os.environ.get("BITFAB_API_KEY"))
```
When no key is passed (or it resolves empty), the SDK falls back to reading `BITFAB_API_KEY` from the environment, again at first use. For standalone scripts where a run that emits no traces should be treated as a failure rather than silently skipped, set `strict`:
```python theme={null}
# Raises on the first traced call if no key resolves, instead of disabling quietly
Bitfab(api_key=os.environ.get("BITFAB_API_KEY"), strict=True)
```
If you load env with dotenv in a script, prefer loading it before the module graph is imported, for example `dotenv run -- python script.py`, so every module-level read sees the key.
## Tracing
**Trace the whole workflow, not just its entrypoint.** A single `@span` around the outer function records one input and one output for everything inside it, which leaves replay mocking, per-step diagnosis, and prompt iteration with nothing to work on. Spans exist only where you create them: nesting is automatic, but only between spans that exist.
Give a step its own span when any of these is true:
* **It calls a model.** Always. This is the span you iterate on, compare across experiments, and attach graders to.
* **It reads external mutable state** (DB query, HTTP `GET`, object storage, vector search, cache). These are the spans you will want to mock on replay.
* **It writes external state** (DB write, queue publish, email, charge, file write). Mark these to mock on replay so a replayed trace does not repeat the side effect.
* **It transforms the model output** (parsing, validation, ranking, formatting), so a quality regression points at the model or at your post-processing.
* **It retries or loops**, one span per attempt or iteration, so a trace shows how many attempts it really took.
Skip trivial in-memory helpers, per-item work inside a large loop (wrap the loop or the batch), and internals a [framework integration](/frameworks/overview) already captures.
Worked examples, replay-mocking decisions, and common pitfalls: [Instrumentation](/instrumentation).
### Opt-in and opt-out tracing
Bitfab has two ways to decide what a trace records. Pick one per workflow, and start with opt-out.
| | Opt-out **(recommended)** | Opt-in |
| ----------------- | ------------------------------------------------------------------- | ------------------------------- |
| Decorators | `@trace` on the root, `@node` on a descendant that needs policy | `@span` |
| Recorded | The root and every first-party function beneath it, unless excluded | Only the functions you decorate |
| Replay boundaries | The `@trace` root; `@node(mock_on_replay=True)` marks a descendant | Every `@span` |
| Status | **Experimental**, Python 3.12+ | Stable |
Opt-out gets a workflow traced in one decorator and you narrow from there, which is faster and less error-prone than deciding every boundary up front. It is still experimental, so use `@span` when you want a small exact set of spans or you are below Python 3.12, where the root span still records and its descendants are skipped. See [Instrumentation](/instrumentation) for the full comparison.
The two are never combined in one call stack. A `@span` entered beneath an active `@trace`, or a `@trace` or `@node` entered beneath an active `@span`, raises `MixedTracingError` naming both decorators. Inside a `@trace` subtree, configure a step with `@node`; inside a `@span` workflow, add more `@span` decorators.
### Declaring the trace function key
#### Using `get_function()` to Link Spans
Declare the trace function key once and link multiple spans together:
```python theme={null}
order_service = bitfab.get_function("order-processing")
@order_service.span(mock_on_replay=True)
def load_order(order_id: str) -> Order:
return db.orders.find_by_id(order_id)
@order_service.span(type="llm")
def classify_order(order: Order) -> Classification:
return client.responses.parse(model=MODEL, input=build_prompt(order), text_format=Classification)
@order_service.span()
def validate_order(classification: Classification) -> dict:
return {"valid": classification.confidence > 0.8}
# The root calls the decorated steps, so they record as its children
@order_service.span(type="agent")
def process_order(order_id: str) -> dict:
order = load_order(order_id)
classification = classify_order(order)
return validate_order(classification)
```
Calling `process_order(id)` records one trace with four spans:
```
process_order
├── load_order
├── classify_order
└── validate_order
```
Decorating only `process_order` would record the same work as a single node, with the model call, the database read, and the validation collapsed into the root's input and output.
#### Multi-File Projects
For projects with instrumented functions spread across multiple files, create a dedicated file that initializes Bitfab and exports the function. Import it wherever you need to instrument.
```python theme={null}
# lib/bitfab_client.py -- single source of truth
import os
from bitfab import Bitfab
bitfab = Bitfab(api_key=os.environ["BITFAB_API_KEY"])
order_service = bitfab.get_function("order-processing")
```
```python theme={null}
# services/validate_order.py
from lib.bitfab_client import order_service
@order_service.span()
def validate_order(order_id: str) -> dict:
return {"valid": True}
```
```python theme={null}
# services/process_order.py
from lib.bitfab_client import order_service
from services.validate_order import validate_order
@order_service.span()
def process_order(order_id: str) -> dict:
validate_order(order_id)
return {"order_id": order_id}
```
Spans from different files are automatically linked as parent-child when one decorated function calls another.
#### Using `@bitfab.span()` Directly
For a single span without linking to a function group:
```python theme={null}
@bitfab.span("one-off-operation")
def standalone_task() -> str:
return "done"
```
#### Automatic Nesting
Spans nest automatically based on call stack:
```python theme={null}
@bitfab.span("outer", type="agent")
def outer():
inner() # Becomes a child of "outer"
@bitfab.span("inner", type="function")
def inner():
pass
```
For reusable helpers that should appear only inside an existing trace, use `capture_when="nested"`:
```python theme={null}
@bitfab.span("workflow", type="function", capture_when="nested")
def helper(value: str) -> str:
return transform(value)
@bitfab.span("workflow")
def workflow(value: str) -> str:
return helper(value)
helper("standalone") # Runs normally without creating a trace
workflow("nested") # Captures helper as a child
```
### Content capture from the sim plan
A trace function's sim plan can turn content capture off for individual spans. The client reads your organization's sim plan from `GET /api/sdk/sim-plan` in the background and applies it to every span it sends, so no code change is needed. A node is matched by the trace's root trace function key plus the span's name:
* A span whose content is off still records its name, type, timing, errors, contexts, and links to nested traces, but not its inputs and outputs, and it carries `content_off_by_simulation_plan: true` so Bitfab knows why the content is missing.
* Spans recorded by a framework integration (the OpenAI Agents processor, the LangGraph handler and integration, the Claude Agent SDK handler) keep their content: every span carries a `span_origin` record (the SDK name and version, and `instrumentation.name` saying what recorded it), and the plan cannot turn those off.
* The read starts as soon as you wrap a function or call `get_function`, on a background thread with a five second timeout, and refreshes about once a minute while spans flow. A traced call never waits on it.
* A span sent before the first read has succeeded is held back and sent once the plan arrives, with the plan applied, so no span ever leaves the process with content the plan turned off, not even the first one. Held spans go out on `flush_traces()`, on `close()`, and at exit, as soon as the plan has loaded, and while records are held those paths wait up to the five second read timeout for the plan before giving up on them. At most 1,000 records are held per client, and past that the oldest are dropped with a one-time warning.
* A trace's completion waits behind whichever of that trace's spans are held, and goes out right away when none are, so a trace made only of framework spans is never held back.
* A failed read is retried every ten seconds while a record is held, and otherwise on the next span. Before the first success that keeps spans held; after it the last plan stays in effect. A plan change takes effect within one refresh.
* A server with no sim plan feature answers the read with 404, which counts as an empty plan loaded: nothing is held and nothing is stripped.
* Pass `simulation_plan=False` to `Bitfab(...)` to turn the plan off entirely (no read, nothing held back, nothing stripped), the same as setting `BITFAB_DISABLE_SIM_PLAN` to any value that is not empty or whitespace.
### Subtree Tracing
**Experimental.** `trace()` is new and its behavior may change in a future
release. Requires Python 3.12 or newer. `span()` is the stable decorator.
`span()` records the function you decorate. `trace()` records that function **and every one of your own functions it calls**, at any depth, without decorating them:
```python theme={null}
@bitfab.trace("ticket-triage", type="agent")
def triage(ticket: dict) -> dict:
normalized = normalize_ticket(ticket) # recorded, not decorated
signals = extract_signals(normalized) # recorded, not decorated
return build_response(normalized, signals)
```
That single decorator produces the whole tree:
```
triage
├─ normalize_ticket
│ ├─ redact_emails
│ └─ collapse_whitespace
├─ extract_signals
│ ├─ detect_keywords
│ └─ measure_length
└─ build_response
└─ render_summary
```
Capture is scoped to the traced call: it turns on when `triage` is entered and off when it returns, so the rest of your application is unaffected.
Use `node()` when one discovered function needs the same policy controls as an explicit span without becoming an independent instrumentation boundary:
```python theme={null}
@bitfab.node(name="Generate reply", type="llm")
def generate_reply(prompt: str) -> str:
return call_model(prompt)
@bitfab.node(mock_on_replay=False)
def apply_new_policy(value: str) -> str:
return value.strip()
@bitfab.node(capture=False)
def implementation_detail(value: str) -> str:
return normalize(value)
@bitfab.trace("ticket-triage", type="agent", mock_on_replay_default=True)
def triage(ticket: dict) -> dict:
prompt = apply_new_policy(implementation_detail(ticket["body"]))
return {"reply": generate_reply(prompt)}
```
`node()` is consumed only by the enclosing `trace()`. It never creates a span or trace when the function runs by itself. Its API mirrors the applicable `span()` options (`name`, `type`, `test_run_id`, `mock_on_replay`, and `finalize`), while `capture` controls whether `trace()` includes the node. `trace(mock_on_replay_default=True)` makes replay mocking the default for configured nodes under `mock="marked"`; a node with `mock_on_replay=False` overrides it. The trace option is off by default. With `capture=False`, the function is omitted and its captured descendants attach to the nearest captured parent. Recorded-output mocking requires a captured node, so combining `capture=False` with `mock_on_replay=True` raises `ValueError`.
**What gets recorded.** Only functions you wrote, meaning the package directory containing the decorated function. The standard library, site-packages, and Bitfab's own code are never recorded. Lambdas, generator expressions, and decorator wrappers are skipped too, since they add spans without adding a step you would recognize in your own call tree.
**Bounds.** A traced subtree stops at `max_depth` (default 30) and `max_spans` (default 500), so a hot loop cannot produce an unbounded trace. Hitting either limit logs a one-time warning naming the function, because a trace that stops partway looks like code that never ran.
```python theme={null}
@bitfab.trace(
"ticket-triage",
max_depth=10, # stop recording below this call depth
max_spans=100, # stop recording after this many descendants
exclude=["parse_row"], # never record these function names
include_wrappers=True, # record decorator wrappers too (off by default)
)
def triage(ticket: dict) -> dict:
...
```
On Python 3.11 and earlier, the decorated function still records its own span
exactly as `span()` would, and a one-time warning explains that descendants
were skipped. Your code runs the same either way.
**Nested `trace()` roots.** A nested `trace()` root starts a separate trace while its full subtree also appears in every outer `trace()` capture. Each trace gets its own span IDs and the same complete structure it would have recorded alone. Two nested roots therefore double span volume in the nested region; each additional active `trace()` root records another independent copy. A `node()` inside the nested region keeps its configured name, type, `test_run_id`, and finalized output in every enclosing trace's copy, with `finalize` running once, and so do the LangGraph integration's tool and invoke spans. Framework handler spans (the OpenAI Agents processor, the Claude Agent SDK handler, the LangGraph callback handler) attach inside the innermost trace only. The outer trace's span for the nested root carries `nested_trace_id`, `nested_trace_function_key`, and `nested_root_span_id`, and the nested root span carries `enclosing_trace_id`, `enclosing_span_id`, and `enclosing_trace_function_key`, so the two traces point at each other. In the trace viewer the enclosing trace's span for the nested root shows a lip naming the nested trace function. It opens that trace at its root span. The nested trace's root shows a lip back to the enclosing trace function that opens the span that started it. Inside a replay item or `seed_trace` a nested root starts no trace of its own: the item's trace records it as an ordinary descendant.
Use nested `trace()` roots when both boundaries need to stand alone as complete traces:
```python theme={null}
@bitfab.trace("ticket-detail")
def build_ticket_detail(ticket: dict) -> dict:
return enrich_ticket(ticket)
@bitfab.trace("ticket-workflow")
def process_ticket(ticket: dict) -> dict:
return build_ticket_detail(ticket)
process_ticket(ticket)
# Creates two independent traces:
# ticket-workflow -> build_ticket_detail -> enrich_ticket
# ticket-detail -> enrich_ticket
```
**No `span()` inside `trace()`, no `trace()` inside `span()`.** `trace()` and `node()` are the opt-out tracing surface; `span()` is the opt-in surface. Entering one beneath the other raises `MixedTracingError`:
```python theme={null}
@bitfab.span("ticket-detail", type="function")
def build_ticket_detail(ticket: dict) -> dict:
return enrich_ticket(ticket)
@bitfab.trace("ticket-workflow", type="agent")
def process_ticket(ticket: dict) -> dict:
return build_ticket_detail(ticket)
process_ticket(ticket)
# MixedTracingError: Opt-in and opt-out tracing can't be mixed: @span (opt-in)
# was entered inside a @trace call (opt-out). Inside a @trace subtree,
# configure a function with @node instead, or trace this workflow with @span only.
```
Inside a `trace()` subtree, give a step its own name, type, or replay policy with `node()`. The check runs only while tracing is active: with `capture_enabled=False` and outside a replay item, decorated functions run as written.
Framework handlers need no decorators of their own inside a `trace()` subtree. Spans from the OpenAI Agents tracing processor, the Claude Agent SDK handler, and the LangGraph callback handler join the trace and attach under the nearest captured call, keeping their own span types (`llm`, `agent`, `function`).
The LangGraph integration's tool and invoke wrappers (`get_langgraph_integration`) adapt at call time. Outside a `trace()` subtree they open opt-in spans, exactly as before. To get opt-out tracing around a graph, put `@trace` on the function that calls it. Everything beneath becomes opt-out, framework spans included: tool spans behave like `node()` calls, attaching under the nearest captured call, counting toward `max_spans` and `max_depth`, and inheriting `mock_on_replay_default`.
**Limitations.** Work dispatched to `ThreadPoolExecutor` or `threading.Thread` inside a traced call is not captured, since capture rides contextvars. Async-generator capture is released between yielded items, so stopping iteration does not leave capture active; a generator abandoned before it finishes still records no span. Unconfigured descendants remain typed `function` and are not replay mock targets. Use `node()` for descendant naming, typing, capture, finalization, and replay-mocking policy.
### Tracing Across Threads
Span nesting rides Python contextvars, which do not reach `ThreadPoolExecutor.submit` / `loop.run_in_executor` work items or `threading.Thread` targets: a decorated function called there roots its own single-span trace, and replay mocking never fires for it. If instrumented functions are dispatched that way (common in agent tool executors), enable propagation on the client:
```python theme={null}
bitfab = Bitfab(trace_across_threads=True)
```
`True`: on. `False`: off. `None` (default): on only when `BITFAB_TRACE_ACROSS_THREADS=1` is set. Process-global once installed. `asyncio` tasks and `asyncio.to_thread` need nothing; pre-created queue consumer tasks and other processes are out of its reach.
#### Span Options
**Parameters:**
* `trace_function_key` (required): String identifier for grouping spans
* `name` (optional): Display name. Defaults to the function's qualified name (`Order.process` for a method, `process` for a plain function or a closure), then the trace function key
* `type` (optional): Span type. Defaults to `"custom"`. A label only, used to organize and filter spans in the dashboard; it does not change how the span is traced, replayed, or evaluated
* `capture_when` (optional): `"always"` (default) or `"nested"`. Nested-only spans are captured under an active parent and run untraced when called standalone. Unknown values warn once and default to `"always"`
* `finalize` (optional): `Callable[[result], serializable]`. Record a serializable view of a non-serializable result (a live stream). See [Tracing streaming functions](#tracing-streaming-functions)
**Span Types:**
```python theme={null}
SpanType = Literal[
"llm", # LLM calls
"agent", # Agent workflows
"function", # Function calls
"guardrail", # Safety checks
"handoff", # Human handoffs
"custom" # Default
]
```
**Examples:**
```python theme={null}
# Function name is automatically captured as span name
@bitfab.span("order-processing")
def process_order(order_id: str) -> dict:
return {"order_id": order_id}
# Span name: "process_order"
# Override with name option
@bitfab.span("order-processing", name="OrderProcessor")
def process_order(order_id: str) -> dict:
return {"order_id": order_id}
# Span name: "OrderProcessor"
# Set span type
@bitfab.span("safety-check", type="guardrail")
def check_content(content: str) -> dict:
return {"safe": True}
```
#### Tracing Streaming Functions
A streaming function hands chunks to the caller as they arrive; the raw stream isn't serializable as a trace output, and consuming it to record a summary would break streaming. The `finalize` option records a serializable, replayable view of the stream while the caller still receives every chunk.
Because Python streams are single-consumer (unlike a JS stream you can tee), the non-destructive way to trace streaming is an **async generator** that `yield`s its chunks. The span collects the chunks as they pass through to the caller, and `finalize` turns the collected chunks into a summary. Use the prebuilt `finalizers.openai_chunks` or `finalizers.anthropic_events`:
```python theme={null}
from bitfab import finalizers
@bitfab.span("chat", type="llm", finalize=finalizers.openai_chunks)
async def chat(messages):
stream = await client.chat.completions.create(
model="gpt-4o", messages=messages, stream=True
)
async for chunk in stream:
yield chunk # caller still receives every chunk
# The span records { text, finish_reason, usage, tool_calls } in the background.
```
`finalize` may also be a plain callable that builds whatever shape you want from the collected chunks:
```python theme={null}
@bitfab.span("chat", type="llm", finalize=lambda chunks: {"text": "".join(
c.choices[0].delta.content or "" for c in chunks
)})
async def chat(messages):
...
async for chunk in stream:
yield chunk
```
For a non-generator function, `finalize` receives the return value instead of the collected chunks and is applied inline before the span is recorded (awaited on an async span). The caller's return value is always the raw result, but a live single-consumer stream returned here will be blocked on and consumed, so use an async generator for streaming, and reserve the non-generator form for plain return values or results with non-destructive accessors. A `finalize` that raises records an error on the span instead of crashing the host. Inputs to the wrapped function must still be serializable for the trace to replay.
#### Span Context
Use `get_current_span()` to get a handle to the active span, then call `.add_context()` to attach contextual key-value pairs from inside a traced function -- useful for runtime values like request IDs, computed scores, or dynamic context:
```python theme={null}
from bitfab import get_current_span
@bitfab.span("order-processing", type="function")
def process_order(order_id: str) -> dict:
user_id = get_current_user()
get_current_span().add_context({"user_id": user_id, "order_id": order_id})
return {"order_id": order_id, "status": "completed"}
```
Each `add_context` call pushes the entire dictionary as one entry. Multiple calls accumulate entries:
```python theme={null}
get_current_span().add_context({"user_id": "u-123"})
get_current_span().add_context({"request_id": "req-789"})
# Result: contexts: [{"user_id": "u-123"}, {"request_id": "req-789"}]
```
`get_current_span().id` and `.trace_id` expose the canonical Bitfab span and trace IDs. Both are empty strings outside a span context.
#### Span Prompt
Use `get_current_span()` to set the prompt string on the current span. This is stored in `span_data.prompt` and is useful for capturing the exact prompt text sent to an LLM:
```python theme={null}
from bitfab import get_current_span
@bitfab.span("classification", type="llm")
def classify_text(text: str) -> str:
prompt = f"Classify the following text: {text}"
get_current_span().set_prompt(prompt)
result = llm.complete(prompt)
return result
```
The prompt is metadata only. It records the prompt text for display and reference in the dashboard; it does not send the prompt to any model or change what the span executes. The last `set_prompt` call wins -- it overwrites any previously set prompt on the span. Calling `set_prompt` outside a span context is a no-op (it never crashes).
#### Framework Integrations
Bitfab provides automatic tracing for popular AI frameworks. See the dedicated guides for full API references:
Callback tracing plus Experimental (alpha) `ToolNode` output mocking for replay
Trace processor for agent runs
Auto-capture prompts and LLM metadata
Capture LLM turns, tool calls, and subagents
#### Trace Context
Use `get_current_trace()` to set context that applies to the entire trace (all spans within a single execution). This is useful for grouping traces by session or attaching trace-level metadata:
```python theme={null}
from bitfab import get_current_trace
@bitfab.span("order-processing", type="function")
def process_order(order_id: str) -> dict:
trace = get_current_trace()
# Set session ID (stored as database column, filterable in dashboard)
trace.set_session_id("session-123")
# Name the trace (its title in Bitfab, searchable and filterable)
trace.set_name(f"Order {order_id}")
# Set trace metadata (stored in raw trace data)
trace.set_metadata({"region": "us-west-2", "environment": "production"})
# Add context entries (stored as key-value pairs, accumulates across calls)
trace.add_context({"workflow": "checkout-flow", "batch_id": "batch-2024-01"})
return {"order_id": order_id, "status": "completed"}
```
* `set_session_id(id)` -- Groups traces by user session. Stored as a database column for efficient filtering.
* `set_name(name)` -- The trace's title in Bitfab, and a field you can search and filter on. Use it for the case, ticket, or record the run is about. Unset, the trace is titled by its trace function key.
* `set_metadata(dict)` -- Arbitrary key-value metadata on the trace. Merges with existing metadata.
* `add_context(dict)` -- Key-value context entries. Accumulates across multiple calls.
#### Dropping a Trace
Call `.drop()` on the current-trace handle to discard the in-flight trace. Once flagged, spans that complete afterward are not uploaded at all, and the flag rides out on the completion payload, so when the trace completes the server scrubs any payloads that already raced out (the trace, its external trace, and sibling spans), deletes the archived S3 objects, and marks it `dropped` instead of `completed`, keeping only a skeleton audit row. Use it to discard runs you never want stored (health checks, test traffic) or a run you know carries sensitive data.
```python theme={null}
from bitfab import get_current_trace
@bitfab.span("order-processing", type="function")
def process_order(order_id: str) -> dict:
if is_health_check(order_id):
get_current_trace().drop()
return {"order_id": order_id, "status": "completed"}
```
* Safe to call outside a trace (a no-op), and never raises into your application.
#### Detached Trace
Use `client.get_trace(trace_id)` to get a handle to a trace that has already closed. This lets you add context, merge metadata, or set the session ID from any process, thread, or agent that knows the trace ID, with no shared in-memory state.
```python theme={null}
trace = client.get_trace(trace_id)
trace.add_context({"refund_status": "approved"})
trace.set_metadata({"region": "us-west"})
trace.set_session_id("session_xyz")
trace.set_name("Ticket 4521")
```
The `trace_id` is Bitfab's canonical trace ID, the same UUID exposed by `get_current_span().trace_id` for native SDK traces and used in Bitfab trace URLs. All methods are blocking, like `get_trace_span()`: each returns once the server has applied the change, so a later read always observes the write. They raise if the server rejects the update, and are silent no-ops when the client is disabled.
* `add_context(context)` -- Appends a context entry. Existing entries are preserved.
* `set_metadata(metadata)` -- Shallow-merges new keys into existing metadata.
* `set_session_id(session_id)` -- Replaces any existing session ID.
* `set_name(name)` -- Replaces any existing trace name.
#### Read One Persisted Span
Use `get_trace_span` to fetch one span without loading the full trace. Both the trace ID and exact span ID are canonical Bitfab IDs; ingestion source IDs are not accepted. Repeated name matches default to the last span.
```python theme={null}
span = client.get_trace_span(trace_id, name="GenerateAnswer")
first = client.get_trace_span(
trace_id, name="GenerateAnswer", occurrence="first"
)
exact = client.get_trace_span(trace_id, id=span_id)
```
`occurrence` also accepts a zero-based integer. A missing trace or span returns `None`.
#### Error Handling
Errors are captured in the span and re-raised:
```python theme={null}
@bitfab.span("risky-service")
def risky():
raise ValueError("error")
try:
risky()
except ValueError:
pass
# Span records error and timing
```
Each error is classified by source. Errors raised by your code are recorded with `error_source: "code"`. SDK-internal errors are recorded with `source: "sdk"`. Both appear in the span's `errors` array in the Bitfab dashboard.
#### Flushing Traces
```python theme={null}
from bitfab import flush_traces
if not flush_traces(timeout=30.0):
raise RuntimeError("Bitfab traces were not delivered before the deadline")
```
The return value is `False` when an export fails or the deadline expires.
Traces also flush automatically on process exit via an `atexit` hook.
#### OpenTelemetry Transport
| Variable | Default | Purpose |
| -------------------------------- | --------- | --------------------------------------------- |
| `BITFAB_OTEL_EXPORT_CONCURRENCY` | `32` | Concurrent direct requests, `1` through `64`. |
| `BITFAB_OTEL_MAX_REQUEST_BYTES` | `3000000` | Request-size target. Can only be lowered. |
The Python SDK lazily creates one private OpenTelemetry provider and bounded
`BatchSpanProcessor` per client; it does not replace your
application's global OTel provider, and an unused client starts no OTel worker.
Decorators and framework handlers submit the same replay-safe Bitfab payloads
through a transport interface. Batches are sent to Bitfab as OTLP/JSON. Each
carrier is encoded once and the request body is assembled from those encodings,
so a batch is never re-encoded to measure its size.
Live and replay traces share the same OTel pipeline. Before completing a replay
test run, the SDK flushes OTel and uses delivery acknowledgments to confirm
every carrier that reached Bitfab. If any delivery is uncertain, it polls
Bitfab until every submitted replay trace completion and expected span count is
persisted.
For the full ownership model, carrier format, live and replay flows, batching
limits, lifecycle, and failure semantics, see
[OpenTelemetry Transport Architecture](/otel-architecture).
The SDK partitions count-based OTel exports into requests of at most about
3 MB. Each request contains at most eight carriers and is packed using their
exact encoded size. Set
`BITFAB_OTEL_MAX_REQUEST_BYTES` to a positive integer no greater than
`3000000` to use a smaller target for a stricter proxy; unsafe
values warn and fall back to `3000000`. An oversized carrier gets its own request
and uses the compression and trimming fallback described below. If it still
cannot fit or ingress rejects it with HTTP 413, the SDK reports an export
failure. Flush and shutdown share one total caller-supplied deadline.
If Bitfab rejects malformed carriers from an otherwise valid direct batch,
the standard OTLP `partialSuccess` response is logged with the rejected-span
count and reason.
A single span may use up to 7,800,000 carrier bytes when its dedicated request
gzips below the 3,000,000-byte wire target and remains below the 8,000,000-byte
decompressed ingress limit. The carrier is the payload re-escaped into the OTLP
attribute. If it does not compress enough, compression is unavailable, or it
exceeds the raw ceiling, the SDK replaces its largest fields with
`` placeholders until it fits the
2,800,000-byte fallback budget. The trim is recorded on the span's `errors` so
the trace is flagged as incomplete.
For transient clients in long-running processes, use `Bitfab` as a context
manager or call `client.close(timeout=30.0)` when finished. Closing is
idempotent: it flushes and shuts down only that client's OTel worker, which is
also reused by framework handlers created from the client. A shared application
client can remain open and will still shut down automatically at process exit.
### Replay
A trace is replayable when its root span has serializable inputs, or when the workflow is instrumented through a [framework handler](#replaying-handler-instrumented-functions) (whose recorded root input is itself serializable). One of these must hold for replay to work.
Replay historical traces through a function and create a test run with comparison data. This is useful for testing changes to your functions against real production inputs.
```python theme={null}
@bitfab.span("my-function-key")
def my_function(text: str) -> dict:
return {"processed": text.upper()}
result = bitfab.replay(my_function, limit=5)
# Or replay specific traces by ID
result = bitfab.replay(my_function, trace_ids=["trace-abc", "trace-def"])
print(f"Test Run: {result['test_run_url']}")
for item in result["items"]:
print(f" Input: {item['input']}")
print(f" Result: {item['result']}")
print(f" Original: {item['original_output']}")
print(f" Duration (ms): {item['original_duration_ms']}")
print(f" Tokens: {item['tokens']}") # {"input", "output", "cached", "total"} or None
print(f" Model: {item['model']}")
```
Pass `replay()` **either** an already-`@span`-decorated function (it carries its trace function key, so it runs as-is) **or**, with an explicit key, a plain callable that re-invokes a raw entrypoint (which `replay()` wraps for you). Do not pass a plain closure that itself calls a `@span`-decorated function: `replay()` wraps the closure as the root span while the inner decorated function records its own span underneath, nesting a duplicate. If your root is already decorated, pass it directly: `bitfab.replay(my_function, limit=5)`.
**Parameters:**
* `fn` (required): The function to replay. Two call forms: `replay(decorated_fn)` reads the trace function key from the `@span` decorator; `replay("key", fn)` takes an explicit key with any plain callable (the SDK wraps it internally). That wrapper's root span belongs to neither tracing surface, so the callable may call a `@trace` root, which nests beneath it. Use the explicit-key form for handler-instrumented functions with no decorated root in the app; see **Replaying handler-instrumented functions** below.
* `limit` (optional): Maximum number of recent traces to replay. Default: `5`; maximum: `5,000`. Ignored when `trace_ids`, `dataset_id` or `dataset_ids` is passed, since an explicit ID list or a dataset already determines how many traces replay. To replay part of a dataset selection, name the members to run in `trace_ids`.
* `trace_ids` (optional): List of trace IDs to replay (max 100). The ID count determines how many traces replay, and `limit` is ignored when both are passed. Passed alongside a dataset selector it pins which members of that selection replay, and the server rejects any ID none of those datasets contains.
* `name` (optional): Display name for the resulting experiment/test run.
* `max_concurrency` (optional): Maximum items processed in parallel. `1` for sequential, `None` for unlimited. Default: `10`
* `attempts` (optional): How many times to replay each trace, 1 to 100. Every attempt is its own replay trace under the same experiment, so the experiment reports per-attempt pass rates and flags traces whose attempts disagree. Default `1`.
* `concurrency` (optional): A `ReplayConcurrency(attempts=..., primitive=..., max_concurrency=...)` carrying the two settings above plus the primitive that runs them. Passing it alongside `attempts` or `max_concurrency` raises rather than merging, so the two spellings can never disagree. `primitive="async"` is the default and is exactly today's behavior: every work item is a coroutine in one process, `max_concurrency` defaulting to 10. `primitive="process"` gives each work item its own child interpreter, with `max_concurrency` defaulting to 4 and `None` refused, and is the only primitive that accepts `on_item_finish_in_child_process`. Reach for it when your replayed code keeps its world in process-global state, such as a settings module read once at import, a per-item database, or a module-level registry, so that two items cannot safely share one process. The unit is the work item rather than the attempt, so N traces at K attempts is N x K child processes taken from a single queue. Because process mode re-runs the replay command once per item, it works only under `bitfab-replay --registry` and raises from a plain `client.replay(...)` call. Under process mode it also carries `memory_throttle`, on by default, which launches a child only once the machine has memory for it so a run on a loaded laptop degrades to fewer children in parallel rather than being killed by the operating system. It never raises concurrency above `max_concurrency` and never holds back the only child in flight, so a machine under lasting pressure still finishes one item at a time. Set `BITFAB_REPLAY_MEMORY_THROTTLE=off` to turn it off for a single run.
* `code_change_description` (optional): Rationale for the code change being tested in this replay (stored on the experiment); when supplied alone, it is preserved while files are captured automatically
* `code_change_files` (optional): List of edited files, each as `{"path": str, "before": str, "after": str}` (use `""` for newly created or deleted files); omit to capture automatically or pass `None` to suppress capture
* `mock` (optional): Mock strategy for descendant spans: `"marked"` (default), `"none"`, or `"all"`.
* `mock_override` (optional): One `MockOverride`, or a list of them, that substitutes a supplied output for matched spans. Per-call overrides take precedence over registered overrides and the base `mock` strategy.
* `experiment_group_id` (optional): UUID string that groups multiple replay runs into a single experiment batch. Pass the same ID across successive `replay()` calls to link them together in the dashboard.
* `dataset_id` (optional): Dataset UUID. Replays that dataset's traces and durably attributes the resulting experiment to it, and `limit` is ignored because the dataset determines the item count. Pass `trace_ids` alongside it to replay only those members.
* `dataset_ids` (optional): Dataset UUIDs, for benchmarking one function against several corpora in a single run. Replays the union of their traces, graded by the union of their graders, and attributes the experiment to every one of them. Pass one dataset through `dataset_id` and several through this.
* `grader_ids` (optional): Array of grader UUIDs (max 100) attached directly to this replay run, independent of the dataset's own graders. The resulting experiment is graded by the union of these and the dataset's runnable graders. Use it to grade a single run with a check you don't want to add to the dataset permanently. Each id must be an active grader in the same organization and trace function, or the replay is rejected with a 400. A replay with no dataset can still carry graders this way.
* `adapt_inputs` (optional): Hook to reshape recorded inputs onto the function's current signature when its shape changed after the traces were captured. See **Adapting inputs after a signature change** below.
* `on_item_start` (optional): Callback fired when a worker begins processing an item, before replay setup and customer code run. Pair it with `on_item_finish` to distinguish queued items from in-flight items whose callback has not returned. A raising callback never crashes the run.
* `on_item_finish` (optional): Callback fired exactly once per item as it finishes, always with that item plus running totals (original/source trace id, the server replay `trace_id` read back per trace off the OTLP ingest response and surfaced as each item finishes (its trace is flushed on finish, so the id is in hand at the callback, not only after the whole run), input, result, original output, error, duration, tokens/model metadata). It never emits a whole-run completion event. Use it to render live progress or start evaluating completed items while replay runs. A raising callback never crashes the run. The deprecated `on_progress` callback receives the same per-item events plus its legacy item-less terminal `complete` event, and is ignored when both are supplied. The installed `bitfab-replay` command passes the SDK's ready-made `report_replay_progress` callback as both `on_item_start` and `on_item_finish`; it writes lifecycle events to stderr, which the plugin uses to identify in-flight traces, report finished items, and write per-item result files while stdout remains available for direct-run `ReplayResult` JSON.
* `db_branch` (optional): `True` or a `DbBranchOptions` dict. `db_branch=True` requests a DB branch per replay item with the mirror's own sizing; pass a dict to tune it, and `False` or omission leaves branching off. Each replay worker resolves its branch from the source trace's captured snapshot reference, so `max_concurrency` also bounds live branches. `get_current_replay_branch()` hands the branch to you inside the replayed function, and the SDK releases it after the item. The accessor returns `None` when no branch was resolved (e.g. the trace predates snapshot capture, or DB branching isn't configured), so `branch.database_url if branch else os.environ["DATABASE_URL"]` falls back to your live database. The keys tune the branch: `min_cu` and `max_cu` are the compute's autoscaling floor and ceiling (0.25 to 56). Equal values pin a fixed size, allowed up to 56; an autoscaling range may not span more than 8 CU or exceed 16 CU; setting them equal pins the size, so one item can't post a better number purely because it ran against an already-scaled endpoint. `warmup_sql` is appended to the branch's readiness check, so the cache is warm before your function sees the branch and the warm-up is never charged to the replayed call. Omit them and the branch keeps the mirror's own defaults.
**Returns:**
```python theme={null}
{
"items": [
{
"input": [...], # The inputs passed to fn
"result": ..., # What fn returned
"original_output": ..., # What the original trace produced
"error": None | str, # Error message if fn raised
"duration_ms": int | None, # How long THIS replay took, in ms
"original_duration_ms": int | None, # The original trace's duration
"original_tokens": {...} | None, # Original trace token usage
"original_model": str | None, # Original model name
"tokens": { # Replayed run token usage, or None
"input": int | None,
"output": int | None,
"cached": int | None,
"total": int | None,
} | None,
"model": str | None, # Deprecated alias for original_model
"db_branch_timings": dict | None, # Per-phase branch provisioning timings
"trace_id": str | None, # Server trace ID for the replayed execution
"db_snapshot_ref": dict | None, # The source trace's snapshot pin, if any
"trace_outline": dict | None, # The replayed trace's span tree, no payloads
"original_trace_outline": dict | None, # The original trace's span tree, no payloads
}
],
"test_run_id": "...",
"test_run_url": "..."
}
```
Replay waits for every submitted trace completion and expected span count to be persisted server-side before completing the test run, so `trace_id` is a real server trace ID for completed items. Persistence is checked by Bitfab after the shared OTel pipeline is flushed. If that barrier times out, `replay()` raises a `RuntimeError` and does not finalize an incomplete run. If NO otherwise-completed item's trace persisted (uploads wholesale failed, or the replayed function isn't decorated with `@span`), `replay()` also raises instead of silently returning `None` trace IDs. If only SOME completed items are absent from the final mapping, those items get `None` trace IDs with a logged error and the rest of the run is returned intact.
Anything describing the trace being replayed carries the `original_` prefix: `original_duration_ms`, `original_model`, and `original_tokens`. Unprefixed fields are the replay's own: `duration_ms` is how long this run's call took, and `tokens` is the **replayed run's** usage (the same numbers Studio's experiments view shows). Comparing `tokens["total"]` against `original_tokens["total"]` tells you how your change moved cost. Each field is `None` when it wasn't captured.
The same rule names the two trace outlines: `original_trace_outline` is the original trace's span tree and `trace_outline` is the replayed trace's. An outline carries each span's name, type, nesting, order, duration, tokens, model, errors, and whether it was mocked, and no inputs or outputs, so it is small enough to keep on every item. Both are filled in at completion (they are `None` in `on_item_finish` and against older servers) so a grader can compare the path the replay took against the original's, not only its output. The shape is documented in the [Python reference](/reference/python).
`model` remains as a deprecated alias for `original_model`. Note that `duration_ms` changed meaning: it used to report the original trace's duration and now reports the replay's.
#### Replaying handler-instrumented functions
Workflows instrumented through a framework handler (`get_langgraph_callback_handler`, `get_langchain_callback_handler`, `get_claude_agent_handler`, `get_openai_agent_handler`) have no `@span`-decorated root in the application code: the handler (or run wrapper) records the framework invocation itself as the root span, with the framework's own input (a LangGraph initial state, an agent prompt, the run input) as the recorded root input. **These traces are fully replayable.** Pass the handler's trace function key explicitly, plus any plain callable that re-invokes the framework entrypoint:
The OpenAI Agents SDK uses `get_openai_agent_handler(key).wrap_run(agent, input)` (a drop-in for `Runner.run`) for the replayable root; the bare `get_openai_tracing_processor` captures internals only and records an empty-input root. The Claude Agent SDK handler needs a hint: the prompt is not present in the message stream, so pass it explicitly (`wrap_query(stream, input=prompt)`, or `wrap_response(stream, input=prompt)`) for the handler to record a replayable root.
```python theme={null}
# scripts/replay.py
from my_app.agent import graph # the compiled LangGraph graph
from my_app.bitfab_client import bitfab # same client as instrumentation
handler = bitfab.get_langgraph_callback_handler("my-agent") # same key
def replay_my_agent(state):
config = {"callbacks": [handler], "configurable": build_replay_config()}
return graph.invoke(state, config=config)
result = bitfab.replay("my-agent", replay_my_agent, limit=10)
```
How it fits together:
* `replay("key", fn)` fetches the handler-recorded production traces under the key and wraps `fn` in a span under that key internally, so each replayed invocation records a trace tied to the test run. No decorator needed; the key is the only link between the production traces and the replay callable.
* When the SDK auto-wraps a plain callable this way, a recorded dict root input (e.g. a LangGraph state) is passed to `fn` as a **single positional argument** (matching the TypeScript SDK) and reported faithfully on `item["input"]`. Decorated functions keep the decorated-path keyword-args semantics even when a matching key is also passed.
* Attaching the handler inside the callable makes the replayed graph's node/LLM/tool spans nest under the replay span, so replay traces have the same tree as production ones.
* The callable rebuilds runtime wiring the trace never captured: framework `config`, dependency objects, API keys. Put every unsafe call made by that wiring behind a replay-mockable marked span. Use a no-op value only for a replay-only callback slot with no recorded call to mock.
**Older SDKs** (before explicit-key replay): decorate a wrapper in the replay script with the same key instead: `@bitfab.span("my-agent")` on `def replay_my_agent(**state)` (on that path the recorded dict splats into keyword args and `item["input"]` reports `[]`), then call `bitfab.replay(replay_my_agent, limit=10)`.
#### Mocking child spans during replay
For the workflow-level guide, see [Replay Mocking](/replay-mocking).
When iterating on a root function, child spans sometimes fail in your local environment for reasons unrelated to the code under test: a paid API key is missing, an external service is flaky, or a production-only DB row isn't seeded locally. The `mock` keyword lets the child return its recorded output so the root function can still run.
Three strategies on `replay()`:
* **`"marked"`** (default): only descendants declared with `mock_on_replay=True` are short-circuited; everything else runs real. This is the iteration-friendly mode.
* **`"none"`**: every child span runs real code.
* **`"all"`**: every matched recorded descendant span returns its historical output. The root function still runs real; a missing or exhausted child occurrence fails the item closed. Useful for a quick sanity-check against recorded data; not the recommended iteration strategy because changes to matched descendants won't execute.
Async-generator spans are not mockable yet. When `mock="all"`, `mock="marked"` selects one, or an override matches, the replay item fails without iterating the real generator. Move unsafe work inside the generator to a mockable sync or coroutine descendant.
Per-span opt-in via the `mock_on_replay` kwarg on `@client.span(...)`:
```python theme={null}
article_pipeline = bitfab.get_function("process-article")
@article_pipeline.span(name="fetch-article-from-db", mock_on_replay=True)
def fetch_article_from_db(article_id: str) -> Article:
return db.articles.find_by_id(article_id)
@article_pipeline.span(name="summarize-article")
def summarize_article(article: Article) -> Summary:
# Real summarization, no flag -- this is what we're iterating on.
return Summary(...)
@article_pipeline.span(name="process-article")
def process_article(article_id: str) -> Summary:
return summarize_article(fetch_article_from_db(article_id))
# During replay, fetch-article-from-db returns its recorded output;
# summarize-article runs real so you can iterate on it.
result = bitfab.replay(process_article, limit=10)
```
`mock_on_replay` is a per-span tag at definition time -- it has no effect outside replay, and it's read by the default `mock="marked"` strategy. The root function always runs real code; only descendants can be mocked. When a strategy selects a child for mocking but no historical occurrence is available, the item errors and the real child does not execute.
#### Injecting custom values with overrides
A **mock override** substitutes a value you supply for a matched span, so downstream real code runs against it -- for "what if this step returned X" experiments without editing the traced code. An override is a `MockOverride(match, value)`: `match` selects spans by structural metadata (`node.span_name`, `node.type`, `node.trace_function_key`, `node.original_span_id`); `value` is a flat value injected as-is, or a callable that returns one.
The decorated function's trace function key selects the workflow's historical root traces; it does not by itself identify the descendant to override. Because the example above binds every call to `process-article`, match the descendant by its span name. If your calls use separate trace function keys, matching `node.trace_function_key` is also valid.
```python theme={null}
from bitfab import MockOverride
result = bitfab.replay(
process_article,
mock="none", # run everything real...
mock_override=MockOverride(
# ...except this span, which gets the value you supply
match=lambda node: node.span_name == "fetch-article-from-db",
value={"id": "fixed", "title": "Fixed title"}, # flat value
),
)
```
A callable `value` receives a context with the live positional `inputs`, the live keyword `kwargs` (empty when the call used none), and `get_original_output()` (synchronous in Python) to tweak the recorded output instead of replacing it:
```python theme={null}
value=lambda ctx: {**ctx.get_original_output(), "score": 1}
```
Under `marked`/override replay the recorded output is fetched lazily on first access, so `get_original_output()` (and a marked span's own recorded output) may block on a short HTTP request. Replay offloads that fetch off the event loop for async spans, so concurrent items are not stalled. A **synchronous** span tagged `mock_on_replay` (or matched by an override), when called from an **async** replay root, cannot offload and does the fetch on the loop thread, briefly serializing concurrent items. Make such a span `async`, or use `mock="all"` (eager, no per-span fetch), to avoid it.
Register overrides on the client to apply them to every replay (object or ordered form), and reset with `clear_mock_overrides()`:
```python theme={null}
bitfab.register_mock_override(
MockOverride(match=lambda node: node.type == "llm", value={"label": "refund"})
)
# Ordered form (equivalent): bitfab.register_mock_override(match, value)
bitfab.clear_mock_overrides()
```
Use the keyed form when one registration belongs to a known trace function. The second argument can be a resolver or `MockOverride`; for an override, both the registered trace function key and its `match` predicate must match:
```python theme={null}
client.register_mock_override(
"classify-intent",
lambda ctx: {"label": str(ctx.inputs[0])},
)
client.register_mock_override(
"shared-workflow",
MockOverride(
match=lambda node: node.span_name == "Classifier",
value={"label": "refund"},
),
)
```
Pass one callable directly for a client-wide resolver that routes by trace function key. Return `NO_MOCK_OVERRIDE` to decline a span; `None` remains a valid mocked output:
```python theme={null}
from bitfab import NO_MOCK_OVERRIDE
client.register_mock_override(
lambda ctx: (
{"label": str(ctx.inputs[0])}
if ctx.node.trace_function_key == "classify-intent"
else NO_MOCK_OVERRIDE
)
)
```
Precedence per span: per-call `mock_override`, then registered overrides, then the base `mock` strategy. `NO_MOCK_OVERRIDE` continues at the next override, then falls back to that base strategy. Pass a single `MockOverride`, resolver, or list.
#### Adapting inputs after a signature change
Replay deserializes each trace's inputs exactly as they were captured against the function's signature **at trace time**, then calls the current function with them. If the signature drifted since capture (a param renamed, reordered, folded into a dict, or a new required arg added), `fn(*args, **kwargs)` no longer lines up and raises. The `adapt_inputs` hook reshapes the recorded inputs onto the current signature so replay can still run:
```python theme={null}
# Recorded as (user_id, limit); current signature is (opts: dict).
def adapt(args, kwargs, ctx):
user_id, limit = args
return [{"user_id": user_id, "limit": limit}], {}
result = bitfab.replay(my_function, adapt_inputs=adapt)
```
The hook receives the deserialized `(args, kwargs)` plus a per-trace `ctx` (`{"original_trace_id", "original_span_id", "metadata"}`, with deprecated `source_*` aliases; `metadata` is the original trace's stored metadata, what `seed_trace` or `get_current_trace().set_metadata` put on it, merged so that the caller's keys win over the metadata an integration's own trace export carries for the same trace, and is fetched only when a hook is registered) and returns the `(args, kwargs)` actually passed to the function. The returned `args` is what `item["input"]` reports. It runs once per item, **inside the same error boundary as the function**: if it raises, that item's `error` is set and the run continues, so one unmappable trace never crashes the batch.
`ctx["original_trace_id"]` (the original Bitfab trace ID) lets a table-driven adapter look up a per-trace transform. That's the escape hatch for reshapes that need judgement rather than mechanical rearrangement: compute the adapted inputs per trace up front, then have the hook look them up by `original_trace_id`, keeping replay deterministic instead of calling a model mid-replay.
When the new signature has a genuinely new **required** input with no analog in the recorded trace, don't fabricate one -- there's nothing faithful to map it to. Leave those traces unmapped (let them raise) rather than inventing test inputs.
For anything beyond a one-liner, keep the adapter in its own file next to the replay registry and import it:
```python theme={null}
# scripts/replay_adapters/extraction.py
def adapt_inputs(args, kwargs, ctx):
user_id, limit = args
return [{"user_id": user_id, "limit": limit}], {}
```
```python theme={null}
# scripts/replay.py
from replay_adapters.extraction import adapt_inputs
bitfab.replay(my_function, limit=limit, adapt_inputs=adapt_inputs)
```
That keeps the transform versioned and reviewable alongside the function it adapts, and you add the import only when a drift actually needs it.
#### Attaching a Code Change
Each replay creates an experiment (test run). When you're iterating on a function and replaying after every edit, attach the change so the dashboard can show *exactly what was edited* alongside the results. Read each file before editing, edit, then read it again -- the two strings go straight into `code_change_files`. There's no diff format to construct.
```python theme={null}
with open("src/foo.py") as f:
before = f.read()
# ...edit src/foo.py...
with open("src/foo.py") as f:
after = f.read()
result = bitfab.replay(
my_function,
code_change_description="fix off-by-one in retry logic",
code_change_files=[{"path": "src/foo.py", "before": before, "after": after}],
)
```
Both options are optional and independent -- you can pass just `code_change_description` for a quick rationale-only annotation, or just `code_change_files` to record the literal edits.
If you omit `code_change_files`, `replay()` falls back to capturing your working-tree diff against the trunk merge-base (best-effort, only inside a git repo), so an experiment still shows a diff. This fallback uses Git rename detection: a renamed-and-edited file is compared once under its destination path, while an unchanged rename adds no content diff. A supplied `code_change_description` is preserved while the files are captured. Passing `code_change_files` explicitly always wins and is the way to record a precise per-edit before/after. To opt out for one replay run, pass `code_change_files=None` (and optionally `code_change_description=None` if you also want no description). Set `BITFAB_DISABLE_CODE_CHANGE_CAPTURE` to turn the fallback off for every replay in the process.
**Notes:**
* In the `replay(fn)` form the function must be decorated with `@span` -- the trace function key is read from the decorator. When the production code has a decorated root, **pass the decorated function itself**, not an undecorated wrapper around it; the `@span` attribute is what identifies the trace key. An undecorated wrapper has no key, so `replay()` wraps it as the root and the inner decorated function then records its own span underneath, nesting a duplicate. For nested decorators (e.g. `@retry(@cache(@span(fn)))`), pass the outermost -- replay walks the `__wrapped__` chain to find `@span`. For handler-instrumented functions with no decorated root, use the explicit-key form `replay("key", fn)` with any plain callable (see **Replaying handler-instrumented functions** above). Passing an explicit key that contradicts the decorator's key raises.
* **For decorated methods on classes**, pass the unbound function on the class (`MyClass.method`) to replay traces for all instances, or a bound method on a specific instance (`instance.method`) to replay through that instance's state. Both resolve to the same trace function key.
* **Use a single `Bitfab` client across instrumentation and replay.** If your instrumented module constructs `Bitfab()` at import and your replay registry constructs another, they do not share registered trace functions -- import the client from the instrumented module (or a shared singleton) rather than constructing a new one in the registry.
* The function can be sync or async (async functions are detected and run automatically)
* If the function raises an error for one input, replay continues with the remaining inputs
* Each replay creates a test run visible in the Bitfab dashboard
* Works through nested decorators (e.g. `@retry`, `@cache`) -- walks the `__wrapped__` chain to find `@span`
#### Replay Output Contract
Replay results are typically consumed by automation (CI logs, code reviewers, and coding agents). When `BITFAB_REPLAY_RESULT_PATH` is set, `bitfab.replay()` automatically writes the full `ReplayResult` JSON to that file. For direct/manual runs, **emit the full `ReplayResult` as a single stdout JSON block** so a consumer can `json.loads` it and reason about every field, including the per-item `original_duration_ms`, `original_tokens`, `original_model`, `tokens`, `original_trace_outline`, and `trace_outline`. Never print only lengths, counts, hashes, or truncated previews, and never replace the JSON block with ad-hoc per-field log lines.
Recommended script tail:
```python theme={null}
result = bitfab.replay(my_function, limit=limit)
# Human-readable summary goes to stderr, so stdout stays pure JSON.
print(f"Test run: {result['test_run_url']}", file=sys.stderr)
print(f"Items: {len(result['items'])}", file=sys.stderr)
# Then: full structured dump, ready for json.loads. The SDK serializer retains
# useful fields from trace_error and replay_error exception objects.
print(serialize_replay_result(result))
```
The dumped object includes every item's `input`, `result`, `original_output`, `error`, structured `trace_error` and `replay_error`, `duration_ms`, `original_duration_ms`, `original_tokens`, `original_model`, `tokens`, `model`, `db_branch_timings`, `trace_outline`, `original_trace_outline`, and `trace_id`, plus `test_run_id` and `test_run_url`. Import `serialize_replay_result` from `bitfab`; `json.dumps(..., default=str)` reduces exceptions to strings and loses their structured fields. When the Bitfab plugin runs this script, it sets `BITFAB_REPLAY_RESULT_PATH`; the SDK writes the same structured JSON there, and the plugin reads that file into the replay run's `.bitfab/replays//events.jsonl` while writing large per-item payloads under `.bitfab/replays//items/`.
**Per-item errors are part of the contract.** If the wrapped function raises while executing the replayed trace, `bitfab.replay` retains the actual exception in `item['trace_error']`, copies its message to `item['error']`, leaves `item['result']` as `None`, and continues. If replay setup fails before the function starts (for example database warmup or input loading), the actual exception is instead in `item['replay_error']`. A database branch resolution failure is a `DbBranchReplayError`; inspect its `code`, message, and `original_trace_id` to distinguish failures such as `branch_create_failed`, `snapshot_from_replaced_origin`, and `invalid_snapshot_ref` without parsing `item['error']`. A lease-endpoint HTTP, timeout, or network failure uses `lease_request_failed` and retains the original client exception as `cause`; unexpected resolver failures use `internal_error`. Treat either error kind as **unreplayable**, not as a failing output. If the whole run later raises, `ReplayError.items` still contains every collected item and `ReplayError.cause` retains the whole-run exception.
**Don't swallow per-item errors in the script.** A custom `try/except` that returns a placeholder turns infra failures into fake successes. Let the SDK record them. The only allowed top-level `except` is a fatal handler around `main()` that exits non-zero, so callers can tell a whole-replay crash from a clean run with some unreplayable items.
**Input serialization caveat.** Replay deserializes historical span inputs and passes them back to your function. This works for strings, numbers, and plain dicts. If your span wraps a function that takes hydrated domain objects (ORM models, class instances, DB records), they won't round-trip through serialization -- move the span to where inputs are IDs or plain data and let the function fetch objects internally, or reshape arguments in the wrapper.
#### Replay Registry
Create a small registry module. Your project owns only the imports and registrations; installing the SDK also installs the standard `bitfab-replay` executable, which owns command-line flags, lifecycle progress, code-change loading, result serialization, and summary output.
```python theme={null}
from dotenv import load_dotenv
from bitfab import ReplayRegistry
from lib.bitfab_client import bitfab
from scripts.replay_mocks import create_search_mock
from services.extraction import extract_memories
from services.search import search_documents
load_dotenv()
registry = ReplayRegistry()
registry.register("extraction", bitfab, extract_memories)
registry.register(
"search",
bitfab,
search_documents,
# Plain handler functions need the key explicitly. A @span-decorated
# function carries its key, so trace_function_key can be omitted.
trace_function_key="my-search-pipeline",
mock="marked",
options_factory=lambda ctx: {
"mock_override": create_search_mock(ctx.params.get("scenario"))
},
)
```
Keep non-trivial executable configuration in a sibling module and import it into the registry. For example, `replay_mocks.py` can export a `MockOverride` factory whose value callable uses live replay inputs and caller-supplied parameters. `options_factory` receives JSON values loaded from `--params ` and repeated `--param name=value`; direct parameters override file values.
Run the executable with `bitfab-replay --registry scripts/replay_registry.py `. The registry module must define the variable `registry`.
| Flag | Value | Effect |
| ------------------------------------ | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `--limit` | `N` | Most traces to select. It bounds a `--trace-ids` or `--dataset-ids` selection instead of replacing it |
| `--trace-ids` | `id1,id2` | Replay exactly these traces. Mutually exclusive with `--dataset-ids` |
| `--dataset-ids` | `uuid1,uuid2` | Replay the membership of one or more datasets, as their deduped union. Mutually exclusive with `--trace-ids`. `--dataset-id` is the same flag |
| `--name` | `NAME` | Title for the resulting experiment |
| `--attempts` | `N` | Replay each selected trace N times in one run (1 to 100, default 1) |
| `--concurrency`, `--max-concurrency` | `N` | Items in flight at once |
| `--dry-run` | | Resolve every item's inputs and stop without calling the function |
| `--experiment-group-id` | `UUID` | Add this run to an existing experiment group |
| `--grader-ids` | `id1,id2` | Attach graders to this run only |
| `--only-with-assertions` | | Replay only the selected traces that carry an assertion |
| `--code-change` | `PATH` | Load a code-change description from a file |
| `--no-code-change` | | Record no code change, overriding a registry default |
| `--mock` | `none\|all\|marked` | Which recorded child spans return their historical output |
| `--db-branch`, `--no-db-branch` | | Turn per-item database branching on or off |
| `--seed` | `cases.jsonl` | Run the file's cases once each and record them instead of replaying. See [Seeding traces](#seeding-traces) |
| `--params` | `PATH` | JSON file of values passed to `options_factory` |
| `--param` | `name=value` | One value passed to `options_factory`. Repeatable, and overrides `--params` |
| `-h`, `--help` | | Print usage |
Pass static function-specific behavior such as `adapt_inputs`, `mock_override`, or `db_branch` to `register`; build parameterized behavior with `options_factory`. Command-line values override overlapping scalar defaults without removing unrelated executable options. Unknown registry option names fail when the module loads instead of reaching replay as misspelled keywords.
#### Grading each attempt as it finishes
`on_item_finish` is the one lifecycle callback a registry entry can set. The command runs its own progress reporter first, then calls yours with the same finished item. That is the earliest point a replay verdict can be written: the row is keyed by lineage, so the server only resolves it once the attempt's own trace has been flushed.
```python theme={null}
def grade(progress):
item = progress["item"]
target = dict(
original_trace_id=item["original_trace_id"],
attempt=item["attempt"],
test_run_id=progress["test_run_id"],
)
if item["error"] is not None:
client.labels.skip(**target)
return
assertions = client.traces.get_assertions(item["original_trace_id"])
assessments = [
{
key: assertion[key]
for key in (
"assertion",
"passCriteria",
"failCriteria",
"targetOnEvaluatedTrace",
)
}
for assertion in assertions["assertions"]
]
verdict = judge(assessments, item["result"])
client.labels.save(label=verdict.passed, annotation=verdict.summary, **target)
registry = ReplayRegistry().register(
"support-agent", client, run_agent, on_item_finish=grade
)
```
It is skipped under `--dry-run`, since nothing ran and there is no replay trace to grade, and a callback that raises is reported on stderr naming the trace and attempt instead of failing the run.
Passing `assertion_id` scores one assertion instead of the whole trace. Judging each assertion on its own gives you a verdict per row rather than a single pass or fail for the attempt.
```python theme={null}
def grade_each_assertion(progress):
item = progress["item"]
target = dict(
original_trace_id=item["original_trace_id"],
attempt=item["attempt"],
test_run_id=progress["test_run_id"],
)
assertions = client.traces.get_assertions(item["original_trace_id"])
for assertion in assertions["assertions"]:
assessment = {
key: assertion[key]
for key in (
"assertion",
"passCriteria",
"failCriteria",
"targetOnEvaluatedTrace",
)
}
verdict = judge_one(assessment, item["result"])
client.labels.save(
label=verdict.passed,
annotation=verdict.summary,
assertion_id=assertion["id"],
**target,
)
```
`humanNote` is returned with each assertion so your tooling can display people-only context. It is written through MCP or Studio, not SDK saves. Do not include it in the object sent to an LLM judge; verdict evidence is the assertion, its pass/fail criteria, its target, and the evaluated trace.
`skip` and `archive` take `assertion_id` too, so an assertion whose target could not be resolved is withheld on its own without touching the others.
#### Grading inside the process that replayed the item
`on_item_finish` on the registry entry runs in the process that owns the run. Under `primitive="process"` that is the parent, which never executed a line of your code, so whatever the replayed run left in memory is gone by the time it fires. Assertions the run accumulated, a sandbox database it opened, a module-level registry it populated: none of it exists there.
`ReplayConcurrency` takes `on_item_finish_in_child_process` for that case. It runs in the child interpreter that replayed the item, after the item's spans are confirmed delivered and its result file is written, and before that child exits. The name says where it runs, and it lives on the concurrency object because that object is what creates the child, so both halves of the answer are visible at the point you declare it.
```python theme={null}
def grade_where_it_ran(event):
item = event["item"]
verdict = judge(collected_assertions(), item["result"])
client.labels.save(
label=verdict.passed,
annotation=verdict.summary,
trace_id=item["trace_id"],
)
registry = ReplayRegistry().register(
"support-agent",
client,
run_agent,
concurrency=ReplayConcurrency(
primitive="process", on_item_finish_in_child_process=grade_where_it_ran
),
)
```
It is refused under any other primitive, since there is no child process to run it in, and the error points you at the registry option instead. That is deliberate: a hook whose whole purpose is the child should not quietly relocate when you change how work is fanned out.
`item["trace_id"]` is the server's own ID for the replay trace this item produced, so a verdict can be keyed by it directly instead of by `original_trace_id` plus `attempt` plus `test_run_id`. It is `None` only when delivery could not be confirmed in time, which leaves lineage as the key.
##### Setting both
The two hooks compose rather than compete, and setting both is the expected shape.
| | `on_item_finish` | `on_item_finish_in_child_process` |
| ---------------------------------- | -------------------------------------------- | --------------------------------- |
| Set on | the registry entry, or `replay()` | `ReplayConcurrency` |
| Runs in | the process that owns the run | the child that replayed the item |
| Payload | `ReplayItemFinishProgress` | `ReplayItemFinishEvent` |
| Running totals | `completed`, `total`, `succeeded`, `errored` | none, a child ran one item |
| Sees the replayed run's memory | no, under `primitive="process"` | yes |
| Fires for an item whose child died | yes | no, that child never reached it |
| Available under | every primitive | `primitive="process"` only |
Each fires exactly once per item. The child's runs first, because the parent only learns an item exists once its child has exited, and the child runs the hook before exiting. So a slow judge in the child delays that item's progress event in the parent, and never the reverse.
Split the work along what each one can see. The child grades, because it is the only place the run's own state exists. The parent streams, because running totals are only countable where every item lands, and it is the only place an item whose child died shows up at all, which is where a crashed attempt gets recorded as skipped.
```python theme={null}
def stream_progress(progress):
dashboard.update(done=progress["completed"], total=progress["total"])
registry = ReplayRegistry().register(
"support-agent",
client,
run_agent,
on_item_finish=stream_progress,
concurrency=ReplayConcurrency(
primitive="process", on_item_finish_in_child_process=grade_where_it_ran
),
)
```
`ReplayItemFinishEvent` carries `test_run_id` and the finished `item`, and both keys also appear on `ReplayItemFinishProgress`, so one function can serve either hook if it reads nothing else.
##### Operational notes
Anything the child hook prints goes to that child's log, which the command surfaces only when the item fails. Record verdicts through the labels API rather than stdout.
Both hooks are skipped under `--dry-run`, since nothing ran and there is no replay trace to grade. A callback that raises is reported instead of failing the run, naming the trace and attempt, and each reports under its own name, so you can tell `on_item_finish_in_child_process failed` from `on_item_finish failed`. A child hook's failure is forwarded to the command's own stderr as well, since the child's log is otherwise read only when the item itself failed.
A child writes its result file before running the hook, so the parent's copy of the item never depends on what the hook does. The one case the child hook can be skipped is a child killed in the window between writing its result and finishing the hook.
Cross-item state does not belong in the child hook. Each child sees only its own item, so a running tally kept there counts to one. Keep those in the parent hook or read them off the returned `ReplayResult`.
### Seeding traces
Replay needs a trace to replay. Until production has produced one, there is nothing to select, so a corpus you already hold (a dataset export, a spreadsheet, hand-written cases) cannot be run. `seed_trace` runs your function once against a case and records that run as a replayable **original** trace, returning its trace ID.
```python theme={null}
from bitfab import Bitfab
from services.agent import run_ticket
bitfab = Bitfab(api_key="...", capture_enabled=False)
trace_id = bitfab.seed_trace(
"agent-turn",
run_ticket,
kwargs={"ticket_id": "T-1"},
metadata={"case_uid": "c-1", "suite": "smoke"},
name="T-1",
)
bitfab.replay(run_ticket, trace_ids=[trace_id])
```
The recorded trace carries the root span, the full first-party subtree, the real inputs, and whatever the run produced as the output. Capture stays off for everything else, so a seeding script does not have to run with tracing on for the rest of the process. The trace lands under `agent-turn` with `ingestion_type: seeded`, `replay` selects it like any captured trace, and every replay of it links back as `original_trace_id`.
Because a seeded trace has a full recorded subtree, replay mocking works on it exactly as it does on a captured trace. It carries no database pin, so `db_branch` refuses it.
`fn` resolves the same way it does for `replay`. A decorated function records under its own key, which must match the key you pass, and a plain callable is wrapped under the key here. An exception is recorded on the root span, the trace still persists, and the exception is re-raised. A call that records nothing (no API key resolved, or `fn` is a generator) raises rather than handing back an ID replay could never find.
Call `seed_trace` from synchronous code. An `async def` function runs to completion on a fresh event loop, and calling from inside a running loop raises. With `trace_across_threads=True`, spans from worker threads inside the call nest under the seeded root.
`name` is the trace's title and a searchable, filterable field. Put the case's own label there (a ticket ID, a dataset row name) so the seeded trace can be found by it. `metadata` is stored on the trace and handed to a later replay's `adapt_inputs` hook as `ctx["metadata"]`, so a case's provenance rides with the trace instead of through the recorded inputs. If the function also emits its own trace through an integration that exports trace metadata, the caller's metadata is merged onto that export and wins on any shared key.
#### Re-seeding a trace
A trace whose recorded run is wrong (it errored, or the world it ran against has moved on) can be re-seeded. `reseed_trace` reads the trace's recorded inputs, name, session, and metadata, runs the function once the way `seed_trace` does, and asks Bitfab to adopt that run under the same trace id.
```python theme={null}
result = bitfab.reseed_trace("agent-turn", run_ticket, trace_id="3f2a...")
result["trace_id"], result["previous_run_trace_id"]
```
The trace keeps its id, labels, assertions, dataset membership, name, and metadata, so anything you stored against it still names the same case. The previous run is kept as its own trace, `previous_run_trace_id`, with `reseedOfTraceId` pointing back at the case. Nothing is mocked and no experiment is created; a re-seed is a seed, not a replay. A run that raises is recorded but never adopted, so the trace is untouched, and Bitfab rejects a run that comes from another function or already belongs to a dataset. Graders on the datasets holding the trace re-run afterwards, and default replay selection skips previous runs.
From the shell, `bitfab-seed --from-trace [,...]` does the same through the replay registry:
```bash theme={null}
bitfab-seed --registry scripts/replay_registry.py extraction --from-trace 3f2a...
```
A re-seed runs the function exactly as production does, side effects included, so treat it like running the code, not like a replay.
#### Seeding a whole cases file
`seed_from_registry` seeds through an already-registered pipeline, reusing its client, callable, and trace function key, so every case runs through the exact function the later replay selects. The installed command does the same from the shell:
```bash theme={null}
bitfab-replay --registry scripts/replay_registry.py extraction --seed cases.jsonl
```
Each line of `cases.jsonl` is a JSON object with an `input` list plus optional `kwargs`, `metadata`, and `session_id`. `input` and `kwargs` are the call itself, recorded as-is. A case carrying `expected` is rejected, because the output is what the run produced. The registration's `adapt_inputs` is a replay hook and is not run at seed time, so a seeded trace is never adapted twice.
Replay reports a seeded item exactly as it reports a captured one, since a seeded output is a real run rather than an assertion. Each item's source ingestion type is available on the item, and a source with none reads as captured. See the [reference](/reference/python#seed-trace) for full signatures.
### Advanced Configuration
```python theme={null}
Bitfab(
api_key: str, # Required
service_url: str | None = None, # Default: https://bitfab.ai
env_vars: dict[str, str] | None = None, # For local function execution
capture_enabled: bool = True, # Capture traced calls
simulation_plan: bool = True, # Read and apply the sim plan
baml_client: Any = None # Generated BAML client (for wrap_baml)
)
```
* `env_vars`: Pass LLM provider API keys for local execution (e.g., `{"OPENAI_API_KEY": "..."}`)
* `capture_enabled`: When `False`, decorated functions still execute normally but no spans are sent. Replay records inside each item regardless, and `seed_trace` records the one call it runs regardless, so one client with capture off can still replay and seed. `enabled` is a deprecated alias.
* `simulation_plan`: When `False`, the sim plan is never read and content capture is never narrowed. See [Content capture from the sim plan](#content-capture-from-the-sim-plan).
* `baml_client`: The generated BAML client instance (e.g., `b` from `baml_client`). See [BAML framework guide](/frameworks/baml) for full usage.
## Datasets
`client.datasets` creates, reads, and modifies datasets programmatically, with the same operations your coding agent reaches through the Bitfab MCP tools. A dataset is a named bucket of traces under one trace function. Experiments replay against it and its graders score its members.
```python theme={null}
saved = bitfab.datasets.save(
"checkout-agent",
"Refund failures",
description="Checkout runs where the refund was declined",
)
dataset_id = saved["dataset"]["id"]
added = bitfab.datasets.add_traces(dataset_id, [trace_id])
if added["skippedTraceIds"]:
print("not in this trace function:", added["skippedTraceIds"])
trace_ids = bitfab.datasets.list_traces(dataset_id)["traceIds"]
bitfab.datasets.add_graders(dataset_id, [grader_id])
run = bitfab.datasets.rerun_graders(dataset_id)["run"]
print(run["status"], run["result"])
```
`save` is an upsert on the dataset name within its trace function, so re-running a script does not accumulate duplicates. Membership and grader calls accept up to 100 ids and report ids they skipped rather than failing the whole call. `remove_traces` only drops membership. Traces are never deleted. `rerun_graders` waits for the run by default (90 seconds, configurable) and returns whatever state it last saw. Pass `wait=False` to return immediately and poll with `get_grader_rerun`. See the [reference](/reference/python#datasets) for every method and result type.
## Labels
`client.labels` writes pass/fail verdicts and reads them back, the same operations your coding agent reaches through the `save_agent_labels`, `save_human_labels`, and `get_trace_labels` MCP tools. A verdict says how a run that already happened turned out. Written per assertion, it is stored and read back per assertion, so a judge inside a replay process can score each assertion on its own and verify what landed without opening Studio.
```python theme={null}
result = client.traces.get_assertions(item.original_trace_id)
client.labels.save_all(
[
{
"originalTraceId": item.original_trace_id,
"attempt": item.attempt,
"assertionId": assertion["id"],
"label": judge(assertion, output),
"annotation": explain(assertion, output),
}
for assertion in result["assertions"]
],
test_run_id=test_run_id,
)
labels = client.labels.get(item.trace_id)
for verdict in labels["assertions"]:
print(verdict["assertion"], verdict["label"], verdict["annotation"])
graded = client.graders.get_labels(trace_ids=[item.trace_id])
```
`save` and `save_all` write the agent's verdicts, which start unapproved until a person approves them in Studio. `save_human` and `save_human_all` write verdicts that are validated on write, for cases a person has already decided, such as a production bug captured as a regression test. Both batches are all-or-nothing: a trace outside the organization, a repeated target, or an assertion that is not active on its trace rejects the call and writes nothing. `get` and `get_all` return each trace's effective verdict plus one row per scored assertion. `graders.get_labels` is the per-grader breakdown the effective verdict folds together. Approving a verdict is not on this surface, or on MCP, by design. See the [reference](/reference/python#labels) for every method and type.
# Go SDK Reference
Source: https://docs.bitfab.ai/reference/go
Pure API reference for the bitfab-go module.
The module is `github.com/Project-White-Rabbit/bitfab-go`. It requires Go 1.25 or later. Runtime dependencies are `github.com/google/uuid`, `go.opentelemetry.io/otel`, `go.opentelemetry.io/otel/trace`, and `go.opentelemetry.io/otel/sdk`.
## Framework Integrations
No framework-native adapters are shipped for Go yet. Instrument Go code manually with `client.Span` or `client.Start`. See the [Go SDK guide](/go-sdk) for the walkthrough. The [Frameworks overview](/frameworks/overview) page tracks framework coverage across every SDK.
## Package Constants
```go theme={null}
const DefaultServiceURL = "https://bitfab.ai"
const Version = ""
const MaxSerializedValueBytes = 7_800_000
const MaxSpanCarrierBytes = 2_800_000
type CaptureWhen string
const (
CaptureWhenAlways CaptureWhen = "always"
CaptureWhenNested CaptureWhen = "nested"
)
```
`Version` is the value attached to SDK requests. The two byte limits are transport safeguards. The SDK enforces them automatically. They're exported so a caller can validate capture sizes before tracing.
## Constructor
### `NewClient`
```go theme={null}
func NewClient(apiKey string, opts ...Option) *Client
```
| Param | Type | Description |
| -------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `apiKey` | `string` | Falls back to `BITFAB_API_KEY` when empty or whitespace. If that's also empty, tracing warns and disables itself, unless `WithStrict(true)` is set |
| `opts` | `...Option` | Functional options |
### `Option`
```go theme={null}
type Option func(*Client)
```
#### `WithServiceURL`
```go theme={null}
func WithServiceURL(url string) Option
```
#### `WithEnabled`
```go theme={null}
func WithEnabled(enabled bool) Option
```
When `enabled` is `false`, `Span` still executes the callback. `Start` returns a no-op `*ActiveSpan`. No data is sent to the API.
#### `WithAPIKey`
```go theme={null}
func WithAPIKey(apiKey string) Option
```
It's equivalent to the `apiKey` argument of `NewClient`. Use it to construct a client purely from options. Whichever one is set last wins.
#### `WithStrict`
```go theme={null}
func WithStrict(strict bool) Option
```
Makes an unresolvable API key a fatal misconfiguration. `NewClient` panics instead of disabling tracing. It's off by default. That way, a missing telemetry key never crashes the host app. Turn it on in standalone programs where an untraced run is a failure you want surfaced immediately.
### Transport environment variables
| Variable | Default | Description |
| -------------------------------- | --------- | ------------------------------------------------------------------------------------ |
| `BITFAB_OTEL_MAX_REQUEST_BYTES` | `3000000` | Request-size target for OTLP/JSON exports. Accepts positive integers up to `3000000` |
| `BITFAB_OTEL_EXPORT_CONCURRENCY` | `32` | Concurrent direct requests per export window. Accepts `1` through `64` |
### Commit ref environment variables
| Variable | Effect |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `BITFAB_DISABLE_COMMIT_REF` | Set to any value to send no `commit_ref` at all. Neither the platform variables nor `git` are consulted |
| `BITFAB_COMMIT_SHA` | The commit sent as `commit_ref.sha` on every root trace. Wins over every deploy platform variable and over `git` |
| `VERCEL_GIT_COMMIT_SHA`, `GITHUB_SHA`, `RAILWAY_GIT_COMMIT_SHA`, `RENDER_GIT_COMMIT`, `SOURCE_VERSION`, `CF_PAGES_COMMIT_SHA`, `CI_COMMIT_SHA`, `BUILD_SOURCEVERSION`, `CIRCLE_SHA1` | Read in that order when `BITFAB_COMMIT_SHA` is unset, together with the same platform's branch and repository variables when it has them. The first one set wins, and `git` is never run |
### `CommitRef`
```go theme={null}
type CommitRef struct {
SHA string `json:"sha"`
Branch *string `json:"branch"`
Dirty *bool `json:"dirty"`
Remote *string `json:"remote"`
RootSHA *string `json:"root_sha"`
}
```
The commit the traced code was running at, sent as `commit_ref` on every root trace completion. `sha` is the commit. `branch` is the checked-out branch, or `nil` when detached or unknown. `dirty` is `true` when the working tree had uncommitted or untracked changes, `false` when it was clean, and `nil` when the SDK could not tell, which is always the case when the ref came from environment variables rather than `git`. `remote` is the `origin` URL reduced to `host/owner/repo` with any credentials removed, so a CI checkout token never reaches the trace. `root_sha` is the repository's first commit, so two checkouts of the same repository match even without a remote.
Resolution order is `BITFAB_COMMIT_SHA`, then the deploy platform's build variables, then `git` in the process's working directory (see [Commit ref environment variables](#commit-ref-environment-variables)). Environment resolution is synchronous and free. The `git` path runs once per process in a goroutine, with a two second timeout per command, so it never sits on the code path that ran the traced function. A trace that completes before it lands ships without a `commit_ref`, and a process with neither variables nor a repository never sends one. The result, including a negative one, is memoized for the life of the process. Set `BITFAB_DISABLE_COMMIT_REF` to opt the process out entirely. Nil pointer fields serialize as JSON `null`.
## `type Client`
Opaque struct. Construct via `NewClient`.
## Span lookup types
```go theme={null}
type SpanLookup struct {
ID string
Name string
Occurrence SpanOccurrence
}
const FirstSpanOccurrence SpanOccurrence = "first"
const LastSpanOccurrence SpanOccurrence = "last"
func SpanOccurrenceAt(index int) SpanOccurrence
```
```go theme={null}
type CapturedSpan struct {
ID string
TraceID string
ParentSpanID *string
Name *string
Type string
Input any
Output any
Contexts []ContextEntry
Prompt *string
Metadata map[string]any
Metrics map[string]any
Errors any
StartedAt *time.Time
EndedAt *time.Time
}
```
### `(c *Client) Span`
```go theme={null}
func (c *Client) Span(
ctx context.Context,
traceFunctionKey string,
fn SpanFunc,
opts ...SpanOption,
) (any, error)
```
Executes `fn` inside a traced span. `fn`'s return value is captured as the span output.
**Returns:** `fn`'s `(any, error)` return values.
**Errors:** Only `fn`'s own error. It's captured on the span. It's also returned to the caller. Tracing never fails the call. An unknown `WithType` value degrades to `custom` with a one-time warning, instead of returning an error. An internal instrumentation failure runs `fn` untraced instead of crashing the host.
### `(c *Client) Start`
```go theme={null}
func (c *Client) Start(
ctx context.Context,
traceFunctionKey string,
spanName string,
opts ...SpanOption,
) (context.Context, *ActiveSpan)
```
Start/End-style span for instrumenting existing functions. Always call `defer span.End()`.
Returns a child context that carries the span. It also returns an `*ActiveSpan` for recording data. When `enabled` is `false`, it returns the original context instead. The returned `*ActiveSpan` is then a zero value whose methods are all no-ops.
Spans exist only where you create them. Nesting between spans is automatic, but only among spans that already exist. Wrapping just the outermost function, for example, records a trace with a single node. See [Instrumentation](/instrumentation) for more.
### `(c *Client) FlushTraces`
```go theme={null}
func (c *Client) FlushTraces(timeout time.Duration) bool
```
Drains this client's pending span deliveries, up to `timeout`. Returns `false` when an export failed or the deadline expired. Use it for a mid-run flush, such as before reading a span back with `GetTraceSpan`.
### `(c *Client) Close`
```go theme={null}
func (c *Client) Close(timeout time.Duration) bool
```
Flushes pending spans. Then permanently shuts down this client's OpenTelemetry worker. It's idempotent. It returns `false` when an export failed or the deadline expired. A closed client no longer records spans.
**Go has no `atexit`. Always `defer client.Close(...)` in `main()`.**
### `(c *Client) GetFunction`
```go theme={null}
func (c *Client) GetFunction(traceFunctionKey string) *Function
```
### `(c *Client) GetTraceSpan`
```go theme={null}
func (c *Client) GetTraceSpan(
ctx context.Context,
traceID string,
lookup SpanLookup,
) (*CapturedSpan, error)
```
Fetches one persisted span without loading its trace. `traceID` is the canonical Bitfab trace ID. Set exactly one of the span's Bitfab `SpanLookup.ID` or `SpanLookup.Name`. Name lookup defaults to the last match. Use `FirstSpanOccurrence` or `SpanOccurrenceAt(index)` to override it. A miss returns `nil, nil`.
## Datasets
```go theme={null}
type DatasetsClient struct{ /* unexported */ }
func (d *DatasetsClient) Save(ctx context.Context, params SaveDatasetParams) (*SaveDatasetResult, error)
func (d *DatasetsClient) List(ctx context.Context, params ListDatasetsParams) ([]Dataset, error)
func (d *DatasetsClient) Get(ctx context.Context, datasetID string) (*Dataset, error)
func (d *DatasetsClient) ListTraces(ctx context.Context, datasetID string) (*DatasetTraceIDs, error)
func (d *DatasetsClient) AddTraces(ctx context.Context, datasetID string, traceIDs []string) (*AddDatasetTracesResult, error)
func (d *DatasetsClient) RemoveTraces(ctx context.Context, datasetID string, traceIDs []string) (*RemoveDatasetTracesResult, error)
func (d *DatasetsClient) AddGraders(ctx context.Context, datasetID string, graderIDs []string) (*AddDatasetGradersResult, error)
func (d *DatasetsClient) RemoveGraders(ctx context.Context, datasetID string, graderIDs []string) (*RemoveDatasetGradersResult, error)
func (d *DatasetsClient) RerunGraders(ctx context.Context, datasetID string, options RerunGradersOptions) (*RerunGradersResult, error)
func (d *DatasetsClient) GetGraderRerun(ctx context.Context, datasetID, runID string) (*GraderRerun, error)
```
The request and response types are:
```go theme={null}
type DatasetGraderRef struct {
ID string
Name *string
}
type Dataset struct {
ID string
TraceFunctionKey string
Name string
Description *string
TraceCount int
Graders []DatasetGraderRef
CreatedAt string
UpdatedAt string
}
type SaveDatasetParams struct {
TraceFunctionKey string
Name string
Description string
}
type SaveDatasetResult struct {
Dataset Dataset
Created bool
}
type ListDatasetsParams struct {
TraceFunctionKey string
}
type DatasetTraceIDs struct {
DatasetID string
TraceIDs []string
}
type AddDatasetTracesResult struct {
Dataset Dataset
AddedTraceIDs []string
AlreadyPresentTraceIDs []string
SkippedTraceIDs []string
}
type RemoveDatasetTracesResult struct {
Dataset Dataset
RemovedTraceIDs []string
NotPresentTraceIDs []string
}
type AddDatasetGradersResult struct {
Dataset Dataset
AddedGraderIDs []string
AlreadyAssignedGraderIDs []string
SkippedGraderIDs []string
}
type RemoveDatasetGradersResult struct {
Dataset Dataset
RemovedGraderIDs []string
NotAssignedGraderIDs []string
}
type GraderRerunStatus string
const (
GraderRerunPending GraderRerunStatus = "pending"
GraderRerunRunning GraderRerunStatus = "running"
GraderRerunCompleted GraderRerunStatus = "completed"
GraderRerunErrored GraderRerunStatus = "errored"
)
func (s GraderRerunStatus) Terminal() bool
type GraderRerunProgress struct {
CompletedTraces int
TotalTraces int
GraderCount int
}
type GraderRerunResult struct {
TracesGraded int
GradersRun int
}
type GraderRerun struct {
ID string
Status GraderRerunStatus
GraderIDs []string
Progress *GraderRerunProgress
Result *GraderRerunResult
Error *string
CreatedAt string
UpdatedAt string
}
type RerunGradersOptions struct {
GraderIDs []string
NoWait bool
Timeout time.Duration
PollInterval time.Duration
}
type RerunGradersResult struct {
Run GraderRerun
JoinedExisting bool
}
```
`client.Datasets` holds a `*DatasetsClient` for the authenticated organization. It exposes the same operations the Bitfab MCP tools expose to a coding agent.
* `Save` is an upsert keyed on `(TraceFunctionKey, Name)`. `Created` is `true` for a new dataset and `false` when an existing one was updated. An empty `Description` leaves an existing description untouched.
* `List` scopes to `params.TraceFunctionKey` when set. The zero value lists every dataset in the organization.
* `Dataset` carries `ID`, `TraceFunctionKey`, `Name`, `Description` (`*string`), `TraceCount`, `Graders` (`[]DatasetGraderRef` with `ID` and `Name`), `CreatedAt`, and `UpdatedAt`.
* `ListTraces` returns `DatasetTraceIDs{DatasetID, TraceIDs}`. This is the same membership a replay selects with `ReplayOptions.DatasetID`.
* `AddTraces` and `AddGraders` each accept 1 to 100 ids. They report partial acceptance instead of failing outright. An id outside the organization, or under another trace function, comes back in `SkippedTraceIDs` or `SkippedGraderIDs`. An id already present comes back in `AlreadyPresentTraceIDs` or `AlreadyAssignedGraderIDs`.
* `RemoveTraces` never deletes a trace. It only removes its membership in the dataset. Ids that were not members come back in `NotPresentTraceIDs`. `RemoveGraders` reports `NotAssignedGraderIDs` the same way.
* `RerunGraders` re-scores every trace in the dataset. `RerunGradersOptions.GraderIDs` defaults to every assigned grader. Passing an unassigned id fails the call. With the zero-value options, `RerunGraders` waits for the run to finish. It polls every `PollInterval`, which defaults to 1 second, up to `Timeout`, which defaults to 90 seconds. It returns the last `Run` state it saw. Set `NoWait` to return as soon as the run is queued instead. A request that matches an in-flight run joins it, reported as `JoinedExisting`. Cancelling `ctx` while waiting returns `ctx.Err()`.
* `GraderRerun` has `Status` (a `GraderRerunStatus`, one of `GraderRerunPending`, `GraderRerunRunning`, `GraderRerunCompleted`, `GraderRerunErrored`, with `Terminal()`), `GraderIDs`, `Progress` (`*GraderRerunProgress` while running), `Result` (`*GraderRerunResult` when completed), and `Error`.
* A dataset id from another organization fails with a 404 status error.
## Replay
### `(c *Client) Replay`
```go theme={null}
func (c *Client) Replay(
ctx context.Context,
traceFunctionKey string,
fn any,
options *ReplayOptions,
) (ReplayResult, error)
```
`Replay` fetches historical traces for `traceFunctionKey`. It re-runs their recorded inputs through `fn`. It waits for every emitted trace to persist. Then it completes the resulting experiment.
`fn` must be a non-nil function. It may take `context.Context` as its first parameter, followed by any number of JSON-decodable typed parameters. It may return zero or more values, plus an optional final `error`. A single non-error return becomes `ReplayItem.Result`. Multiple returns become `[]any` instead. Replay catches a panic from `fn`. It stores the panic as that item's `TraceError`.
Every invocation is automatically wrapped in a root span under `traceFunctionKey`. A production function that uses `Start` and `End` receives the replay root's context. As a result, its existing spans become children of that root.
### `type ReplayOptions`
```go theme={null}
type ReplayOptions struct {
Limit int
TraceIDs []string
Name string
MaxConcurrency int
CodeChangeDescription *string
CodeChangeFiles []CodeChangeFile
DisableCodeChangeCapture bool
Mock MockStrategy
MockOverrides []MockOverride
DBBranch *DBBranchOptions
ExperimentGroupID string
DatasetID string
DatasetIDs []string
GraderIDs []string
AdaptInputs ReplayInputAdapter
OnItemStart func(ReplayItemStartProgress)
OnItemFinish func(ReplayItemFinishProgress)
}
```
`Limit` defaults to `5`. It accepts values from `1` through `5000`. `MaxConcurrency` defaults to `10`. `TraceIDs` accepts at most `100` IDs. When it's present, it determines the item count on its own. `Limit` is then omitted from the start request.
When `CodeChangeFiles` is nil, replay first looks for `BITFAB_CODE_CHANGE_PATH`. If that's not set, it captures the rename-aware Git diff against trunk instead. An explicit slice always wins over the automatic capture. Set `DisableCodeChangeCapture`, or the `BITFAB_DISABLE_CODE_CHANGE_CAPTURE` environment variable, to opt out entirely. `BITFAB_CODE_CHANGE_BASE` overrides trunk detection.
`Mock` defaults to `MockMarked`. `MockNone` and `MockAll` select the other two strategies. `MockOverrides` take precedence over overrides registered on the client. Both take precedence over the base strategy. Mock interception only applies to closure-style child `Client.Span` calls, because those calls own execution. Manual `Start`/`End` instrumentation cannot prevent caller-owned code from running.
`DatasetID` and `DatasetIDs` are the same selector at two counts, so set one of them: one dataset on `DatasetID`, several on `DatasetIDs`. Several replays the union of their traces, deduped where they overlap, graded by the union of their graders, and attributes the experiment to every one of them, so it appears under each dataset's experiments.
A non-nil `DBBranch` enables a historical database branch. An empty `DBBranchOptions` uses the connected project's defaults. Set `MinCU`, `MaxCU`, or `WarmupSQL` to tune provisioning instead. Resolution happens inside the bounded item workers.
### Replay mock types
```go theme={null}
type MockStrategy string
type MockSource string
const (
MockNone MockStrategy = "none"
MockAll MockStrategy = "all"
MockMarked MockStrategy = "marked"
MockSourceRecorded MockSource = "recorded"
MockSourceOverride MockSource = "override"
)
type MockOverride struct {
Match NodeMatcher
Value any
Resolve MockValueFunc
}
type MockOverrideContext struct {
Node SpanNodeMeta
Inputs []any
GetOriginalOutput func() (any, error)
}
type SpanNodeMeta struct {
TraceFunctionKey string
SpanName string
Type string
OriginalSpanID string
}
type NodeMatcher func(SpanNodeMeta) bool
type MockValueFunc func(MockOverrideContext) (any, error)
```
`Resolve` takes precedence over `Value`. A nil `Value` is a valid flat override. The first matching override wins. `GetOriginalOutput` fetches the matched historical output lazily. It memoizes that output per replay item.
```go theme={null}
func (c *Client) RegisterMockOverride(override MockOverride) error
func (c *Client) ClearMockOverrides()
```
Per-call overrides are evaluated before client-registered overrides. A mocked span upload carries `mocked: true`, `mockTarget: "output"`, and a `mockSource` of `recorded` or `override`.
### Bound replay functions
```go theme={null}
func BindReplayFunction(traceFunctionKey string, fn any) ReplayFunction
func (f *Function) BindReplay(fn any) ReplayFunction
func (f *Function) Replay(ctx context.Context, fn any, options *ReplayOptions) (ReplayResult, error)
```
Go function values cannot carry decorator metadata. These APIs explicitly bind a callable to its declared key. That lets `Client.Replay` reject a mismatch before starting the experiment. Plain callables remain valid for handler-instrumented workflows with no declared Go root.
### Historical database branch
```go theme={null}
type DBBranchOptions struct {
MinCU float64
MaxCU float64
WarmupSQL string
}
type ReplayBranch struct {
NeonBranchID string
EnvKey string
ExpiresAt string
SnapshotTimestamp string
ProviderConsoleURL string
ReadOnly *bool
Region string
TraceID string
Extra map[string]any
}
type DBSnapshotRef struct {
Provider string
SDKWallClockBeforeFn string
Origin string
}
type DBBranchTimings struct {
StartedAt string
ProjectResolveMS *float64
BranchCreateMS *float64
ConnectionURIMS *float64
ComputeConnectMS *float64
BaseProbeMS *float64
WarmupMS *float64
TotalMS float64
}
type DBBranchReplayError struct {
Code string
Message string
OriginalTraceID string
Cause error
}
func GetCurrentReplayBranch(ctx context.Context) *ReplayBranch
func (branch *ReplayBranch) DatabaseURL() string
```
Inside a replay item whose source trace has a resolvable snapshot, `GetCurrentReplayBranch` returns that item's `*ReplayBranch`. Outside replay, or when the source trace has no resolvable snapshot, it returns nil instead. `DatabaseURL()` is the only accessor that exposes the connection string. Calling it also marks the trace's `db_snapshot_usage.accessed` flag. JSON encoding and `String()` both omit the URL. The branch also exposes its Neon branch ID, environment key, expiration, snapshot timestamp, console URL, read-only flag, region, and source trace ID as direct fields. Any other field the server adds later comes through `Extra`.
Resolution failures produce a `*DBBranchReplayError` on the item's `ReplayError`. It preserves `Code`, `OriginalTraceID`, and `Cause`. `ReplayItem.DBSnapshotRef` and `ReplayItem.DBBranchTimings` expose the historical pin and the server-measured provisioning phases.
### Replay input adaptation
```go theme={null}
type ReplayInputAdapter func(inputs []any, ctx AdaptContext) ([]any, error)
type AdaptContext struct {
OriginalTraceID string
OriginalSpanID string
SourceTraceID string // deprecated alias
SourceSpanID string // deprecated alias
}
```
The adapter runs before typed parameter decoding. Its returned slice is passed positionally to `fn`. That same slice is stored as `ReplayItem.Input`. An adapter error becomes the item's `ReplayError`. It does not stop other items from running.
### Replay results and errors
```go theme={null}
type ReplayResult struct {
Items []ReplayItem
TestRunID string
TestRunURL string
}
type ReplayItem struct {
TraceID *string
OriginalTraceID string
OriginalSpanID string
SourceTraceID string // deprecated alias
SourceSpanID string // deprecated alias
Input []any
Result any
OriginalOutput any
Error *string
TraceError error
ReplayError error
DurationMS *int64
OriginalDurationMS *int64
OriginalTokens *TokenUsage
OriginalModel *string
Tokens *TokenUsage
Model *string // deprecated original-model alias
DBSnapshotRef *DBSnapshotRef
DBBranchTimings *DBBranchTimings
TraceOutline *TraceOutline // the replayed trace's span tree, no payloads
OriginalTraceOutline *TraceOutline // the original trace's span tree, no payloads
}
type TraceOutline struct {
TraceID string
Name *string
Status string
TraceFunctionKey *string
DurationMS *int64
SpanCount int
Spans []TraceOutlineSpan // root spans in start order
}
type TraceOutlineSpan struct {
SpanID string
Name *string
Type string // "llm", "agent", "function", "guardrail", "handoff", or "custom"
TraceFunctionKey *string
DurationMS *int64
Tokens *TokenUsage
Model *string
Errors []TraceOutlineSpanError
Mocked bool // served from the recorded output instead of re-executing
Children []TraceOutlineSpan
}
type TraceOutlineSpanError struct {
Source string
Error string
Step *string
}
type ReplayError struct {
Message string
Items []ReplayItem
TestRunID string
TestRunURL string
Cause error
}
```
`ReplayItem` carries everything about one replayed trace, including:
* `Input` (the adapted recorded input), `Result`, and `OriginalOutput`
* A compatible `Error` message, plus the actual `TraceError` and `ReplayError`
* The replay's own `DurationMS`, plus the original trace's duration, token, and model measurements
* The replay's own token usage
* The original trace and span lineage
* The new server `TraceID`
* The two trace outlines
`TraceOutline` is the replayed trace's span tree. `OriginalTraceOutline` is the original trace's span tree. Neither carries inputs or outputs. Each span in an outline records its name, type, nesting, start order, duration, tokens, model, errors, and whether it was mocked.
Both outlines are nil until the run completes, so they're also nil in every `OnItemFinish` event. They're nil against an older server that doesn't build them, too. `TraceOutline` is nil for one more case, an item whose replay produced no trace at all.
These outlines exist for grading. Comparing the two trees shows whether a replay reached its output by the same path as the original.
Per-item function and setup errors stay on their items. A run-wide persistence or completion failure returns `*ReplayError`. Use `errors.As` and `errors.Unwrap` to inspect it without losing the partial items it collected.
### Replay progress and serialization
```go theme={null}
type ReplayItemStartProgress struct {
Type string
TestRunID string
Started int
Completed int
Total int
Succeeded int
Errored int
Item AdaptContext
}
type ReplayItemFinishProgress struct {
TestRunID string
Completed int
Total int
Succeeded int
Errored int
Item ReplayItem
}
func ReportReplayProgress(progress any)
func SerializeReplayResult(result ReplayResult) (string, error)
```
`OnItemStart` fires when a worker begins an item. This happens before replay loads its inputs, prepares mocks, or resolves a database branch. Its `Item` field carries only the original trace and span lineage at that point, because the rest of `ReplayItem` doesn't exist yet.
`OnItemFinish` fires exactly once per item, as it finishes. Items finish in completion order, not input order. `OnItemFinish` carries the full `ReplayItem`.
Both callbacks report running totals, `Completed`, `Succeeded`, `Errored`, and `Total`. Replay doesn't know pass or fail at finish time, since verdicts are assigned later. The totals only split runs that completed without error from runs that errored.
A panicking callback is recovered. It never crashes the run.
`ReportReplayProgress` writes one `@@bitfab:progress {json}` line to stderr. Pass it from both callbacks:
```go theme={null}
options := &bitfab.ReplayOptions{
OnItemStart: func(p bitfab.ReplayItemStartProgress) {
bitfab.ReportReplayProgress(p)
},
OnItemFinish: func(p bitfab.ReplayItemFinishProgress) {
bitfab.ReportReplayProgress(p)
},
}
```
`OnItemFinish` flushes the finished trace. When that trace's delivery is acknowledged, `OnItemFinish` fills the event's server `TraceID` directly from the successful ingestion response. If per-item delivery can't be confirmed, `TraceID` stays nil in the event instead. The run-wide persistence barrier then falls back to the status endpoint before filling in the final result. Replay aggregates token usage only at completion. Because of that, token usage appears only in the final `ReplayResult`, not in progress events. A panic inside a callback is swallowed.
If `BITFAB_REPLAY_RESULT_PATH` is set, a successful `Replay` writes the structured result JSON there with one trailing newline.
## Payload serialization helpers
```go theme={null}
func MarshalSpanPayload(payload map[string]any) ([]byte, error)
func UnmarshalSpanPayload[T any](data []byte) (T, error)
```
These helpers expose the SDK's JSON round trip for applications that need to preflight a captured value. Normal tracing and replay do not require calling them.
Standard interface methods such as `Error`, `Unwrap`, `MarshalJSON`, `String`, and `Format` are documented with their owning replay types, rather than as independent entry points. `TraceState` is exported. This lets current-span handles share state across package boundaries. It's transport plumbing, though, not an application-facing API.
## `type SpanFunc`
```go theme={null}
type SpanFunc func(ctx context.Context) (any, error)
```
## `type SpanOption`
```go theme={null}
type SpanOption func(*spanConfig)
```
### `WithName`
```go theme={null}
func WithName(name string) SpanOption
```
Defaults to `traceFunctionKey` for `Span`, or the `spanName` arg for `Start`.
### `WithType`
```go theme={null}
func WithType(spanType string) SpanOption
```
One of `"llm"`, `"agent"`, `"function"`, `"guardrail"`, `"handoff"`, `"custom"`. An unknown value warns once. It then degrades to `"custom"` without changing the user's result. Defaults to `"custom"`.
### `WithMockOnReplay`
```go theme={null}
func WithMockOnReplay(mock bool) SpanOption
```
Marks a closure-style child `Span` for recorded-output substitution under `MockMarked`. It has no effect outside replay. A selected occurrence that is absent from the historical tree returns an error without executing the real callback.
### `WithMockOutputType`
```go theme={null}
func WithMockOutputType[T any]() SpanOption
```
Decodes a recorded or overridden JSON output into `T` before a mocked closure-style `Span` returns it. Use it for structs, slices, and other concrete outputs that would otherwise arrive as JSON-shaped `map[string]any` or `[]any` values. Primitive and deliberately dynamic outputs do not need it.
### `WithFunctionName`
```go theme={null}
func WithFunctionName(name string) SpanOption
```
Recorded as `span_data.function_name`.
### `WithInput`
```go theme={null}
func WithInput(args ...any) SpanOption
```
One arg stored directly. Multiple args stored as a slice. Only relevant to `Span`. For `Start`, use `ActiveSpan.SetInput`.
### `WithCaptureWhen`
```go theme={null}
func WithCaptureWhen(captureWhen CaptureWhen) SpanOption
```
`CaptureWhenNested` records the span only when `ctx` contains an active Bitfab parent span. Without a parent, `Span` runs the callback untraced. `Start` returns the original context with a no-op `ActiveSpan` instead. The default is `CaptureWhenAlways`. An unknown value warns once. It then falls back to that default.
## `type Function`
```go theme={null}
type Function struct { /* ... */ }
```
Obtained via `(*Client).GetFunction(key)`. Fluent wrapper that binds `traceFunctionKey`.
### `(f *Function) Span`
```go theme={null}
func (f *Function) Span(ctx context.Context, fn SpanFunc, opts ...SpanOption) (any, error)
```
### `(f *Function) Start`
```go theme={null}
func (f *Function) Start(ctx context.Context, spanName string, opts ...SpanOption) (context.Context, *ActiveSpan)
```
## `type ActiveSpan`
Returned by `Start`. All methods are safe on `nil`, safe under `recover()`, and idempotent where noted.
### `(s *ActiveSpan) SetInput`
```go theme={null}
func (s *ActiveSpan) SetInput(args ...any)
```
One arg stored directly. Multiple args stored as a slice. No-op on `nil` receiver.
### `(s *ActiveSpan) SetOutput`
```go theme={null}
func (s *ActiveSpan) SetOutput(output any)
```
### `(s *ActiveSpan) SetError`
```go theme={null}
func (s *ActiveSpan) SetError(err error)
```
### `(s *ActiveSpan) AddContext`
```go theme={null}
func (s *ActiveSpan) AddContext(context map[string]any)
```
Appends the map as one entry on `span_data.contexts`. No-op when `context == nil`.
### `(s *ActiveSpan) SetPrompt`
```go theme={null}
func (s *ActiveSpan) SetPrompt(prompt string)
```
Overwrites `span_data.prompt`. No-op on empty string.
### `(s *ActiveSpan) End`
```go theme={null}
func (s *ActiveSpan) End()
```
Idempotent via `sync.Once`. Sends the span in a background goroutine. Any panic inside the send path is recovered. That keeps the host app from crashing.
## Trace-Level API
### `type ContextEntry`
```go theme={null}
type ContextEntry = map[string]any
```
### `type CurrentSpan`
```go theme={null}
type CurrentSpan struct { /* ... */ }
func GetCurrentSpan(ctx context.Context) *CurrentSpan
func (cs *CurrentSpan) ID() string
func (cs *CurrentSpan) TraceID() string
```
Returns the active span's canonical Bitfab span and trace IDs. `GetCurrentSpan` returns `nil` outside a span. Both methods are safe on a `nil` receiver. They return an empty string in that case.
### `type CurrentTrace`
```go theme={null}
type CurrentTrace struct { /* ... */ }
```
Returned by `GetCurrentTrace`.
#### `(ct *CurrentTrace) TraceID`
```go theme={null}
func (ct *CurrentTrace) TraceID() string
```
Returns the canonical Bitfab trace ID. It's safe on a `nil` receiver. It returns an empty string in that case.
#### `(ct *CurrentTrace) SetSessionID`
```go theme={null}
func (ct *CurrentTrace) SetSessionID(sessionID string)
```
#### `(ct *CurrentTrace) SetName`
```go theme={null}
func (ct *CurrentTrace) SetName(name string)
```
Sets the trace's title in Bitfab. The title is a searchable and filterable field, stored in the trace's `name` column. Unset, the trace is titled by its trace function key instead. Empty strings are ignored. It's safe on a `nil` receiver.
#### `(ct *CurrentTrace) SetMetadata`
```go theme={null}
func (ct *CurrentTrace) SetMetadata(metadata map[string]any)
```
Shallow-merges with existing trace metadata. Later keys win.
#### `(ct *CurrentTrace) AddContext`
```go theme={null}
func (ct *CurrentTrace) AddContext(context map[string]any)
```
Appends. Accumulates across calls.
#### `(ct *CurrentTrace) Drop`
```go theme={null}
func (ct *CurrentTrace) Drop()
```
Flags the trace to be dropped. Once the flag is set, no spans that complete afterward are uploaded at all. The flag itself rides along on the trace's completion payload. At completion, the server does the rest of the cleanup:
* It scrubs any payloads that already raced out ahead of the flag, meaning the trace, its external trace, and any sibling spans
* It deletes the archived S3 objects
* It marks the trace `dropped` instead of `completed`, keeping only a skeleton audit row
`Drop` is safe on a `nil` receiver. That makes it a no-op, so calling `GetCurrentTrace(ctx).Drop()` outside a span does nothing. `Drop` never panics.
### `GetCurrentTrace`
```go theme={null}
func GetCurrentTrace(ctx context.Context) *CurrentTrace
```
Returns `nil` when `ctx` carries no active span. Callers must nil-check.
## Concurrency Model
* Nested span context is carried on `context.Context`. It's safe across goroutines that inherit that context
* Goroutines that do **not** inherit the context will not see the parent span
* Spans are queued on a private OpenTelemetry batching worker. Each client gets its own worker, started lazily on the first span it sends. Queueing itself never blocks. `FlushTraces` and `Close` are what drain the worker
* Trace state is stored in a mutex-protected package-level map, keyed by `traceID`
## Error Behavior Summary
| Situation | Behavior |
| -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Empty `apiKey` | Falls back to `BITFAB_API_KEY`. If that's also empty, normal mode logs a warning and disables tracing. `WithStrict(true)` makes `NewClient` panic instead |
| `SpanFunc` returns error | Captured on span with `error_source: "code"`, returned to caller |
| Invalid `WithType` value | Warns once. Records the span as `custom`. User code still runs normally |
| Panic inside user code | Not recovered. Your code panics. The SDK only recovers panics on its own send path |
| Span transport failure | Swallowed. User's return value / error passes through |
| `ActiveSpan` methods on `nil` receiver | No-op |
| `End()` called multiple times | Only the first call has effect (`sync.Once`) |
| `GetCurrentTrace(ctx)` outside a span | Returns `nil` |
# HTTP Endpoints
Source: https://docs.bitfab.ai/reference/http
SDK-facing HTTP endpoints. Every SDK speaks this wire protocol.
All SDKs POST to the same set of endpoints under `{serviceUrl}/api/sdk/`. This page documents the wire protocol so you can integrate directly without using an SDK, or debug SDK behavior.
Base URL defaults to `https://bitfab.ai`.
## Authentication
All endpoints require the API key in the `Authorization` header:
```
Authorization: Bearer {apiKey}
```
Missing, empty, or invalid keys return `401 Unauthorized`.
## Request Compression
Request bodies may be sent uncompressed or compressed with `gzip` (also accepted
as `x-gzip`) or `deflate`, declared with the `Content-Encoding` header. This is
what lets an OpenTelemetry Collector export to Bitfab with its default `gzip`
compression without extra configuration.
Any other `Content-Encoding`, including `zstd`, `snappy`, and `br`, returns
`415 Unsupported Media Type`.
A compressed body that expands beyond 8,000,000 bytes returns
`413 Payload Too Large`. Bitfab's SDKs target at most 3,000,000 bytes on the
wire and never send more than 8,000,000 decompressed bytes.
Beneath that, every SDK lets a single span's whole payload (its input, output,
contexts, prompt, and metadata together) use up to 7,800,000 carrier bytes when
its single-span request gzips below the 3,000,000-byte wire target. The carrier
is the payload re-escaped into its OTLP attribute. If the request does not
compress enough, compression is disabled, or preparing it fails, the SDK
replaces the largest fields with ``
placeholders until the carrier fits the 2,800,000-byte fallback budget. See
[the OTel architecture](/otel-architecture) for the full behavior.
The SDKs gzip their own requests once a body reaches 8,192 bytes and gzip makes
it smaller. Smaller or incompressible requests are sent uncompressed.
Set `BITFAB_DISABLE_COMPRESSION` to any value to send everything uncompressed.
## Common Request Fields
The SDKs append these fields to most request bodies:
| Field | Type | Description |
| ------------ | -------- | ----------------------------------------------------- |
| `sdkVersion` | `string` | The SDK package version |
| `source` | `string` | Origin tag - `"typescript-sdk"`, `"python-sdk"`, etc. |
## Endpoints
### `POST /api/sdk/functions/lookup`
Look up a function by name. Blocking.
**Request:**
```json theme={null}
{ "name": "my-function-key" }
```
**Response 200:**
```json theme={null}
{
"id": "uuid",
"name": "my-function-key",
"versionId": "uuid",
"versionNumber": 3,
"prompt": "...",
"providers": [ /* ProviderDefinition[] */ ]
}
```
**Response (not found):** `{ "id": null }` - SDKs raise a typed error pointing users to `/functions`.
### `POST /api/sdk/functions/{functionId}/traces`
Record a trace produced by a server-side function execution (e.g. `client.call(...)`). Fire-and-forget.
**Request:**
```json theme={null}
{
"result": "...",
"source": "typescript-sdk",
"inputs": { "...": "..." },
"rawCollector": { /* BAML collector snapshot, optional */ },
"sdkVersion": "..."
}
```
**Response:** `200 OK`.
### `POST /api/sdk/externalSpans`
Record a single span produced by `withSpan` / `@span` / `Span` / `bitfab_span`. Fire-and-forget.
**Request:**
```json theme={null}
{
"id": "canonical Bitfab span UUID",
"traceId": "canonical Bitfab trace UUID",
"type": "sdk-function",
"source": "typescript-sdk-function",
"sourceTraceId": "uuid",
"traceFunctionKey": "my-function-key",
"rawSpan": {
"id": "uuid",
"trace_id": "uuid",
"parent_id": "uuid | null",
"started_at": "ISO8601",
"ended_at": "ISO8601",
"input_source_span_id": "uuid | null",
"span_data": {
"name": "...",
"type": "llm|agent|function|guardrail|handoff|custom",
"input": {/* json */},
"output": {/* json */},
"input_meta": {/* superjson meta */},
"output_meta": {/* superjson meta */},
"function_name": "...",
"error": "...",
"contexts": [{}],
"prompt": "..."
}
},
"testRunId": "uuid (optional, set during replay)",
"sdkVersion": "..."
}
```
**Response:** `200 OK`.
### `POST /api/sdk/otel/v1/traces`
Record a batch from the TypeScript, Python, Ruby, or Go SDK, or from an OpenTelemetry
Collector, using the standard OTLP/JSON `ExportTraceServiceRequest` shape.
Those SDKs enrich each OTel span with two string attributes:
* `bitfab.operation`: `external_span`, `external_trace`, or `internal_trace`
* `bitfab.payload`: the JSON-encoded Bitfab replay/rendering payload
The endpoint validates each enriched payload with the same schemas used by
`externalSpans`, `externalTraces`, and `functions/{functionId}/traces`. Because
OTLP carriers have no URL to read path parameters from, an `internal_trace`
carrier holds that route's body plus a `functionId` field. Ordinary OTel spans
without those Bitfab attributes are ignored. A fully successful OTLP/JSON
response is `{}`. If one or more enriched spans are malformed or unsupported,
valid carriers in the same request are still ingested and the response uses the
standard `partialSuccess` object with `rejectedSpans` and `errorMessage`.
Those SDKs keep each request at most around 3 MB even when OTel's
count-based processor produces a larger candidate batch. Direct requests also
contain at most 128 carriers in TypeScript or eight in Python, Ruby, and Go. Set
`BITFAB_OTEL_MAX_REQUEST_BYTES` to a positive integer no greater than
`3000000` to use a smaller request target. An oversized carrier gets its own
request. The SDK tries compression and, if needed, trims the largest payload
fields to placeholders before reporting an export failure if the carrier still
cannot fit. HTTP 413 responses from ingress are reported as export failures.
See [Batching and payload size](/otel-architecture#batching-and-payload-size)
for the wire limits and trimming behavior.
When configuring a Collector, set its `otlphttp` exporter endpoint to
`https://bitfab.ai/api/sdk/otel`, use JSON encoding, and supply the Bitfab API
key as an `Authorization: Bearer ...` exporter header. The exporter appends
`/v1/traces`.
See [OpenTelemetry Transport Architecture](/otel-architecture)
for the carrier model, direct and Collector flows, size partitioning, and
replay persistence barrier.
### `POST /api/sdk/externalTraces`
Record the completion of a root span's trace. Sent once per trace when the outermost span ends. Fire-and-forget.
The request's top-level `id` is the canonical Bitfab trace UUID. The nested raw trace ID remains an ingestion correlation value.
**Request:**
```json theme={null}
{
"type": "sdk-function",
"source": "typescript-sdk-function",
"traceFunctionKey": "my-function-key",
"externalTrace": {
"id": "uuid",
"started_at": "ISO8601",
"ended_at": "ISO8601",
"name": "Ticket 4521 (optional)",
"ingestion_type": "captured | seeded (optional)",
"metadata": {/* optional */},
"contexts": [{/* optional */}]
},
"completed": true,
"sessionId": "... (optional)",
"sdkVersion": "..."
}
```
`name` is the trace's title: what Bitfab shows for the trace, and a field `search_traces` and the trace list filter match on. Unset, the trace is titled by its trace function key. `workflow_name` is accepted as a legacy alias of `name` (older SDKs sent the trace function key there, and the OpenAI Agents SDK emits it natively); when both are present, `name` wins.
`ingestion_type` records how the trace came to exist and is stored on the trace's `ingestion_type` column. `seeded` marks a trace written by `seedTrace` / `seed_trace` from a case rather than produced by production traffic. It is read back on replay item selection, and a seeded trace is refused a database branch because it carries no snapshot pin. An absent, unknown, or malformed value resolves to `captured`, so a trace from an older SDK is never mistaken for a seeded one. `rawTrace` is accepted as an alias of `externalTrace`.
### `GET /api/sdk/sim-plan`
Read the organization's sim plan decisions about content capture, as the SDKs do in the background. Blocking, and safe to call once a minute. The SDKs give the read a five second timeout and send `Connection: close`.
**Response:**
```json theme={null}
{
"nodes": [
{ "traceFunctionKey": "support-agent", "name": "Agent.plan", "captureContent": false }
]
}
```
Only nodes whose content capture someone turned off (in Studio or over the `save_sim_plan` MCP tool) are listed, so every entry carries `captureContent` false; a node absent from the list has content capture on. A node whose content capture is locked (recorded by a framework integration, the root of its traces, imported from another platform, or named by Bitfab) is never listed, even when a stale decision for it is stored, so the SDKs never strip it. A node is matched by the trace function key of the trace's root span plus the span's `span_data.name`. When a node has `captureContent` false, the SDK sends the span without `span_data.input`, `input_meta`, `output`, `output_meta` (and Python's `input_serialized` and `output_serialized`) and sets `span_data.content_off_by_simulation_plan` to `true`. The server stores that mark on the span's `content_off_by_simulation_plan` column, so a span with no payload because of the sim plan is never counted as a span that recorded nothing. The span tree omits such spans unless they recorded an error; the sim plan page still counts them.
Every span from the TypeScript and Python SDKs also carries a `span_origin` record beside `span_data`, holding the SDK `name` (`bitfab.sdk.typescript` or `bitfab.sdk.python`), its `version`, and `instrumentation.name`, which says what produced the span: `span` (`withSpan` / `@span`), `trace` (`withTrace` / `@trace` and the subtree beneath it), `openai-agents`, `langgraph`, `claude-agent-sdk`, or `vercel-ai` (the framework integrations). The server stores `instrumentation.name` on the span's `instrumentation` column and the SDK name and version on its `span_origin_name` and `span_origin_version` columns. Spans recorded before this change have all three empty, so they resolve their instrumentation from the trace source at read time and show no SDK name or version. Spans recorded by a framework integration always keep their content, and so does a node that is the root of its traces. The sim plan refuses to turn content off for such a node, and the SDKs never strip them.
### `GET /api/sdk/traces/{traceId}/span`
Fetch one persisted span without loading the full trace. `traceId` and the optional exact `id` selector are canonical Bitfab UUIDs. Select exactly one of:
* `id={canonicalSpanId}` for an exact match
* `name={spanName}` with optional `occurrence=first|last|{zeroBasedIndex}`; the default is `last`
The response includes the span's canonical `id`, `traceId`, and `parentSpanId`, plus its name, type, input, output, contexts, prompt, metadata, metrics, errors, and timestamps. It never exposes ingestion source IDs.
### `PATCH /api/sdk/traces/{traceId}`
Update a detached current trace by its canonical Bitfab ID. The body accepts `appendContexts`, `mergeMetadata`, `setSessionId`, and `setName`. Blocking: SDKs call this synchronously and surface a rejection to the caller.
### `POST /api/sdk/replay/start`
Begin a replay session. Blocking. Timeout: 30 s on the client side.
**Request:**
```json theme={null}
{
"traceFunctionKey": "my-function-key",
"name": "Prompt candidate",
"limit": 10,
"traceIds": ["uuid", "..."],
"datasetId": "uuid",
"datasetIds": ["uuid", "..."],
"graderIds": ["uuid", "..."]
}
```
`name` is an optional display name stored on the resulting experiment/test run. `limit` (1-5,000, default 5) caps how many recent traces are fetched. When `traceIds` is present (max 100 entries) the ID list determines the count and `limit` is ignored; the field stays accepted because older SDKs always send a defaulted value. When `datasetId` or `datasetIds` (max 50) is present, the datasets' full trace lists determine the count and `limit` is ignored. `datasetIds` selects several datasets in one run: the run replays the union of their traces, deduped where they overlap, is graded by the union of their graders, and is attributed to every one of them, so it appears under each dataset's experiments. `datasetId` is the single-dataset spelling and is folded in when both arrive. Every dataset is validated against the organization and the replay's trace function; a foreign or mismatched ID returns a 400. `graderIds` (max 100) attaches graders directly to this run, independent of the dataset's own graders; the resulting experiment is graded by the union of these and the dataset's runnable graders at completion. Each id must be an active grader in the same organization and trace function, or the replay returns a 400. A replay with no dataset can still carry graders this way.
**Response 200:**
```json theme={null}
{
"testRunId": "uuid",
"testRunUrl": "/trace-functions/my-function-key/traces?testRunId=uuid",
"items": [
{
"originalTraceId": "uuid",
"originalSpanId": "uuid",
"sourceTraceId": "uuid (deprecated alias for originalTraceId)",
"sourceSpanId": "uuid (deprecated alias for originalSpanId)",
"originalDurationMs": 1750,
"originalTokens": {
"input": 80,
"output": 20,
"cached": 5,
"total": 100
},
"originalModel": "claude-sonnet-4-5",
"durationMs": 1750,
"tokens": {
"input": 80,
"output": 20,
"cached": 5,
"total": 100
},
"model": "claude-sonnet-4-5"
}
]
}
```
Each `items[]` entry carries reference metrics from the original trace being replayed, under `originalDurationMs` (end-to-end wall time in integer milliseconds), `originalTokens` (an object with `input`/`output`/`cached`/`total` counts), and `originalModel` (the model id). The unprefixed `durationMs`, `tokens`, and `model` are deprecated aliases carrying the same original-trace values, for SDKs that predate the rename. Any field may be `null` when the underlying trace didn't capture it. The replayed run's own token usage is returned later by `/api/sdk/replay/complete` (see below), once its spans are persisted.
### `POST /api/sdk/replay/status`
Read a replay test run without finalizing it. The Python and Ruby SDKs use this as a
server-authoritative persistence barrier after flushing its OTel pipeline.
```json theme={null}
{
"testRunId": "uuid",
"expectedSpanCounts": {
"sdk-generated-replay-trace-id": 4
}
}
```
`expectedSpanCounts` is optional. When supplied, `traceIds` includes an entry
only after that trace has reached a final status and at least the expected
number of spans is persisted. When omitted, `traceIds` contains every replay
trace currently associated with the test run, preserving the polling contract
used by older clients.
### `POST /api/sdk/replay/complete`
Signal that a replay session finished.
**Request:**
```json theme={null}
{ "testRunId": "uuid" }
```
**Response:**
```json theme={null}
{
"id": "uuid",
"status": "completed",
"traceIds": { "": "" },
"tokens": {
"": { "input": 80, "output": 20, "cached": 5, "total": 100 }
},
"traceOutlines": {
"": {
"traceId": "",
"name": "book_flight",
"status": "completed",
"traceFunctionKey": "book-flight",
"durationMs": 1810,
"spanCount": 2,
"spans": [
{
"spanId": "root-span-id",
"name": "book_flight",
"type": "agent",
"traceFunctionKey": "book-flight",
"durationMs": 1810,
"tokens": null,
"model": null,
"errors": null,
"mocked": false,
"children": [
{
"spanId": "child-span-id",
"name": "search_flights",
"type": "llm",
"traceFunctionKey": "book-flight",
"durationMs": 640,
"tokens": { "input": 80, "output": 20, "cached": 5, "total": 100 },
"model": "claude-sonnet-4-5",
"errors": null,
"mocked": true,
"children": []
}
]
}
]
}
},
"originalTraceOutlines": {
"": { "traceId": "", "spanCount": 2, "spans": [] }
},
"traceCount": 1
}
```
`traceIds` maps each SDK-generated replay trace id to the persisted server trace id. `tokens` is each replay trace's token usage, keyed by server trace id and aggregated from the freshly-uploaded replay spans (the same source the experiments view reads), so it's the replayed run's cost rather than the original's; a trace with no token data maps to `null`. `traceCount` is how many traces the server persisted for the run. The SDK maps `tokens` onto each item to populate its replay `tokens`.
`traceOutlines` is each replay trace's outline keyed by server trace id, and `originalTraceOutlines` is each original trace's outline keyed by the original trace id (the item's `originalTraceId`). A trace outline is the span tree with no inputs or outputs: per span its `spanId` (the SDK's span id), `name`, `type` (`llm`, `agent`, `function`, `guardrail`, `handoff`, or `custom`), `traceFunctionKey`, `durationMs`, `tokens`, `model`, `errors` (a list of `{ source, error, step? }` or `null`), `mocked` (true when the span was served from the recorded output instead of re-executing), and `children` in start order; the trace level carries `name`, `status`, `traceFunctionKey`, `durationMs`, and `spanCount`. The SDK maps these onto each item's `traceOutline` and `originalTraceOutline` so graders can compare the path a replay took against the original's.
### `GET /api/sdk/externalSpans/{id}`
Fetch a specific external span by ID. Used by replay internals. Returns the `rawSpan` object.
### Datasets
Datasets are named buckets of traces scoped to one trace function. Every endpoint below is
organization-scoped through the API key. A dataset id from another organization returns a
404\. Add and remove calls report partial acceptance in their response instead of failing
the whole request: ids the organization does not own, or that belong to a different trace
function, come back under a `skipped` list.
The dataset object returned by every endpoint:
```json theme={null}
{
"id": "uuid",
"traceFunctionKey": "my-function-key",
"name": "Refund failures",
"description": "Checkout runs where the refund was declined",
"traceCount": 42,
"graders": [{ "id": "uuid", "name": "Correctness" }],
"createdAt": "2026-08-30T17:04:11.000Z",
"updatedAt": "2026-08-30T17:04:11.000Z"
}
```
### `GET /api/sdk/datasets`
List datasets. Pass `?traceFunctionKey=my-function-key` to scope to one function, or omit it
for every dataset in the organization.
For bounded responses, pass `limit` (1–100), then send the returned `nextCursor`
unchanged as `cursor` on the next request. Keep the same `traceFunctionKey` filter
across pages. Results are ordered by most recently updated, with the dataset ID
breaking ties. Stop when `nextCursor` is `null`.
**Paged response 200:** `{ "datasets": [ ...dataset ], "nextCursor": "...", "hasMore": true }`
Requests without `limit` or `cursor` retain the complete-list response for older
clients. Current TypeScript, Python, Ruby, and Go SDKs request bounded pages and
collect them automatically.
**Response 200:** `{ "datasets": [ ...dataset ] }`
### `POST /api/sdk/datasets`
Create a dataset, or update the one that already carries this name under the same trace
function. The upsert key is `(traceFunctionKey, name)`. On an update, an omitted
`description` keeps the existing one.
**Request:**
```json theme={null}
{
"traceFunctionKey": "my-function-key",
"name": "Refund failures",
"description": "Optional"
}
```
**Response 200:** `{ "dataset": { ... }, "created": true }`
### `GET /api/sdk/datasets/{id}`
**Response 200:** `{ "dataset": { ... } }`
### `GET /api/sdk/datasets/{id}/traces`
The ids of every trace in the dataset, which is the same membership a replay with
`datasetId` selects.
**Response 200:** `{ "datasetId": "uuid", "traceIds": ["uuid", "..."] }`
Pass `limit` (1–100) for a bounded page, then pass `nextCursor` unchanged as
`cursor` to continue. Trace IDs are ordered ascending. A cursor-only request
defaults to 100 items. Stop when `nextCursor` is `null`.
**Paged response 200:** `{ "datasetId": "uuid", "traceIds": ["uuid"], "nextCursor": "uuid", "hasMore": true }`
Requests without `limit` or `cursor` retain the complete membership response.
All four SDKs fetch the pages automatically and preserve their existing return
shape. Pagination reads current membership; changes made during traversal can
affect the result.
### `POST /api/sdk/datasets/{id}/traces`
Add traces to the dataset. Between 1 and 100 ids per call. Traces must belong to the
organization and to the dataset's trace function; the rest are reported under
`skippedTraceIds`. Adding a trace that is already a member is a no-op reported under
`alreadyPresentTraceIds`.
**Request:** `{ "traceIds": ["uuid", "..."] }`
**Response 200:**
```json theme={null}
{
"dataset": { ... },
"addedTraceIds": ["uuid"],
"alreadyPresentTraceIds": [],
"skippedTraceIds": []
}
```
### `POST /api/sdk/datasets/{id}/removeTraces`
Remove traces from the dataset. The traces themselves are never deleted; only their
membership in this dataset is. Ids that were not members come back under
`notPresentTraceIds`.
**Request:** `{ "traceIds": ["uuid", "..."] }`
**Response 200:** `{ "dataset": { ... }, "removedTraceIds": ["uuid"], "notPresentTraceIds": [] }`
### `POST /api/sdk/datasets/{id}/graders`
Assign graders to the dataset. Graders must belong to the organization and to the
dataset's trace function; the rest come back under `skippedGraderIds`. Already-assigned
graders are reported under `alreadyAssignedGraderIds`.
**Request:** `{ "graderIds": ["uuid", "..."] }`
**Response 200:**
```json theme={null}
{
"dataset": { ... },
"addedGraderIds": ["uuid"],
"alreadyAssignedGraderIds": [],
"skippedGraderIds": []
}
```
### `POST /api/sdk/datasets/{id}/removeGraders`
**Request:** `{ "graderIds": ["uuid", "..."] }`
**Response 200:** `{ "dataset": { ... }, "removedGraderIds": ["uuid"], "notAssignedGraderIds": [] }`
### `POST /api/sdk/datasets/{id}/rerunGraders`
Re-run graders over every trace in the dataset. `graderIds` is optional and defaults to
every grader assigned to the dataset; a grader that is not assigned returns a 400. A dataset
with no graders assigned returns a 400. Only one re-run can be active per dataset: a request
that matches the active run's graders joins it (`joinedExisting: true`), while a different
selection returns a 400 until the active run finishes. The call returns as soon as the run
is queued; poll the `GET` below for progress.
**Request:** `{ "graderIds": ["uuid", "..."] }` (optional body)
**Response 200:**
```json theme={null}
{
"run": {
"id": "uuid",
"status": "pending",
"graderIds": ["uuid"],
"progress": null,
"result": null,
"error": null,
"createdAt": "2026-08-30T17:04:11.000Z",
"updatedAt": "2026-08-30T17:04:11.000Z"
},
"joinedExisting": false
}
```
### `GET /api/sdk/datasets/{id}/rerunGraders`
The dataset's active re-run, or the run named by `?runId=`. `status` is one of `pending`,
`running`, `completed`, or `errored`. While running, `progress` carries
`{ completedTraces, totalTraces, graderCount }`. When completed, `result` carries
`{ tracesGraded, gradersRun }`. When errored, `error` carries the message.
**Response 200:** `{ "run": { ... } }` or `{ "run": null }` when nothing is active.
### Assertions
An assertion says what a trace SHOULD do the next time it is replayed, as opposed to a
label, which is a verdict on a run that already happened. Every endpoint is
organization-scoped through the API key, and a trace from another organization returns a
404\.
Write assertions against the ORIGINAL trace. A replay that carries none of its own reads
its original's through the replay lineage, and the response names the original in
`inheritedFrom`.
The assertion object returned by every endpoint:
```json theme={null}
{
"id": "uuid",
"traceId": "uuid",
"assertion": "The itinerary returned lands before 9am local time",
"passCriteria": "arrival timestamp is before 09:00",
"failCriteria": null,
"targetOnEvaluatedTrace": { "kind": "output" },
"source": "agent",
"createdByUserId": "uuid",
"author": {
"id": "uuid",
"fullName": "Ada Lovelace",
"email": "ada@example.com",
"imageUrl": "https://..."
},
"createdAt": "2026-09-02T05:44:35.000Z",
"updatedAt": "2026-09-02T05:44:35.000Z"
}
```
`targetOnEvaluatedTrace` names what on the trace under evaluation the assertion checks. It
is `null` for the whole trace, `{ "kind": "output" }` for the final output, or
`{ "kind": "span", "name": "search_flights", "occurrence": "last" }` for one span.
`occurrence` accepts `"first"`, `"last"` (the default), or a 0-based index. Targets are span
names, never span ids, because an id captured on the original resolves to nothing on the
replay. An assertion whose target cannot be found on the trace being evaluated is errored,
never passed.
### `GET /api/sdk/traces/{traceId}/assertions`
Read a trace's assertions. Returns `{ "assertions": [...], "inheritedFrom": null }`, or
the original trace's id in `inheritedFrom` when this trace is a replay reading its
original's.
### `POST /api/sdk/traces/assertions`
Create or edit assertions across one or many traces. A call carries 1 to 500 traces, 1 to
50 assertions per trace, and at most 1000 assertions in total.
```json theme={null}
{
"updates": [
{
"traceId": "uuid",
"assertions": [
{
"assertion": "The itinerary returned lands before 9am local time",
"passCriteria": "arrival timestamp is before 09:00",
"targetOnEvaluatedTrace": { "kind": "output" }
}
]
}
],
"source": "agent"
}
```
Returns `{ "assertions": [...] }`, one flat list covering every trace in the batch, each
row carrying its own `traceId`.
Pass an entry's `id` to edit an existing assertion and omit it to add a new one, so two
callers adding different assertions to one trace never overwrite each other. On an edit,
an omitted field is preserved and an explicit `null` clears it. An edit keeps the original
author and source. `source` defaults to `"agent"`. The author is recorded from the API
key's user and is required, so every assertion has one.
The whole batch is written in one transaction. A trace the organization does not own, an
`id` that names an assertion on a different trace, or a total over the 1000-assertion cap
rejects the request and writes nothing.
### `POST /api/sdk/traces/{traceId}/assertions/archive`
Archive assertions by id. Archiving hides them from every read and keeps the rows for
audit; it never deletes a trace or an assertion outright.
```json theme={null}
{ "assertionIds": ["uuid"] }
```
Returns `{ "archived": ["uuid"] }`.
### `POST /api/sdk/traces/labels`
Record pass/fail verdicts, 1 to 200 per call. These are the same writes the
`save_agent_labels` MCP tool performs, reachable from a replay process rather than a
coding-agent session.
```json theme={null}
{
"testRunId": "uuid",
"labels": [
{
"originalTraceId": "uuid",
"attempt": 0,
"label": false,
"annotation": "picked the 9am flight",
"confidence": "Medium"
}
]
}
```
Key a direct verdict by `traceId`. Key a replay verdict by `originalTraceId` plus the
top-level `testRunId`, adding `attempt` when the experiment ran each trace more than once;
the server resolves it to the replay trace through the lineage, so a caller never needs a
server-generated replay trace id. `{ "skip": true }` withholds a verdict and
`{ "archive": true }` clears a previous one. Use `skip` for an attempt that crashed, was
punctured, or fell back to a schema default, and for an assertion whose target could not
be resolved, where a FAIL would read as a behavior regression rather than a check that never
ran.
Returns `{ "labels": [{ "key": "uuid", "traceId": "uuid", "action": "set" }] }`, where `key`
echoes the id you addressed the verdict by and `action` is one of `set`, `skipped`,
`archived`, or `no-active-label`.
### `GET /api/sdk/traces/labels`
Read verdicts back. Pass `traceIds` as a comma-separated list, 1 to 100 per call.
```
GET /api/sdk/traces/labels?traceIds=uuid,uuid
```
Returns one entry per trace that belongs to this organization, in the order the ids were
given, plus the ids that did not resolve:
```json theme={null}
{
"labels": [
{
"traceId": "uuid",
"labelStatus": "labeled",
"label": false,
"annotation": "one check failed",
"approved": false,
"passed": 1,
"failed": 1,
"assertions": [
{
"assertionId": "uuid",
"assertion": "Lands before 9am",
"labelStatus": "labeled",
"label": true,
"annotation": "arrived 06:40",
"confidence": "High",
"labelSource": "agent",
"approved": false
}
]
}
],
"notFound": ["uuid"]
}
```
`label`, `annotation`, and `approved` describe the trace's effective verdict. `assertions`
carries one entry per scored assertion, keyed by the same `assertionId` the write used, so
a verdict written per assertion is readable per assertion rather than only as the
`passed` / `failed` tally. A trace scored only as a whole returns an empty `assertions`.
### `POST /api/sdk/traces/labels/human`
Record human-authored verdicts, 1 to 200 per call. These are the same writes the
`save_human_labels` MCP tool performs. Unlike `POST /api/sdk/traces/labels`, which writes
agent suggestions that start unapproved, these are validated on write and satisfy
`search_traces` `validated: true` immediately.
```json theme={null}
{
"labels": [
{
"traceId": "uuid",
"assertionId": "uuid",
"label": true,
"annotation": "checked by hand",
"confidence": "High"
}
]
}
```
Use this only when a human decided the verdict. An agent's own first-pass guesses belong on
`POST /api/sdk/traces/labels` so they keep the approve-or-edit loop. Approving an existing
agent verdict is not available on any programmatic surface, by design: it happens in Studio.
The call is all-or-nothing, matching `POST /api/sdk/traces/labels`. A trace id outside the
organization, the same target twice in one batch, or an `assertionId` that is not active on
its trace fails the whole request with a `400` and writes nothing.
Returns `{ "labels": [{ "traceId": "uuid", "assertionId": null, "label": true, "action": "set" }] }`.
### `GET /api/sdk/graderLabels`
Read the individual verdicts each automated grader recorded, one row per grader per trace.
The same breakdown the `get_grader_labels` MCP tool returns.
```
GET /api/sdk/graderLabels?traceIds=uuid,uuid
GET /api/sdk/graderLabels?graderId=uuid&limit=50
```
Pass `traceIds` (1 to 100, comma separated) to see every grader's verdict on those traces,
`graderId` to see one grader's most recent verdicts across traces (newest first), or both to
narrow. Passing neither is a `400`. `limit` applies only when reading by grader alone, and
defaults to 50 with a maximum of 200.
Returns `{ "labels": [{ "traceId": "uuid", "graderId": "uuid", "graderName": "...", "graderStatus": "active", "label": false, "labelReason": "...", "failureDiagnostic": "...", "labelConfidence": "High", "source": "live_grader", "evaluatedAt": "..." }] }`. `source` is `human` for a verdict a person recorded on the grader and `live_grader` for one a grader run produced; it is a different enum from the `labelSource` on trace labels.
## Serialization
* Inputs and outputs in `span_data.input` / `span_data.output` are serialized using [superjson](https://github.com/flightcontrolhq/superjson) by the TypeScript SDK. The `input_meta` / `output_meta` fields are the superjson meta descriptors.
* Python, Ruby, and Go SDKs use JSON-compatible fallbacks. Objects that don't round-trip through JSON are coerced to string via the SDK's `serialize` helper.
## Error Responses
| Status | Meaning |
| ------ | --------------------------------------------------------- |
| `200` | Success |
| `400` | Malformed request body |
| `401` | Missing/invalid API key |
| `403` | API key valid but lacks access to the referenced resource |
| `404` | Function / span / trace ID not found |
| `429` | Rate limit exceeded |
| `5xx` | Server error |
SDKs swallow all non-2xx responses on fire-and-forget endpoints and log to stderr. Blocking endpoints (`lookup`, `replay/start`) raise typed errors.
# Reference Overview
Source: https://docs.bitfab.ai/reference/overview
Dense, canonical API reference with signatures, types, and semantics only. No tutorials.
This section is a **pure reference**. It holds exhaustive signatures, parameters, return values, and semantics for every public API across every Bitfab SDK. No narrative, no "how to use", no walkthroughs.
For tutorials and getting started, see the individual SDK guides:
[TypeScript](/typescript-sdk), [Python](/python-sdk), [Ruby](/ruby-sdk), [Go](/go-sdk).
## Pages
* [TypeScript SDK Reference](/reference/typescript): `@bitfab/sdk` npm package
* [Python SDK Reference](/reference/python): `bitfab-py` PyPI package
* [Ruby SDK Reference](/reference/ruby): `bitfab` RubyGem
* [Go SDK Reference](/reference/go): `github.com/Project-White-Rabbit/bitfab-go` module
* [Span Types](/reference/span-types): common `SpanType` enum
* [HTTP Endpoints](/reference/http): trace ingestion and lookup endpoints the SDKs call
* [OpenTelemetry Transport Architecture](/otel-architecture): batching, delivery, and replay persistence boundaries
## Instrumentation Primitives
The client exposes three instrumentation primitives, not just spans. A span is an explicit boundary you write yourself. A trace records a root plus every first-party call beneath it, with no decorators on those calls. A node applies trace-owned policy (name, type, capture, replay mocking) to one call the trace discovered, and never creates a span or trace on its own.
| Primitive | TypeScript | Python | Ruby | Go |
| ----------------------------- | ---------------------------------------------------------------------------------- | ---------------------------------- | -------------------------------------- | -------------------------------- |
| Explicit span | [`withSpan`](/reference/typescript#withspan), [`span`](/reference/typescript#span) | [`span`](/reference/python#span) | [`Bitfab::Traceable`](/reference/ruby) | [`Span`, `Start`](/reference/go) |
| Automatic subtree root | [`withTrace`, `trace`](/reference/typescript#trace-withtrace) | [`trace`](/reference/python#trace) | - | - |
| Configure one discovered call | [`withNode`](/reference/typescript#withnode), [`node`](/reference/typescript#node) | [`node`](/reference/python#node) | - | - |
Subtree capture is experimental. TypeScript needs a compatible [`@bitfab/transform` adapter](/typescript-sdk#experimental-subtree-tracing) and Python needs 3.12 or newer. Without them the root span still records and descendants are skipped.
## Trace Seeding
Seeding writes a replayable trace from a case you already hold, so `replay` has something to select before production has produced a trace for it. A seeded trace carries `ingestion_type: seeded` and no database pin, so a database branch is refused for it.
| Capability | TypeScript | Python | Ruby | Go |
| --------------------------------- | ------------------------------------------------------------------------------ | ---------------------------------------------------------------- | :--: | :-: |
| Seed by running the function once | [`seedTrace(key, fn, options)`](/reference/typescript#seeding-by-running-once) | [`seed_trace(key, fn, ...)`](/reference/python#seed-trace) | - | - |
| Seed from a case without running | [`seedTrace(key, case)`](/reference/typescript#seedtrace) | - | - | - |
| Seed through a replay registry | [`seedFromRegistry`](/reference/typescript#seedfromregistry) | [`seed_from_registry`](/reference/python#seed-from-registry) | - | - |
| Seed a cases file from the shell | `bitfab-replay --seed ` (add `--run` to execute each case) | `bitfab-replay --seed ` (always executes each case) | - | - |
Seeding records its one call whether capture is on or off, so a seeding script can run against a capture-off client and record exactly the calls it seeds and nothing else. A run-seeded trace has a full recorded subtree, so replay mocking works on it as it does on a captured trace. A case-seeded trace (TypeScript only) has no child spans, so there is nothing recorded to mock. See the [TypeScript](/typescript-sdk#seeding-traces) and [Python](/python-sdk#seeding-traces) guides.
## Datasets
Every SDK exposes the dataset operations the Bitfab MCP tools give a coding agent. Scripts and CI jobs use them to build or maintain datasets with no agent in the loop. All four namespaces cover the same ten operations over the same [HTTP routes](/reference/http#datasets).
| Namespace | TypeScript | Python | Ruby | Go |
| -------------------------------------- | --------------------------------------------------- | ----------------------------------------------- | --------------------------------------------- | ------------------------------------------- |
| Client property | [`client.datasets`](/reference/typescript#datasets) | [`client.datasets`](/reference/python#datasets) | [`client.datasets`](/reference/ruby#datasets) | [`client.Datasets`](/reference/go#datasets) |
| Upsert on `(trace function key, name)` | `save` | `save` | `save` | `Save` |
| Read | `list`, `get`, `listTraces` | `list`, `get`, `list_traces` | `list`, `get`, `list_traces` | `List`, `Get`, `ListTraces` |
| Membership | `addTraces`, `removeTraces` | `add_traces`, `remove_traces` | `add_traces`, `remove_traces` | `AddTraces`, `RemoveTraces` |
| Graders | `addGraders`, `removeGraders` | `add_graders`, `remove_graders` | `add_graders`, `remove_graders` | `AddGraders`, `RemoveGraders` |
| Grader re-run | `rerunGraders`, `getGraderRerun` | `rerun_graders`, `get_grader_rerun` | `rerun_graders`, `get_grader_rerun` | `RerunGraders`, `GetGraderRerun` |
These semantics hold across all four SDKs.
* `save` is an upsert. It reports whether it created the dataset.
* Adds are idempotent and accept 1 to 100 ids.
* An id from outside the organization, or under another trace function, comes back as skipped rather than failing the call.
* Removing a trace drops its membership and never deletes the trace.
* A grader re-run waits up to 90 seconds by default. Pass an option to return immediately instead.
* A dataset id from another organization is a 404.
See [Datasets](/primitives/datasets).
## Assertions and labels
An assertion says what a trace SHOULD do the next time it is replayed. A label is the verdict on a run that already happened. TypeScript and Python expose both. A replay process can read what the case was supposed to do, judge its own run, and write the verdict back, with no coding agent in the loop. Ruby and Go expose neither namespace. `client.labels` keys a replay verdict by attempt, and neither SDK has replay attempts, so there is nothing for the surface to express there.
| Namespace | TypeScript | Python | Ruby | Go |
| ---------------------------- | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- | ---- | --- |
| Client property | [`client.traces`](/reference/typescript#traces), [`client.labels`](/reference/typescript#labels) | [`client.traces`](/reference/python#traces), [`client.labels`](/reference/python#labels) | n/a | n/a |
| Read assertions | `getAssertions` | `get_assertions` | n/a | n/a |
| Manage assertion categories | `assertionCategories.save/get/list/delete` | `assertion_categories.save/get/list/delete` | n/a | n/a |
| Assign an assertion category | `category_assertion_id` on assertion saves | `category_assertion_id` on assertion saves | n/a | n/a |
| Write assertions | `saveAssertions`, `saveAssertionsAll`, `archiveAssertions` | `save_assertions`, `save_assertions_all`, `archive_assertions` | n/a | n/a |
| Write verdicts | `save`, `saveAll` | `save`, `save_all`, `skip`, `archive` | n/a | n/a |
These semantics hold across TypeScript and Python.
* Assertions are written against the original trace. A replay with none of its own reads its original's assertions, naming that trace in `inheritedFrom`.
* Saving with an entry's id edits that assertion in place.
* `saveAssertionsAll` / `save_assertions_all` takes one update per trace, so covering many traces is one request in one transaction. A rejected batch writes nothing. The singular delegates to it.
* A field omitted from that save keeps its previous value.
* Editing an assertion never changes its original author, which is always the API key's user.
* A verdict is keyed by `traceId`, or by `originalTraceId` plus `testRunId` and `attempt` for a replay.
* `skip` withholds a verdict for a crashed or punctured attempt, and for an assertion whose target could not be resolved. A FAIL in either case would read as a behavior regression rather than a check that never ran.
Same [HTTP routes](/reference/http#assertions) as the MCP tools.
## Replay execution and results
Every SDK returns a common set of fields on each replay result.
* The replayed output and the original output.
* Distinct trace-code and replay-setup errors.
* Duration and token comparisons.
* Source lineage.
* Database branch metadata.
* Original and replayed trace outlines.
A trace outline holds structure and measurements but no inputs or outputs, so graders can compare execution paths without reloading full trace payloads.
| Capability | TypeScript | Python | Ruby | Go |
| ----------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------- | ----------------- | ---------------- |
| Replay each selected trace multiple times | `attempts` (1-100) | `attempts` or `ReplayConcurrency.attempts` (1-100) | - | - |
| Async/thread concurrency limit | `maxConcurrency` | `max_concurrency` / `ReplayConcurrency(primitive="async")` | `max_concurrency` | `MaxConcurrency` |
| Isolated process concurrency | - | `ReplayConcurrency(primitive="process")`, registry command only | - | - |
| Grade an item inside the process that replayed it | - | `ReplayConcurrency(on_item_finish_in_child_process=...)`, process primitive only | - | - |
| Hold a child back until the machine has memory for it | - | `ReplayConcurrency(memory_throttle=...)`, process primitive only, on by default | - | - |
| Resolve inputs without calling the function | `dryRun` | `dry_run` | - | - |
| Per-item start and finish callbacks | yes | yes | yes | yes |
| Original and replayed trace outlines | yes | yes | yes | yes |
## Framework Integrations
Per-framework signatures (handlers, processors, wrappers) are documented in the **[Frameworks](/frameworks/overview)** section. Language-specific signatures are mirrored in each SDK reference's **Framework Integrations** subsection.
| Framework | TypeScript reference | Python reference | Ruby | Go |
| ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | :--: | :-: |
| [LangGraph / LangChain](/frameworks/langgraph) | [`getLangGraphCallbackHandler`](/reference/typescript#framework-integrations) | [`get_langgraph_callback_handler`](/reference/python#framework-integrations) | - | - |
| [OpenAI Agents SDK](/frameworks/openai-agents) | [`getOpenAiTracingProcessor`](/reference/typescript#framework-integrations) + [`getOpenAiAgentHandler`](/reference/typescript#framework-integrations) | [`get_openai_tracing_processor`](/reference/python#framework-integrations) + [`get_openai_agent_handler`](/reference/python#framework-integrations) | - | - |
| [BAML](/frameworks/baml) | [`wrapBAML`](/reference/typescript#wrapbaml) | [`wrap_baml`](/reference/python#wrap-baml) | - | - |
| [Claude Agent SDK](/frameworks/claude-agent-sdk) | [`getClaudeAgentHandler`](/reference/typescript#framework-integrations) | [`get_claude_agent_handler`](/reference/python#framework-integrations) | - | - |
| [Vercel AI SDK](/frameworks/vercel-ai-sdk) | [`getVercelAiMiddleware`](/reference/typescript#framework-integrations) | - | - | - |
## Invariants Across SDKs
Behavior that is guaranteed to match across all SDKs:
| Invariant | Behavior |
| ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Empty / missing API key | Tracing disabled, warning logged, wrapped functions still execute, and no spans are sent. TypeScript, Python, Ruby, and Go can opt into strict mode, which fails instead of degrading |
| Capture off (TypeScript: `captureEnabled: false`, Python: `capture_enabled=False`, Ruby: `enabled: false`, Go: `WithEnabled(false)`) | Same as above. In TypeScript and Python the flag turns off capture, not the tracing wiring. Wrapped functions keep their wrapper and trace function key. They still record inside replay items and inside the one call `seedTrace` / `seed_trace` runs. Ruby and Go are plain on/off switches on sending, with no seeding to record inside. `enabled` is a deprecated alias in TypeScript and Python, and it warns once |
| Span dispatch | Fire-and-forget. Errors in span transport never propagate to host code |
| Nested spans | Child spans inherit `traceId`. Parent-child relationships are tracked via context |
| Nested-only capture | `"nested"` records a span only with an active parent. A standalone call executes normally without creating a root trace. Default: `"always"` |
| Trace ID | UUID v4. Generated at root span, inherited by descendants |
| Span ID | UUID v4. Unique per span |
| Root span | The outermost span in a call tree. Emits trace completion signal when it ends |
| Errors in user code | Captured on the span (`error` field) then re-raised/re-thrown |
| Default `serviceUrl` | `https://bitfab.ai` |
| Default span `type` | `"custom"`. A label only. Does not affect tracing, replay, or evaluation |
| `addContext` | Each call appends one entry. Multiple calls accumulate |
| `setPrompt` | Metadata only. Does not change what the span executes. Last write wins. Calling outside a span is a no-op |
| `setSessionId` | Stored as a DB column on the trace. Filterable in dashboard |
| `setMetadata` | Merges with existing metadata. Later values win per-key. Metadata the caller recorded wins over the metadata an integration's own trace export carries for the same trace |
## Common Span Types
Every SDK accepts the same enum for span type:
```
"llm" | "agent" | "function" | "guardrail" | "handoff" | "custom"
```
See [Span Types](/reference/span-types) for full semantics.
# Python SDK Reference
Source: https://docs.bitfab.ai/reference/python
Pure API reference for the bitfab-py PyPI package.
Package: `bitfab-py` (imported as `bitfab`). Python ≥ 3.10.
## Module Exports
```python theme={null}
from bitfab import (
BITFAB_PROGRESS_PREFIX,
AdaptContext,
AddDatasetGradersResult,
AddDatasetTracesResult,
AllowedEnvVars,
Bitfab,
BitfabFunction,
BitfabClaudeAgentHandler,
BitfabLangChainCallbackHandler,
BitfabLangGraphCallbackHandler,
BitfabLangGraphIntegration,
BitfabOpenAIAgentHandler,
BitfabTracingProcessor,
CaptureWhen,
CapturedSpan,
CodeChangeFile,
ConcurrencyPrimitive,
CurrentSpan,
CurrentTrace,
Dataset,
DatasetGraderRef,
DatasetTraceIds,
DatasetsClient,
DetachedTrace,
MixedTracingError,
DbBranchLease,
DbBranchOptions,
DbBranchReplayError,
DbSnapshotRef,
GraderRerun,
GraderRerunProgress,
GraderRerunResult,
GraderRerunStatus,
LabelAction,
LabelConfidence,
LabelOutcome,
LabelUpdate,
LabelsClient,
MockOverride,
MockOverrideCtx,
MockOverrideInput,
MockOverrideResolver,
MockValue,
NO_MOCK_OVERRIDE,
NodeMatcher,
OutputTarget,
RemoveDatasetGradersResult,
RemoveDatasetTracesResult,
ReplayBranch,
ReplayConcurrency,
ReplayError,
ReplayItem,
ReplayItemFinishEvent,
ReplayItemStartProgress,
ReplayItemFinishProgress,
ReplayProgress,
ReplayRegistration,
ReplayRegistry,
ReplayRegistryContext,
ReplayResult,
RerunGradersResult,
SaveAssertion,
SaveDatasetResult,
SpanNodeMeta,
SpanOccurrence,
SpanTarget,
SpanType,
TraceAssertion,
TraceAssertionSource,
TraceAssertionsResult,
TraceAssertionsUpdate,
TraceTarget,
TraceTargetOccurrence,
TracesClient,
finalizers, # finalizers.openai_chunks, finalizers.anthropic_events
flush_traces,
get_current_replay_branch,
get_current_span,
get_current_trace,
report_replay_progress,
seed_from_registry,
serialize_replay_result,
)
```
All adapter symbols can be imported without installing their framework. The matching optional dependency is only required once the adapter enters that framework's runtime surface. The OpenAI Agents adapters need `openai-agents`. The LangGraph and LangChain adapters need `langchain-core` or `langgraph`. The Claude Agent SDK adapter needs `claude-agent-sdk`.
## Type Aliases
```python theme={null}
SpanType = Literal["llm", "agent", "function", "guardrail", "handoff", "custom"]
CaptureWhen = Literal["always", "nested"]
ConcurrencyPrimitive = Literal["async", "process"]
MockStrategy = Literal["none", "all", "marked"]
```
## `class Bitfab`
### `__init__`
```python theme={null}
Bitfab(
api_key: str | Callable[[], str | None] | None = None,
service_url: Optional[str] = None,
env_vars: Optional[AllowedEnvVars] = None,
capture_enabled: bool = True,
baml_client: Any = None,
strict: bool = False,
trace_across_threads: bool | None = None,
enabled: bool | None = None,
)
```
| Param | Type | Default | Description |
| ---------------------- | ------------------------------------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `api_key` | `str \| Callable[[], str \| None] \| None` | `None` | Resolved lazily at first use. Falls back to `BITFAB_API_KEY` when omitted or empty |
| `service_url` | `Optional[str]` | `"https://bitfab.ai"` | Base URL |
| `env_vars` | `AllowedEnvVars` | `{}` | LLM provider keys for `call()` |
| `capture_enabled` | `bool` | `True` | When `False`, decorated functions run without recording a span. Nothing is sent. Replay still records inside each item. `seed_trace` still records the one call it runs. Decorators always return a wrapper, so `replay(fn)` and the replay registry find the trace function key either way |
| `baml_client` | `Any` | `None` | Generated BAML client for `wrap_baml()` without explicit client arg |
| `strict` | `bool` | `False` | Raise on the first traced call when no API key resolves, instead of disabling tracing quietly |
| `enabled` | `bool \| None` | `None` | Deprecated alias for `capture_enabled`. Warns once. Combines as `capture_enabled = capture_enabled and enabled`. `enabled=False` disables capture even when `capture_enabled=True` |
| `trace_across_threads` | `bool \| None` | `None` | Nests spans from worker threads and thread pools under the trace that submitted the work, so replay mocks fire on them too. `True` turns this on. `False` turns it off. `None` is the default, and turns it on only when `BITFAB_TRACE_ACROSS_THREADS` is `1`, `true`, or `yes` (case-insensitive). It wraps `ThreadPoolExecutor.submit` (which covers `loop.run_in_executor`) and `threading.Thread.start`. The wrap is process-global once installed, and a later `False` never uninstalls it. It excludes pool-internal worker threads and the SDK's own OTel transport threads, so a long-lived dispatch loop is never pinned to whichever trace spawned it. It carries only the SDK's own context. `asyncio.to_thread` passes through unaffected, and it does not reach pre-created queue consumers or other processes. Every span records `span_data.runtime` (`thread_id`, `thread_name`, and `parent_thread_id` when the parent span's thread is known). Spans dispatched by the wrapper also record `submit_thread_id`/`submit_thread_name`, the thread that called `submit()`/`start()` |
### Client properties
```python theme={null}
client.api_key: str | None
client.capture_enabled: bool
client.enabled: bool # deprecated alias
```
`api_key` resolves the configured string or callable. It does not apply the `BITFAB_API_KEY` fallback, and it never warns. `capture_enabled` is the effective state, so it resolves the fallback key too, the same way the first traced call would. Under `strict=True`, reading `capture_enabled` without a key raises `RuntimeError`. `enabled` warns once and returns the same value as `capture_enabled`.
### Transport environment variables
| Variable | Default | Description |
| -------------------------------- | --------- | ------------------------------------------------------------------------------------ |
| `BITFAB_OTEL_MAX_REQUEST_BYTES` | `3000000` | Request-size target for OTLP/JSON exports. Accepts positive integers up to `3000000` |
| `BITFAB_OTEL_EXPORT_CONCURRENCY` | `32` | Concurrent direct requests per export window. Accepts `1` through `64` |
### Commit ref environment variables
| Variable | Effect |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `BITFAB_DISABLE_COMMIT_REF` | Set to any value to send no `commit_ref` at all. Neither the platform variables nor `git` are consulted |
| `BITFAB_COMMIT_SHA` | The commit sent as `commit_ref.sha` on every root trace. Wins over every deploy platform variable and over `git` |
| `VERCEL_GIT_COMMIT_SHA`, `GITHUB_SHA`, `RAILWAY_GIT_COMMIT_SHA`, `RENDER_GIT_COMMIT`, `SOURCE_VERSION`, `CF_PAGES_COMMIT_SHA`, `CI_COMMIT_SHA`, `BUILD_SOURCEVERSION`, `CIRCLE_SHA1` | Read in that order when `BITFAB_COMMIT_SHA` is unset, together with the same platform's branch and repository variables when it has them. The first one set wins, and `git` is never run |
### `span`
```python theme={null}
def span(
trace_function_key: str,
*,
name: Optional[str] = None,
type: SpanType = "custom",
capture_when: CaptureWhen = "always",
test_run_id: Optional[str] = None,
mock_on_replay: bool = False,
finalize: Optional[Callable[[Any], Any]] = None,
) -> Callable[[Callable[P, T]], Callable[P, T]]
```
Decorator. Wraps the decorated function (sync or async) with a span.
**Returns:** a decorator that returns a function with the same signature.
**Semantics:**
* When `capture_enabled=False`, the wrapper runs the function without recording, except inside a replay item
* `name` defaults to the function's qualified name: `Order.process` for a method, `process` for a plain function or a closure (the enclosing function is dropped, since it is already the parent in the tree). If the function has no name, for example a lambda, it falls back to `trace_function_key`. The raw `__name__` still travels separately as `function_name`
* `capture_when="nested"` records the span only when another Bitfab span is active. Without a parent, the decorated function runs normally and does not create a root trace. The default is `"always"`. An unknown value warns once and falls back to that default
* Nested spans propagate via `contextvars.ContextVar`. This is safe across `asyncio.gather`, threads, and sync/async boundaries
* Spans exist only where you create them. Nesting between them is automatic, but only among spans that exist. One wrapper around just the outermost function records a single-node trace. See [Instrumentation](/instrumentation)
* `span()` is the opt-in tracing surface. `trace()` and `node()` are the opt-out surface. A `span()` entered beneath an active `trace()` raises `MixedTracingError`, a `RuntimeError` subclass. Configure a function inside a subtree with `node()` instead
* Exceptions are recorded on the span, then re-raised
* `test_run_id` is rarely set directly. `replay()` injects it through a replay context
* `finalize` records a serializable view of a streaming result as the span output. The raw result is always returned to the caller unchanged. On an **async generator**, `finalize` receives the list of yielded chunks. That list is assembled after iteration, once the caller has already streamed every chunk, so this is non-blocking and non-destructive. On a plain sync or async function, `finalize` receives the return value instead and runs inline before the span is recorded, so a live single-consumer stream is blocked on and consumed. Prefer an async generator for streaming to avoid that. `finalize` may be `async` on async or async-generator spans, and must be sync on sync spans. A `finalize` that raises records an error instead of crashing. Pair it with `finalizers.openai_chunks` or `finalizers.anthropic_events`. See [Tracing streaming functions](/python-sdk#tracing-streaming-functions)
### `trace`
**Experimental.** New API. Behavior may change in a future release. Requires
Python 3.12+ for subtree capture.
```python theme={null}
def trace(
trace_function_key: str,
*,
name: Optional[str] = None,
type: SpanType = "custom",
mock_on_replay_default: bool = False,
max_depth: int = 30,
max_spans: int = 500,
exclude: Collection[str] = (),
include_wrappers: bool = False,
) -> Callable[[Callable[P, T]], Callable[P, T]]
```
Decorator. Records a span for the decorated function and for every first-party function it calls, at any depth, without those functions being decorated.
**Returns:** a decorator that returns a function with the same signature.
**Semantics:**
* The root span behaves exactly as `span()`. Descendant spans are typed `"function"`. They are named by their qualified name, for example `Order.process`, with any `.` prefix stripped
* First-party means the package directory containing the decorated function. That directory is found by walking up while `__init__.py` exists. Standard library code, site-packages, and Bitfab's own code never record spans
* Capture is scoped to the traced call. `sys.monitoring` events are installed on entry and removed on exit. Code outside a traced call is unaffected
* Lambdas, generator expressions, comprehensions, module-level (``) frames, and decorator wrappers are skipped. A decorator wrapper here means a function taking only `*args, **kwargs`. `include_wrappers=True` records wrappers instead of skipping them
* Skipping is transparent. A skipped wrapper never becomes a parent. The function it wraps parents to the real caller instead
* `max_depth` and `max_spans` bound a subtree. Exceeding either stops recording, without affecting the call itself. It also logs a one-time warning per traced function, so a truncated trace is not mistaken for code that never ran
* Free-threaded builds warn once. Subtree capture is unverified on free-threaded builds. The root span is unaffected either way
* `self` / `cls` are stripped from recorded inputs. Exceptions are recorded on the failing frame using `PY_UNWIND`, then re-raised
* Sync, async, and async-generator roots are all supported. Suspension does not close a span. A coroutine's span spans its full lifetime
* On Python 3.11 and earlier, the root span is still recorded. A one-time warning explains that descendants were skipped
* A nested `trace()` root starts its own independent trace. Its complete subtree also appears in every outer `trace()` capture. Each copy has separate span IDs, and the same structure the decorator would record alone. Two active roots double the span volume in the region they share. A `node()` and the LangGraph integration's spans keep their configured name, type, `test_run_id`, and finalized output in every enclosing copy, with `finalize` running once; framework handler spans attach inside the innermost trace only. The outer trace's span for the nested root carries `nested_trace_id`, `nested_trace_function_key`, and `nested_root_span_id`, and the nested root span carries `enclosing_trace_id`, `enclosing_span_id`, and `enclosing_trace_function_key`. Inside a replay item or `seed_trace` a nested root starts no trace of its own; the item's trace records it as an ordinary descendant
* `trace()` and `node()` are the opt-out tracing surface. `span()` is the opt-in surface. A `trace()` root entered beneath an active `span()` raises `MixedTracingError`. The reverse raises too. A `span()` entered beneath a `trace()` also raises. The check only runs while tracing is active, meaning capture is on or the call is inside a replay item. The root span that `replay("key", fn)` wraps around an undecorated callable belongs to neither surface. A `trace()` root called from that callable nests beneath it instead
* Work dispatched to `ThreadPoolExecutor` / `threading.Thread` is not captured. An abandoned generator records no span
* Async-generator capture is released between yielded items. This holds even when a caller stops consuming without calling `aclose()`
* Automatically discovered descendants are not replay mock targets unless configured with `node(mock_on_replay=True)`. Use `node()` when a descendant needs trace-owned capture policy
* `mock_on_replay_default=True` makes replay mocking the default for configured `node()` descendants under `mock="marked"`. A node with `mock_on_replay=False` overrides it. The trace option defaults to `False`. Unconfigured descendants remain non-mockable, because interpreter monitoring cannot short-circuit them
### `node`
```python theme={null}
def node(
*,
name: Optional[str] = None,
type: SpanType = "custom",
capture: bool = True,
test_run_id: Optional[str] = None,
mock_on_replay: Optional[bool] = None,
finalize: Optional[Callable[[Any], Any]] = None,
) -> Callable[[Callable[P, T]], Callable[P, T]]
```
Trace-only configuration for a function discovered beneath `@client.trace`. Its options intentionally mirror the applicable `span()` options. `node()` never creates a span or trace by itself. It inherits the enclosing trace function key.
**Semantics:**
* Outside an active `trace()` call, the function runs normally with no capture or replay behavior. This does not hold when a `span()` is active. `node()` belongs to the opt-out surface, so entering it beneath an opt-in `span()` raises `MixedTracingError`
* `capture=True` (default) lets the enclosing trace capture the node once, with the requested `name`, `type`, `test_run_id`, and `finalize` behavior
* `capture=False` omits the node. Captured descendants attach to its nearest captured parent
* `mock_on_replay=True` makes a captured, configured descendant return its recorded output under the default `mock="marked"` strategy. `None` inherits the enclosing trace's `mock_on_replay_default`. `False` overrides that default. Replay `mock="all"`, `mock="none"`, and mock overrides all behave exactly as they do for `span()`
* `capture=False` with `mock_on_replay=True` raises `ValueError`, because an uncaptured node has no recorded output
* Sync, async, and async-generator functions follow the corresponding `span()` behavior. This includes the existing async-generator replay-mocking limitation
* Requires the Python 3.12+ subtree capture used by `trace()`. On earlier versions, the root still records. Configured descendants run normally and untraced
### `get_function`
```python theme={null}
def get_function(trace_function_key: str) -> BitfabFunction
```
### `wrap_baml`
Framework integration → see [BAML framework guide](/frameworks/baml) for examples.
```python theme={null}
# Form 1: uses baml_client from constructor
def wrap_baml(
method: Callable[..., Any],
*,
on_collector: Callable[[Any], None] | None = None,
) -> Callable[..., Any]
# Form 2: explicit client
def wrap_baml(
baml_client: Any,
method: Callable[..., Any],
*,
on_collector: Callable[[Any], None] | None = None,
) -> Callable[..., Any]
```
**Returns:** an `async` wrapper with the same signature as `method`, plus a `.collector` attribute holding the most recent call's BAML `Collector` (`None` before the first call or if `baml-py` is not installed). Pass `on_collector` to receive the `Collector` after each call.
**Raises:**
* `ValueError` if form 1 is used without `baml_client` in constructor
* `ValueError` if the method has no `__name__`
**Semantics:**
* If `baml-py` is not installed, the method is called directly without instrumentation
* Otherwise calls `get_current_span().set_prompt(...)` and `get_current_span().add_context({...})` with extracted metadata
* Must be invoked inside a `@span`-decorated function for the prompt/context to attach to anything
### `get_trace`
```python theme={null}
def get_trace(trace_id: str) -> DetachedTrace
```
Returns a `DetachedTrace` handle for annotating a trace after its root span has closed, from any process or thread.
**Raises:** `ValueError` if `trace_id` is not a canonical Bitfab trace ID.
**Semantics:**
* All methods on the returned handle are blocking, like `get_trace_span`. Each one returns only once the server has applied the change
* When `capture_enabled=False`, methods return immediately without sending, except inside a replay item
* The server returns 404 if no trace exists with that ID. The method raises rather than logging
### `get_trace_span`
```python theme={null}
def get_trace_span(
trace_id: str,
*,
id: Optional[str] = None,
name: Optional[str] = None,
occurrence: Literal["first", "last"] | int = "last",
) -> Optional[CapturedSpan]
```
Fetches one persisted span without loading its trace. `trace_id` is the canonical Bitfab trace ID. Exactly one of the span's Bitfab `id` or `name` is required. Numeric occurrences are zero-based in start-time order. Returns `None` when no trace or span matches.
**Raises:**
* `ValueError` when `id` and `name` are both given, or both omitted
* `ValueError` when `trace_id` or `id` is not a valid Bitfab ID
* `ValueError` when `name` is empty or not a string
* `ValueError` when `occurrence` is not `"first"`, `"last"`, or a non-negative integer. `occurrence=True` raises too, even though `bool` is a Python `int` subtype
### `datasets`
```python theme={null}
client.datasets.save(trace_function_key: str, name: str, description: str | None = None) -> SaveDatasetResult
client.datasets.list(trace_function_key: str | None = None) -> list[Dataset]
client.datasets.get(dataset_id: str) -> Dataset
client.datasets.list_traces(dataset_id: str) -> DatasetTraceIds
client.datasets.add_traces(dataset_id: str, trace_ids: list[str]) -> AddDatasetTracesResult
client.datasets.remove_traces(dataset_id: str, trace_ids: list[str]) -> RemoveDatasetTracesResult
client.datasets.add_graders(dataset_id: str, grader_ids: list[str]) -> AddDatasetGradersResult
client.datasets.remove_graders(dataset_id: str, grader_ids: list[str]) -> RemoveDatasetGradersResult
client.datasets.rerun_graders(dataset_id: str, grader_ids: list[str] | None = None, *, wait: bool = True, timeout: float = 90.0, poll_interval: float = 1.0) -> RerunGradersResult
client.datasets.get_grader_rerun(dataset_id: str, run_id: str | None = None) -> GraderRerun | None
```
```python theme={null}
class DatasetGraderRef(TypedDict):
id: str
name: str | None
class Dataset(TypedDict):
id: str
traceFunctionKey: str
name: str
description: str | None
traceCount: int
graders: list[DatasetGraderRef]
createdAt: str
updatedAt: str
class SaveDatasetResult(TypedDict):
dataset: Dataset
created: bool
class DatasetTraceIds(TypedDict):
datasetId: str
traceIds: list[str]
class AddDatasetTracesResult(TypedDict):
dataset: Dataset
addedTraceIds: list[str]
alreadyPresentTraceIds: list[str]
skippedTraceIds: list[str]
class RemoveDatasetTracesResult(TypedDict):
dataset: Dataset
removedTraceIds: list[str]
notPresentTraceIds: list[str]
class AddDatasetGradersResult(TypedDict):
dataset: Dataset
addedGraderIds: list[str]
alreadyAssignedGraderIds: list[str]
skippedGraderIds: list[str]
class RemoveDatasetGradersResult(TypedDict):
dataset: Dataset
removedGraderIds: list[str]
notAssignedGraderIds: list[str]
GraderRerunStatus = Literal["pending", "running", "completed", "errored"]
class GraderRerun(TypedDict):
id: str
status: GraderRerunStatus
graderIds: list[str]
progress: GraderRerunProgress | None
result: GraderRerunResult | None
error: str | None
createdAt: str
updatedAt: str
class RerunGradersResult(TypedDict):
run: GraderRerun
joinedExisting: bool
```
Dataset operations for the authenticated organization, the same operations the Bitfab MCP tools expose to a coding agent. Results are `TypedDict`s whose keys match the HTTP response (`traceCount`, `addedTraceIds`, and so on).
* `save` is an upsert keyed on `(trace_function_key, name)`. `created` is `True` for a new dataset, and `False` when an existing one was updated instead. `description=None` leaves an existing description untouched.
* `list` takes an optional trace function key. Without it, every dataset in the organization is returned.
* `Dataset` carries `id`, `traceFunctionKey`, `name`, `description`, `traceCount`, `graders` (`[{"id", "name"}]`), `createdAt`, and `updatedAt`.
* `list_traces` returns `{"datasetId", "traceIds"}`, the same membership a replay with `dataset_id` selects.
* `add_traces` and `add_graders` each accept 1 to 100 ids. They report partial acceptance rather than rejecting the whole call. Ids outside the organization, or under another trace function, come back in `skippedTraceIds` or `skippedGraderIds`. Ids already present come back in `alreadyPresentTraceIds` or `alreadyAssignedGraderIds`.
* `remove_traces` never deletes a trace, only its membership in the dataset. Ids that were not members come back in `notPresentTraceIds`. `remove_graders` reports `notAssignedGraderIds` the same way.
* `rerun_graders` re-scores every trace in the dataset. `grader_ids` defaults to every assigned grader. An unassigned id rejects the call. It waits up to `timeout` seconds, polling every `poll_interval` seconds, and returns the last `run` seen. Pass `wait=False` to return as soon as the run is queued instead. A request that matches an in-flight run joins that run, and `joinedExisting` is `True` in that case.
* `get_grader_rerun` returns the dataset's active grader re-run, or the run named by `run_id`. It returns `None` when nothing is active, or when the named run does not belong to this dataset.
* `GraderRerun` has `status` (`pending` | `running` | `completed` | `errored`), `graderIds`, `progress` (`{completedTraces, totalTraces, graderCount}` while running), `result` (`{tracesGraded, gradersRun}` when completed), and `error`.
* A dataset id from another organization raises `requests.HTTPError` with a 404.
### `traces`
```python theme={null}
client.traces.get_assertions(trace_id: str) -> TraceAssertionsResult
client.traces.save_assertions(trace_id: str, assertions: list[SaveAssertion], source: TraceAssertionSource = "agent") -> list[TraceAssertion]
client.traces.save_assertions_all(updates: list[TraceAssertionsUpdate], source: TraceAssertionSource = "agent") -> list[TraceAssertion]
client.traces.archive_assertions(trace_id: str, assertion_ids: list[str]) -> list[str]
```
An assertion says what should happen when a trace is replayed. Attach assertions to the ORIGINAL trace. `get_assertions` returns `{"assertions": [...], "inheritedFrom": ...}`. Reading a replay that has no assertions of its own returns the nearest ancestor's instead. That lookup resolves through the replay lineage, and names the ancestor trace in `inheritedFrom`. Writing assertions through a replay trace id is refused, and the error names the original to retry with. `targetOnEvaluatedTrace` narrows what the assertion checks on the trace under evaluation. It is either `{"kind": "output"}` or a span identified by name and occurrence. Omit it to check the whole trace. Targets are span names, never span ids, because a span id captured on the original resolves to nothing on the replay. Saving with an entry's `id` edits that existing assertion, so two callers adding different assertions to one trace never overwrite each other.
`save_assertions` writes one trace. `save_assertions_all` takes one update per trace and writes them all in a single request, so a publisher covering hundreds of traces makes one call. The singular delegates to it, so both go through the same route.
```python theme={null}
saved = client.traces.save_assertions_all(
[
{"traceId": trace_id, "assertions": [{"assertion": "Departs before 9am"}]}
for trace_id in trace_ids
]
)
```
The server writes the whole batch in one transaction, so a rejected batch writes nothing and there is no half-saved state to reconcile. Results come back as one flat list covering every trace, each row carrying its own `traceId`. A batch takes up to 500 traces, up to 50 assertions per trace, and at most 1000 assertions in total. Passing an empty list writes nothing and sends no request.
`humanNote` is people-only context attached to the assertion. SDK reads return it for display, while MCP and Studio can write it. It must never be used as evidence when judging a replay.
```python theme={null}
TraceTargetOccurrence = Literal["first", "last"] | int
TraceAssertionSource = Literal["human", "agent"]
class OutputTarget(TypedDict):
kind: Literal["output"]
class SpanTarget(TypedDict):
kind: Literal["span"]
name: str
occurrence: NotRequired[TraceTargetOccurrence]
TraceTarget = OutputTarget | SpanTarget
class SaveAssertion(TypedDict):
category_assertion_id: NotRequired[str | None]
id: NotRequired[str]
assertion: str
passCriteria: NotRequired[str | None]
failCriteria: NotRequired[str | None]
targetOnEvaluatedTrace: NotRequired[TraceTarget | None]
class TraceAssertion(TypedDict):
category_assertion_id: str | None
category: AssertionCategorySummary | None
id: str
traceId: str
assertion: str
humanNote: str | None
passCriteria: str | None
failCriteria: str | None
targetOnEvaluatedTrace: TraceTarget | None
source: TraceAssertionSource
createdAt: str
updatedAt: str
class TraceAssertionsResult(TypedDict):
assertions: list[TraceAssertion]
inheritedFrom: str | None
class TraceAssertionsUpdate(TypedDict):
traceId: str
assertions: list[SaveAssertion]
```
### `assertion_categories`
```python theme={null}
client.assertion_categories.save(title: str, *, id: str | None = None, description: str | None = None) -> AssertionCategory
client.assertion_categories.get(id: str) -> AssertionCategory
client.assertion_categories.list() -> list[AssertionCategory]
client.assertion_categories.delete(id: str) -> AssertionCategory
class AssertionCategorySummary(TypedDict):
id: str
title: str
description: str
class AssertionCategory(AssertionCategorySummary):
organizationId: str
createdAt: str
updatedAt: str
```
Categories group assertions within the API key's organization. `save` creates without an `id` and updates with one. The title is required. Omit `description`, or pass `None`, to preserve it on an update. Pass an empty string to clear it. `list` returns categories ordered by title. `delete` returns the deleted category and clears its assignments. Assertions and verdicts remain intact.
Pass `category_assertion_id` on entries in `save_assertions` or `save_assertions_all` to assign a category. Omit the field to preserve an existing assignment. Pass `None` to clear it. Assertion reads and saves return `category_assertion_id` and the nullable `category` summary. Inherited assertions include the same category metadata.
```python theme={null}
category = client.assertion_categories.save(
"Mandate compliance", description="Checks explicit user instructions"
)
assertion = client.traces.save_assertions(
trace_id,
[{"assertion": "Escalates conflicting calendar requests", "category_assertion_id": category["id"]}],
)[0]
client.traces.save_assertions(
trace_id,
[{"id": assertion["id"], "assertion": assertion["assertion"], "category_assertion_id": None}],
)
client.assertion_categories.delete(category["id"])
```
### `labels`
```python theme={null}
client.labels.save(label: bool, annotation: str, trace_id: str | None = None, original_trace_id: str | None = None, attempt: int | None = None, confidence: LabelConfidence | None = None, test_run_id: str | None = None, assertion_id: str | None = None) -> LabelOutcome
client.labels.save_all(labels: list[LabelUpdate], test_run_id: str | None = None) -> list[LabelOutcome]
client.labels.skip(trace_id: str | None = None, original_trace_id: str | None = None, attempt: int | None = None, test_run_id: str | None = None, assertion_id: str | None = None) -> LabelOutcome
client.labels.archive(trace_id: str | None = None, original_trace_id: str | None = None, attempt: int | None = None, test_run_id: str | None = None, assertion_id: str | None = None) -> LabelOutcome
client.labels.save_human(label: bool, annotation: str, trace_id: str, confidence: LabelConfidence | None = None, assertion_id: str | None = None) -> HumanLabelOutcome
client.labels.save_human_all(labels: list[HumanLabelUpdate]) -> list[HumanLabelOutcome]
client.labels.get(trace_id: str) -> TraceLabels | None
client.labels.get_all(trace_ids: list[str]) -> list[TraceLabels]
```
Writes the same pass/fail verdicts the `save_agent_labels` MCP tool writes. It runs from inside a replay process instead of a coding-agent session. Key a replay verdict by `original_trace_id` plus the `test_run_id` it ran under. Add `attempt` when the experiment ran each trace more than once. Use `skip` for an attempt that crashed, was punctured, or fell back to a schema default. Also use `skip` for an assertion whose target could not be resolved. Recording a FAIL in either case would read as a behavior regression rather than a check that never ran.
Omit `assertion_id` and the verdict scores the whole trace. Pass one and the verdict scores that single assertion. A per-assertion verdict and a whole-trace verdict can both sit on the same trace. `save_all` carries the same field as `assertionId` on each entry, so one batch can mix both forms.
**Raises:** `ValueError` from `save`, `skip`, and `archive` unless exactly one of `trace_id` or `original_trace_id` is given. `save_all` takes prebuilt `LabelUpdate` entries and does not run that check.
```python theme={null}
LabelConfidence = Literal["VeryLow", "Low", "Medium", "High", "VeryHigh"]
LabelAction = Literal["set", "archived", "no-active-label", "skipped"]
class LabelUpdate(TypedDict):
traceId: NotRequired[str]
originalTraceId: NotRequired[str]
attempt: NotRequired[int]
assertionId: NotRequired[str]
label: NotRequired[bool]
annotation: NotRequired[str]
confidence: NotRequired[LabelConfidence]
skip: NotRequired[bool]
archive: NotRequired[bool]
class LabelOutcome(TypedDict):
key: str
traceId: str
action: LabelAction
```
`save_human` and `save_human_all` write the verdicts `save_human_labels` writes over MCP. They are validated on write with no approval step, so they satisfy `search_traces` `validated: true` immediately. Use them only when a human decided the verdict, such as capturing a known production bug as a regression case. An agent's own first-pass guesses go through `save` so they keep the approve-or-edit loop. Approving an existing agent verdict is not on this surface at all: that happens in Studio, by a person.
Like `save_all`, the batch is all-or-nothing. A trace outside the organization, a repeated target, or an `assertion_id` that is not active on its trace rejects the whole call and writes nothing, so a raised error never leaves part of the batch committed.
```python theme={null}
class HumanLabelUpdate(TypedDict):
traceId: str
assertionId: NotRequired[str]
label: bool
annotation: str
confidence: NotRequired[LabelConfidence]
class HumanLabelOutcome(TypedDict):
traceId: str
assertionId: str | None
label: bool
action: Literal["set"]
```
`get` and `get_all` read verdicts back. Each trace carries its effective verdict plus one row per scored assertion, keyed by the same `assertion_id` the write used, so a per-assertion verdict is verifiable per assertion rather than as a passed/failed tally. `get` returns `None` when the trace is not in this organization. One `get_all` call accepts up to 100 ids.
```python theme={null}
LabelStatus = Literal["labeled", "skipped", "unlabeled"]
LabelSource = Literal["human", "agent"]
class AssertionVerdict(TypedDict):
assertionId: str
assertion: str | None
labelStatus: LabelStatus
label: bool | None
annotation: str | None
confidence: LabelConfidence | None
labelSource: LabelSource
approved: bool
class TraceLabels(TypedDict):
traceId: str
labelStatus: LabelStatus
label: bool | None
annotation: str | None
approved: bool
passed: int
failed: int
assertions: list[AssertionVerdict]
```
### `graders`
```python theme={null}
client.graders.get_labels(trace_ids: list[str] | None = None, grader_id: str | None = None, limit: int | None = None) -> list[GraderLabel]
```
Reads the individual verdicts each automated grader recorded, one row per grader per trace, the same breakdown the `get_grader_labels` MCP tool returns. Pass `trace_ids` to see every grader's verdict on those traces, `grader_id` to see one grader's most recent verdicts across traces, or both to narrow. This is the per-grader detail that `labels.get` does not carry, since that returns one grader-agnostic verdict per trace.
**Raises:** `ValueError` when neither `trace_ids` nor `grader_id` is given.
```python theme={null}
GraderLabelSource = Literal["human", "live_grader"]
class GraderLabel(TypedDict):
traceId: str
graderId: str
graderName: str | None
graderStatus: GraderStatus
label: bool | None
labelReason: str | None
failureDiagnostic: str | None
labelConfidence: LabelConfidence | None
source: GraderLabelSource
evaluatedAt: str | None
```
Under `bitfab-replay`, this loop runs in one of two hooks. The registry entry's `on_item_finish` runs in the process that owns the run, and is the default place for it. `ReplayConcurrency`'s `on_item_finish_in_child_process` runs in the child that replayed the item, and is the only option when the judge needs state that run left in memory. Both fire once the attempt's own trace has been flushed, which is when a lineage-keyed verdict row resolves and when the replay trace ID becomes available.
Reading the assertions, judging the replay, and writing the verdict back is the whole loop:
```python theme={null}
result = client.traces.get_assertions(item.original_trace_id)
verdict = judge(result["assertions"], replay_output)
if verdict is None:
client.labels.skip(
original_trace_id=item.original_trace_id,
attempt=item.attempt,
test_run_id=run.test_run_id,
)
else:
client.labels.save(
label=verdict.passed,
annotation=verdict.summary,
original_trace_id=item.original_trace_id,
attempt=item.attempt,
test_run_id=run.test_run_id,
)
```
### Replay registry and command
```python theme={null}
registry = ReplayRegistry()
registry.register("checkout", bitfab, run_checkout)
registry.register(
"agent",
bitfab,
run_agent,
trace_function_key="support-agent",
mock="marked",
adapt_inputs=adapt_inputs,
options_factory=lambda ctx: {
"mock_override": create_support_agent_mock(ctx.params.get("scenario"))
},
)
```
```python theme={null}
class ReplayRegistry:
@property
def names(self) -> tuple[str, ...]: ... # registered command names, in registration order
def register(
self,
name: str,
client: Bitfab,
fn: Callable[..., Any],
*,
trace_function_key: str | None = None,
options_factory: Callable[[ReplayRegistryContext], dict[str, Any]] | None = None,
**options: Any,
) -> ReplayRegistry: ...
def get(self, name: str) -> ReplayRegistration: ...
```
`ReplayRegistry.register` stores the exact production callable, static per-function replay behavior, and an optional `options_factory`. It returns `self`, so calls chain. A `@span`-decorated function supplies its key automatically. A plain handler root passes `trace_function_key` explicitly instead. `**options` accepts the same keyword arguments as `replay()`: `limit`, `trace_ids`, `name`, `max_concurrency`, `code_change_description`, `code_change_files`, `experiment_group_id`, `dataset_id`, `dataset_ids`, `grader_ids`, `only_with_assertions`, `mock`, `mock_override`, `adapt_inputs`, `db_branch`, `dry_run`, `attempts`, `concurrency`, and `on_item_finish`. The child's own grading hook is not among them. It is set on the `ReplayConcurrency` passed as `concurrency`, since only that object creates a child to run it in.
The factory receives JSON values from `--params` and `--param`. It can construct executable options such as `mock_override`. Values passed directly with `--param` override values loaded from a `--params` file. Unknown option names are rejected when the registry loads.
`on_item_finish` is the one lifecycle callback a registry entry can set. The command runs its own progress reporter first, then calls yours with the same finished item, which is where a replay verdict is written. It runs in the process that owns the run, so under `primitive="process"` it runs in the parent, once that item's child has exited. It is also the only hook that fires for an item whose child died without producing a result. It is skipped under `--dry-run`, since nothing ran. A callback that raises is reported on stderr as `on_item_finish failed` and never fails the run.
To grade inside the child instead, or in addition, see `on_item_finish_in_child_process` on [`ReplayConcurrency`](#replayconcurrency).
`ReplayRegistry.get(name)` returns the stored `ReplayRegistration`, carrying `client`, `fn`, `trace_function_key`, `options`, and `options_factory`. `.names` lists registered command names in registration order. The package installs `bitfab-replay`. Run it as `bitfab-replay --registry [options]`. The registry module must define `registry`.
**Raises:**
* `ValueError` for an empty or duplicate `name`
* `ValueError` for a non-callable `options_factory`
* `ValueError` for a non-callable `on_item_finish`
* `ValueError` for a plain `fn` with no `trace_function_key` and no `@span` key of its own
* `ValueError` for an option name outside the accepted set above
| Flag | Value | Effect |
| ------------------------------------ | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--limit` | `N` | Most traces to select (default 10). It bounds whatever selection is in effect, whether that selector came from the command line or the registry, truncating an ID list and pinning the first N members of the selected datasets ordered by ID |
| `--trace-ids` | `id1,id2` | Replay exactly these traces. Mutually exclusive with `--dataset-ids` |
| `--dataset-ids` | `uuid1,uuid2` | Replay the membership of one or more datasets, as their deduped union. Mutually exclusive with `--trace-ids`. `--dataset-id` is the same flag |
| `--name` | `NAME` | Title for the resulting experiment |
| `--attempts` | `N` | Replay each selected trace N times in one run (1 to 100, default 1) |
| `--concurrency`, `--max-concurrency` | `N` | Items in flight at once |
| `--dry-run` | | Resolve every item's inputs and stop without calling the function |
| `--experiment-group-id` | `UUID` | Add this run to an existing experiment group |
| `--grader-ids` | `id1,id2` | Attach graders to this run only |
| `--only-with-assertions` | | Replay only the selected traces that carry an assertion. Narrows `--limit`, `--trace-ids` or `--dataset-ids`. Omitting it leaves a registry-set `only_with_assertions` in place |
| `--code-change` | `PATH` | Load a code-change description from a file |
| `--no-code-change` | | Record no code change, overriding a registry default |
| `--mock` | `none\|all\|marked` | Which recorded child spans return their historical output |
| `--db-branch` | | Turn on per-item database branching, keeping a registry-configured `db_branch` mapping (`min_cu`/`max_cu`/`warmup_sql`) if one is already set |
| `--no-db-branch` | | Force per-item database branching off, overriding any registry default |
| `--seed` | `cases.jsonl` | Run the file's cases once each and record them instead of replaying. Ignores `--params`/`--param`, because the registration's `options_factory` never runs during seeding |
| `--params` | `PATH` | JSON file of values passed to `options_factory` |
| `--param` | `name=value` | One value passed to `options_factory`. Repeatable, and overrides `--params` |
| `-h`, `--help` | | Print usage |
A run whose selection matched no traces exits non-zero rather than reporting a clean run of zero items.
`--seed ` runs each case once through the same registration and records the run, instead of replaying. Each line is a JSON object with an `input` list, plus optional `kwargs`, `metadata`, and `session_id`. A JSON array of those objects works too. A case's `input` and `kwargs` are the call itself, recorded as-is. A case carrying `expected` is rejected, since the output is what the run produced. The registration already holds the client, the callable, and the trace function key, so the case runs against the exact function the later replay selects. The registration's `adapt_inputs` is not run at seed time. It is a replay hook. A replay of the seeded trace applies it then, with the case's `metadata` available on `ctx`.
### Replay auto-capture environment variables
| Variable | Effect |
| ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `BITFAB_CODE_CHANGE_PATH` | Path to a JSON file of `{"description", "files"}`, written by the Bitfab plugin. `replay()` reads it instead of running `git diff` when `code_change_files` is omitted |
| `BITFAB_CODE_CHANGE_BASE` | Git ref to diff against, instead of searching the default trunk candidates (`origin/HEAD`, `origin/main`, `origin/master`, `main`, `master`) |
| `BITFAB_DISABLE_CODE_CHANGE_CAPTURE` | Set to opt every replay in this process out of automatic code-change capture |
| `BITFAB_REPLAY_RESULT_PATH` | When set by the Bitfab plugin, `replay()` writes the full final `ReplayResult` JSON to this path automatically, in addition to its return value |
### Replay memory throttle environment variables
These apply under `primitive="process"`, where `memory_throttle` decides when a child may launch.
| Variable | Effect |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `BITFAB_REPLAY_MEMORY_THROTTLE` | Set to `off`, `0`, `false` or `no` to launch every child as soon as a worker is free, whatever the machine's memory looks like |
| `BITFAB_REPLAY_CHILD_MEMORY_MB` | Memory to expect one child to reach, in MB. Used until a child has finished and been measured, after which the measured peak takes over. Defaults to 2048 |
| `BITFAB_REPLAY_MEMORY_FLOOR_MB` | Memory to leave free rather than hand to a new child, in MB. Defaults to 2048 |
`primitive="process"` re-execs `bitfab-replay --registry` once per work item. Each item runs in its own child interpreter. The environment for that interpreter is captured before the registry module's own import-time side effects run. A child that runs longer than 40 minutes is killed. It is then recorded as a failed replay item. To grade an item inside the child that ran it, set `on_item_finish_in_child_process` on the `ReplayConcurrency`.
### `register_mock_override`
```python theme={null}
def register_mock_override(
match_or_override: str | NodeMatcher | MockOverride | MockOverrideResolver,
value: Any = ...,
) -> None
```
Registers an instance-scoped override for every subsequent replay. Accepted forms are `register_mock_override(MockOverride(...))`, `register_mock_override(match, value)`, `register_mock_override(trace_function_key, resolver_or_override)`, and `register_mock_override(resolver)`. A keyed resolver runs only for child spans with that trace function key. A keyed `MockOverride` also applies its matcher. A global resolver can route on `ctx.node.trace_function_key`. Return `NO_MOCK_OVERRIDE` to continue to lower-priority overrides and the base mock strategy. `None` remains a valid mocked output. Per-call overrides take precedence over registered overrides.
**Raises:**
* `ValueError` when a keyed call's second argument is neither a `MockOverride` nor a callable resolver
* `ValueError` when a `MockOverride` is passed together with a second `value` argument
* `ValueError` when a bare, non-callable `match_or_override` is given with no `value`
### `clear_mock_overrides`
```python theme={null}
def clear_mock_overrides() -> None
```
Removes every override registered on this client.
### `replay`
```python theme={null}
def replay(
fn_or_key: Callable | str | None = None,
fn: Optional[Callable] = None, # required when fn_or_key is a key string, or when fn_or_key is omitted (legacy replay(fn=decorated_fn))
*,
limit: Optional[int] = None, # default 5; max 5,000; ignored with trace_ids/datasets
trace_ids: Optional[list[str]] = None, # max 100; with a dataset selector it pins which members replay
name: str | None = None, # display name for the resulting experiment/test run
max_concurrency: Optional[int] = 10,
attempts: int = 1, # 1-100; replay each trace this many times, attempt-major
concurrency: ReplayConcurrency | None = None, # supersedes attempts + max_concurrency
code_change_description: str | None = ...,
code_change_files: list[CodeChangeFile] | None = ..., # omit for rename-aware git diff vs trunk; None disables capture
mock: MockStrategy = "marked",
mock_override: MockOverrideInput | list[MockOverrideInput] | None = None,
adapt_inputs: Callable[[list[Any], dict[str, Any], AdaptContext], tuple[list[Any], dict[str, Any]]] | None = None,
experiment_group_id: str | None = None,
db_branch: DbBranchOptions | bool | None = None, # True requests a DB branch per replay item
dry_run: bool = False, # resolve every item's inputs and call nothing
dataset_id: str | None = None, # durably attributes the experiment to a dataset
dataset_ids: list[str] | None = None, # several datasets: replays their union, attributed to each
grader_ids: list[str] | None = None, # graders attached to this run, unioned with the dataset's at grading (max 100)
only_with_assertions: bool = False, # narrow the selection to traces carrying an assertion
on_item_start: Callable[[ReplayItemStartProgress], None] | None = None,
on_item_finish: Callable[[ReplayItemFinishProgress], None] | None = None,
on_progress: Callable[[ReplayProgress], None] | None = None, # deprecated compatibility callback
process_launcher: ProcessLauncher | None = None, # internal: wired by bitfab-replay --registry for primitive="process"
) -> ReplayResult
```
`only_with_assertions` narrows whatever `limit`, `trace_ids` or `dataset_ids` selected to the traces that carry at least one assertion, so a run measuring assertion outcomes does not pay to re-execute traces nothing can grade. The server applies it as part of selection, which is what makes it compose with `limit`: `limit=10, only_with_assertions=True` is the ten most recent traces that HAVE assertions, not the ten most recent filtered down to however few do. An archived assertion does not count, and a selection that narrows to nothing replays nothing (the CLI exits non-zero, as it does for any empty selection).
`mock_override` accepts a `MockOverride`, a global resolver, or a list of either. The first matching entry whose value is not `NO_MOCK_OVERRIDE` wins. Per-call overrides run before registered overrides. Both take precedence over `mock`.
`on_item_start` fires when a worker begins processing an item, before replay setup or customer code runs. It carries `type="started"`, running lifecycle totals, and the historical trace and span identity.
Pair it with `on_item_finish`. `on_item_finish` fires exactly once per item, as that item finishes. It carries running totals (`completed`, `total`, `succeeded`, `errored`), `test_run_id`, and the required finished `item`. It never represents whole-run completion.
The finished `item` carries these fields:
* `trace_id`: the server replay `traces.id`, read back off the ingest response and surfaced as the item finishes. Its trace is flushed on finish. `trace_id` is `None` only if that flush could not confirm delivery in time
* `original_trace_id`: the original historical trace being replayed. `source_trace_id` is a deprecated alias for the same value
* `input`, `result`, `original_output`, `error`, `trace_error`, and `replay_error`
* `duration_ms`: this replay's own duration
* `original_duration_ms`, `original_tokens`, and `original_model`: describe the trace being replayed
* `tokens`, `model`, `db_snapshot_ref`, and `db_branch_timings`
* `trace_outline` and `original_trace_outline`: both `None` at this point, filled in at completion
Replay doesn't know pass or fail yet. Verdicts are assigned later. The totals only split ran-ok from errored.
After each item executes, the SDK flushes the shared OTel pipeline. It finishes from delivery acknowledgments once every carrier is confirmed. Only ambiguous delivery falls back to final-status and expected-span-count polling, before the test run is finalized.
A raising lifecycle callback never crashes the run.
The deprecated `on_progress` callback receives the same per-item events. It may additionally receive its legacy, item-less terminal `complete` event. It is ignored when `on_item_finish` is also provided.
Pass the ready-made `report_replay_progress` reporter as both `on_item_start` and `on_item_finish`. The Bitfab plugin uses its stderr events to identify in-flight traces and to write finished per-item result files.
There are two call forms.
`replay(decorated_fn)` reads the trace function key from the `@span` decorator. It raises if `decorated_fn` is undecorated.
`replay("key", fn)` takes an explicit key plus any plain callable. The SDK wraps `fn` in a span under that key internally. This explicit-key form is how handler-instrumented workflows replay: LangGraph, LangChain, the Claude Agent SDK, and OpenAI Agents. Those workflows have no decorated root in the app.
A legacy keyword form, `replay(fn=decorated_fn)`, is also preserved. `fn_or_key` used to be positionally named `fn`, so this spelling still works. Prefer `replay(decorated_fn)` in new code.
When replay auto-wraps a plain callable, it passes a recorded dict root input, for example a LangGraph state, as a single positional argument. This matches the TypeScript SDK. A decorated function always gets the keyword-args splat instead, whether or not a redundant matching key is also passed. An explicit key that contradicts the decorator's own key raises.
`max_concurrency=None` means unlimited. `max_concurrency=1` means sequential. When `trace_ids` is passed, `limit` is ignored, with a warning, because the explicit ID list already determines how many traces replay.
`attempts` (1 to 100, default 1) replays each selected trace that many times inside the same experiment. Attempts run attempt-major. Each attempt is its own replay trace, with its own correlation id, verdict, tokens, and cost. `ReplayItem.attempt` reports which attempt an item is.
`concurrency` is the options-object form of the same controls: `ReplayConcurrency(attempts=..., primitive=..., max_concurrency=...)`. Passing `concurrency` alongside the `attempts` or `max_concurrency` arguments raises, rather than merging the two.
* `primitive="async"` is the default. It runs every work item as a coroutine in one process. `max_concurrency` defaults to 10
* `primitive="process"` runs each work item in its own child interpreter. `max_concurrency` defaults to 4, and `None` is refused. Only this primitive accepts `on_item_finish_in_child_process` and `memory_throttle`
Process mode is for a replay target whose world lives in process-global state, such as a settings module, a per-item database, or a module-level registry. Two items cannot share one process in that case. The unit of concurrency is the work item, not the attempt. N traces at K attempts is N x K child processes, all drawn from one queue.
Process mode re-execs the replay command once per item. It therefore only runs under `bitfab-replay --registry`. A bare `replay()` call raises instead, because it holds a live function object with no re-exec target.
`process_launcher` is the internal seam `bitfab-replay --registry` uses to supply that re-exec mechanism. Its type is not exported. Calling `replay()` directly should never pass it.
`db_branch=True` requests a DB branch per replay item, sized however the mirror project is sized. `False`, or omitting `db_branch`, leaves branching off. Each bounded worker resolves its own branch, including a separate branch per attempt. `get_current_replay_branch()` exposes that branch inside `fn`.
`db_branch` also accepts a mapping to tune the branch, for example `db_branch={"min_cu": 2, "max_cu": 2, "warmup_sql": "SELECT 1;"}`. Every key in that mapping is optional.
* `min_cu` and `max_cu` are the compute's autoscaling floor and ceiling, in Neon Compute Units (0.25 to 56). Equal values pin a fixed size, up to 56, which keeps items comparable. An autoscaling range may not span more than 8 CU, and may not exceed a ceiling of 16 CU
* `warmup_sql` is appended to the branch's readiness check. It runs before `fn` sees the branch, so its time is not charged to the replayed call. Invalid warm-up SQL fails the branch, rather than quietly handing back a cold one
Omit `db_branch`'s keys entirely and the branch keeps the mirror project's own defaults.
Async-generator spans are not mockable. If `mock="all"`, `mock="marked"` selects one, or a `mock_override` matches, the item errors before the real generator is iterated.
A trace replays only when its root span has serializable inputs. It also replays when it was instrumented through a [framework handler](/python-sdk#replaying-handler-instrumented-functions), whose recorded root input is serializable. If the original inputs were stubbed as non-serializable at capture time, the trace cannot be replayed.
`dry_run=True` resolves every item's inputs through selection, deserialization, and `adapt_inputs`. It then stops without calling `fn`. Each item reports the exact `(args, kwargs)` the function would have received. This is the cheap way to check that recorded inputs still fit the current signature. The persistence barrier is skipped, since a run that executed nothing produces no traces.
### `seed_trace`
```python theme={null}
def seed_trace(
trace_function_key: str,
fn: Callable,
*,
args: Sequence[Any] = (),
kwargs: Mapping[str, Any] | None = None,
metadata: Mapping[str, Any] | None = None,
session_id: str | None = None,
name: str | None = None,
) -> str
```
Runs `fn` once and records the execution as an original trace. It returns the trace ID, for use with `replay(trace_ids=[...])`. Capture stays off. This records exactly one call, with the same semantics capture-on would give it: a root span, a first-party subtree under the `trace` decorator's bounds, and no mocking. The recorded input is the real call. The output is what the run produced. The trace lands under `trace_function_key`, with `ingestion_type: seeded`. `replay` selects it like any other trace. Each replay of it links back as `original_trace_id`.
`fn` resolves exactly as it does for `replay`. A decorated function records under its own key. That key must match `trace_function_key`. A plain callable is wrapped under `trace_function_key` instead. An exception is recorded on the root span. The trace still persists. The exception is re-raised. A call that records nothing, for example because no API key resolved or `fn` is a generator, raises instead of returning an ID that replay could never find.
`metadata` is stored on the trace. It is handed to a later replay's `adapt_inputs` hook as `ctx["metadata"]`. This lets a case's provenance, such as its id, suite, or source row, ride with the trace instead of through the recorded inputs. If the traced function also emits its own trace through an integration, the caller's metadata is merged onto that export and wins on any shared key, so the provenance recorded here is what replay reads back. `name` is the trace's title and a searchable field. Put the case's own label there, such as a ticket id or a dataset row name, so the seeded trace can be found by it.
Call it from synchronous code. An `async def` `fn` runs to completion on a fresh event loop. Calling it from inside an already-running loop raises. With `trace_across_threads=True`, spans from worker threads inside the call nest under the seeded root.
A seeded trace carries no database pin. `db_branch` refuses it as a result. It has a full recorded subtree, though. Replay mocking works on it exactly as it does for a captured trace.
```python theme={null}
bitfab = Bitfab(capture_enabled=False)
trace_id = bitfab.seed_trace(
"agent-turn",
run_ticket,
kwargs={"ticket_id": "T-1"},
metadata={"case_uid": "c-1", "suite": "smoke"},
)
bitfab.replay(run_ticket, trace_ids=[trace_id], adapt_inputs=lambda args, kwargs, ctx: (args, kwargs))
```
### `reseed_trace`
```python theme={null}
def reseed_trace(
trace_function_key: str,
fn: Callable,
*,
trace_id: str,
) -> ReseedResult
```
Re-seeds one trace. Reads the trace's recorded root inputs, name, session, and metadata from Bitfab, runs `fn` once exactly as `seed_trace` would, then asks Bitfab to adopt that run under `trace_id`. Returns `{"trace_id", "previous_run_trace_id"}`. The trace keeps its id, labels, assertions, dataset membership, name, and metadata. The previous run is kept as its own trace with `reseedOfTraceId` pointing back at `trace_id`. Nothing is mocked and no test run is created.
`fn` resolves as it does for `replay` and `seed_trace`, and `trace_function_key` must match the trace's own key. A run that raises is recorded but not adopted, and the exception is re-raised. Bitfab rejects a run from another function, one that already belongs to a dataset, or a trace that is itself a previous run. Graders on the datasets holding the trace are re-queued, and default replay selection skips previous runs.
### `reseed_from_registry`
```python theme={null}
def reseed_from_registry(
registry: ReplayRegistry,
pipeline: str,
trace_ids: Sequence[str],
) -> dict[str, Any]
```
Re-seeds each trace through an already-registered pipeline, using the registration's client, callable, and trace function key. Returns `{"pipeline", "traceFunctionKey", "reseeded": [{"traceId", "previousRunTraceId"}]}`. The installed `bitfab-seed --registry --from-trace [,...]` command calls it.
### `seed_from_registry`
```python theme={null}
def seed_from_registry(
registry: ReplayRegistry,
pipeline: str,
cases: Sequence[Mapping[str, Any]],
) -> dict[str, Any]
```
Runs each case once through an already-registered pipeline and records it. It uses the registration's client, callable, and trace function key. Each case is a mapping with an `input` list, plus optional `kwargs`, `metadata`, and `session_id`. `input` and `kwargs` are the call itself, recorded as-is. The registration's `adapt_inputs` is a replay hook. It is not run at seed time. A seeded trace is therefore never adapted twice. Returns `{"pipeline", "traceFunctionKey", "traceIds"}`.
### `call`
```python theme={null}
def call(method_name: str, **kwargs: Any) -> Any
```
Executes a server-configured BAML function locally using `env_vars`.
**Raises:** `ValueError` when the function lookup fails. An execution error re-raises whatever exception the BAML call itself raised, not necessarily `ValueError`.
### Framework Integrations
Handlers returned by these methods plug into each framework's
callback, processor, or hook surface. They emit Bitfab spans automatically.
They also reuse the owning `Bitfab` client's lazy OTel worker, so
`Bitfab.close()` releases the decorator and the framework transport together.
Directly constructed LangGraph or Claude handlers are the exception. They own
their own transport, and expose `close(timeout=30.0)` plus a context manager.
For usage examples and semantics, see the per-framework guides.
#### `get_langgraph_callback_handler`
```python theme={null}
def get_langgraph_callback_handler(trace_function_key: str) -> BitfabLangGraphCallbackHandler
```
Returns a LangChain/LangGraph `BaseCallbackHandler`. Pass via `config={"callbacks": [handler]}` when invoking. The handler-created root is replayable from the framework input. A separate `@span` root is only needed for meaningful surrounding application work. Retriever calls are captured too, as `function`-type spans carrying the query and returned documents. See [LangGraph framework guide](/frameworks/langgraph).
Aliased as `get_langchain_callback_handler(trace_function_key)` for plain LangChain projects. The returned handler and its behavior are identical. The handler class is also exported as `BitfabLangChainCallbackHandler`.
#### `get_langgraph_integration`
```python theme={null}
def get_langgraph_integration(
trace_function_key: str,
*,
mock_tools_on_replay: bool | list[str] | tuple[str, ...] = True,
) -> BitfabLangGraphIntegration
integration.create_invoker(graph) -> Callable[..., Any]
integration.create_async_invoker(graph) -> Callable[..., Awaitable[Any]]
```
**Experimental (alpha).** Returns the LangGraph integration.
Pass `wrap_tool_call` and `awrap_tool_call` directly to LangGraph's native `ToolNode` constructor.
`create_invoker(graph)` returns a sync callable. It adds `callback_handler` through LangGraph's public `with_config()` API. It preserves invocation-time config and any existing callbacks. It records only the graph input as the replayable root input. Use `create_async_invoker(graph)` instead for async-only graphs.
Use the lower-level `callback_handler` and `wrap_invoke(fn)` when meaningful application work around the graph invocation needs to live inside the trace.
Under an active `trace()` subtree, the integration's tool and invoke spans behave as `node()` calls instead of opening opt-in spans. This gives them nearest-frame parenting, span budgets, and `mock_on_replay_default` inheritance. A wrapped graph can therefore run inside opt-out tracing without raising `MixedTracingError`.
Integration-managed tools are marked for replay mocking by default. Recorded `ToolMessage` and `Command` results are reconstructed with the current tool-call ID. An expected tool output that is missing during replay fails closed, instead of running the live tool.
Calls are matched by tool name and occurrence order. Repeated concurrent calls to the same tool are not yet recommended, as a result.
Install the `langgraph` extra. See the [LangGraph framework guide and current limitations](/frameworks/langgraph#current-alpha-limitations).
```python theme={null}
from langgraph.prebuilt import ToolNode
integration = bitfab.get_langgraph_integration("support-agent")
tools = [lookup_customer, search_docs]
tool_node = ToolNode(
tools,
wrap_tool_call=integration.wrap_tool_call,
awrap_tool_call=integration.awrap_tool_call,
)
graph = build_support_graph(tools=tools, tool_node=tool_node)
run_support_agent = integration.create_invoker(graph)
result = run_support_agent(
{"messages": [{"role": "user", "content": "Help with order 123"}]},
config={"configurable": {"thread_id": "customer-456"}},
)
bitfab.replay("support-agent", run_support_agent, limit=10)
```
#### `get_openai_tracing_processor`
```python theme={null}
def get_openai_tracing_processor() -> BitfabOpenAITracingProcessor
```
Constructing the processor does not require `openai-agents`. Registering it with `agents.add_trace_processor` does require it. It captures agent internals. Pair it with `get_openai_agent_handler` for a replayable root. See [OpenAI Agents framework guide](/frameworks/openai-agents).
#### `get_openai_agent_handler`
```python theme={null}
def get_openai_agent_handler(trace_function_key: str) -> BitfabOpenAIAgentHandler
```
Returns a handler. Its `wrap_run(agent, input, **run_kwargs)` is a drop-in for `Runner.run`. It records a keyed, replayable root span carrying the run input, and the tracing processor's spans nest underneath it. For streamed runs, `wrap_run_streamed(agent, input, **run_kwargs)` is an async-generator drop-in for `Runner.run_streamed`. Iterate it to consume the same stream events while the run is traced. `bitfab.replay()` re-runs the non-streaming `wrap_run`, not this async generator. Both methods skip opening their own root span when already inside an enclosing Bitfab span, such as a replay auto-wrap or the caller's own `@span`. In that case they run the call directly, so the tracing processor nests under the existing root instead of doubling it. See [OpenAI Agents framework guide](/frameworks/openai-agents).
#### `get_claude_agent_handler`
```python theme={null}
def get_claude_agent_handler(trace_function_key: str) -> BitfabClaudeAgentHandler
```
Returns a handler exposing `instrument_options(options)`, `wrap_response(stream, input=...)`, and `wrap_query(stream, input=...)` for the Claude Agent SDK. Pass `input=prompt` to the wrap call to record a replayable root span. When a Bitfab span is already active, no second root opens. The handler's spans nest under the existing one instead. See [Claude Agent SDK framework guide](/frameworks/claude-agent-sdk).
#### `wrap_baml`
See the [BAML framework guide](/frameworks/baml) for examples. Full signature under [`wrap_baml`](#wrap-baml) above.
## `class BitfabFunction`
Returned from `client.get_function(key)`.
```python theme={null}
def span(
*,
name: Optional[str] = None,
type: SpanType = "custom",
capture_when: CaptureWhen = "always",
mock_on_replay: bool = False,
finalize: Optional[Callable[[Any], Any]] = None,
) -> Callable[[Callable[P, T]], Callable[P, T]]
def get_claude_agent_handler() -> BitfabClaudeAgentHandler
def get_langgraph_callback_handler() -> BitfabLangGraphCallbackHandler
def get_langchain_callback_handler() -> BitfabLangGraphCallbackHandler
def get_langgraph_integration(
*,
mock_tools_on_replay: bool | list[str] | tuple[str, ...] = True,
) -> BitfabLangGraphIntegration
def wrap_baml(
method_or_client: Any,
method: Optional[Callable] = None,
) -> Callable[..., Any]
```
Delegates to the parent `Bitfab` instance, using the bound `trace_function_key`. The `get_*_handler` methods reuse that bound key, so a `span` root and the handler share it. This is the documented same-key nesting pattern, and it needs no repeated string. The experimental `get_langgraph_integration()` also binds the key. Pass its sync and async hooks directly to LangGraph's native `ToolNode`. Then create the graph entry point with `create_invoker()` or `create_async_invoker()`. `wrap_baml` is the exception. It opens no span and uses no key. It enriches the *current* span instead. Call it inside a function that is already wrapped by this handle's `span`.
## Module Functions
### `finalizers`
```python theme={null}
def finalizers.openai_chunks(chunks: list[Any]) -> dict[str, Any]
def finalizers.anthropic_events(events: list[Any]) -> dict[str, Any]
```
Prebuilt `finalize=` helpers for streaming spans. `openai_chunks` assembles OpenAI chat-completion chunks into `text`, `finish_reason`, `usage`, and `tool_calls`. `usage` is present only when the request set `stream_options={"include_usage": True}`. `anthropic_events` assembles Anthropic stream events into `text`, `stop_reason`, and `usage`. Both helpers are duck-typed. Both tolerate malformed or unfamiliar events. Neither requires the provider package at runtime.
### `get_current_span()`
```python theme={null}
def get_current_span() -> CurrentSpan | _NoOpCurrentSpan
```
Returns a no-op object when called outside a span context. `id` and `trace_id` return `""`. The other methods do nothing. Never raises.
### `get_current_trace()`
```python theme={null}
def get_current_trace() -> CurrentTrace | _NoOpCurrentTrace
```
Returns a no-op object when called outside a span context.
### `get_current_replay_branch()`
```python theme={null}
def get_current_replay_branch() -> ReplayBranch | None
class ReplayBranch:
database_url: str # connection string for this item's branch
neon_branch_id: str # the provider's own id for this branch
env_key: str # env var name your app reads, e.g. "DATABASE_URL"
expires_at: str # ISO-8601
snapshot_timestamp: str | None # ISO-8601, the instant the branch is pinned to
provider_console_url: str | None
read_only: bool | None
region: str | None # e.g. "aws-us-east-1"
trace_id: str # the historical trace this item replays
```
Call it inside the replayed function to get the branch resolved for the item currently running. It returns `None` outside a replay item. It also returns `None` for an item whose source trace carried no DB snapshot reference. That is the fallback path: `url = branch.database_url if branch else os.environ["DATABASE_URL"]`.
The value object is immutable, and scoped to one item. It is built from the replay `ContextVar`, so concurrent items each see their own branch. Reading `database_url` marks the trace as having used the branch, reported as `accessed`. The other attributes inspect the branch without exposing the connection string, and deliberately do not mark it as accessed. `repr()` redacts the URL.
Every field the service puts on the lease is copied onto the branch, under its snake\_case name. A field added server-side is therefore readable before you upgrade the SDK. `database_url` is the sole exception. It is the credential, and the only member that may mark the branch as accessed.
### `flush_traces(timeout: float = 30.0)`
```python theme={null}
def flush_traces(timeout: float = 30.0) -> bool
```
Forces the private OpenTelemetry batch processor to export pending spans. It
also waits for any remaining legacy mutation requests. Together, these wait up
to `timeout` seconds. Returns `True` when the queued exports completed
successfully within that deadline. Returns `False` when delivery failed, or the
flush timed out. Use it before process exit in short-lived scripts.
### `Bitfab.close(timeout: float = 30.0)`
Flushes pending requests and permanently shuts down this client's private OTel
transports, within one total deadline. The method is idempotent. It returns
`False` if delivery or shutdown misses the deadline. `Bitfab` also supports
`with Bitfab(...) as client:`, which calls `close()` on context exit. Use
either form when a long-running process creates transient clients. Shared
clients may otherwise remain open until the process-wide exit hook runs.
## Classes (Context Handles)
### `CurrentSpan`
```python theme={null}
class CurrentSpan:
@property
def id(self) -> str
@property
def trace_id(self) -> str
def add_context(self, context: dict[str, Any]) -> None
def set_prompt(self, prompt: str) -> None
```
| Method | Semantics |
| ------------- | ------------------------------------------------------------------------------------------- |
| `id` | Canonical Bitfab span ID |
| `trace_id` | UUID string |
| `add_context` | Appends the dict as one entry to `span_data.contexts`. Non-dict input ignored. Never raises |
| `set_prompt` | Overwrites `span_data.prompt`. Non-string input ignored. Never raises |
### `CurrentTrace`
```python theme={null}
class CurrentTrace:
def set_session_id(self, session_id: str) -> None
def set_name(self, name: str) -> None
def set_metadata(self, metadata: dict[str, Any]) -> None
def add_context(self, context: dict[str, Any]) -> None
def drop(self) -> None
```
`set_session_id` groups traces from the same user session. Unlike the other setters, it does not validate that `session_id` is a non-empty string.
`set_name` sets the trace's title in Bitfab. This is a searchable and filterable field, stored on the trace's `name` column. A trace with no name set is titled by its trace function key instead. Empty strings are ignored.
`set_metadata` shallow-merges with existing metadata, with later keys winning.
`add_context` accumulates entries.
`drop` flags the trace to be dropped. Once flagged, spans that complete afterward are not uploaded at all. The flag rides out on the completion payload. At completion, the server scrubs any payloads that already raced out, meaning the trace itself, its external trace, and its sibling spans. It deletes the archived S3 objects. It marks the trace `dropped` instead of `completed`, keeping only a skeleton audit row. `drop` is a no-op outside a span, and never raises.
### `DetachedTrace`
```python theme={null}
class DetachedTrace:
trace_id: str # read-only property
def add_context(self, context: dict[str, Any]) -> None
def set_metadata(self, metadata: dict[str, Any]) -> None
def set_session_id(self, session_id: str) -> None
def set_name(self, name: str) -> None
```
Returned by `client.get_trace(trace_id)`, where `trace_id` is the canonical Bitfab trace ID. Its methods send to the server immediately, and block until it responds. A later read therefore always observes the write. They raise if the server rejects the update. They are silent no-ops if the client is disabled, or if input validation fails. Validation otherwise mirrors `CurrentTrace`, with one exception. `DetachedTrace.set_session_id` does validate a non-empty string, unlike `CurrentTrace.set_session_id`.
## TypedDicts & Dataclasses
### `AllowedEnvVars`
```python theme={null}
class AllowedEnvVars(TypedDict, total=False):
OPENAI_API_KEY: str
```
### `CapturedSpan`
```python theme={null}
class CapturedSpan(TypedDict):
id: str
traceId: str
parentSpanId: str | None
name: str | None
type: str
input: Any
output: Any
contexts: list[dict[str, Any]]
prompt: str | None
metadata: dict[str, Any]
metrics: dict[str, Any] | None
errors: Any
startedAt: str | None
endedAt: str | None
```
Returned by `get_trace_span`. `SpanOccurrence` is `Literal["first", "last"] | int`.
### `ReplayConcurrency`
```python theme={null}
@dataclass(frozen=True)
class ReplayConcurrency:
attempts: int = 1
primitive: ConcurrencyPrimitive = "async"
max_concurrency: int | None = 10 # effective default; 4 for process mode
on_item_finish_in_child_process: Callable[[ReplayItemFinishEvent], None] | None = None
memory_throttle: bool = True # effective default; False for async mode
```
`attempts` must be between 1 and 100. Async mode accepts `max_concurrency=None` for unlimited work. Process mode requires a positive bound instead. Passing `concurrency=` together with the legacy `attempts=` or `max_concurrency=` parameters raises, rather than choosing one over the other.
`memory_throttle` holds a child back until the machine has memory for it, so a run on a loaded laptop finishes with fewer children in parallel instead of being killed by the operating system. Free memory must cover every running child's remaining growth, plus the new child, plus a reserve, and swap must not be near full. Each running child is charged its full expected peak rather than its current resident size, because a child that started a moment ago has not grown yet. That per-child estimate starts from a default and then calibrates in both directions off the resident size actually measured for children that have finished.
It only ever holds launches back. It never raises concurrency above `max_concurrency`, and it never holds back the only child in flight, so a machine under lasting memory pressure still finishes the run one item at a time. Where memory cannot be read it admits normally.
It defaults to `True` under `primitive="process"` and `False` under `primitive="async"`. Passing `memory_throttle=True` alongside `primitive="async"` raises, because every async item shares one process and there is nothing to hold back. Lower `max_concurrency` there instead.
`on_item_finish_in_child_process` is the child's own item-finish hook. It runs in the child interpreter that replayed the item, which is the only place the replayed run's in-memory state still exists, after that item's spans are confirmed delivered so `item["trace_id"]` is in hand and after its result file is written, and before that child exits.
```python theme={null}
class ReplayItemFinishEvent(TypedDict):
test_run_id: str
item: ReplayItem
```
It carries no running totals, because a child ran one item and cannot count the rest of the run. Both of its keys also appear on `ReplayItemFinishProgress`, so one callback can serve the registry hook and this one.
It does not replace the registry entry's `on_item_finish`, and setting both is the expected shape.
| | `on_item_finish` | `on_item_finish_in_child_process` |
| ---------------------------------- | --------------------------------- | --------------------------------- |
| Set on | the registry entry, or `replay()` | `ReplayConcurrency` |
| Runs in | the process that owns the run | the child that replayed the item |
| Payload | `ReplayItemFinishProgress` | `ReplayItemFinishEvent` |
| Running totals | yes | no |
| Sees the replayed run's memory | no, under `primitive="process"` | yes |
| Fires for an item whose child died | yes | no |
| Available under | every primitive | `primitive="process"` only |
Each fires exactly once per item, and the child's runs first. The parent only learns an item exists once its child has exited, and the child runs its hook before exiting, so a slow judge in the child delays that item's progress event in the parent and never the reverse. Cross-item state belongs in the parent hook or in the returned `ReplayResult`, since each child sees only its own item.
It is refused under any primitive other than `process`, since no child process exists to run it in. It is skipped under `--dry-run`. A callback that raises is reported as `on_item_finish_in_child_process failed`, naming the trace and attempt, instead of failing the item. The message is written to the child's stderr and forwarded to the command's own stderr, so a judge that throws is visible even though a successful item's child log is never read, and the child writes its result file before running the hook. A child that dies before reaching the hook is still reported to the registry's `on_item_finish` in the parent, which is where a crashed attempt is recorded as skipped.
### `TokenUsage`
```python theme={null}
class TokenUsage(TypedDict):
input: int | None
output: int | None
cached: int | None
total: int | None
```
The shape of `ReplayItem.tokens` and `ReplayItem.original_tokens`, and of their `ReplayProgressItem` equivalents. Not exported from `bitfab`. Referenced here only for the field's type.
### `ReplayItem`
```python theme={null}
class ReplayItem(TypedDict):
input: list[Any]
result: Any
original_output: Any
error: str | None # compatible message for either error kind
trace_error: BaseException | None # actual exception from the replayed trace
replay_error: BaseException | None # actual exception from replay setup
duration_ms: int | None # how long THIS replay took
original_duration_ms: int | None # the original trace's duration
original_tokens: TokenUsage | None # the original trace's token usage
original_model: str | None # the original trace's model
ingestion_type: str | None # how the source trace came to exist ("captured" or "seeded"); None on older servers, which only served captured traces
tokens: TokenUsage | None # the replayed run's usage (compare vs original_tokens)
model: str | None # deprecated alias for original_model
trace_id: str | None
original_trace_id: str | None
original_span_id: str | None
source_trace_id: str | None # deprecated alias for original_trace_id
source_span_id: str | None # deprecated alias for original_span_id
db_snapshot_ref: dict | None # the source trace's snapshot pin, if any
attempt: int # 0-based attempt index within this experiment
db_branch_timings: dict | None # per-phase branch provisioning timings
trace_outline: dict | None # the replayed trace's span tree, no payloads
original_trace_outline: dict | None # the original trace's span tree, no payloads
```
`trace_outline` and `original_trace_outline` are the replayed and the original trace's span trees, with no inputs or outputs. They are passed through from the server as-is, in camelCase keys, the same shape as the `traceOutlines` entries in the [HTTP reference](/reference/http). Each trace outline carries `traceId`, `name`, `status`, `traceFunctionKey`, `durationMs`, `spanCount`, and `spans`. Each span in turn carries `spanId`, `name`, `type`, `traceFunctionKey`, `durationMs`, `tokens`, `model`, `errors`, `mocked`, and `children`. Both fields are `None` on progress items, on items whose replay produced no trace (`trace_outline` only), and against older servers. They exist for grading. Compare the two trees to tell whether a replay reached its output by the same path.
### `ReplayResult`
```python theme={null}
class ReplayResult(TypedDict):
items: list[ReplayItem]
test_run_id: str
test_run_url: str
attempts: int
```
### `CodeChangeFile`
```python theme={null}
class CodeChangeFile(TypedDict):
path: str
before: str # "" for a newly created file
after: str # "" for a deleted file
```
One file edited as part of a code change, passed in `replay(code_change_files=[...])` or read back from a `BITFAB_CODE_CHANGE_PATH` file. `path` is relative to the repo root, or any consistently used root.
### `AdaptContext`
```python theme={null}
class AdaptContext(TypedDict):
original_trace_id: str | None
original_span_id: str
metadata: dict[str, Any]
source_trace_id: str | None # deprecated alias for original_trace_id
source_span_id: str # deprecated alias for original_span_id
```
Passed as the third argument to `adapt_inputs`. `original_trace_id` is the Bitfab trace ID of the trace being replayed. It lets a table-driven adapter look up per-trace adapted inputs. `original_span_id` is the external span ID the recorded inputs were read from. `metadata` is the original trace's stored metadata, meaning whatever `seed_trace` or `get_current_trace().set_metadata` recorded on it. It is empty when the trace carries none. When the traced function also emits its own trace through an integration that exports trace metadata, the caller's metadata is merged with that export and wins on any shared key, so an integration no longer replaces the provenance the caller stored. Keys the integration set are kept alongside it, so this can carry keys the caller never wrote. Requires v0.52.3 or later. This lets an adapter read a seeded case's provenance without it being smuggled through the recorded inputs.
### Replay lifecycle progress
```python theme={null}
class ReplayItemStartProgress(TypedDict):
type: Literal["started"]
test_run_id: str
started: int
completed: int
total: int
succeeded: int
errored: int
item: dict[str, str | int | None] # lineage plus 0-based attempt
class ReplayItemFinishProgress(TypedDict):
test_run_id: str
completed: int
total: int
succeeded: int
errored: int
item: ReplayProgressItem
```
`ReplayProgressItem` mirrors `ReplayItem`. It makes the fields that are unavailable during execution optional. `ReplayProgress` is the deprecated compatibility shape. It additionally supports a terminal `type="complete"` event, with an optional `result`.
### `DbSnapshotRef`
```python theme={null}
class DbSnapshotRef(TypedDict, total=False):
sdkWallClockBeforeFn: str
```
The snapshot pin attached to every root trace. This is the value `ReplayItem.db_snapshot_ref` carries. `sdkWallClockBeforeFn` is the ISO wall-clock timestamp the SDK observed immediately before invoking the wrapped function. The server-side resolver uses that timestamp as the snapshot instant. No provider is captured here. The provider is resolved later, at replay time. It uses camelCase keys, since the ref goes to the server as-is.
### `CommitRef`
```python theme={null}
class CommitRef(TypedDict):
sha: str
branch: str | None
dirty: bool | None
remote: str | None
root_sha: str | None
```
The commit the traced code was running at, sent as `commit_ref` on every root trace completion. `sha` is the commit. `branch` is the checked-out branch, or `None` when detached or unknown. `dirty` is `True` when the working tree had uncommitted or untracked changes, `False` when it was clean, and `None` when the SDK could not tell, which is always the case when the ref came from environment variables rather than `git`. `remote` is the `origin` URL reduced to `host/owner/repo` with any credentials removed, so a CI checkout token never reaches the trace. `root_sha` is the repository's first commit, so two checkouts of the same repository match even without a remote.
Resolution order is `BITFAB_COMMIT_SHA`, then the deploy platform's build variables, then `git` in the process's working directory (see [Commit ref environment variables](#commit-ref-environment-variables)). Environment resolution is synchronous and free. The `git` path runs once per process on a background thread with a two second timeout per command, so it never sits on the thread that ran the traced function. A trace that completes before it lands ships without a `commit_ref`, and a process with neither variables nor a repository never sends one. The result, including a negative one, is memoized for the life of the process. Set `BITFAB_DISABLE_COMMIT_REF` to opt the process out entirely.
### `DbBranchLease`
```python theme={null}
class DbBranchLease(TypedDict, total=False):
neonBranchId: str
envKey: str
databaseUrl: str
expiresAt: str
snapshotTimestamp: str
providerConsoleUrl: str
readOnly: bool
region: str
```
The per-item database branch the Bitfab service resolved from the source trace's `db_snapshot_ref`. It is carried on the replay context with camelCase keys, because it arrives straight off the wire. `neonBranchId` is the literal Neon branch id. `snapshotTimestamp` is the instant the branch was pinned to. It is echoed back in `db_snapshot_usage` on the replayed trace's completion. Inside a replayed function, read the branch through [`get_current_replay_branch()`](#get-current-replay-branch) instead of this raw lease. That accessor returns a `ReplayBranch`.
### `DbBranchOptions`
```python theme={null}
class DbBranchOptions(TypedDict, total=False):
min_cu: float
max_cu: float
warmup_sql: str
```
Passed as `client.replay(db_branch={...})`, to tune how each item's branch is sized and warmed. Every key is optional. `db_branch=True` branches with the mirror project's own sizing instead. `min_cu` is the autoscaling floor, in Neon Compute Units (0.25 to 56). `max_cu` is the ceiling. Equal to `min_cu`, it pins a fixed size. Otherwise, a later item can run against an already-scaled-up endpoint and post a better number for the same code. `warmup_sql` is appended to the branch's readiness check. It runs before `fn` sees the branch, so its time is not charged to the replayed call. Invalid SQL fails the branch, rather than silently leaving it cold.
### `MockOverride`
```python theme={null}
@dataclass(frozen=True)
class MockOverride:
match: NodeMatcher
value: MockValue
NodeMatcher = Callable[[SpanNodeMeta], bool]
MockOverrideResolver = Callable[[MockOverrideCtx], Any]
MockOverrideInput = Union[MockOverride, MockOverrideResolver]
```
One `(match, value)` pair. The first override whose `match` returns `True` wins for a given span. A `MockOverrideResolver` is the keyless form. It runs for every child span. It returns `NO_MOCK_OVERRIDE` to decline a given span. `mock_override` accepts either form, or a list of them.
### `MockOverrideCtx`
```python theme={null}
@dataclass
class MockOverrideCtx:
node: SpanNodeMeta
inputs: list[Any]
kwargs: dict[str, Any]
get_original_output: Callable[[], Any]
```
Passed to a `MockValue` callable, and to a resolver. `inputs` and `kwargs` are the live positional and keyword arguments passed to the wrapped function on this run. This lets an override compute from what the changed code actually asked for. `get_original_output` is synchronous. It returns this span's original recorded output, deserialized. It is memoized for the replay item. It raises if the span has no recorded counterpart in the replayed trace.
### `SpanNodeMeta`
```python theme={null}
@dataclass(frozen=True)
class SpanNodeMeta:
trace_function_key: str
span_name: str
type: str
original_span_id: str | None = None
```
The structural identity of a span during replay, passed to a [`MockOverride`](#mockoverride) match predicate. It carries no output payload. Matching runs on structural metadata only. `span_name` is the resolved name: the `name` option, else the function's qualified name (`Order.process` for a method), falling back to `trace_function_key`. `original_span_id` is this span's id in the original trace. It is `None` when the live span has no recorded counterpart, such as a span the changed code newly introduced.
### `MockValue`
```python theme={null}
MockValue = Union[Callable[[MockOverrideCtx], Any], Any]
```
What an override supplies for a matched span: either a flat value used as-is, or a callable that receives a [`MockOverrideCtx`](#mockoverridectx) and returns the value. `None` is a legitimate mocked output. Returning it substitutes `None`, rather than declining. Return `NO_MOCK_OVERRIDE` to decline instead.
A trace error means the replayed function started and raised. A replay error means Bitfab could not invoke it at all, for example because database warmup, input loading, or mock preparation failed. If delivery or finalization later fails, `replay()` raises `ReplayError`. Its `items`, `test_run_id`, `test_run_url`, and `cause` preserve the partial result and the original whole-run exception.
If database branch resolution fails, `replay_error` is a `DbBranchReplayError`. It carries `code`, the server message, `original_trace_id`, and an optional `cause`. Resolver codes such as `branch_create_failed`, `snapshot_from_replaced_origin`, `invalid_snapshot_ref`, `seeded_trace_has_no_snapshot`, and `internal_error` remain available in memory, in progress events, in result files, and in `ReplayError.items`. (`seeded_trace_has_no_snapshot` means the source was seeded, so it pinned no database instant.) HTTP, timeout, and network failures while requesting a lease use `lease_request_failed` instead, with the original client exception as `cause`.
```python theme={null}
try:
client.replay(process_document)
except ReplayError as error:
print(error.cause)
for item in error.items:
print(item["original_trace_id"], item["trace_error"] or item["replay_error"])
```
### `serialize_replay_result`
```python theme={null}
def serialize_replay_result(result: ReplayResult) -> str
```
Returns indented JSON, while preserving structured fields from `trace_error` and `replay_error`. That includes `DbBranchReplayError.code`, `original_trace_id`, and nested `cause`. Use it for direct-run stdout, instead of `json.dumps(..., default=str)`. That alternative reduces exceptions to plain strings.
## Error Behavior Summary
| Situation | Behavior |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Empty `api_key` | Resolved lazily, including a `BITFAB_API_KEY` fallback. Without `strict`, this warns and disables capture. With `strict=True`, the first traced call raises `RuntimeError` instead |
| Opt-in and opt-out tracing in one call stack | `MixedTracingError`. The inner function does not run |
| `@span` transport failure | Swallowed. User's return value / exception passes through |
| Decorated function raises | Span records `error` and `error_source: "code"`, exception re-raised |
| `add_context` / `set_prompt` with invalid input | Silently ignored |
| `call()` cannot find the function | `ValueError` with URL to `/functions` |
| `call()` finds the function but there's no prompt | `ValueError` with URL to `/functions/{id}` |
| `call()` raises during execution | Whatever exception BAML raised, re-raised as-is (not necessarily `ValueError`) |
| `wrap_baml` missing client | `ValueError` |
| `replay("key", fn)` with a plain callable (no `@span`) | Auto-wrapped under the key, so spans link to the test run (not an error) |
| `replay(fn)` with `fn` undecorated and no explicit key | `ValueError` (decorate with `@span`, or pass an explicit key) |
| `replay("key", fn)` with `fn` decorated under a different key | `ValueError` (key mismatch) |
| `replay()` fails to deliver or finalize after items settle | `ReplayError` with collected `items`, test-run identifiers, and original `cause` |
| `replay()`'s requested database branch cannot be resolved | Item `replay_error` is `DbBranchReplayError` with the resolver code, message, and original trace ID |
| `get_trace_span` with both or neither of `id`/`name`, an invalid ID, an empty `name`, or an invalid `occurrence` | `ValueError` |
| `register_mock_override` with malformed keyed, paired, or bare-value arguments | `ValueError` |
| `ReplayRegistry.register()` with an empty or duplicate name, a non-callable `options_factory`, no resolvable trace function key, or an unknown option name | `ValueError` |
| `ReplayConcurrency` with `attempts` outside 1-100, an unknown `primitive`, `primitive="process"` with `max_concurrency=None`, or a non-callable `on_item_finish_in_child_process` | `ValueError` |
| `ReplayConcurrency(on_item_finish_in_child_process=...)` under any primitive but `"process"` | `ValueError` (no child process exists to run it in) |
| `client.labels.save` / `.skip` / `.archive` with neither or both of `trace_id`/`original_trace_id` | `ValueError` |
| Use a framework adapter without its optional dependency | The adapter fails at the framework boundary with an installation-oriented error. Importing `bitfab` itself remains safe |
# Ruby SDK Reference
Source: https://docs.bitfab.ai/reference/ruby
Pure API reference for the bitfab RubyGem.
The gem is called `bitfab`. It requires Ruby 3.4 or newer. It uses a global singleton client pattern (`Bitfab.configure` / `Bitfab.client`).
## Framework Integrations
No framework-native adapters are shipped for Ruby yet. Instrument Ruby code manually via `Bitfab::Traceable`, `Bitfab::Traceable.wrap`, or a bound `Bitfab::BitfabFunction`. See the [Ruby SDK guide](/ruby-sdk). Framework coverage status lives in the [Frameworks overview](/frameworks/overview).
## Module Layout
```ruby theme={null}
Bitfab # top-level module & singleton facade
Bitfab::Client # the underlying client class
Bitfab::BitfabFunction # fluent wrapper bound to one trace_function_key
Bitfab::Traceable # mixin for declarative span tracing
Bitfab::Datasets # dataset namespace reached as client.datasets
Bitfab::Subtree # TracePoint subtree capture (internal)
Bitfab::CurrentSpan # span handle
Bitfab::CurrentTrace # trace handle
Bitfab::NoOpCurrentSpan # returned outside a span context
Bitfab::NoOpCurrentTrace # returned outside a span context
Bitfab::Replay # replay orchestration (internal)
Bitfab::ReplayContext # thread-local replay context shared by same-thread fibers (internal)
Bitfab::ReplayBranch # per-trace DB branch exposed during replay
Bitfab::DbBranchReplayError # a requested database branch could not be resolved
Bitfab::ReplayError # whole-run replay failure carrying partial items and test-run identifiers
Bitfab::ReplayRegistry # project-owned replay root registry
Bitfab::ReplayCommand # installed bitfab-replay command implementation
Bitfab::ReplayCli # argument parsing and execution for bitfab-replay (internal)
Bitfab::NO_MOCK_OVERRIDE # resolver sentinel that continues mock resolution
Bitfab::SpanContext # thread-local span stack (internal)
Bitfab::TraceState # trace state storage (internal)
Bitfab::Transport # span/trace delivery seam (internal)
Bitfab::Otel # OpenTelemetry batch transport (internal)
Bitfab::MOCK_STRATEGIES # %w[none all marked]
Bitfab::BITFAB_PROGRESS_PREFIX # wire prefix used by report_replay_progress
```
## Module-Level API
### `Bitfab.configure(api_key: nil, service_url: nil, enabled: true, strict: false)`
Creates the global client. Subsequent calls replace it.
| Param | Type | Default |
| -------------- | ------------------- | --------------------- |
| `api_key:` | `String, Proc, nil` | `nil` |
| `service_url:` | `String, nil` | `"https://bitfab.ai"` |
| `enabled:` | `Boolean` | `true` |
| `strict:` | `Boolean` | `false` |
The key is resolved lazily, at the first traced call. If you passed a `Proc`, it is called at that point. A missing or empty configured value falls back to `ENV["BITFAB_API_KEY"]`. In normal mode, a missing key warns. It also disables tracing. With `strict: true`, a missing key raises instead. `enabled: false` disables tracing outright. It does not resolve or warn about the key at all.
### `Bitfab.client`
Returns the configured `Bitfab::Client`. Raises `RuntimeError("Bitfab not configured...")` if `configure` has not been called.
### `Bitfab.client_or_nil`
Returns the configured `Bitfab::Client`, or `nil` if `configure` has not been called. The traced-call path for `bitfab_span` resolves the client this way, not through `Bitfab.client`. As a result, a method invoked before `Bitfab.configure` runs untraced instead of raising.
### `Bitfab.reset!`
Resets the global client to `nil`. Intended for tests.
### `Bitfab.current_span`
Returns the active `CurrentSpan`, or `Bitfab::NO_OP_SPAN` outside a span context.
### `Bitfab.current_trace`
Returns the active `CurrentTrace`, or `Bitfab::NO_OP_TRACE` outside a span context.
### `Bitfab.current_replay_branch`
Returns the `Bitfab::ReplayBranch` resolved for the replay item currently running. Returns `nil` outside a replay item. It also returns `nil` when the item's source trace carried no DB snapshot reference. Call it inside the replayed method, then fall back to your normal connection string:
```ruby theme={null}
branch = Bitfab.current_replay_branch
url = branch ? branch.database_url : ENV["DATABASE_URL"]
```
The branch is a frozen, per-item value object. It exposes `database_url`, `neon_branch_id`, `env_key`, `expires_at`, `snapshot_timestamp`, `provider_console_url`, `read_only`, `region`, and `trace_id`. It is built from the thread-local replay context, so parallel items each see their own branch. Reading `database_url` marks the trace as having used the branch, reported as `accessed`. The other readers inspect the branch without exposing the connection string, and deliberately leave that mark untouched. `inspect` redacts the URL.
Every field the service puts on the lease gets a reader under its snake\_case name. A field added server-side is readable before you upgrade the SDK. `database_url` is the one exception. It carries the credential. It is also the only member whose read marks the branch as accessed.
`as_json` and `to_json` serialize only those exposed fields. Without this, ActiveSupport would serialize the object's instance variables by default. That would put the connection string into any Rails log line or API response that touched a branch. Because of this override, a serialized branch is safe to hand around. `database_url` still stays available to code that explicitly asks for it.
### `Bitfab.report_replay_progress(progress)`
A ready-made replay lifecycle callback. Pass it into `replay` as both `on_item_start:` and `on_item_finish:`. It writes lifecycle events to stderr, prefixed with `Bitfab::BITFAB_PROGRESS_PREFIX`. The Bitfab plugin reads those events to identify in-flight traces and finished results. stdout stays free for direct-run `ReplayResult` JSON. It never raises.
### `Bitfab.serialize_replay_result(result)`
Returns indented JSON. It preserves structured fields from `:trace_error` and `:replay_error`, including `Bitfab::DbBranchReplayError#code`, `original_trace_id`, and nested `cause`. Use it for direct-run stdout instead of `JSON.pretty_generate(result)`. `JSON.pretty_generate(result)` raises when a result contains an exception object.
### `Bitfab.flush_traces(timeout: 30)`
Waits for queued spans and trace completions from every live client to reach the server within one total deadline. Returns `true` when everything landed and `false` on delivery failure or timeout. It does not close the clients.
## `class Bitfab::Client`
### `SPAN_TYPES`
```ruby theme={null}
Bitfab::Client::SPAN_TYPES # => %w[llm agent function guardrail handoff custom]
```
### Readable client state
```ruby theme={null}
client.api_key # configured String/nil, or the result of the configured Proc
client.service_url
client.enabled # effective state; lazily resolves the key
client.datasets
```
`api_key` does not apply the `ENV["BITFAB_API_KEY"]` fallback. It also never warns. `enabled` reports the effective state. Reading it resolves the key, the same way the first traced call would. `service_url` and `datasets` are attribute readers.
### `get_trace_span`
```ruby theme={null}
get_trace_span(trace_id, id: nil, name: nil, occurrence: "last")
```
Fetches one persisted span without loading its trace. `trace_id` is the canonical Bitfab trace ID. Exactly one of the span's Bitfab `id` or `name` is required. `occurrence` accepts `"first"`, `"last"`, or a zero-based integer. It defaults to `"last"`. Returns `nil` when no trace or span matches.
### `datasets`
```ruby theme={null}
client.datasets.save(trace_function_key:, name:, description: nil)
client.datasets.list(trace_function_key: nil)
client.datasets.get(dataset_id)
client.datasets.list_traces(dataset_id)
client.datasets.add_traces(dataset_id, trace_ids)
client.datasets.remove_traces(dataset_id, trace_ids)
client.datasets.add_graders(dataset_id, grader_ids)
client.datasets.remove_graders(dataset_id, grader_ids)
client.datasets.rerun_graders(dataset_id, grader_ids: nil, wait: true, timeout: 90, poll_interval: 1)
client.datasets.get_grader_rerun(dataset_id, run_id: nil)
```
Dataset operations for the authenticated organization. These are the same operations the Bitfab MCP tools expose to a coding agent. Every method returns the parsed JSON response as a `Hash` with string keys matching the HTTP API (`"traceCount"`, `"addedTraceIds"`, and so on).
* `save` is an upsert keyed on `(trace_function_key, name)`. `"created"` is `true` for a new dataset. It is `false` when an existing one was updated instead. A `nil` description leaves an existing description untouched.
* `list` takes an optional trace function key. Without it, every dataset in the organization is returned.
* A dataset carries `"id"`, `"traceFunctionKey"`, `"name"`, `"description"`, `"traceCount"`, `"graders"` (`[{"id", "name"}]`), `"createdAt"`, and `"updatedAt"`.
* `list_traces` returns `{"datasetId", "traceIds"}`. This is the same membership a replay with `dataset_id:` selects.
* `add_traces` and `add_graders` accept 1 to 100 ids. They report partial acceptance instead of rejecting the whole call. Ids outside the organization, or under another trace function, come back in `"skippedTraceIds"` / `"skippedGraderIds"`. Ids already present come back in `"alreadyPresentTraceIds"` / `"alreadyAssignedGraderIds"`.
* `remove_traces` never deletes a trace, only its membership in the dataset. Ids that were not members come back in `"notPresentTraceIds"`. `remove_graders` reports `"notAssignedGraderIds"` the same way.
* `rerun_graders` re-scores every trace in the dataset. `grader_ids:` defaults to every assigned grader. An unassigned id rejects the call. It waits up to `timeout:` seconds, polling every `poll_interval:` seconds. It returns the last `"run"` seen. Pass `wait: false` to return as soon as the run is queued. A request matching an in-flight run joins it instead, reporting `"joinedExisting"` as `true`.
* A run has `"status"` (`pending` | `running` | `completed` | `errored`), `"graderIds"`, `"progress"` (`{completedTraces, totalTraces, graderCount}` while running), `"result"` (`{tracesGraded, gradersRun}` when completed), and `"error"`.
* A dataset id from another organization raises `Net::HTTPError` with a 404.
### `initialize(api_key: nil, service_url: nil, enabled: true, strict: false)`
Same semantics as `Bitfab.configure`.
### `flush(timeout: 30)`
Waits for the spans and traces this client queued to be delivered, without closing it. Returns `true` when everything landed inside the deadline.
### `close(timeout: 30)`
Flushes, then permanently shuts down this client's OpenTelemetry batch worker. Returns `true` when everything queued was delivered inside the deadline. Later submissions on the closed client are dropped with a one-time warning. A shared client does not need this. `at_exit` already shuts remaining transports down.
### Replay registry and command
```ruby theme={null}
REGISTRY = Bitfab::ReplayRegistry.new
REGISTRY.register("checkout", CheckoutService.new, :run)
REGISTRY.register(
"agent",
AgentService.new,
:run,
trace_function_key: "support-agent",
mock: "marked",
adapt_inputs:,
options_factory: ->(ctx) {
{mock_override: SupportAgentMocks.build(ctx.params["scenario"])}
}
)
```
`Bitfab::ReplayRegistry#register` stores the exact production receiver method. It also stores static per-function replay behavior and an optional `options_factory:`. A `bitfab_span` method supplies its trace function key automatically. A plain handler method must pass `trace_function_key:` explicitly.
Pass `client:` to route this entry's replay through a specific client. It defaults to `Bitfab.client`. Unlike `Traceable.wrap`'s `client:`, which resolves on every call, this one resolves once, when `register` is called.
The factory receives JSON values from `--params` and `--param`. It can use them to construct executable options such as `mock_override:`. A direct `--param` value overrides one loaded from a `--params` file.
Registered and factory-produced options are limited to these names:
* `limit`, `trace_ids`, `name`, `max_concurrency`
* `code_change_description`, `code_change_files`
* `experiment_group_id`, `dataset_id`, `dataset_ids`, `grader_ids`
* `mock`, `mock_override`, `adapt_inputs`, `db_branch`
An unknown option name is rejected, either when the registry loads or when the factory runs. Lifecycle callbacks are excluded from this list because the installed command owns progress reporting.
The gem installs the `bitfab-replay` executable. Run it as `bundle exec bitfab-replay --registry [options]`. The registry module must define `REGISTRY`.
| Flag | Value | Effect |
| ------------------------------------ | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--limit` | `N` | Most traces to select (default 10). It bounds whatever selection is in effect, whether that selector came from the command line or the registry, truncating an ID list and pinning the first N members of the selected datasets ordered by ID |
| `--trace-ids` | `id1,id2` | Replay exactly these traces. Mutually exclusive with `--dataset-ids` |
| `--dataset-ids` | `uuid1,uuid2` | Replay the membership of one or more datasets, as their deduped union. Mutually exclusive with `--trace-ids`. `--dataset-id` is the same flag |
| `--name` | `NAME` | Title for the resulting experiment |
| `--concurrency`, `--max-concurrency` | `N` | Items in flight at once |
| `--experiment-group-id` | `UUID` | Add this run to an existing experiment group |
| `--grader-ids` | `id1,id2` | Attach graders to this run only |
| `--code-change` | `PATH` | Load a code-change description from a file |
| `--no-code-change` | | Record no code change, overriding a registry default |
| `--mock` | `none\|all\|marked` | Which recorded child spans return their historical output |
| `--db-branch`, `--no-db-branch` | | Turn per-item database branching on or off |
| `--params` | `PATH` | JSON file of values passed to `options_factory:` |
| `--param` | `name=value` | One value passed to `options_factory:`. Repeatable, and overrides `--params` |
| `-h`, `--help` | | Print usage |
### `register_mock_override`
```ruby theme={null}
register_mock_override(override)
register_mock_override(match, value)
register_mock_override(trace_function_key, resolver_or_override)
register_mock_override(match: matcher, value: value)
```
Registers an instance-scoped override for every subsequent replay. A keyed resolver runs only for child spans with that trace function key. A keyed `{ match:, value: }` hash also applies its own matcher. A global resolver can route on `ctx[:node][:trace_function_key]`. Return `Bitfab::NO_MOCK_OVERRIDE` to continue to lower-priority overrides and the base mock strategy. `nil` remains a valid mocked output. Per-call overrides take precedence over registered overrides.
### `clear_mock_overrides`
Removes every override registered on this client.
### `replay(receiver, method_name, trace_function_key:, limit: nil, trace_ids: nil, name: nil, max_concurrency: 10, code_change_description: Replay::CODE_CHANGE_UNSET, code_change_files: Replay::CODE_CHANGE_UNSET, mock: "marked", mock_override: nil, adapt_inputs: nil, db_branch: nil, experiment_group_id: nil, dataset_id: nil, dataset_ids: nil, grader_ids: nil, on_item_start: nil, on_item_finish: nil, on_progress: nil)`
| Param | Type | Description |
| -------------------------- | ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `receiver` | `Object` or `Class` | Instance for instance methods. Class for class methods |
| `method_name` | `Symbol` | Method to replay |
| `trace_function_key:` | `String` | Trace function key for this method. It must match the key the method is traced under, via `bitfab_span` or `Bitfab::Traceable.wrap`. A contradicting key raises `ArgumentError` |
| `limit:` | `Integer, nil` | Max recent traces to replay (default `5`, maximum 5,000). Ignored when `trace_ids:` or a dataset selector is passed |
| `trace_ids:` | `Array, nil` | Optional explicit trace IDs. Max 100. Passed alongside `dataset_id:` or `dataset_ids:` it pins which members of that selection replay, and the server rejects any ID none of those datasets contains. The ID count determines how many traces replay |
| `name:` | `String, nil` | Optional display name for the resulting experiment/test run |
| `max_concurrency:` | `Integer, nil` | Max parallel threads (default `10`, `nil` means unlimited) |
| `code_change_description:` | `String, nil` | Optional rationale for the code change being tested. It is stored on the experiment. When supplied by itself, it is preserved. Files are still captured automatically in that case |
| `code_change_files:` | `Array, nil` | Optional list of edited files, each `{ path:, before:, after: }`. Omit it for a rename-aware working-tree diff against trunk. Pass `nil` instead to suppress capture entirely |
| `mock:` | `String` | Mock strategy for child spans. One of `"marked"` (default), `"none"`, or `"all"`. Raises `ArgumentError` on any other value |
| `mock_override:` | `Hash, Proc, Array, nil` | Per-call `{ match:, value: }` overrides or global resolvers. The first matching result other than `Bitfab::NO_MOCK_OVERRIDE` wins. These run before registered overrides and the base `mock:` strategy |
| `adapt_inputs:` | `#call, nil` | Reshapes recorded positional and keyword arguments for the method's current signature |
| `experiment_group_id:` | `String, nil` | Optional UUID that groups multiple replay runs into a single experiment batch |
| `dataset_id:` | `String, nil` | Optional UUID of the dataset this replay runs against. Mutually exclusive with `dataset_ids:`. Alone it replays the dataset's full membership, and with `trace_ids:` only those members. Durably attributes the experiment to that dataset |
| `dataset_ids:` | `Array, nil` | Optional UUIDs of the datasets this replay runs against. Mutually exclusive with `dataset_id:`. Replays the union of their traces, or only the selected members when combined with `trace_ids:`. Uses the union of their graders and attributes the experiment to every selected dataset |
| `grader_ids:` | `Array, nil` | Optional UUIDs of graders attached directly to this run, up to 100. They are unioned with the dataset's runnable graders when the experiment is graded. Each grader must be active, in the same organization and trace function. Otherwise the replay is rejected with a 400 |
| `db_branch:` | `Boolean, Hash, nil` | Optional. `db_branch: true` requests a DB branch for each replay item, using the mirror's own sizing. `false` or `nil` leaves branching off. Each bounded worker resolves its own branch. As a result, `max_concurrency` also bounds how many branches are live at once. `Bitfab.current_replay_branch` exposes the branch inside the replayed method. Its keys tune the branch, for example `db_branch: {min_cu: 2, max_cu: 2, warmup_sql: "SELECT 1;"}`. All of these keys are optional. Symbol or string keys both work. This matters because a hash built from JSON or YAML arrives string-keyed. `min_cu` and `max_cu` set the compute's autoscaling floor and ceiling, from 0.25 to 56. Equal `min_cu`/`max_cu` values pin a fixed size, up to 56. This keeps items comparable to each other. An autoscaling range may not span more than 8 CU. It may also not exceed 16 CU overall. `warmup_sql` is appended to the branch's readiness check, warming the cache before the replayed method sees the branch. That way, warm-up time is not charged to the call itself. Invalid warm-up SQL fails the branch, rather than quietly handing back a cold one. Omitted keys leave the mirror's own defaults in place |
| `on_item_start:` | `#call, nil` | Optional callback fired when a worker begins each item, before replay setup or customer code runs. It carries `type: "started"`, running lifecycle totals, and the historical trace/span identity. Pair it with `on_item_finish:` to distinguish queued work from in-flight work. A raising callback never crashes the run |
| `on_item_finish:` | `#call, nil` | Optional callback fired exactly once per item, as it finishes. It receives a running-totals hash: `{ test_run_id:, completed:, total:, succeeded:, errored:, item: }`. Every invocation carries `:item`. The callback itself never represents whole-run completion. `item[:trace_id]` is the server replay `traces.id`. It is read back off the ingest response. It is surfaced once the item finishes. Its trace is flushed on finish. `item[:trace_id]` is `nil` only if that flush could not confirm delivery in time. `item[:original_trace_id]` is the original historical trace being replayed. `:source_trace_id` is a deprecated alias for it. The item also carries `input`, `result`, `original_output`, `error`, `trace_error`, `replay_error`, `duration_ms`, `tokens`, `model`, and `db_snapshot_ref`. Use this callback to render live progress while the replay runs. Its running totals only split ran-ok from errored. Pass or fail isn't known yet at this point. A raising callback never crashes the run. Pass `Bitfab.method(:report_replay_progress)` as both lifecycle callbacks. The Bitfab plugin uses its stderr events to identify in-flight traces. It also uses them to write finished per-item result files |
| `on_progress:` | `#call, nil` | Deprecated compatibility callback. It receives the same per-item events, plus its own legacy item-less terminal `"complete"` event. It is ignored when both `on_progress:` and `on_item_finish:` are provided |
**Returns:** a `Hash` with the keys `:items`, `:test_run_id`, and `:test_run_url`.
Each item in `:items` carries these fields:
* `:input`, `:result`, `:original_output`, `:error`, `:trace_error`, `:replay_error`
* `:duration_ms` (`Integer, nil`), this replay's own duration
* `:original_duration_ms`, `:original_tokens`, `:original_model`
* `:db_branch_timings` (`Hash, nil`)
* `:tokens` (`{ input:, output:, cached:, total: }` or `nil`)
* `:model` (`String, nil`)
* `:trace_id`, `:original_trace_id`, `:original_span_id` (each `String, nil`)
* the deprecated aliases `:source_trace_id` and `:source_span_id`
* `:db_snapshot_ref` (`Hash, nil`), the source trace's snapshot pin, if any
* `:trace_outline` and `:original_trace_outline` (`Hash, nil`)
`:trace_outline` and `:original_trace_outline` are the replayed and the original trace's span trees, with no inputs or outputs. The server passes them through as-is, with string camelCase keys, in the shape of the `traceOutlines` entries in the [HTTP reference](/reference/http). Both are `nil` on progress items and against older servers. `:trace_outline` is also `nil` when the replay produced no trace.
`:trace_error` retains the actual exception raised while executing the replayed trace. `:replay_error` retains the actual exception raised before the method could even run, such as a database warmup or input-loading failure. `:error` holds the compatible message for either one.
A database branch resolver failure surfaces as a `Bitfab::DbBranchReplayError`, carrying `code`, the server message, and `original_trace_id`. This is true both in progress JSON and in `Bitfab::ReplayError#items`. HTTP, timeout, and network failures use the code `lease_request_failed`, with the original client exception as `cause`. Malformed refs use `invalid_snapshot_ref`. Unexpected resolver failures use `internal_error`.
`:duration_ms` is how long this replay took. `:tokens` is the replayed run's own token usage. The trace being replayed is described separately, by `:original_duration_ms`, `:original_tokens`, and `:original_model`. `:model` is a deprecated alias for `:original_model`. Compare `:tokens` against `:original_tokens` to see the cost delta.
`:trace_id` is the new replay trace's server id. It becomes available to `on_item_finish` once that item's trace has been flushed. It stays `nil` if delivery could not be confirmed, or against an older server. `:original_trace_id` and `:original_span_id` identify the original trace and root span being replayed.
If trace delivery or finalization fails after items settle, `replay` raises `Bitfab::ReplayError`. Its `items`, `test_run_id`, and `test_run_url` retain the partial result. Its `cause` retains the original whole-run exception.
**Mock strategies:**
* `"marked"`: only child spans declared with `mock_on_replay: true` return historical output. If a selected occurrence is unavailable, the item errors without executing the real child. Default.
* `"none"`: every child span runs real code.
* `"all"`: every matched recorded child span returns its historical output. Only the root runs real code. A missing or exhausted occurrence fails the item closed.
Child calls are matched by `trace_function_key` and span name. Repeated calls with the same key and name are distinguished by call order. Calls that share a trace function key but have different span names are tracked independently.
### Replay auto-capture environment variables
| Variable | Effect |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `BITFAB_CODE_CHANGE_PATH` | Path to a JSON file of `{ "description", "files" }`, written by the Bitfab plugin. `replay` reads it instead of running `git diff` when `code_change_files:` is omitted |
| `BITFAB_CODE_CHANGE_BASE` | Git ref to diff against, instead of searching the default trunk candidates (`origin/HEAD`, `origin/main`, `origin/master`, `main`, `master`) |
| `BITFAB_DISABLE_CODE_CHANGE_CAPTURE` | Set to opt every replay in this process out of automatic code-change capture |
| `BITFAB_REPLAY_RESULT_PATH` | When set by the Bitfab plugin, `replay` writes the full final `ReplayResult` JSON to this path automatically, in addition to its return value |
### `get_function(trace_function_key)`
Returns a `Bitfab::BitfabFunction` bound to `trace_function_key`. Mirrors `client.get_function` in the Python SDK and `client.getFunction` in TypeScript.
### `node(klass, method_name, name: nil, type: "custom", capture: true, test_run_id: nil, mock_on_replay: false, finalize: nil)`
Client-bound external-class form of `bitfab_node`. The method is configured only beneath an active `bitfab_trace` owned by this client and otherwise runs unchanged.
### `execute_span(...)` (internal)
Called by `Traceable`. Not intended for direct use.
### Transport environment variables
| Variable | Default | Description |
| -------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------ |
| `BITFAB_OTEL_MAX_REQUEST_BYTES` | `3000000` | Request-size target for OTLP/JSON exports. Accepts positive integers up to `3000000` |
| `BITFAB_OTEL_EXPORT_CONCURRENCY` | `32` | Concurrent direct requests per export window. Accepts `1` through `64` |
| `BITFAB_DISABLE_COMPRESSION` | unset | Set to send every request body uncompressed instead of gzip-encoding bodies of 8,192 bytes or more that compress smaller |
### Commit ref environment variables
| Variable | Effect |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `BITFAB_DISABLE_COMMIT_REF` | Set to any value to send no `commit_ref` at all. Neither the platform variables nor `git` are consulted |
| `BITFAB_COMMIT_SHA` | The commit sent as `commit_ref.sha` on every root trace. Wins over every deploy platform variable and over `git` |
| `VERCEL_GIT_COMMIT_SHA`, `GITHUB_SHA`, `RAILWAY_GIT_COMMIT_SHA`, `RENDER_GIT_COMMIT`, `SOURCE_VERSION`, `CF_PAGES_COMMIT_SHA`, `CI_COMMIT_SHA`, `BUILD_SOURCEVERSION`, `CIRCLE_SHA1` | Read in that order when `BITFAB_COMMIT_SHA` is unset, together with the same platform's branch and repository variables when it has them. The first one set wins, and `git` is never run |
### `CommitRef`
```ruby theme={null}
{
"sha" => String,
"branch" => String | nil,
"dirty" => true | false | nil,
"remote" => String | nil,
"root_sha" => String | nil
}
```
The commit the traced code was running at, sent as `commit_ref` on every root trace completion. `sha` is the commit. `branch` is the checked-out branch, or `nil` when detached or unknown. `dirty` is `true` when the working tree had uncommitted or untracked changes, `false` when it was clean, and `nil` when the SDK could not tell, which is always the case when the ref came from environment variables rather than `git`. `remote` is the `origin` URL reduced to `host/owner/repo` with any credentials removed, so a CI checkout token never reaches the trace. `root_sha` is the repository's first commit, so two checkouts of the same repository match even without a remote.
Resolution order is `BITFAB_COMMIT_SHA`, then the deploy platform's build variables, then `git` in the process's working directory (see [Commit ref environment variables](#commit-ref-environment-variables)). Environment resolution is synchronous and free. The `git` path runs once per process on a background thread, with a two second timeout per command, so it never sits on the code path that ran the traced function. A trace that completes before it lands ships without a `commit_ref`, and a process with neither variables nor a repository never sends one. The result, including a negative one, is memoized for the life of the process. Set `BITFAB_DISABLE_COMMIT_REF` to opt the process out entirely. `Bitfab::CommitRef.current` returns the hash, or `nil` while unresolved.
## `class Bitfab::BitfabFunction`
Fluent wrapper bound to a single `trace_function_key`, returned by `Bitfab::Client#get_function`.
### `attr_reader :trace_function_key`
### `wrap(klass, method_name, name: nil, type: "custom", capture_when: "always", mock_on_replay: false)`
Wraps `method_name` on `klass` with span tracing under the bound `trace_function_key`. Delegates to `Bitfab::Traceable.wrap`.
### `trace(klass, method_name, name: nil, type: "custom", max_depth: 30, max_spans: 500, exclude: [], include_wrappers: false)`
Experimental. Wraps `method_name` as the root and captures its first-party Ruby call subtree via `TracePoint`. Delegates to `Bitfab::Traceable.trace`; the bound key is used for the root and every captured call.
## `module Bitfab::Traceable`
Mixin for declarative instance-method tracing.
### `include Bitfab::Traceable`
Extends the including class with `ClassMethods` below.
### Class Methods (after `include`)
#### `bitfab_function(key)`
Sets the default `trace_function_key` for all `bitfab_span` declarations in this class.
#### `bitfab_span(method_name, trace_function_key: nil, name: nil, type: "custom", capture_when: "always", mock_on_replay: false)`
Wraps `method_name` with span tracing.
Spans exist only where you create them. Nesting is automatic, but only between spans that already exist. One wrapper around the outermost function records a single-node trace. See [Instrumentation](/instrumentation).
| Param | Type | Default |
| --------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `method_name` | `Symbol` | required |
| `trace_function_key:` | `String, nil` | falls back to the class-level `bitfab_function` key |
| `name:` | `String, nil` | defaults to `method_name.to_s` |
| `type:` | `String` | `"custom"` (one of `SPAN_TYPES`) |
| `capture_when:` | `String, Symbol` | `"always"` / `:always` (default). `"nested"` / `:nested` records only when there's an active parent span. Otherwise, it runs the method untraced. Unknown values warn once and default to `"always"` |
| `mock_on_replay:` | `Boolean` | `false`. When `true`, replay returns this span's historical output, under the default `"marked"` mock strategy |
Supports three call styles:
1. **Before `def`**: registered via `method_added` hook, wrapped when method is defined
2. **Inline**: `bitfab_span def foo ... end, type: "function"` (Ruby's `def` returns `:foo`)
3. **After `def`**: wraps immediately if the method already exists
Raises `RuntimeError` if no `trace_function_key` is provided and no class-level `bitfab_function` was set.
#### `bitfab_trace(method_name, trace_function_key: nil, name: nil, type: "custom", max_depth: 30, max_spans: 500, exclude: [], include_wrappers: false)`
Experimental. Records the root method and every first-party Ruby method it calls, at any depth, without declaring those methods separately. The root uses `name` and `type`; descendants use their method names and type `"function"`.
Capture is active only while the root runs, including while a returned `Enumerator` is consumed. It follows recursive, module, inherited, `define_method`, singleton-class, cross-file, and same-thread child Fiber calls. An alias is named and matched by `exclude` using the invoked alias rather than its original definition name. Ruby standard-library, dependency-gem, Bitfab SDK, synthetic dynamic-evaluation, and child-thread methods are excluded. An installed gem containing the traced root is treated as first-party, without admitting neighboring gems or SDK internals. Blocks and lambdas are not method-call events and are not recorded. Methods whose only parameters are `*args`, `**kwargs`, and `&block` are treated as forwarding decorator wrappers and skipped unless `include_wrappers: true`.
Child Fibers created during capture inherit node policies, trace identity, parenting, and depth limits through Ruby Fiber storage, with separate call stacks. Pre-existing Fibers and `Fiber.new(storage: nil)` do not inherit this context. After an inner capture ends, child Fibers retain any still-active outer capture, with their existing parenting and depth limits. Inherited context expires once all enclosing captures end. While a root remains active, hitting `max_spans` stops new capture but allows already-open spans in every Fiber to finish. Independently traced Fibers remain isolated during interleaving, including when one root finishes before another.
When capture ends, already-open automatic calls in suspended Fibers are emitted once with a nil output and `Subtree capture ended before the call returned` as the error, unless a real error was already observed. Their inputs and parenting are retained. The SDK does not resume Fibers, and later resumption cannot add spans to the ended capture. If hot reload moves the root to an unresolved source, the last valid first-party boundary and its exclusions remain in use.
A returned `Enumerator` defers capture completion until iteration finishes or raises. Calls already paused in child Fibers retain their inputs, parenting, node context, depth and span-budget state across that handoff. Between the initial return and consumption, the session is inactive and its deferred frames are detached from the thread dispatcher. Calls still unfinished when iteration ends receive the incomplete-capture error described above. An Enumerator that is never consumed remains pending; the SDK does not force iteration. Automatically captured methods that return Enumerators record a `` output placeholder without consuming them; the traced root records the values yielded during consumption.
A nested `bitfab_trace` root with a different trace function key starts an independent trace while its complete subtree also appears in every outer `bitfab_trace` capture. The traces have distinct trace and span IDs, including during replay, and the same structure each root records alone. Two active roots therefore double span volume in their shared region. A recursive invocation or nested annotation with the same key stays inside the active trace instead of starting another root.
`max_depth` and `max_spans` cap descendant capture; hitting either warns once per trace function key. `exclude` contains unqualified method names as strings or symbols. Automatically captured spans are observations, not replay mock targets. A descendant declared with `bitfab_span` records exactly once through its explicit wrapper, remains replay-mockable, and parents automatically captured calls beneath it.
The SDK prepares the project boundary and exclusions when the method is declared, refreshes the boundary if hot reload moves the root definition, caches canonical source paths, and shares one `TracePoint` dispatcher across active roots on a calling thread. After `max_spans` is exhausted, it stops collecting metadata and values immediately, then stops processing events for that session once its captured calls finish. Each active root still captures and serializes its own copy of every overlapping call. Keep this experimental API for discovery; prefer explicit `bitfab_span` declarations on production hot paths.
Supports the same before-`def`, inline, and after-`def` declaration styles as `bitfab_span`:
```ruby theme={null}
class OrderService
include Bitfab::Traceable
bitfab_function "order-processing"
bitfab_trace :process_order, type: "agent", exclude: [:healthcheck]
def process_order(order_id)
order = load_order(order_id)
summarize(order)
end
end
```
The wrapper preserves the root method's public, protected, or private visibility.
#### `bitfab_node(method_name, name: nil, type: "custom", capture: true, test_run_id: nil, mock_on_replay: false, finalize: nil)`
Experimental. Configures a method only when it is discovered beneath an active `bitfab_trace`. It never creates a standalone span or trace and inherits the active root's trace function key, limits, parent, and lifecycle.
| Param | Type | Default |
| ----------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `method_name` | `Symbol` | required |
| `name:` | `String, nil` | defaults to `method_name.to_s` |
| `type:` | `String` | `"custom"` (one of `SPAN_TYPES`) |
| `capture:` | `Boolean` | `true`; `false` omits the node while captured descendants attach to its nearest visible parent |
| `test_run_id:` | `String, nil` | optional test-run attribution |
| `mock_on_replay:` | `Boolean` | `false`; when `true`, marked replay reuses the node's recorded output |
| `finalize:` | `#call, nil` | transforms the recorded output without changing the method's return value; failures are recorded on the node and do not escape |
Captured nodes count toward the enclosing root's `max_depth` and `max_spans`. A root-level `exclude` wins over the node configuration for that root. In different-key overlapping roots, each root remains independent: the innermost owner uses the configured node span while outer roots keep their automatically captured copy. `capture: false` applies to every active root because it is an unconditional property of the method. `capture: false, mock_on_replay: true` raises `ArgumentError` because an omitted node has no recorded output.
Supports the same before-`def`, inline, and after-`def` declaration styles as `bitfab_span` and preserves public, protected, or private visibility.
### Module-Level Wrapping
#### `Bitfab::Traceable.wrap(klass, method_name, trace_function_key:, name: nil, type: "custom", capture_when: "always", mock_on_replay: false, client: nil)`
Wraps a method on a class you do not own. Uses `Module#prepend`.
| Param | Type |
| --------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `klass` | `Class` or `Module` |
| `method_name` | `Symbol` |
| `trace_function_key:` | `String` |
| `name:` | `String, nil` |
| `type:` | `String` |
| `capture_when:` | `String, Symbol` (`"always"` / `:always` or `"nested"` / `:nested`). Unknown values warn once and default to `"always"` |
| `mock_on_replay:` | `Boolean` |
| `client:` | `Bitfab::Client, nil`. Route through a specific client instead of resolving the global client on every call |
#### `Bitfab::Traceable.trace_function_key_for(receiver, method_name)`
Returns the trace function key carried by a method wrapped with `bitfab_span` / `Traceable.wrap`, or `nil` when the receiver method has no Bitfab wrapper. Replay and replay registries use it to reject key mismatches.
#### `Bitfab::Traceable.trace(klass, method_name, trace_function_key:, name: nil, type: "custom", max_depth: 30, max_spans: 500, exclude: [], include_wrappers: false)`
Experimental external-class form of `bitfab_trace`. Uses `Module#prepend`; its options have the same semantics as the class macro.
#### `Bitfab::Traceable.node(klass, method_name, name: nil, type: "custom", capture: true, test_run_id: nil, mock_on_replay: false, finalize: nil)`
Experimental external-class form of `bitfab_node`. Uses `Module#prepend`; pass `client:` internally or prefer `client.node(...)` when configuration must belong to a specific client.
## `class Bitfab::CurrentSpan`
```ruby theme={null}
def id # => canonical Bitfab span ID
def trace_id # => String
def add_context(context) # Hash → appends one entry; ignores non-Hash; never raises
def set_prompt(prompt) # String → overwrites; ignores non-String; never raises
```
## `class Bitfab::CurrentTrace`
```ruby theme={null}
def set_session_id(session_id) # String
def set_name(name) # String → the trace's title, searchable; empty strings ignored
def set_metadata(metadata) # Hash → shallow-merge, later keys win
def add_context(context) # Hash → appends entry
def drop # flags the trace to be dropped at completion
```
All methods swallow exceptions internally. They never raise.
`drop` flags the trace to be dropped. Once flagged, spans that complete afterward are not uploaded at all. The flag itself still rides out on the completion payload. At completion, the server scrubs any payloads that already raced out, meaning the trace itself, its external trace, and any sibling spans. It deletes the archived S3 objects. It marks the trace `dropped` instead of `completed`, keeping only a skeleton audit row. Calling `drop` outside a span, on `NO_OP_TRACE`, is a no-op.
## `class Bitfab::NoOpCurrentSpan` / `NoOpCurrentTrace`
Singleton instances exposed as `Bitfab::NO_OP_SPAN` and `Bitfab::NO_OP_TRACE`. All methods are no-ops. `id` and `trace_id` return `""`.
## Thread & Concurrency Model
* Span stack is stored in fiber-local `Thread.current[:__bitfab_span_stack]`. It is **not** propagated to child threads. Traced methods that return an `Enumerator` bridge their parent span into the same-thread `Enumerator.new` / `enum_for` source fiber. This keeps nested spans and replay mocks as descendants while the stream is consumed.
* Replay context lives in true thread-variables instead (`Thread.current.thread_variable_get` / `thread_variable_set`). This context includes the mock tree, the DB branch lease, and the test run id. Unlike the fiber-local span stack, it persists across every fiber on the same thread. It still never propagates to child threads or forked processes.
* Trace state is stored in a mutex-protected module-level hash (`Bitfab::TraceState`), keyed by `trace_id`.
* Spans and trace completions are queued on the client's private OpenTelemetry `BatchSpanProcessor`. They are delivered by its batch worker. Every other API call is blocking.
* `Bitfab.flush_traces(timeout:)` waits for every live client's queued spans. It returns `false` when delivery fails or the deadline expires. `client.close(timeout:)` shuts a single client's worker down permanently.
* Subtree capture reuses one `TracePoint` dispatcher per calling thread and keeps a separate method stack per fiber and active different-key root. Nested roots share the dispatcher, but every root still filters and serializes its own copy of overlapping calls, so capture work grows with the number of overlapping roots. Same-key nested annotations and recursion stay in the active session. Calls made on child threads are not captured.
## Error Behavior Summary
| Situation | Behavior |
| ----------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Empty `api_key` | Resolved lazily, including an `ENV["BITFAB_API_KEY"]` fallback. Without `strict`, it warns and disables tracing. With `strict: true`, the first traced call raises `RuntimeError` instead |
| `bitfab_span`-wrapped method called before `Bitfab.configure` | Runs untraced via `Bitfab.client_or_nil`. No error |
| `Traceable.wrap` / `BitfabFunction#wrap`-wrapped method called before `Bitfab.configure`, with no `client:` bound | `RuntimeError("Bitfab not configured...")`. This path resolves `Bitfab.client`, not `client_or_nil` |
| `bitfab_span` method raises | Span records `error` and `error_source: "code"`. The exception is re-raised |
| `bitfab_trace` descendant raises | Descendant records the exception; root behavior and exception propagation are unchanged |
| `bitfab_node` finalizer raises | Node records `finalize failed: ...`; the original method result is still returned |
| `add_context` / `set_prompt` with invalid input | Silently ignored. Never raises |
| Span transport failure | Swallowed, though it warns once. The host app never crashes. `flush_traces` returns `false` |
| Span submitted after `client.close` | Warned once and dropped |
| `Bitfab.client` before `configure` | `RuntimeError("Bitfab not configured...")` |
| `bitfab_span` without `trace_function_key` | `RuntimeError` |
| `replay`: `trace_function_key:` differs from the method's declared key | `ArgumentError` (key mismatch) |
| `replay`: delivery or finalization fails after items settle | `Bitfab::ReplayError` with collected `items`, test-run identifiers, and original `cause` |
| `replay`: requested database branch cannot be resolved | Item `:replay_error` is `Bitfab::DbBranchReplayError` with the resolver code, message, and original trace ID |
# Span Types
Source: https://docs.bitfab.ai/reference/span-types
Canonical SpanType enum and semantics shared across all SDKs.
The `SpanType` enum is identical across every Bitfab SDK. It is a string literal union; passing any other value is an error.
The span type is a **label only**. It is used to organize, filter, and display spans in the dashboard. It does **not** change how a span is traced, replayed, mocked, or evaluated. Two spans with identical code behave identically regardless of their type; picking `"llm"` versus `"function"` versus `"custom"` never alters runtime behavior.
## Values
| Value | Semantic |
| ------------- | -------------------------------------------------------------------------- |
| `"llm"` | A direct LLM API call. Prompt and model metadata are expected |
| `"agent"` | An autonomous orchestrator that makes decisions and dispatches other spans |
| `"function"` | A deterministic tool or function call |
| `"guardrail"` | A safety, validation, or policy check |
| `"handoff"` | An agent-to-agent transfer or human handoff |
| `"custom"` | Default when unspecified. Application-specific tracing |
## Default
When omitted, every SDK defaults to `"custom"`.
## Per-SDK Representation
| SDK | Type |
| ---------- | ---------------------------------------------------------------------------------------------------------------- |
| TypeScript | `export type SpanType = "llm" \| "agent" \| "function" \| "guardrail" \| "handoff" \| "custom"` |
| Python | `SpanType = Literal["llm", "agent", "function", "guardrail", "handoff", "custom"]` |
| Ruby | `Bitfab::Client::SPAN_TYPES = %w[llm agent function guardrail handoff custom]` - passed as a `String` to `type:` |
| Go | `string` passed to `WithType(spanType string) SpanOption`. No compile-time enum |
## Invalid Values
* TypeScript: caught at compile time by the type system
* Python: not runtime-validated; the value is forwarded to the backend as-is
* Ruby: not runtime-validated; the value is forwarded to the backend as-is
* Go: not validated; the string is forwarded to the backend as-is
Unknown values sent to the backend are rejected by the trace ingestion endpoint.
# TypeScript SDK Reference
Source: https://docs.bitfab.ai/reference/typescript
Pure API reference for the @bitfab/sdk npm package.
Package: `@bitfab/sdk`. Dual ESM/CJS. Node.js ≥ 18 and modern browsers.
## Module Exports
```typescript theme={null}
// Values
export { Bitfab, BitfabError, BitfabFunction, DbBranchReplayError, MixedTracingError, ReplayError, NO_MOCK_OVERRIDE, getCurrentReplayBranch, getCurrentSpan, getCurrentTrace } from "@bitfab/sdk"
export { BitfabClaudeAgentHandler, BitfabLangChainCallbackHandler, BitfabLangGraphCallbackHandler, BitfabLangGraphIntegration, BitfabOpenAIAgentHandler, BitfabOpenAITracingProcessor, BitfabVercelAiHandler } from "@bitfab/sdk"
export { DatasetsClient, HttpClient, LabelsClient, TracesClient, defineReplayRegistry, finalizers, flushTraces, reportReplayProgress, seedFromRegistry, serializeReplayResult } from "@bitfab/sdk"
export { __version__, DEFAULT_SERVICE_URL, BITFAB_PROGRESS_PREFIX, SUPPORTED_PROVIDERS } from "@bitfab/sdk"
// Types
export type {
ActiveSpanContext,
AdaptContext,
AdaptInputsFn,
AddDatasetGradersResult,
AddDatasetTracesResult,
AllowedEnvVars,
ArchiveAssertionsParams,
BamlExecutionResult,
BitfabConfig,
BitfabLanguageModelMiddleware,
CaptureSurface,
CaptureWhen,
CapturedSpan,
CodeChangeFile,
CurrentSpan,
CurrentTrace,
Dataset,
DatasetGraderRef,
DatasetTraceIds,
DbBranchOptions,
DbBranchTimings,
DbSnapshotConfig,
DbSnapshotProvider,
DbSnapshotRef,
DetachedTrace,
GraderRerun,
GraderRerunProgress,
GraderRerunResult,
GraderRerunStatus,
LabelAction,
LabelConfidence,
LabelOutcome,
LabelUpdate,
LangGraphIntegrationOptions,
ListDatasetsParams,
MockOverride,
MockOverrideCtx,
MockOverrideInput,
MockOverrideResolver,
MockStrategy,
MockValue,
NodeMatcher,
NodeMethodDecorator,
NodeOptions,
ProviderDefinition,
RemoveDatasetGradersResult,
RemoveDatasetTracesResult,
ReplayBranch,
ReplayItem,
ReplayItemFinishProgress,
ReplayItemStartProgress,
ReplayOptions,
ReplayOptionsFactory,
ReplayProgress,
ReplayProgressItem,
ReplayRegistration,
ReplayRegistry,
ReplayRegistryContext,
ReplayRegistryOptions,
ReplayResult,
RerunGradersOptions,
RerunGradersResult,
SaveAssertion,
SaveAssertionsAllParams,
SaveAssertionsParams,
SaveDatasetParams,
SaveDatasetResult,
SeedCase,
SeedCaseOptions,
SeedResult,
SeedRunOptions,
SpanLookup,
SpanMethodDecorator,
SpanMethodDecoratorContext,
SpanNodeMeta,
SpanOccurrence,
SpanOptions,
SpanType,
TokenUsage,
TraceIngestionType,
TraceAssertion,
TraceAssertionsResult,
TraceAssertionSource,
TraceAssertionsUpdate,
TraceOutline,
TraceOutlineSpan,
TraceOutlineSpanError,
TraceResponse,
TraceTarget,
TraceTargetOccurrence,
TracingProcessor,
VercelCallParams,
VercelGenerateResult,
VercelStreamResult,
WrapBAMLOptions,
WrappedBamlFn,
} from "@bitfab/sdk"
```
## Constants
| Export | Type | Value |
| --------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------- |
| `DEFAULT_SERVICE_URL` | `string` | `"https://bitfab.ai"` |
| `__version__` | `string` | Current package version |
| `NO_MOCK_OVERRIDE` | unique `symbol` | Sentinel returned by a mock override resolver to continue to lower-priority overrides and the base mock strategy |
| `SUPPORTED_PROVIDERS` | `readonly ["neon"]` | Database snapshot providers `DbSnapshotConfig.provider` accepts |
| `HttpClient` | `class` | The transport `DatasetsClient` is constructed over. Exported for advanced callers. The client wires it for you |
## `class Bitfab`
### `new Bitfab(config: BitfabConfig)`
| Param | Type | Default | Description |
| ----------------------- | ------------------------------------------------------------ | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `config.apiKey` | `string \| (() => string \| null \| undefined) \| undefined` | `undefined` | Resolved lazily at first traced use. A missing or empty value falls back to `BITFAB_API_KEY`. If neither resolves, tracing warns and disables itself. Set `strict` to throw instead |
| `config.serviceUrl` | `string` | `"https://bitfab.ai"` | Base URL for Bitfab API |
| `config.timeout` | `number` | `120000` | HTTP request timeout (ms) |
| `config.envVars` | `AllowedEnvVars` | `{}` | LLM provider keys for `call()` (only `OPENAI_API_KEY`) |
| `config.captureEnabled` | `boolean` | `true` | When `false`, wrapped functions run without recording a span. Nothing is sent. Replay still records inside each item. `seedTrace` records the one call it runs. `withSpan`, `withTrace`, `trace`, `node`, and `withNode` always return a wrapper, so `replay` and the replay registry find the trace function key either way |
| `config.enabled` | `boolean` | `true` | Deprecated alias for `config.captureEnabled`. Warns once |
| `config.strict` | `boolean` | `false` | Throw `BitfabError` on the first traced call if no API key resolves, instead of disabling capture |
| `config.bamlClient` | `unknown` | `null` | Generated BAML client (for `wrapBAML()` without explicit client arg) |
| `config.dbSnapshot` | `DbSnapshotConfig` | `undefined` | Per-trace database snapshot configuration. Currently accepts `{ provider: "neon" }`. When omitted, the provider is resolved at replay time |
### `captureEnabled`
```typescript theme={null}
client.captureEnabled: boolean
```
Effective capture state. Reading it resolves the API key lazily, including the `BITFAB_API_KEY` fallback. It is `false` when capture was explicitly disabled or no key resolves. Under `strict: true`, reading it without a key throws `BitfabError` instead.
### `withSpan`
```typescript theme={null}
withSpan(
traceFunctionKey: string,
optionsOrFn: SpanOptions | ((...args: TArgs) => TReturn),
maybeFn?: (...args: TArgs) => TReturn,
): (...args: TArgs) => TReturn
```
Wraps a function so that each invocation produces a span.
**Returns:** a function with the same signature as the input.
**Semantics:**
* Returns a wrapper whether or not capture is on. With `captureEnabled: false`, the wrapper runs the function untraced. It sends nothing, except inside a replay item or the one call `seedTrace` runs
* Span `name` defaults to the function's qualified name, resolved per call: `Order.process` when the function runs as a method of an `Order` instance (or `Order.build` for a static method), `process` for a plain function, then `traceFunctionKey` when the function is anonymous. The raw `fn.name` still travels separately as `function_name`
* Span `type` defaults to `"custom"`
* `testRunId` links the span to a test run. If the span starts a trace, that trace is linked too
* `captureWhen: "nested"` records the span only when another Bitfab span is active. Without a parent, the wrapped function runs normally. It does not create a root trace. The default is `"always"`. An unknown value warns once and falls back to that default
* Input arguments are serialized via superjson. This preserves type information
* The return value is serialized as output, whether it is a plain value or a resolved Promise
* A value that fails to serialize is stubbed instead of dropping the span. A warning notes it may not be replayable. A span whose encoded payload still exceeds the per-request byte ceiling ships anyway, with its largest fields stubbed first and its identifying fields stubbed last. This way an oversized span degrades instead of vanishing
* `SpanOptions.finalize?: (result) => unknown | Promise` records a serializable view of a non-serializable result, such as a live stream. The raw result is returned to the caller unchanged. `await finalize(result)` is recorded as the span output instead. It runs in the background. A throwing `finalize` records an error instead of crashing. It is ignored for async-generator results. Pair it with the exported `finalizers.aiSdk` for the Vercel AI SDK, or `finalizers.readableStream`
* An async-generator span remains open until iteration completes, returns early, or throws. A nested span created inside the generator body inherits that span. To include spans created by the consumer in the same trace, wrap the controller that owns the iteration in an outer root span
* A thrown error is recorded on the span, in the `error` and `error_source: "code"` fields. It is then re-thrown
* Spans nest automatically, via `AsyncLocalStorage` on Node or a module-level stack as a browser fallback
* Spans exist only where you create them. Nesting is automatic, but only between spans that exist. One wrapper around the outermost function records a single-node trace. See [Instrumentation](/instrumentation)
* `withSpan()` is the opt-in tracing surface. `trace()`/`withTrace()` and `node()`/`withNode()` are the opt-out surface. A `withSpan()` entered beneath an active subtree trace throws `MixedTracingError`. Configure a discovered call with `node()`/`withNode()` instead
* The browser fallback does not isolate concurrent async chains. `Promise.all` with independent `withSpan` calls may see the wrong parent
### `trace` / `withTrace`
```typescript theme={null}
trace(traceFunctionKey: string, options?: TraceOptions): TraceMethodDecorator
withTrace(
traceFunctionKey: string,
fn: (...args: TArgs) => TReturn,
): (...args: TArgs) => TReturn
withTrace(
traceFunctionKey: string,
options: TraceOptions,
fn: (...args: TArgs) => TReturn,
): (...args: TArgs) => TReturn
interface TraceOptions {
name?: string
type?: SpanType
mockOnReplayDefault?: boolean
maxDepth?: number
maxSpans?: number
exclude?: readonly string[] | ReadonlySet
includeWrappers?: boolean
}
```
Experimental automatic subtree roots. With a compatible [`@bitfab/transform` adapter](/typescript-sdk#experimental-subtree-tracing), every discovered first-party call beneath the root records its full inputs, output, and thrown error by default. Without a transform, only the normal rich root span is recorded.
`TraceOptions` controls the root `name` and `type`. The root name defaults to the function's qualified name, resolved per call (`Pipeline.run` for a decorated method), then the trace function key when the function is anonymous. It also controls `maxDepth` (default `30`), `maxSpans` (default `500`), qualified or simple `exclude` names, and `includeWrappers` (default `false`). An excluded call is omitted. Its captured descendants attach to the nearest captured parent instead.
A subtree root entered beneath another subtree root starts its own independent trace while every enclosing root keeps recording. The nested root's function and its whole subtree appear in each enclosing trace with separate span IDs and the shape that root would record alone, and each root applies its own `maxDepth`, `maxSpans`, `exclude`, and capture policy to its copy, while `node()` and `withNode()` configuration applies in every copy, including `testRunId` and the finalized output, with `finalize` running once per call. Framework integration spans attach inside the innermost trace only. The enclosing trace's span for the nested root carries `nested_trace_id`, `nested_trace_function_key`, and `nested_root_span_id`. The nested root span carries `enclosing_trace_id`, `enclosing_span_id`, and `enclosing_trace_function_key`. Inside `replay()` or `seedTrace()` a nested root starts no trace of its own and the item's trace records it as an ordinary descendant.
Set `mockOnReplayDefault: true` on the trace options to make replay mocking the default for every automatically captured node. This applies under the default `mock: "marked"` strategy. A configured node with `mockOnReplay: false` overrides that default and runs live. The option defaults to `false`. It does not change `mock: "all"` behavior.
A confirmed Studio capture policy can narrow rich content to selected function IDs, once the policy loads. No policy, or a failed initial policy request, leaves full capture enabled. `node({ capture: false })` and `exclude` remain explicit source-level opt-outs.
`trace()` and `node()` are the opt-out tracing surface. `withSpan()` is the opt-in surface. A subtree root entered beneath an active `withSpan()` throws `MixedTracingError`. So does a `withSpan()` entered beneath a subtree root. Framework integration spans never trip the check. The LangGraph, OpenAI Agents, and Vercel AI SDK wrappers open their spans on the surrounding surface, and the callback-based handlers emit spans directly. The root span that `replay()` and `seedTrace()` wrap around an undecorated callable belongs to neither surface, so a subtree root called from that callable nests beneath it.
### `span`
```typescript theme={null}
span(
traceFunctionKey: string,
options?: SpanOptions,
): SpanMethodDecorator
```
Returns an optional standard ECMAScript method decorator that delegates to `withSpan`.
**Requirements and semantics:**
* Requires TypeScript 5.0 or newer, and a compiler or transpiler that supports the standard ECMAScript decorator transform
* `experimentalDecorators` must be disabled or omitted. The legacy decorator transform is not supported
* `emitDecoratorMetadata` must be disabled or omitted. It is not compatible with standard decorators
* Supports instance, static, and private methods
* Preserves the method receiver, arguments, return type, errors, nesting, and all `SpanOptions`
* `withSpan` remains the recommended default on every TypeScript version. Use it for TypeScript 4.x, standalone functions, class fields, accessors, and legacy-decorator projects
### `node`
```typescript theme={null}
node(options?: NodeOptions): NodeMethodDecorator
```
Configures a transformed class method only when it is discovered beneath an enclosing `trace()` or `withTrace()` call. It never creates a span or trace by itself. It inherits the enclosing trace function key.
**Semantics:**
* Outside an active subtree trace, the method runs normally with no capture or replay behavior. Beneath an active `withSpan()` with no enclosing trace, it throws `MixedTracingError` instead. `node()` belongs to the opt-out surface, so it has no trace to configure there
* `capture: true` (the default) keeps the call captured, with its inputs, output, and errors, even when a confirmed automatic capture policy has not selected it
* `name`, `type`, `testRunId`, `mockOnReplay`, and `finalize` have the same behavior as their `SpanOptions` counterparts
* Under a trace with `mockOnReplayDefault: true`, an omitted node policy inherits the trace default. Setting `mockOnReplay: false` on the node keeps it live instead, under the default `"marked"` replay strategy
* `capture: false` omits the call. Its captured descendants attach to the nearest captured parent instead
* `capture: false` with `mockOnReplay: true` throws `BitfabError`, because an uncaptured call has no recorded output
* Requires a compatible [`@bitfab/transform` integration](/typescript-sdk#experimental-subtree-tracing) to discover the method. Standard and legacy method decorator output are accepted when the transform runs before decorators are lowered
### `withNode`
```typescript theme={null}
withNode(
optionsOrFn: NodeOptions | ((this: This, ...args: TArgs) => TReturn),
maybeFn?: (this: This, ...args: TArgs) => TReturn,
): (this: This, ...args: TArgs) => TReturn
```
Function-oriented equivalent of `node()`. The wrapped function remains eligible for the automatic subtree transform. Without an active `trace()` or `withTrace()` call, it runs normally and never creates a span or trace. Beneath an active `withSpan()`, it throws `MixedTracingError` instead.
The function must have a stable name (a referenced function binding or named function expression). Passing an anonymous inline function throws `BitfabError`, because the transform could not safely bind its configuration to one discovered call.
### `getFunction`
```typescript theme={null}
getFunction(traceFunctionKey: string): BitfabFunction
```
Returns a `BitfabFunction` bound to `traceFunctionKey`.
### `wrapBAML`
Framework integration → see [BAML framework guide](/frameworks/baml) for examples.
```typescript theme={null}
// Form 1: uses bamlClient from constructor
wrapBAML(
method: (...args: TArgs) => Promise,
options?: WrapBAMLOptions,
): WrappedBamlFn
// Form 2: explicit client
wrapBAML(
bamlClient: unknown,
method: (...args: TArgs) => Promise,
options?: WrapBAMLOptions,
): WrappedBamlFn
```
**Returns:** a `WrappedBamlFn`, an async function with a `.collector` property set after each call.
**Throws:**
* `BitfabError` if form 1 is used without `bamlClient` in constructor
* `BitfabError` if the method has no `.name`
**Semantics:**
* If `@boundaryml/baml` is not installed, the method is called directly. `.collector` is `null` in that case
* Otherwise, it creates a BAML `Collector`, calls the method through a tracked client, then:
* Calls `getCurrentSpan().setPrompt(...)` with the rendered messages as JSON
* Calls `getCurrentSpan().addContext({ model, provider, inputTokens, outputTokens, durationMs })`
* The `onCollector` callback fires after each invocation. Errors in the callback are swallowed
### `getTrace`
```typescript theme={null}
getTrace(traceId: string): DetachedTrace
```
Returns a `DetachedTrace` handle for annotating a trace after its root span has closed, from any process or thread.
**Throws:** `BitfabError` if `traceId` is not a canonical Bitfab trace ID.
**Semantics:**
* All methods on the returned handle block (return `Promise`), resolving once the server has applied the change
* When capture is off outside a replay item or a `seedTrace` call, methods return `Promise.resolve()` immediately
* The server returns 404 if no trace exists with that ID. The returned promise **rejects** with a `BitfabError` in that case, rather than logging it
### `getTraceSpan`
```typescript theme={null}
getTraceSpan(traceId: string, lookup: SpanLookup): Promise
```
Fetches one persisted span without loading its trace. `traceId` is the canonical Bitfab trace ID. `lookup` is either `{ id }`, using the span's Bitfab ID, or `{ name, occurrence? }`. `occurrence` is `"first" | "last" | number`. It defaults to `"last"`. Numeric occurrences are zero-based in start-time order. Returns `null` when no trace or span matches.
### `datasets`
```typescript theme={null}
client.datasets.save(params: SaveDatasetParams): Promise
client.datasets.list(params?: ListDatasetsParams): Promise
client.datasets.get(datasetId: string): Promise
client.datasets.listTraces(datasetId: string): Promise
client.datasets.addTraces(datasetId: string, traceIds: string[]): Promise
client.datasets.removeTraces(datasetId: string, traceIds: string[]): Promise
client.datasets.addGraders(datasetId: string, graderIds: string[]): Promise
client.datasets.removeGraders(datasetId: string, graderIds: string[]): Promise
client.datasets.rerunGraders(datasetId: string, options?: RerunGradersOptions): Promise
client.datasets.getGraderRerun(datasetId: string, runId?: string): Promise
```
```typescript theme={null}
interface Dataset {
id: string
traceFunctionKey: string
name: string
description: string | null
traceCount: number
graders: { id: string; name: string | null }[]
createdAt: string
updatedAt: string
}
interface SaveDatasetParams {
traceFunctionKey: string
name: string
description?: string
}
interface SaveDatasetResult { dataset: Dataset; created: boolean }
interface ListDatasetsParams { traceFunctionKey?: string }
interface DatasetTraceIds { datasetId: string; traceIds: string[] }
interface AddDatasetTracesResult {
dataset: Dataset
addedTraceIds: string[]
alreadyPresentTraceIds: string[]
skippedTraceIds: string[]
}
interface RemoveDatasetTracesResult {
dataset: Dataset
removedTraceIds: string[]
notPresentTraceIds: string[]
}
interface AddDatasetGradersResult {
dataset: Dataset
addedGraderIds: string[]
alreadyAssignedGraderIds: string[]
skippedGraderIds: string[]
}
interface RemoveDatasetGradersResult {
dataset: Dataset
removedGraderIds: string[]
notAssignedGraderIds: string[]
}
type GraderRerunStatus = "pending" | "running" | "completed" | "errored"
interface GraderRerun {
id: string
status: GraderRerunStatus
graderIds: string[]
progress: { completedTraces: number; totalTraces: number; graderCount: number } | null
result: { tracesGraded: number; gradersRun: number } | null
error: string | null
createdAt: string
updatedAt: string
}
interface RerunGradersOptions {
graderIds?: string[]
wait?: boolean
timeoutMs?: number
pollIntervalMs?: number
}
interface RerunGradersResult {
run: GraderRerun
joinedExisting: boolean
}
```
Dataset operations for the authenticated organization. These are the same operations the Bitfab MCP tools expose to a coding agent.
* `save` is an upsert keyed on `(traceFunctionKey, name)`. `created` is `true` for a new dataset and `false` when an existing one was updated. An omitted `description` leaves the existing one untouched.
* `list` takes an optional `traceFunctionKey`. Without it, every dataset in the organization is returned.
* `Dataset` carries `id`, `traceFunctionKey`, `name`, `description`, `traceCount`, `graders` (`{ id, name }[]`), `createdAt`, and `updatedAt`.
* `listTraces` returns `{ datasetId, traceIds }`, the same membership a replay with `datasetId` selects.
* `addTraces` and `addGraders` accept 1 to 100 ids. They report partial acceptance instead of rejecting the whole call. An id outside the organization, or under another trace function, comes back in `skippedTraceIds` or `skippedGraderIds`. An id already present comes back in `alreadyPresentTraceIds` or `alreadyAssignedGraderIds`.
* `removeTraces` never deletes a trace, only its membership in the dataset. Ids that were not members come back in `notPresentTraceIds`. `removeGraders` reports `notAssignedGraderIds` the same way.
* `rerunGraders` re-scores every trace in the dataset. `graderIds` defaults to every assigned grader. An unassigned id rejects the call. It waits up to `timeoutMs` (default 90,000), polling every `pollIntervalMs` (default 1,000). It returns the last `run` seen. Pass `wait: false` to return as soon as the run is queued instead. A request matching an in-flight run joins it, reported as `joinedExisting: true`.
* `GraderRerun` has `status` (`pending` | `running` | `completed` | `errored`), `graderIds`, `progress` (`{ completedTraces, totalTraces, graderCount }` while running), `result` (`{ tracesGraded, gradersRun }` when completed), and `error`.
* A dataset id from another organization rejects with a 404 `BitfabError`.
### `traces`
```typescript theme={null}
client.traces.getAssertions(traceId: string): Promise
client.traces.saveAssertions(params: SaveAssertionsParams): Promise
client.traces.saveAssertionsAll(params: SaveAssertionsAllParams): Promise
client.traces.archiveAssertions(params: ArchiveAssertionsParams): Promise
```
An assertion says what should happen when a trace is replayed.
* `getAssertions` returns `{ assertions, inheritedFrom }`. Attach assertions to the ORIGINAL trace. Reading a replay that has none of its own returns the nearest ancestor's assertions instead, resolved through the replay lineage. That ancestor trace is named in `inheritedFrom`. `inheritedFrom` is `null` when the assertions are the trace's own. Writing assertions through a replay trace id is refused, and the error names the original to retry with.
* The common case is to omit `targetOnEvaluatedTrace`. Omitting it checks the whole trace. Set it to narrow the check instead, to either `{ kind: "output" }` or `{ kind: "span", name, occurrence? }`.
* Targets are span names, never span ids, because an id captured on the original resolves to nothing on the replay. `occurrence` accepts `"first"`, `"last"` (the default), or a 0-based index for a span the trace calls more than once.
* `assertion`, `passCriteria`, and `failCriteria` are the same three fields `save_grader` takes, so an assertion that proves out across many traces is promoted into a grader by copying them.
* `humanNote` is people-only context attached to the assertion. SDK reads return it for display, while MCP and Studio can write it. It must never be used as evidence when judging a replay.
* Passing an entry's `id` edits that assertion. Omitting it adds a new one instead. This way two callers adding different assertions to one trace never overwrite each other. `archiveAssertions` hides rows from every read. It keeps them for audit.
* `saveAssertions`' `source` defaults to `"agent"` when omitted, recording that a coding agent, not a person, authored the assertion.
* `saveAssertions` writes one trace. `saveAssertionsAll` takes one update per trace and writes them all in a single request, so a publisher covering hundreds of traces makes one call. The singular delegates to it, so both go through the same route.
* The server writes the whole batch in one transaction, so a rejected batch writes nothing and there is no half-saved state to reconcile. Results come back as one flat array covering every trace, each row carrying its own `traceId`. A batch takes up to 500 traces, up to 50 assertions per trace, and at most 1000 assertions in total. Passing an empty `updates` array writes nothing and sends no request.
```typescript theme={null}
await client.traces.saveAssertionsAll({
updates: traceIds.map((traceId) => ({
traceId,
assertions: [{ assertion: "Departs before 9am" }],
})),
})
```
```typescript theme={null}
type TraceTargetOccurrence = "first" | "last" | number
type TraceTarget =
| { kind: "output" }
| { kind: "span"; name: string; occurrence?: TraceTargetOccurrence }
type TraceAssertionSource = "human" | "agent"
type SaveAssertion = {
category_assertion_id?: string | null
id?: string
assertion: string
passCriteria?: string | null
failCriteria?: string | null
targetOnEvaluatedTrace?: TraceTarget | null
}
type SaveAssertionsParams = {
traceId: string
assertions: SaveAssertion[]
source?: TraceAssertionSource
}
type TraceAssertionsUpdate = {
traceId: string
assertions: SaveAssertion[]
}
type SaveAssertionsAllParams = {
updates: TraceAssertionsUpdate[]
source?: TraceAssertionSource
}
type ArchiveAssertionsParams = {
traceId: string
assertionIds: string[]
}
type TraceAssertion = {
category_assertion_id: string | null
category: AssertionCategorySummary | null
id: string
traceId: string
assertion: string
humanNote: string | null
passCriteria: string | null
failCriteria: string | null
targetOnEvaluatedTrace: TraceTarget | null
source: TraceAssertionSource
createdAt: string
updatedAt: string
}
type TraceAssertionsResult = {
assertions: TraceAssertion[]
inheritedFrom: string | null
}
```
### `assertionCategories`
```typescript theme={null}
client.assertionCategories.save(params: SaveAssertionCategoryParams): Promise
client.assertionCategories.get(id: string): Promise
client.assertionCategories.list(): Promise
client.assertionCategories.delete(id: string): Promise
type SaveAssertionCategoryParams = {
id?: string
title: string
description?: string
}
type AssertionCategorySummary = {
id: string
title: string
description: string
}
type AssertionCategory = AssertionCategorySummary & {
organizationId: string
createdAt: string
updatedAt: string
}
```
Categories group assertions within the API key's organization. `save` creates without an `id` and updates with one. The title is required. Omit `description` to preserve it on an update. Pass an empty string to clear it. `list` returns categories ordered by title. `delete` returns the deleted category and clears its assignments. Assertions and verdicts remain intact.
Pass `category_assertion_id` on entries in `saveAssertions` or `saveAssertionsAll` to assign a category. Omit the field to preserve an existing assignment. Pass `null` to clear it. Assertion reads and saves return `category_assertion_id` and the nullable `category` summary. Inherited assertions include the same category metadata.
```typescript theme={null}
const category = await client.assertionCategories.save({
title: "Mandate compliance",
description: "Checks explicit user instructions",
})
const [assertion] = await client.traces.saveAssertions({
traceId,
assertions: [{ assertion: "Escalates conflicting calendar requests", category_assertion_id: category.id }],
})
await client.traces.saveAssertions({
traceId,
assertions: [{ id: assertion.id, assertion: assertion.assertion, category_assertion_id: null }],
})
await client.assertionCategories.delete(category.id)
```
### `labels`
```typescript theme={null}
client.labels.save(update: LabelUpdate, testRunId?: string): Promise
client.labels.saveAll(updates: LabelUpdate[], testRunId?: string): Promise
client.labels.saveHuman(update: HumanLabelUpdate): Promise
client.labels.saveHumanAll(updates: HumanLabelUpdate[]): Promise
client.labels.get(traceId: string): Promise
client.labels.getAll(traceIds: string[]): Promise
```
Writes the same pass/fail verdicts the `save_agent_labels` MCP tool writes, from inside a replay process rather than a coding-agent session.
* Key a direct verdict by `traceId`. Key a replay verdict by `originalTraceId` plus the top-level `testRunId` it ran under, adding `attempt` when the experiment ran each trace more than once.
* Omit `assertionId` and the verdict scores the whole trace. Pass one and the verdict scores that single assertion. A per-assertion verdict and a whole-trace verdict can both sit on the same trace.
* `{ skip: true }` withholds a verdict. `{ archive: true }` clears a previous one. Skip is the right answer for an attempt that crashed, was punctured, or fell back to a schema default. It is also right for an assertion whose target could not be resolved. A FAIL in those cases would read as a behavior regression, rather than a check that never ran.
* `LabelOutcome.key` echoes the id you addressed the verdict by, so a caller can verify persistence without ever holding a server-generated replay trace id.
```typescript theme={null}
type LabelConfidence = "VeryLow" | "Low" | "Medium" | "High" | "VeryHigh"
type LabelAction = "set" | "archived" | "no-active-label" | "skipped"
type LabelUpdate =
& ({ traceId: string } | { originalTraceId: string; attempt?: number })
& { assertionId?: string }
& (
| { label: boolean; annotation: string; confidence?: LabelConfidence }
| { skip: true }
| { archive: true }
)
type LabelOutcome = {
key: string
traceId: string
action: LabelAction
}
```
`saveHuman` and `saveHumanAll` write the verdicts `save_human_labels` writes over MCP. They are validated on write with no approval step, so they satisfy `search_traces` `validated: true` immediately. Use them only when a human decided the verdict, such as capturing a known production bug as a regression case. An agent's own first-pass guesses go through `save` so they keep the approve-or-edit loop. Approving an existing agent verdict is not on this surface at all: that happens in Studio, by a person.
Like `saveAll`, the batch is all-or-nothing. A trace outside the organization, a repeated target, or an `assertionId` that is not active on its trace rejects the whole call and writes nothing, so a thrown error never leaves part of the batch committed.
```typescript theme={null}
type HumanLabelUpdate = {
traceId: string
assertionId?: string
label: boolean
annotation: string
confidence?: LabelConfidence
}
type HumanLabelOutcome = {
traceId: string
assertionId: string | null
label: boolean
action: "set"
}
```
`get` and `getAll` read verdicts back. Each trace carries its effective verdict plus one row per scored assertion, keyed by the same `assertionId` the write used, so a per-assertion verdict is verifiable per assertion rather than as a passed/failed tally. `get` returns `null` when the trace is not in this organization. One `getAll` call accepts up to 100 ids.
```typescript theme={null}
type LabelStatus = "labeled" | "skipped" | "unlabeled"
type LabelSource = "human" | "agent"
type AssertionVerdict = {
assertionId: string
assertion: string | null
labelStatus: LabelStatus
label: boolean | null
annotation: string | null
confidence: LabelConfidence | null
labelSource: LabelSource
approved: boolean
}
type TraceLabels = {
traceId: string
labelStatus: LabelStatus
label: boolean | null
annotation: string | null
approved: boolean
passed: number
failed: number
assertions: AssertionVerdict[]
}
```
### `graders`
```typescript theme={null}
client.graders.getLabels(params: GetGraderLabelsParams): Promise
```
Reads the individual verdicts each automated grader recorded, one row per grader per trace, the same breakdown the `get_grader_labels` MCP tool returns. Pass `traceIds` to see every grader's verdict on those traces, `graderId` to see one grader's most recent verdicts across traces, or both to narrow. Passing neither raises. This is the per-grader detail that `labels.get` does not carry, since that returns one grader-agnostic verdict per trace.
```typescript theme={null}
type GetGraderLabelsParams = {
traceIds?: string[]
graderId?: string
limit?: number
}
type GraderLabelSource = "human" | "live_grader"
type GraderLabel = {
traceId: string
graderId: string
graderName: string | null
graderStatus: string
label: boolean | null
labelReason: string | null
failureDiagnostic: string | null
labelConfidence: LabelConfidence | null
source: GraderLabelSource
evaluatedAt: string | null
}
```
### Replay registry and command
```typescript theme={null}
export default defineReplayRegistry({
checkout: { client: bitfab, fn: runCheckout },
agent: {
client: bitfab,
fn: runAgent,
traceFunctionKey: "support-agent",
options: { mock: "marked", adaptInputs },
optionsFactory: ({ params }) => ({
mockOverride: createSupportAgentMock(params.scenario),
}),
},
})
```
`defineReplayRegistry` preserves the registry's inferred TypeScript type. Each entry contains a `client`, the exact production `fn`, an optional `traceFunctionKey` for a plain handler root, static `options`, and an optional `optionsFactory({ params })`. A `withSpan`-wrapped function supplies its key automatically. The factory receives JSON values from `--params` and `--param`. It can construct executable options such as `mockOverride`. Direct parameters override file values. Registry options exclude lifecycle callbacks, because the installed command owns progress reporting.
The package installs `bitfab-replay`. Run `bitfab-replay --registry [options]`. The command loads `.ts` registries directly.
| Flag | Value | Effect |
| ------------------------------------ | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--limit` | `N` | Most traces to select (default 10). It bounds whatever selection is in effect, whether that selector came from the command line or the registry, truncating an ID list and pinning the first N members of the selected datasets ordered by ID |
| `--trace-ids` | `id1,id2` | Replay exactly these traces. Mutually exclusive with `--dataset-ids` |
| `--dataset-ids` | `uuid1,uuid2` | Replay the membership of one or more datasets, as their deduped union. Mutually exclusive with `--trace-ids`. `--dataset-id` is the same flag |
| `--name` | `NAME` | Title for the resulting experiment |
| `--attempts` | `N` | Replay each selected trace N times in one run (1 to 100, default 1) |
| `--concurrency`, `--max-concurrency` | `N` | Items in flight at once |
| `--dry-run` | | Resolve every item's inputs and stop without calling the function |
| `--experiment-group-id` | `UUID` | Add this run to an existing experiment group |
| `--grader-ids` | `id1,id2` | Attach graders to this run only |
| `--only-with-assertions` | | Replay only the selected traces that carry an assertion. Narrows `--limit`, `--trace-ids` or `--dataset-ids`. Omitting it leaves a registry-set `onlyWithAssertions` in place |
| `--code-change` | `PATH` | Load a code-change description from a file |
| `--no-code-change` | | Record no code change, overriding a registry default |
| `--mock` | `none\|all\|marked` | Which recorded child spans return their historical output |
| `--db-branch`, `--no-db-branch` | | Turn per-item database branching on or off |
| `--seed` | `cases.jsonl` | Seed the file's cases instead of replaying |
| `--run` | | With `--seed`, run each case once and record the execution |
| `--params` | `PATH` | JSON file of values passed to `optionsFactory` |
| `--param` | `name=value` | One value passed to `optionsFactory`. Repeatable, and overrides `--params` |
| `-h`, `--help` | | Print usage |
Command values override overlapping scalar defaults without removing unrelated executable options. The command wires both lifecycle callbacks to `reportReplayProgress`. It writes the human summary to stderr. It writes the full serialized `ReplayResult` to stdout. A run whose selection matched no traces exits non-zero, rather than reporting a clean run of zero items.
The stderr summary counts items by their source's `ingestionType`. A captured item's recorded output is a previous run, so it counts as **Same** or **Changed**. A seeded item's is the value the case expected, so it counts as **Matched expected** or **Missed expected**. One run can replay both. Both pairs print when both are present. An item with an error counts under **Errors** instead.
`--seed ` seeds cases through the same registration instead of replaying. Each line is a JSON object with an `input` array plus optional `expected`, `metadata`, and `sessionId`. A JSON array of those objects works too. The registration already holds the client, the callable, and the trace function key. Because of that, a seeded case is written against the exact function the later replay selects. A case that cannot supply the function's required arguments is rejected at seed time instead.
Add `--run` to run each case once through the registered function and record the execution, rather than writing the case directly. The output is then what the run produced, so a case carrying `expected` is rejected. The registration's `adaptInputs` is a replay hook. It is not applied at seed time, so a seeded trace is never adapted twice.
### `seedTrace`
```typescript theme={null}
seedTrace(
traceFunctionKey: string,
options: {
input: unknown[]
expected?: unknown
fn?: (...args: any[]) => unknown
metadata?: Record
sessionId?: string
name?: string
spanName?: string
spanType?: SpanType
},
): string
seedTrace(
traceFunctionKey: string,
fn: (...args: TArgs) => TReturn,
options?: {
args?: TArgs
metadata?: Record
sessionId?: string
name?: string
},
): Promise
```
Two forms, chosen by the second argument. Pass a **case** to write a trace without running anything. Pass a **function** to run it once and record that execution.
Writes a replayable trace from a case without running anything. Returns the trace ID for use with `replay({ traceIds: [...] })`. The recorded root span carries `input` as its input and `expected` as its output, so replay reports each item against the value you expected rather than against a previous run.
`name` is the trace's title and a searchable field. Put the case's own label there, such as a ticket id or a dataset row name, so the seeded trace can be found by it. `spanName` labels the root span only.
Use it to turn a corpus you already hold, such as a dataset export, a spreadsheet, or hand-written cases, into traces. Passing `fn` checks the case against the function's required argument count, so a case that could never run fails here instead of at replay. Omit it when the seeding script cannot import the callable.
A trace seeded from a case has no child spans and no database pin. Because of that, replay mocking has nothing recorded to substitute. `dbBranch` refuses it for the same reason. Supply `mockOverride` at replay time for calls that must not run.
#### Seeding by running once
```typescript theme={null}
const bitfab = new Bitfab({ captureEnabled: false })
const traceId = await bitfab.seedTrace("agent-turn", runTicket, {
args: ["T-1"],
metadata: { caseUid: "c-1", suite: "smoke" },
name: "T-1",
})
```
Runs `fn` once. Records the execution as an original trace. Returns the trace ID for use with `replay({ traceIds: [...] })`. Capture stays off. This records exactly one call, with the same semantics capture-on would give it: a root span, a first-party subtree bounded by the enclosing `withTrace` root, and no mocking. The recorded input is the real call. The output is what the run produced. There is no `expected` to pass as a result. The trace lands under `traceFunctionKey`, with `ingestion_type: seeded`. `replay` selects it. Each replay links back to it as `originalTraceId`.
`fn` resolves exactly as it does for `replay`. A `withSpan`-wrapped function records under its own key, and that key must match `traceFunctionKey`. A plain callable is wrapped under the key given here instead. An exception is recorded on the root span. The trace still persists. The exception is re-thrown. A call that records nothing, because no API key resolved, rejects rather than returning an ID replay could never find.
`metadata` is stored on the trace. It is handed to a later replay's `adaptInputs` hook as `ctx.metadata`. This way a case's provenance rides with the trace, instead of through the recorded inputs. If the traced function also emits its own trace through an integration, the caller's metadata is merged onto that export and wins on any shared key, so the provenance recorded here is what replay reads back.
Unlike a trace seeded from a case, this one has a full recorded subtree, so replay mocking works as it does for a captured trace. It still carries no database pin, so `dbBranch` refuses it.
### `reseedTrace`
```typescript theme={null}
reseedTrace(
traceFunctionKey: string,
fn: (...args: any[]) => unknown,
options: { traceId: string },
): Promise<{ traceId: string; previousRunTraceId: string }>
```
Re-seeds one trace. Reads the trace's recorded root inputs, name, session, and metadata from Bitfab, runs `fn` once exactly as `seedTrace` would, then asks Bitfab to adopt that run under `traceId`. The trace keeps its id, labels, assertions, dataset membership, name, and metadata. The previous run is kept as its own trace, `previousRunTraceId`, with `reseedOfTraceId` pointing back at `traceId`. Nothing is mocked and no test run is created.
`fn` resolves as it does for `replay` and `seedTrace`, and `traceFunctionKey` must match the trace's own key. A run that throws is recorded but not adopted, and the error is re-thrown. Bitfab rejects a run from another function, one that already belongs to a dataset, or a trace that is itself a previous run. Graders on the datasets holding the trace are re-queued, and default replay selection skips previous runs.
### `reseedFromRegistry`
```typescript theme={null}
reseedFromRegistry(
registry: ReplayRegistry,
pipeline: string,
traceIds: readonly string[],
): Promise<{
pipeline: string
traceFunctionKey: string
reseeded: { traceId: string; previousRunTraceId: string }[]
}>
```
Re-seeds each trace through an already-registered pipeline, using the registration's client, callable, and trace function key. The installed `bitfab-seed --registry --from-trace [,...]` command calls it; `bitfab-seed --cases [--run]` is the home of what `bitfab-replay --seed` does, and that spelling still works.
### `seedFromRegistry`
```typescript theme={null}
async function seedFromRegistry(
registry: ReplayRegistry,
pipeline: string,
cases: readonly SeedCase[],
options?: { run?: boolean },
): Promise
```
Seeds cases through an already-registered pipeline, using its client, callable, and trace function key. Returns `{ pipeline, traceFunctionKey, traceIds }`. With `{ run: true }`, each case is run once through the registered function. The execution is recorded instead of the case, so a case carrying `expected` is rejected.
### `registerMockOverride`
```typescript theme={null}
registerMockOverride(override: MockOverride): void
registerMockOverride(resolver: MockOverrideResolver): void
registerMockOverride(match: NodeMatcher, value: MockValue): void
registerMockOverride(
traceFunctionKey: string,
override: MockOverride | MockOverrideResolver,
): void
```
Registers an instance-scoped override for every subsequent replay. The keyed form invokes a resolver only for child spans with that trace function key. When its second argument is `{ match, value }`, both the key and matcher must match. A global resolver can route on `ctx.node.traceFunctionKey`. Return `NO_MOCK_OVERRIDE` to continue to the next override and then the replay's base `mock` strategy. A synchronous span requires a synchronous resolver result, including a synchronous sentinel decline. A mixed-tree global resolver can return the sentinel synchronously for sync keys, and a Promise for async keys. `undefined` and `null` remain valid mocked outputs. Per-call overrides take precedence over registered overrides.
### `clearMockOverrides`
```typescript theme={null}
clearMockOverrides(): void
```
Removes every override registered on this client.
### `replay`
```typescript theme={null}
async replay(
traceFunctionKey: string,
fn: (...args: any[]) => TReturn | Promise,
options?: ReplayOptions,
): Promise>
```
| Option | Type | Default |
| ----------------------- | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `limit` | `number` | `5` (maximum 5,000). Ignored when `traceIds`, `datasetId` or `datasetIds` is passed |
| `traceIds` | `string[]` | none. Max 100. Passed alongside `datasetId` or `datasetIds` it pins which members of that selection replay, and the server rejects any ID none of those datasets contains. The ID count determines how many traces replay |
| `name` | `string` | none (display name for the resulting experiment/test run) |
| `maxConcurrency` | `number` | `10` |
| `attempts` | `number` | `1` (1 to 100). Replays each trace this many times, attempt-major, with each attempt as its own replay trace |
| `codeChangeDescription` | `string \| null` | none (preserved when files are captured automatically) |
| `codeChangeFiles` | `CodeChangeFile[] \| null` | Omit for a rename-aware working-tree diff against trunk. `null` disables capture |
| `mock` | `"marked" \| "none" \| "all"` | `"marked"` |
| `mockOverride` | `MockOverrideInput \| MockOverrideInput[]` | none. Pass per-call `{ match, value }` pairs or global resolvers. The first result other than `NO_MOCK_OVERRIDE` wins |
| `adaptInputs` | `(inputs: unknown[], ctx: AdaptContext) => unknown[]` | none. Reshapes recorded inputs for the current function signature. `ctx.metadata` carries the original trace's stored metadata, requested from the server only when an adapter is registered |
| `experimentGroupId` | `string` | none |
| `datasetId` | `string` | none. Mutually exclusive with `datasetIds`. Alone it replays the dataset's full membership, and with `traceIds` only those members. Durably attributes the experiment to a dataset |
| `datasetIds` | `string[]` | none. Mutually exclusive with `datasetId`; pass one dataset through `datasetId` and several through this. Replays the union of their traces, or only the selected members when combined with `traceIds`. Uses the union of their graders and attributes the experiment to every selected dataset |
| `graderIds` | `string[]` | none. Graders attached to this run, unioned with the dataset's at grading. Max 100 |
| `onlyWithAssertions` | `boolean` | none. Narrows whatever `limit`, `traceIds` or `datasetIds` selected to the traces carrying at least one assertion. Applied as part of server-side selection, so `{ limit: 10, onlyWithAssertions: true }` is the ten most recent traces that HAVE assertions, not the ten most recent filtered down. An archived assertion does not count |
| `dbBranch` | `boolean \| DbBranchOptions` | none. `true` requests a DB branch per replay item. An object also sets compute size and cache warm-up |
| `dryRun` | `boolean` | none (resolves every item's inputs and calls nothing, so each item reports the exact arguments `fn` would have received) |
| `onItemStart` | `(p: ReplayItemStartProgress) => void` | none (fires when a worker starts each item) |
| `onItemFinish` | `(p: ReplayItemFinishProgress) => void` | none. Fires exactly once per item, with a required `item`. Never a whole-run event |
| `onProgress` | `(p: ReplayProgress) => void` | Deprecated compatibility callback. Also emits its legacy item-less terminal `complete` event. Ignored when both are provided |
**Returns:** `{ items, testRunId, testRunUrl, attempts }`. See `ReplayResult`.
**Notes:**
* A trace replays only when its root span has serializable inputs. It can also replay when it was instrumented through a [framework handler](/typescript-sdk#replaying-handler-instrumented-functions). That handler's recorded root input is serializable. If the original inputs were stubbed as non-serializable at capture time, the trace cannot be replayed
* `fn` may already be a `withSpan`-wrapped function. That function carries its trace function key and is used as-is. `fn` may instead be a plain callable. `replay()` wraps that callable under the key automatically. Either way, new spans link to the test run via async context. Don't wrap an already-wrapped function in a fresh closure. The closure has no trace function key, so `replay()` wraps the closure as the root, and the inner wrapped function records a second span, nesting a duplicate
* Inputs are deserialized from historical spans and passed positionally
* When `codeChangeFiles` is omitted, `replay()` auto-captures the code change. By default it captures a rename-aware working-tree diff against the nearest trunk ref, tried in this order: `origin/HEAD`, `origin/main`, `origin/master`, `main`, `master`, or `HEAD` when none exist. The diff is bounded to 60 files, 500,000 bytes per file, and 2,000,000 bytes total. Set `BITFAB_CODE_CHANGE_PATH` to point at a JSON file with `description` and/or `files` instead, and that file wins over the git diff. `BITFAB_CODE_CHANGE_BASE` forces which trunk ref to diff against. `BITFAB_DISABLE_CODE_CHANGE_CAPTURE` opts the whole process out of auto-capture. Passing `codeChangeFiles: null` opts out one call instead
* When the Bitfab plugin sets `BITFAB_REPLAY_RESULT_PATH`, `replay()` writes the full `ReplayResult`, using `serializeReplayResult`, to that file after the run completes. A caller does not have to hand-parse stdout as a result. A write failure is logged. It never throws
### `call`
```typescript theme={null}
async call(
methodName: string,
inputs?: Record,
): Promise
```
Executes a server-configured BAML function locally using `envVars`.
**Throws:** `BitfabError` on lookup failure or execution error.
### Framework Integrations
Handlers returned by these methods are framework-native adapters. They plug into each framework's own callback, processor, or hook surface, and emit Bitfab spans automatically. They reuse the owning `Bitfab` client's lazy OTel worker, so `client.close()` releases the `withSpan` and framework transport together. Directly constructed handlers own their transport and expose `close(timeoutMs?)` instead. For usage examples and semantics, see the per-framework guides. Signatures here are canonical.
#### `getLangGraphCallbackHandler`
```typescript theme={null}
getLangGraphCallbackHandler(traceFunctionKey: string): BitfabLangGraphCallbackHandler
```
Returns a duck-typed LangChain/LangGraph callback handler. Pass it in `config.callbacks` when invoking a graph or chain. Root framework invocations are registered immediately as pending external traces. They are completed when the root callback ends. The handler-created root is replayable from the framework input, so a separate `withSpan` root is only needed for meaningful surrounding application work. See [LangGraph framework guide](/frameworks/langgraph).
Aliased as `getLangChainCallbackHandler(traceFunctionKey)` for plain LangChain projects. The returned handler and behavior are identical. The handler class is also exported as `BitfabLangChainCallbackHandler`.
#### `getLangGraphIntegration`
```typescript theme={null}
getLangGraphIntegration(
traceFunctionKey: string,
options?: LangGraphIntegrationOptions,
): BitfabLangGraphIntegration
interface LangGraphIntegrationOptions {
mockToolsOnReplay?: boolean | readonly string[] // default true; a string[] marks only those tool names
}
integration.createInvoker(
graph: LangGraphRunnable,
): (input: TInput, config?: TConfig) => TReturn
```
**Experimental (alpha).** Returns the LangGraph integration. `wrapTools(tools)` wraps each tool's public `invoke()` boundary. `createInvoker(graph)` returns `(input, config?) => output`. It adds `callbackHandler` through LangGraph's public `withConfig()` API. It preserves invocation-time config and existing callbacks. It records only `input` as the replayable root input. Use the lower-level `callbackHandler` and `wrapInvoke(fn)` when meaningful application work around the graph invocation belongs inside the trace. Integration-managed tools are marked for replay mocking by default. Recorded `ToolMessage` and `Command` results are reconstructed with the current tool-call ID. An expected tool output that is missing during replay fails closed, instead of running the live tool. Calls are matched by tool name and occurrence order, so repeated concurrent calls to the same tool are not yet recommended. Install `@langchain/core` and `@langchain/langgraph`. Both are optional SDK peers. See the [LangGraph framework guide and current limitations](/frameworks/langgraph#current-alpha-limitations).
```typescript theme={null}
const integration = bitfab.getLangGraphIntegration("support-agent")
const tools = integration.wrapTools([lookupCustomer, searchDocs])
const graph = buildSupportGraph({ tools })
const runSupportAgent = integration.createInvoker(graph)
const result = await runSupportAgent(
{ messages: [{ role: "user", content: "Help with order 123" }] },
{ configurable: { thread_id: "customer-456" } },
)
await bitfab.replay("support-agent", runSupportAgent, { limit: 10 })
```
#### `getOpenAiTracingProcessor`
```typescript theme={null}
getOpenAiTracingProcessor(): BitfabOpenAITracingProcessor
```
Returns a processor to register with `@openai/agents`' `addTraceProcessor`. This keeps the SDK's default OpenAI exporter. `setTraceProcessors` replaces it instead. The processor captures agent internals. Pair it with `getOpenAiAgentHandler` for a replayable root. See [OpenAI Agents framework guide](/frameworks/openai-agents).
#### `getOpenAiAgentHandler`
```typescript theme={null}
getOpenAiAgentHandler(traceFunctionKey: string): BitfabOpenAIAgentHandler
```
Returns a handler whose `wrapRun(agent, input, options?)` is a drop-in for `@openai/agents`' `run()`. It records a keyed, replayable root span carrying the run input. The tracing processor's spans nest underneath it. Called from inside an already-active span, it runs `run()` directly instead of opening a second root. Pass `{ stream: true }` to `options` for a streamed run instead. The result is handed back immediately. The final output is recorded on the span once the stream drains. First-byte latency is untouched as a result. See [OpenAI Agents framework guide](/frameworks/openai-agents).
#### `getClaudeAgentHandler`
```typescript theme={null}
getClaudeAgentHandler(traceFunctionKey: string): BitfabClaudeAgentHandler
```
Returns a handler exposing `instrumentOptions(options)`, `wrapResponse(stream, opts?)`, and `wrapQuery(stream, opts?)` for the Claude Agent SDK. Wrap `query()`'s async iterator with `wrapQuery`. `wrapResponse` exists for naming symmetry with the Python SDK's wrapper around `ClaudeSDKClient.receiveResponse()`. TypeScript has no equivalent of that method to wrap directly. Pass `{ input: prompt }` to the wrap call to record a replayable root span. See [Claude Agent SDK framework guide](/frameworks/claude-agent-sdk).
#### `getVercelAiMiddleware`
```typescript theme={null}
getVercelAiMiddleware(traceFunctionKey: string): BitfabLanguageModelMiddleware
```
Returns a Vercel AI SDK [language model middleware](https://ai-sdk.dev/docs/ai-sdk-core/middleware). Pass it to the AI SDK's `wrapLanguageModel`, then use the wrapped model with `generateText` / `streamText` / `generateObject` / `streamObject`. Every call is captured as a keyed `llm` span carrying the call parameters as input. Streaming is captured without disturbing the live stream. See [Vercel AI SDK framework guide](/frameworks/vercel-ai-sdk).
The middleware's structural types are exported for callers that wrap or inspect it. All three are open (`[key: string]: unknown`), so provider-specific fields pass through untouched:
```typescript theme={null}
interface VercelCallParams {
prompt?: unknown
[key: string]: unknown
}
interface VercelGenerateResult {
content?: { type: string; text?: string }[]
text?: string // some providers expose a flattened text; preferred when present
usage?: unknown
finishReason?: unknown
[key: string]: unknown
}
interface VercelStreamResult {
stream: ReadableStream
[key: string]: unknown
}
```
`BitfabVercelAiHandler` is the class the middleware is built from, exported for direct construction with `{ traceFunctionKey, withSpan }`.
#### `wrapBAML`
See the [BAML framework guide](/frameworks/baml) for examples. Full signature under [`wrapBAML`](#wrapbaml) above.
## `class BitfabFunction`
Fluent wrapper binding a `traceFunctionKey`. Obtained via `client.getFunction(key)`.
### `withSpan`
```typescript theme={null}
withSpan(
optionsOrFn: SpanOptions | ((...args: TArgs) => TReturn),
maybeFn?: (...args: TArgs) => TReturn,
): (...args: TArgs) => TReturn
```
Delegates to `client.withSpan(boundKey, optionsOrFn, maybeFn)`.
### `span`
```typescript theme={null}
span(options?: SpanOptions): SpanMethodDecorator
```
Returns an optional TypeScript 5.0+ standard method decorator bound to this handle's trace function key. It is equivalent to `client.span(boundKey, options)`. The same standard-transform requirements as `Bitfab#span` apply. `withSpan` remains the recommended default. It is required for TypeScript 4.x projects and non-method callables.
### `getVercelAiMiddleware`
```typescript theme={null}
getVercelAiMiddleware(): BitfabLanguageModelMiddleware
```
Delegates to `client.getVercelAiMiddleware(boundKey)`, reusing the bound key so a `withSpan` root and the middleware share it. See [Nesting with core tracing](/frameworks/vercel-ai-sdk#nesting-with-core-tracing).
### `getClaudeAgentHandler`
```typescript theme={null}
getClaudeAgentHandler(): BitfabClaudeAgentHandler
```
Delegates to `client.getClaudeAgentHandler(boundKey)`, reusing the bound key so a `withSpan` root and the handler share it. See [Nesting with core tracing](/frameworks/claude-agent-sdk#nesting-with-core-tracing).
### `getLangGraphCallbackHandler`
```typescript theme={null}
getLangGraphCallbackHandler(): BitfabLangGraphCallbackHandler
```
Delegates to `client.getLangGraphCallbackHandler(boundKey)`, reusing the bound key so a `withSpan` root and the handler share it. See [Nesting with core tracing](/frameworks/langgraph#nesting-with-core-tracing).
### `getLangChainCallbackHandler`
Alias of `getLangGraphCallbackHandler`. LangChain and LangGraph share one callback system.
### `getLangGraphIntegration`
```typescript theme={null}
getLangGraphIntegration(
options?: LangGraphIntegrationOptions,
): BitfabLangGraphIntegration
```
**Experimental (alpha).** Delegates to `client.getLangGraphIntegration(boundKey, options)`. Pass its wrapped tools to the graph, then call `createInvoker(graph)` to get the callback-configured, replayable entry point.
### `wrapBAML`
Identical signature and semantics to `Bitfab#wrapBAML`. Unlike the `getXHandler()` methods above, it does not use the bound key. It opens no span of its own. It enriches the *current* span instead, so call it inside a function wrapped by this handle's `withSpan`.
## `class BitfabError`
Extends `Error`.
```typescript theme={null}
class BitfabError extends Error {
constructor(message: string, url?: string, status?: number, retryAfterMs?: number)
readonly url?: string
readonly status?: number
readonly retryAfterMs?: number
}
```
Thrown for SDK-originated failures, such as a missing function, missing prompt, or misconfiguration. Also thrown for a request that reaches the server and comes back with a non-2xx response. `status` is the HTTP status code, present only on that non-2xx response failure. It is absent for network failures, timeouts, and an error returned inside an otherwise-successful response body. `retryAfterMs` carries the server's `Retry-After` header in milliseconds, when it sent one. It is present only alongside `status`. Never thrown for transport errors on `withSpan` paths. Those are swallowed.
## `class MixedTracingError`
Extends `Error`.
```typescript theme={null}
class MixedTracingError extends Error {
constructor(message: string)
}
```
Thrown when opt-in tracing (`withSpan`, `span`) and opt-out tracing (`withTrace`, `trace`, `node`, `withNode`) meet in one call stack. The message names which surface was entered inside which and how to resolve it. Unlike other tracing setup failures, it is not swallowed. The wrapped function does not run untraced, because a wiring mistake the check exists to report would otherwise be invisible.
## `class BitfabLangGraphCallbackHandler`
Duck-types LangChain's callback handler interface without importing `@langchain/core`. Obtained via `client.getLangGraphCallbackHandler(key)`. No direct instantiation needed for normal use. It records chain, LLM, tool, and retriever roots as pending traces on start, then completes them on root end. Full callback surface documented in [LangGraph framework guide](/frameworks/langgraph).
## `class BitfabOpenAITracingProcessor`
Implements the OpenAI Agents SDK `TracingProcessor` interface. Obtained via `client.getOpenAiTracingProcessor()`. No direct instantiation needed for normal use. See [OpenAI Agents framework guide](/frameworks/openai-agents).
## `class BitfabOpenAIAgentHandler`
Run wrapper for the OpenAI Agents SDK. Obtained via `client.getOpenAiAgentHandler(key)`. Exposes `wrapRun(agent, input, options?)`, a drop-in for `run()` that records a keyed, replayable root span. See [OpenAI Agents framework guide](/frameworks/openai-agents).
## `class BitfabClaudeAgentHandler`
Handler for the Claude Agent SDK. Obtained via `client.getClaudeAgentHandler(key)`. Exposes `instrumentOptions`, `wrapResponse`, and `wrapQuery`. See [Claude Agent SDK framework guide](/frameworks/claude-agent-sdk) for method signatures and usage.
## Functions
### `finalizers`
```typescript theme={null}
finalizers.aiSdk(result: unknown): Promise>
finalizers.readableStream(
stream: ReadableStream,
onLive: (live: ReadableStream) => void,
): Promise<{ chunks: unknown[] }>
```
Prebuilt `SpanOptions.finalize` helpers. `aiSdk` awaits the Vercel AI SDK's text, usage, finish reason, tool calls, and tool results, without consuming its live stream. It records `{ text, usage, finishReason, toolCalls, toolResults }`. The recorded `usage` prefers the result's `totalUsage` over its `usage` field, when both resolve. Rejected or absent fields become `undefined`. `readableStream` tees a stream. It passes the caller's live branch to `onLive`. It records the other branch as `chunks`. Because a `ReadableStream` is single-consumer, the caller must use the branch received by `onLive`.
### `getCurrentSpan()`
```typescript theme={null}
function getCurrentSpan(): CurrentSpan
```
Returns the innermost active span. Outside a span context, returns a no-op whose `traceId` is `""`.
### `getCurrentTrace()`
```typescript theme={null}
function getCurrentTrace(): CurrentTrace
```
Returns a handle to the active trace. Outside a span context, returns a no-op.
### `flushTraces(timeoutMs?: number)`
```typescript theme={null}
async function flushTraces(timeoutMs?: number): Promise
```
Forces the private OpenTelemetry batch processor to export pending spans. Also waits for any remaining mutation requests. Both share one total deadline. Default `timeoutMs`: `5000`. Use before `process.exit()` in short-lived scripts.
Returns `true` when queued exports completed successfully within the deadline, or `false` when delivery failed or the flush timed out.
### `Bitfab.close(timeoutMs?: number)`
```typescript theme={null}
close(timeoutMs?: number): Promise
```
Flushes pending requests. Permanently shuts down this client's private OTel transport. Both happen within one total deadline (default `30000`). Idempotent. Returns `false` if delivery or shutdown misses the deadline. Use it when a long-running process creates transient clients. Shared clients may remain open until the process-wide exit hook runs instead.
### Transport environment variables
| Variable | Default | Description |
| -------------------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------- |
| `BITFAB_OTEL_MAX_REQUEST_BYTES` | `3000000` | Request-size target for OTLP/JSON exports. Accepts positive integers up to `3000000` |
| `BITFAB_OTEL_EXPORT_CONCURRENCY` | `32` | Concurrent direct requests per export window. Accepts `1` through `64` |
| `BITFAB_DISABLE_COMPRESSION` | unset | When set, request bodies are never gzipped, even above the 8,192-byte threshold where compression normally pays for itself |
### Commit ref environment variables
| Variable | Effect |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `BITFAB_DISABLE_COMMIT_REF` | Set to any value to send no `commit_ref` at all. Neither the platform variables nor `git` are consulted |
| `BITFAB_COMMIT_SHA` | The commit sent as `commit_ref.sha` on every root trace. Wins over every deploy platform variable and over `git` |
| `VERCEL_GIT_COMMIT_SHA`, `GITHUB_SHA`, `RAILWAY_GIT_COMMIT_SHA`, `RENDER_GIT_COMMIT`, `SOURCE_VERSION`, `CF_PAGES_COMMIT_SHA`, `CI_COMMIT_SHA`, `BUILD_SOURCEVERSION`, `CIRCLE_SHA1` | Read in that order when `BITFAB_COMMIT_SHA` is unset, together with the same platform's branch and repository variables when it has them. The first one set wins, and `git` is never run |
### `CommitRef`
```typescript theme={null}
interface CommitRef {
sha: string
branch: string | null
dirty: boolean | null
remote: string | null
rootSha: string | null
}
```
The commit the traced code was running at, sent as `commit_ref` on every root trace completion. `sha` is the commit. `branch` is the checked-out branch, or `null` when detached or unknown. `dirty` is `true` when the working tree had uncommitted or untracked changes, `false` when it was clean, and `null` when the SDK could not tell, which is always the case when the ref came from environment variables rather than `git`. `remote` is the `origin` URL reduced to `host/owner/repo` with any credentials removed, so a CI checkout token never reaches the trace. `root_sha` is the repository's first commit, so two checkouts of the same repository match even without a remote.
Resolution order is `BITFAB_COMMIT_SHA`, then the deploy platform's build variables, then `git` in the process's working directory (see [Commit ref environment variables](#commit-ref-environment-variables)). Environment resolution is synchronous and free. The `git` path runs once per process in a child process off the event loop, with a two second timeout per command, so it never sits on the code path that ran the traced function. A trace that completes before it lands ships without a `commit_ref`, and a process with neither variables nor a repository never sends one. The result, including a negative one, is memoized for the life of the process. Set `BITFAB_DISABLE_COMMIT_REF` to opt the process out entirely. On the wire the field is `root_sha`. In a browser bundle neither environment variables nor `git` exist, so no `commit_ref` is sent.
## Interfaces
### `BitfabConfig`
See [constructor table](#new-bitfabconfig-bitfabconfig).
### `SpanOptions`
```typescript theme={null}
interface SpanOptions {
name?: string // defaults to the qualified function name (Order.process for a method), then traceFunctionKey
type?: SpanType // defaults to "custom"
captureWhen?: CaptureWhen // defaults to "always"
mockOnReplay?: boolean // selected by replay's default "marked" strategy
testRunId?: string // links the span and any trace it starts to a test run
finalize?: (result: any) => unknown | Promise
}
```
`NodeOptions` is `Omit & { capture?: boolean }`. `capture` defaults to `true`. `mockOnReplay` may only select a node whose capture remains enabled.
### `CaptureWhen`
```typescript theme={null}
type CaptureWhen = "always" | "nested"
```
### `SpanType`
```typescript theme={null}
type SpanType = "llm" | "agent" | "function" | "guardrail" | "handoff" | "custom"
```
### `CurrentSpan`
```typescript theme={null}
interface CurrentSpan {
readonly id: string
readonly traceId: string
addContext(context: Record): void
setPrompt(prompt: string): void
}
```
| Method | Semantics |
| ------------ | -------------------------------------------------------------------------------------------------------- |
| `id` | Canonical Bitfab span ID. Empty when outside a span |
| `traceId` | UUID string. Empty when outside a span |
| `addContext` | Pushes the object as a single entry onto `span_data.contexts`. Non-object input is ignored. Never throws |
| `setPrompt` | Overwrites `span_data.prompt`. Non-string input is ignored. Never throws |
### `CurrentTrace`
```typescript theme={null}
interface CurrentTrace {
setSessionId(sessionId: string): void
setName(name: string): void
setMetadata(metadata: Record): void
addContext(context: Record): void
drop(): void
}
```
| Method | Semantics |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `setSessionId` | Stored on the trace's `session_id` DB column |
| `setName` | The trace's title in Bitfab and a searchable, filterable field. Stored on the trace's `name` DB column. Unset, the trace is titled by its trace function key. Empty strings are ignored |
| `setMetadata` | Shallow-merges with existing metadata. Later keys overwrite |
| `addContext` | Appends an entry to `rawData.contexts`. Accumulates across calls |
| `drop` | Flags the trace to be dropped. Once flagged, spans that complete afterward are not uploaded at all. The flag rides out on the completion payload. At completion, the server scrubs any payloads that already raced out, covering the trace itself, its external trace, and sibling spans. The server also deletes the archived S3 objects. The trace is marked `dropped` instead of `completed`, keeping only a skeleton audit row. A no-op outside a span. Never throws |
### `DetachedTrace`
```typescript theme={null}
interface DetachedTrace {
readonly traceId: string
addContext(context: Record): Promise
setMetadata(metadata: Record): Promise
setSessionId(sessionId: string): Promise
setName(name: string): Promise
}
```
Returned by `client.getTrace(traceId)`, where `traceId` is the canonical Bitfab trace ID. Methods have the same semantics as `CurrentTrace`, but send to the server immediately. They block until the server responds. They reject if the update is refused. When capture is off outside a replay item or a `seedTrace` call, all methods return `Promise.resolve()` instead.
### `CapturedSpan`
```typescript theme={null}
interface CapturedSpan {
id: string
traceId: string
parentSpanId: string | null
name: string | null
type: string
input: unknown
output: unknown
contexts: Record[]
prompt: string | null
metadata: Record
metrics: Record | null
errors: unknown
startedAt: string | null
endedAt: string | null
}
```
Returned by `getTraceSpan`. `SpanLookup` is `{ id: string } | { name: string; occurrence?: "first" | "last" | number }`.
### `WrapBAMLOptions`
```typescript theme={null}
interface WrapBAMLOptions {
onCollector?: (collector: unknown) => void
}
```
### `WrappedBamlFn`
```typescript theme={null}
interface WrappedBamlFn {
(...args: TArgs): Promise
collector: unknown | null
}
```
`collector` is `null` before the first call or when `@boundaryml/baml` is unavailable. After each successful call, it holds the BAML `Collector` instance from that invocation.
### `ReplayOptions`
```typescript theme={null}
interface ReplayOptions {
limit?: number // ignored (with a warning) when traceIds is passed
traceIds?: string[] // max 100; the ID count determines how many traces replay
name?: string // display name for the resulting experiment/test run
maxConcurrency?: number // default 10
attempts?: number // default 1; 1-100; replays each trace this many times, attempt-major
codeChangeDescription?: string | null
codeChangeFiles?: CodeChangeFile[] | null // omit for Git diff capture; null disables it
mock?: MockStrategy // "marked" (default), "none", or "all"
mockOverride?: MockOverrideInput | MockOverrideInput[]
adaptInputs?: (inputs: unknown[], ctx: AdaptContext) => unknown[]
dryRun?: boolean // resolve inputs without calling the function
experimentGroupId?: string
dbBranch?: boolean | DbBranchOptions // true requests a DB branch per replay item
datasetId?: string // durably attributes the experiment to a dataset
datasetIds?: string[] // several datasets: replays their union, attributed to each
graderIds?: string[] // graders attached to this run, unioned with the dataset's at grading (max 100)
onlyWithAssertions?: boolean // narrow the selection to traces carrying an assertion
onItemStart?: (progress: ReplayItemStartProgress) => void // each item as processing starts
onItemFinish?: (progress: ReplayItemFinishProgress) => void // each item as it finishes
onProgress?: (progress: ReplayProgress) => void // deprecated compatibility callback
}
// dbBranch: true means "on, mirror defaults"; an object tunes the branch.
interface DbBranchOptions {
minCu?: number // autoscaling floor for the branch's compute, 0.25-56
maxCu?: number // autoscaling ceiling; equal to minCu pins the size.
// A range may not span >8 CU or exceed 16 CU; a pinned
// size is a fixed compute, allowed up to 56.
warmupSql?: string // appended to the branch's readiness check, so it runs
// before your function sees the branch
}
interface ReplayItemFinishProgress {
type?: "item" // omitted on backward-compatible per-item events
testRunId: string // replay test run id
completed: number // items finished so far (succeeded or errored)
total: number // items in the run
succeeded: number // of completed, how many the fn ran without throwing
errored: number // of completed, how many threw
item: ReplayProgressItem // the single trace that just finished
}
interface ReplayProgressItem {
traceId?: string | null // null during the run; the server replay id arrives at completion
originalTraceId: string | null // the original (historical) trace replayed
originalSpanId?: string | null // the original root span the inputs were read from
sourceTraceId: string | null // deprecated alias for originalTraceId
sourceSpanId?: string | null // deprecated alias for originalSpanId
attempt: number // which attempt this item is, under `attempts`
input?: unknown[]
result?: unknown
originalOutput?: unknown
error: string | null // compatible message for either error kind
traceError?: unknown | null // actual exception thrown while executing the replayed trace
replayError?: unknown | null // actual exception thrown by replay setup
durationMs?: number | null // how long this one trace took to replay
originalDurationMs?: number | null
originalTokens?: TokenUsage | null
originalModel?: string | null
ingestionType?: TraceIngestionType
tokens?: TokenUsage | null // null until completion aggregates it
model?: string | null // deprecated alias for originalModel
dbSnapshotRef?: DbSnapshotRef | null
dbBranchTimings?: DbBranchTimings | null
traceOutline?: TraceOutline | null // null until completion builds it
originalTraceOutline?: TraceOutline | null // null until completion builds it
}
/** @deprecated Use ReplayItemFinishProgress. */
interface ReplayProgress {
type?: "item" | "complete"
result?: ReplayResult // terminal complete event only
testRunId?: string
completed: number
total: number
succeeded: number
errored: number
item?: ReplayItemFinishProgress["item"]
}
interface ReplayItemStartProgress {
type: "started"
testRunId: string
started: number
completed: number
total: number
succeeded: number
errored: number
item: {
originalTraceId: string
originalSpanId: string
sourceTraceId: string // deprecated alias
sourceSpanId: string // deprecated alias
attempt: number
}
}
```
`onItemStart` fires when a worker begins processing an item, before replay loads its inputs, prepares mocks or a database branch, or invokes customer code. Pair it with `onItemFinish`. `onItemFinish` fires exactly once per item as it finishes, in completion order rather than input order. Together they distinguish queued items from in-flight items whose callback has not returned. Both callbacks always carry the item. `onItemFinish` never represents whole-run completion. Replay doesn't know pass/fail at finish time, because verdicts are assigned later. The totals only split ran-ok from errored as a result. A throwing lifecycle callback never crashes the run. The deprecated `onProgress` callback receives the same per-item finish events, plus its legacy item-less terminal `complete` event. It is ignored when `onItemFinish` is also provided. The SDK exports a ready-made reporter, `reportReplayProgress`, that accepts either lifecycle event. Pass it as both `onItemStart` and `onItemFinish` to write lifecycle events to stderr. The Bitfab plugin uses start events to identify in-flight traces in its liveness heartbeat. It uses finish events to write per-item result files.
### `getCurrentReplayBranch()`
```typescript theme={null}
function getCurrentReplayBranch(): ReplayBranch | null
interface ReplayBranch {
databaseUrl: string // connection string for this item's branch
neonBranchId: string // the provider's own id for this branch
envKey: string // env var name your app reads, e.g. "DATABASE_URL"
expiresAt: string // ISO-8601
snapshotTimestamp?: string // ISO-8601, the instant the branch is pinned to
providerConsoleUrl?: string
readOnly?: boolean
region?: string // e.g. "aws-us-east-1"
traceId: string // the historical trace this item replays
}
```
Call it inside the replayed function to get the branch resolved for the item currently running. Each bounded replay worker provisions its own branch, so `maxConcurrency` also bounds live branches. With `attempts` set, each attempt resolves its own branch too. It returns `null` outside a replay item, and for an item whose source trace carried no DB snapshot reference. That is the fallback path. Use it like this: `const url = branch?.databaseUrl ?? process.env.DATABASE_URL`.
The value object is immutable and per item, built from the replay context, so parallel items each see their own branch. Reading `databaseUrl` marks the trace as having used the branch, reported as `accessed`. The other fields inspect the branch without exposing the connection string. They deliberately do not mark the branch as accessed. That also means the URL does not appear in `JSON.stringify(branch)`.
Every field the service puts on the lease is copied onto the branch, so one added server-side is readable before you upgrade the SDK. It just won't be in the type yet. `databaseUrl` is the sole exception. It is the credential. It is also the only member that may mark the branch as accessed.
`dbBranch` tunes the branch itself: `{ minCu: 2, maxCu: 2, warmupSql: "SELECT 1;" }`. Setting `minCu` equal to `maxCu` pins the compute, so items stay comparable. `warmupSql` runs inside the branch's readiness check, so warm-up time is never charged to the replayed call. Invalid warm-up SQL fails the branch rather than quietly handing back a cold one. All fields are optional. `dbBranch: true` enables branching. It leaves the mirror's own defaults in place.
### `ReplayResult`
```typescript theme={null}
interface ReplayResult {
items: ReplayItem[]
testRunId: string
testRunUrl: string
attempts: number
}
interface ReplayItem {
input: unknown[]
result: T | undefined
originalOutput: unknown
error: string | null // compatible message for either error kind
traceError: unknown | null // actual exception thrown while executing the replayed trace
replayError: unknown | null // actual exception thrown before the function ran
durationMs: number | null // how long THIS replay took
originalDurationMs: number | null // the original trace's duration
originalTokens: TokenUsage | null // the original trace's token usage
originalModel: string | null // the original trace's model
ingestionType?: TraceIngestionType // how the source trace came to exist ("captured" | "seeded"); absent on older servers, which only served captured traces
tokens: TokenUsage | null // the replayed run's token usage (compare vs originalTokens)
model: string | null // deprecated alias for originalModel
traceId: string | null // the new replay trace's server id, written in after the run completes (null on older servers)
originalTraceId: string // the original (historical) trace being replayed
originalSpanId: string // the original root span the inputs were read from
sourceTraceId: string // deprecated alias for originalTraceId
sourceSpanId: string // deprecated alias for originalSpanId
attempt: number // 1-based attempt number within this experiment
dbSnapshotRef: DbSnapshotRef | null // the source trace's snapshot pin, if any
dbBranchTimings: DbBranchTimings | null // per-phase branch provisioning timings
traceOutline: TraceOutline | null // the replayed trace's span tree, no payloads
originalTraceOutline: TraceOutline | null // the original trace's span tree, no payloads
}
```
`traceOutline` and `originalTraceOutline` are the replayed and the original trace's span trees with no inputs or outputs: each span's name, type, nesting, order, duration, tokens, model, errors, and whether it was mocked. The server builds both when the run completes, so they are `null` on progress items, on items whose replay produced no trace (`traceOutline` only), and against older servers. They exist for grading. Compare the two trees to tell whether a replay reached its output by the same path. That means the same tool calls in the same order, with no new child-span errors and no mocked span that used to run real code.
```typescript theme={null}
interface TraceOutline {
traceId: string // server trace id of the outlined trace
name: string | null
status: string
traceFunctionKey: string | null
durationMs: number | null
spanCount: number
spans: TraceOutlineSpan[] // root spans in start order
}
interface TraceOutlineSpan {
spanId: string
name: string | null
type: string // "llm" | "agent" | "function" | "guardrail" | "handoff" | "custom"
traceFunctionKey: string | null
durationMs: number | null
tokens: TokenUsage | null
model: string | null
errors: TraceOutlineSpanError[] | null
mocked: boolean // served from the recorded output instead of re-executing
children: TraceOutlineSpan[]
}
interface TraceOutlineSpanError {
source: string // "code" for the traced function, "sdk" for capture problems
error: string
step?: string
}
```
Unprefixed fields describe this replay. Anything describing the trace being
replayed carries the `original` prefix.
A trace error means the replayed function started and threw. A replay error means Bitfab could not invoke it, for example because database warmup, input loading, or mock preparation failed. The original thrown value is preserved in the corresponding field. `error` remains its string message, for compatibility and JSON output.
If database branch resolution fails, `replayError` is a `DbBranchReplayError` with `code`, `message`, `originalTraceId`, and an optional `cause`. Resolver codes such as `branch_create_failed`, `snapshot_from_replaced_origin`, `invalid_snapshot_ref`, `seeded_trace_has_no_snapshot` (the source was seeded, so it pinned no database instant), and `internal_error` therefore remain available in memory, progress events, result files, and `ReplayError.items`. HTTP, timeout, and network failures while requesting a lease use `lease_request_failed`, with the original client exception as `cause`.
If trace delivery or test-run finalization fails after items have settled, `replay()` throws `ReplayError`. It exposes `items`, `testRunId`, `testRunUrl`, and `cause`, so callers do not lose the individual failures:
```typescript theme={null}
try {
await bitfab.replay("document-pipeline", processDocument)
} catch (error) {
if (error instanceof ReplayError) {
console.error(error.cause)
for (const item of error.items) {
console.error(item.originalTraceId, item.traceError ?? item.replayError)
}
}
}
```
### `serializeReplayResult`
```typescript theme={null}
function serializeReplayResult(result: ReplayResult): string
```
Returns indented JSON while preserving structured fields from `traceError` and `replayError`, including `DbBranchReplayError.code`, `originalTraceId`, and nested `cause`. Use it for direct-run stdout instead of raw `JSON.stringify`. Raw `JSON.stringify` reduces JavaScript `Error` objects to `{}`.
### `AllowedEnvVars`
```typescript theme={null}
interface AllowedEnvVars {
OPENAI_API_KEY?: string
}
```
Only `OPENAI_API_KEY` is currently allowed.
### `ActiveSpanContext`
```typescript theme={null}
interface ActiveSpanContext {
traceId: string
spanId: string
}
```
Passed to the OpenAI tracing processor's span-linking hook.
### Mock override types
```typescript theme={null}
interface SpanNodeMeta {
traceFunctionKey: string
spanName: string
type: string
originalSpanId?: string
}
interface MockOverrideCtx {
node: SpanNodeMeta
inputs: unknown[]
getOriginalOutput: () => Promise
}
type NodeMatcher = (node: SpanNodeMeta) => boolean
type MockValueFn = (ctx: MockOverrideCtx) => unknown | Promise
type MockValue = MockValueFn | string | number | boolean | bigint | symbol | object | null | undefined
type MockOverrideResolver = (ctx: MockOverrideCtx) => unknown | Promise
interface MockOverride {
match: NodeMatcher
value: MockValue
}
type MockOverrideInput = MockOverride | MockOverrideResolver
```
`SpanNodeMeta` is a span's structural identity during replay. It carries no output payload, so matching runs on metadata only. `spanName` resolves as `options.name ?? fn.name ?? traceFunctionKey`. `originalSpanId` is this span's id in the replayed trace, absent when the live span has no recorded counterpart, such as one the changed code newly introduced.
`MockOverrideCtx.inputs` is the live arguments the changed code passed on this run. `getOriginalOutput()` returns the span's recorded output, memoized per replay item. It rejects when the span has no recorded counterpart. A synchronous span requires a synchronous resolver result, including a synchronous `NO_MOCK_OVERRIDE`.
`null` and `undefined` are legitimate mocked outputs, so returning either substitutes that value rather than declining. Decline with `NO_MOCK_OVERRIDE`.
### Replay supporting types
```typescript theme={null}
type MockStrategy = "none" | "all" | "marked"
type TraceIngestionType = "captured" | "seeded"
type AdaptInputsFn = (inputs: unknown[], ctx: AdaptContext) => unknown[]
interface CodeChangeFile {
path: string
before: string
after: string
}
interface AdaptContext {
originalTraceId: string
originalSpanId: string
sourceTraceId: string // deprecated alias for originalTraceId
sourceSpanId: string // deprecated alias for originalSpanId
metadata: Record
}
```
`AdaptContext.metadata` is the original trace's stored metadata, what `seedTrace` or `getCurrentTrace().setMetadata` put on it. It is requested from the server only when an adapter is registered. When the traced function also emits its own trace through an integration that exports trace metadata, the caller's metadata is merged with that export and wins on any shared key, so an integration no longer replaces the provenance the caller stored. Keys the integration set are kept alongside it, so this can carry keys the caller never wrote. Requires v0.52.3 or later.
### Replay registry types
```typescript theme={null}
type ReplayRegistryOptions = Omit
interface ReplayRegistryContext {
params: Readonly>
}
type ReplayOptionsFactory = (
context: ReplayRegistryContext,
) => ReplayRegistryOptions | Promise
interface SeedCase {
input: unknown[]
expected?: unknown
metadata?: Record
sessionId?: string
}
interface SeedResult {
pipeline: string
traceFunctionKey: string
traceIds: string[]
}
```
Lifecycle callbacks are excluded from `ReplayRegistryOptions` because the installed command owns progress reporting. `ReplayRegistryContext.params` holds the values supplied through `--params` and repeated `--param name=value`.
### Seed option types
```typescript theme={null}
interface SeedCaseOptions {
input: unknown[]
expected?: unknown
fn?: (...args: any[]) => unknown
metadata?: Record
sessionId?: string
name?: string
spanName?: string
spanType?: SpanType
}
interface SeedRunOptions {
args?: TArgs
metadata?: Record
sessionId?: string
name?: string
}
```
The two `seedTrace` overloads. `SeedCaseOptions` writes a trace without running anything. `SeedRunOptions` runs the function once and records it. `spanName` and `spanType` label and type the root span the case form writes. They have no run-form equivalent, because the executed function names its own root span.
### Database snapshot types
```typescript theme={null}
const SUPPORTED_PROVIDERS = ["neon"] as const
type DbSnapshotProvider = (typeof SUPPORTED_PROVIDERS)[number]
interface DbSnapshotConfig {
provider: DbSnapshotProvider
}
```
`DbSnapshotConfig` optionally pins the provider at capture time. It is not required. The provider is otherwise resolved at replay time. See [Database branching](/db-branching#optional-pin-the-provider-typescript).
### `CaptureSurface`
```typescript theme={null}
type CaptureSurface = "opt-in" | "opt-out"
```
Two surfaces can own an active span. `withSpan` / `span` produce `"opt-in"`, while `trace` / `withTrace` / `node` / `withNode` produce `"opt-out"`. Entering one beneath the other throws `MixedTracingError` before the inner function runs. Framework-managed spans and replay/seed root wrappers are neutral and may contain either surface.
## Error Behavior Summary
| Situation | Behavior |
| ------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Empty / missing `apiKey` | Resolved lazily, including a `BITFAB_API_KEY` fallback. Without `strict`, warns and disables capture. With `strict: true`, the first traced call throws `BitfabError` instead |
| Opt-in and opt-out tracing in one call stack | `MixedTracingError`. The inner function does not run |
| Invalid `dbSnapshot.provider` | Constructor throws `BitfabError` |
| `withSpan` transport failure | Swallowed. User's return value / exception passes through |
| User function throws | Span records `error` and `error_source: "code"`. Error is re-thrown |
| `addContext` / `setPrompt` with invalid input | Silently ignored. Never throws |
| `call()`: function not found | `BitfabError` with URL to `/functions` |
| `call()`: no prompt configured | `BitfabError` with URL to `/functions/{id}` |
| `wrapBAML`: missing client | `BitfabError` |
| `replay()`: plain callable `fn` (not `withSpan`-wrapped) | Auto-wrapped under the trace function key, so spans link to the test run (not an error) |
| `replay()`: `fn` wrapped under a key that differs from `traceFunctionKey` | `BitfabError` (key mismatch) |
| `replay()`: requested database branch cannot be resolved | Item `replayError` is `DbBranchReplayError` with the resolver code, message, and original trace ID |
| `replay()`: delivery or finalization fails after items settle | `ReplayError` with collected `items`, test-run identifiers, and original `cause` |
## Module Resolution
* **Node.js ESM:** `dist/index.js`
* **Node.js CJS:** `dist/index.cjs`
* **Browser:** works, but `AsyncLocalStorage`-dependent features degrade (see `withSpan` semantics)
# Replay Mocking
Source: https://docs.bitfab.ai/replay-mocking
Replay production traces while replacing unsafe, expensive, or external child calls with their recorded outputs
Replay runs historical inputs through your current code. Unmocked child calls run real code too: LLM calls, API calls, database fetches, tools, and any other wrapped work.
Replay mocking lets selected child spans return their recorded outputs instead. The root function still runs real code, so you can change the orchestration, prompts, validation, formatting, or downstream logic and test it against production scenarios without paying for every upstream dependency again.
Mocking is the replay safety boundary; the deployment environment is not. Put every unsafe side-effecting call behind a replay-mockable descendant span and mark it. Unmarked calls, root-inline work, and import-time side effects run real code. `mock: "none"` deliberately disables recorded-output mocking.
Available in the **TypeScript, Python, Ruby, and Go** SDKs. Go mocking applies to closure-style `Client.Span` calls, which own execution; manual `Start`/`End` spans cannot prevent caller-owned code from running.
## When to use it
Use replay mocking when a child span is not the thing you are trying to improve:
* An unsafe side effect must not execute again, such as sending an email, charging a card, publishing to a queue, or writing to a database.
* An LLM call is expensive, slow, or rate-limited.
* An external API is flaky or needs credentials you do not have locally.
* A database read depends on production-only rows.
* A retrieval or preprocessing step should stay fixed while you iterate on later logic.
Do not mock the part of the system you are actively changing. If you are improving the summarizer, keep the summarizer real and mock only the setup work around it.
## How it works
Every replay has a mock strategy. If you omit the option, Bitfab uses `"marked"`.
| Strategy | Behavior | Best for |
| ---------- | ----------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| `"marked"` | Only spans tagged at definition time are mocked. | The normal iteration loop, and the default. |
| `"none"` | Every child span runs real code. | Runs where you intentionally want every dependency real and have verified that is safe. |
| `"all"` | Every matched recorded descendant span returns its historical output; a missing occurrence fails the item closed. | Fast sanity checks where only the root logic needs to run. |
The root function always runs real code. Mocking only applies to descendant spans.
Python async-generator spans are not mockable yet. When `mock="all"`, `mock="marked"` selects one, or an override matches, the replay item fails without iterating the real generator. Move unsafe work inside the generator to a mockable sync or coroutine descendant.
Mocking also requires the child to execute as a descendant in the replay context. Python thread pools and `threading.Thread` need `Bitfab(trace_across_threads=True)`; pre-created consumers and other processes are outside that propagation. Ruby replay context is thread-local, so move work dispatched to another thread/process behind a boundary that runs in the replay thread. Traced Ruby methods that return an `Enumerator` bridge their parent span and replay context into same-thread `Enumerator.new` / `enum_for` source fibers. In TypeScript, a synchronous span cannot consume the lazy output used by `mock: "marked"`; use an async/Promise-returning boundary, or eager `mock: "all"` only when freezing every matched recorded child is the intended experiment.
Python's automatically discovered `@bitfab.trace()` descendants cannot be
mocked because interpreter monitoring observes rather than wraps them. Add
`@bitfab.node()` to any descendant that needs a replay boundary, or use
`@bitfab.span()` for an explicit replay boundary.
## Opt a trace into descendant mocking
Subtree tracing can opt into descendant-default mocking without changing the
global replay strategy. Set the trace option to `true`, then annotate only the
code that should keep running with a node-level `false`.
```typescript theme={null}
class SupportAgent {
@bitfab.node({ mockOnReplay: false })
async applyNewPolicy(input: Input): Promise {
return applyNewPolicy(input)
}
@bitfab.trace("support-agent", { mockOnReplayDefault: true })
async run(input: Input): Promise {
return this.applyNewPolicy(input)
}
}
await bitfab.replay("support-agent", agent.run.bind(agent))
```
Every transformed TypeScript descendant is marked by this trace option;
`applyNewPolicy` runs because its `@node` opts out. Without
`mockOnReplayDefault: true`,
subtree mocking remains opt-in exactly as before. Python exposes the analogous
`@trace(..., mock_on_replay_default=True)` option, but only configured `@node()`
descendants can inherit it because interpreter monitoring cannot skip an
unwrapped call. Replay `mock: "all"` retains its original all-descendants meaning.
## Mark child spans
```typescript TypeScript theme={null}
const articlePipeline = bitfab.getFunction("process-article")
const fetchArticle = articlePipeline.withSpan(
{ name: "fetch-article-from-db", mockOnReplay: true },
async (id: string) => db.articles.findById(id),
)
const summarize = articlePipeline.withSpan(
{ name: "summarize-article" },
async (article: Article) => summarizeWithNewPrompt(article),
)
const processArticle = articlePipeline.withSpan(
{ name: "process-article" },
async (id: string) => summarize(await fetchArticle(id)),
)
```
```python Python theme={null}
article_pipeline = bitfab.get_function("process-article")
@article_pipeline.span(name="fetch-article-from-db", mock_on_replay=True)
def fetch_article_from_db(article_id: str) -> Article:
return db.articles.find_by_id(article_id)
@article_pipeline.span(name="summarize-article")
def summarize_article(article: Article) -> Summary:
return summarize_with_new_prompt(article)
@article_pipeline.span(name="process-article")
def process_article(article_id: str) -> Summary:
return summarize_article(fetch_article_from_db(article_id))
```
```ruby Ruby theme={null}
class ArticleProcessor
include Bitfab::Traceable
bitfab_function "process-article"
bitfab_span :fetch_article_from_db, name: "fetch-article-from-db", mock_on_replay: true
def fetch_article_from_db(article_id)
db.articles.find_by_id(article_id)
end
bitfab_span :summarize_article, name: "summarize-article"
def summarize_article(article)
summarize_with_new_prompt(article)
end
bitfab_span :process_article, name: "process-article"
def process_article(article_id)
summarize_article(fetch_article_from_db(article_id))
end
end
```
```go Go theme={null}
func fetchArticleFromDB(ctx context.Context, articleID string) (Article, error) {
value, err := client.Span(
ctx,
"process-article",
func(ctx context.Context) (any, error) {
return db.Articles.FindByID(ctx, articleID)
},
bitfab.WithName("fetch-article-from-db"),
bitfab.WithInput(articleID),
bitfab.WithMockOnReplay(true),
bitfab.WithMockOutputType[Article](),
)
if err != nil {
return Article{}, err
}
return value.(Article), nil
}
```
Go's `Client.Span` returns `any`, and recorded JSON objects otherwise decode as `map[string]any`. Add `WithMockOutputType[T]()` when a mocked child returns a struct, slice, or other concrete type; the SDK decodes the substituted output into `T` before returning it.
## Replay with marked mocks
```typescript TypeScript theme={null}
const result = await bitfab.replay("process-article", processArticle, {
limit: 10,
})
console.log(result.testRunUrl)
```
```python Python theme={null}
result = bitfab.replay(process_article, limit=10)
print(result["test_run_url"])
```
```ruby Ruby theme={null}
processor = ArticleProcessor.new
result = Bitfab.client.replay(
processor, :process_article,
trace_function_key: "process-article",
limit: 10,
)
puts result[:test_run_url]
```
```go Go theme={null}
result, err := client.Replay(
context.Background(),
"process-article",
processArticle,
&bitfab.ReplayOptions{Limit: 10},
)
```
During this replay, `fetch-article-from-db` returns the output recorded in each historical trace. `summarize-article` runs your current code, so the experiment tests the code you are changing.
## What Bitfab matches
Bitfab matches mocked child calls against the recorded span tree for each source trace. Repeated calls to the same span are matched by call order, so the first call gets the first historical output, the second call gets the second, and so on. Current SDKs emit strictly increasing microsecond timestamps so rapid sibling calls retain that order; the service falls back to ingestion order and span ID for legacy traces whose timestamps tie exactly.
If a strategy selects a child for mocking but its recorded occurrence is missing or exhausted, Bitfab fails that replay item without executing the real child. Unselected calls still run real code.
During replay, `mockOverride` / `mock_override` / `MockOverrides` checks each non-root call that passes through a Bitfab span wrapper (`withSpan`, `@span`, `bitfab_span`, or Go's closure-style `Client.Span`). It does not iterate over trace-plan nodes or framework spans that Bitfab only observes.
## Inject custom values (overrides)
Mock overrides are available in the **TypeScript, Python, Ruby, and Go** SDKs.
Marking a span replays its *recorded* output. An override goes further: it substitutes a value **you** supply for a matched span, then lets downstream real code run against that substitution. Use it for "what if this step returned X" experiments, without editing the traced code.
An override is a `match` + `value` pair. `match` selects which calls it applies to (by span name, type, trace function key, or any structural field); `value` is the substitution -- a flat value used as-is, or a function that produces one. Full replacement: `value` becomes the call's output.
The trace function key passed to `replay()` selects the workflow's historical root traces; it does not by itself identify the descendant to override. In this example every call is bound to `process-article`, so the override matches the descendant by its span name. If your calls use separate trace function keys, a matcher can select by that field instead.
```typescript TypeScript theme={null}
const result = await bitfab.replay("process-article", processArticle, {
mock: "none", // run everything real...
mockOverride: {
// ...except this one call, which gets a value you supply
match: (node) => node.spanName === "fetch-article-from-db",
value: { id: "fixed", title: "Fixed title" }, // a flat value
},
})
```
```python Python theme={null}
from bitfab import MockOverride
result = bitfab.replay(
process_article,
mock="none", # run everything real...
mock_override=MockOverride(
# ...except this one call, which gets a value you supply
match=lambda node: node.span_name == "fetch-article-from-db",
value={"id": "fixed", "title": "Fixed title"}, # a flat value
),
)
```
```ruby Ruby theme={null}
result = Bitfab.client.replay(
processor, :process_article,
trace_function_key: "process-article",
mock: "none", # run everything real...
mock_override: {
# ...except this one call, which gets a value you supply
match: ->(node) { node[:span_name] == "fetch-article-from-db" },
value: { id: "fixed", title: "Fixed title" } # a flat value
}
)
```
```go Go theme={null}
result, err := client.Replay(ctx, "process-article", processArticle, &bitfab.ReplayOptions{
Mock: bitfab.MockNone,
MockOverrides: []bitfab.MockOverride{{
Match: func(node bitfab.SpanNodeMeta) bool {
return node.SpanName == "fetch-article-from-db"
},
Value: Article{ID: "fixed", Title: "Fixed title"},
}},
})
```
When `value` is a function it receives the span's **live** replay inputs, plus `get_original_output` to read the recorded output when you want to tweak it rather than replace it wholesale. In the Python and Ruby SDKs the context also carries the live keyword args (`ctx.kwargs` / `ctx[:kwargs]`) alongside the positional `inputs`:
```typescript TypeScript theme={null}
// getOriginalOutput() is async in the TypeScript SDK
value: async ({ getOriginalOutput }) => {
const original = await getOriginalOutput()
return { ...original, score: 1 } // override one field, keep the rest
}
```
```python Python theme={null}
# get_original_output() is synchronous in the Python SDK
value=lambda ctx: {**ctx.get_original_output(), "score": 1}
```
```ruby Ruby theme={null}
# get_original_output is synchronous in the Ruby SDK
value: ->(ctx) { ctx[:get_original_output].call.merge(score: 1) }
```
```go Go theme={null}
Resolve: func(ctx bitfab.MockOverrideContext) (any, error) {
original, err := ctx.GetOriginalOutput()
if err != nil {
return nil, err
}
value := original.(Result)
value.Score = 1
return value, nil
}
```
A flat or synthetic `value` (one that never calls `get_original_output`) fetches no recorded output at all. Pass a single override or an array (first matcher wins). To apply overrides to every replay on a client, register them:
```typescript TypeScript theme={null}
bitfab.registerMockOverride({
match: (node) => node.type === "llm",
value: { label: "refund" },
})
// Ordered form (equivalent): registerMockOverride(match, value)
bitfab.clearMockOverrides() // reset
```
```python Python theme={null}
bitfab.register_mock_override(
MockOverride(match=lambda node: node.type == "llm", value={"label": "refund"})
)
# Ordered form (equivalent): register_mock_override(match, value)
bitfab.clear_mock_overrides() # reset
```
```ruby Ruby theme={null}
Bitfab.client.register_mock_override(
match: ->(node) { node[:type] == "llm" },
value: { label: "refund" }
)
# Positional form (equivalent): register_mock_override(match, value)
Bitfab.client.clear_mock_overrides # reset
```
```go Go theme={null}
err := client.RegisterMockOverride(bitfab.MockOverride{
Match: func(node bitfab.SpanNodeMeta) bool { return node.Type == "llm" },
Value: map[string]any{"label": "refund"},
})
client.ClearMockOverrides()
```
Scope a client registration directly to one trace function key by passing the key first and either a resolver or another match-and-value override second. A keyed resolver only runs for spans with that key. Return the opt-out sentinel to continue to the next override and then the base strategy; ordinary nullish values remain valid mocked outputs. Omit the key to register one global resolver for every child span.
```typescript TypeScript theme={null}
import { NO_MOCK_OVERRIDE } from "@bitfab/sdk"
bitfab.registerMockOverride("checkout-agent", ({ node }) =>
node.spanName === "Classifier"
? { label: "refund" }
: NO_MOCK_OVERRIDE,
)
```
```python Python theme={null}
from bitfab import NO_MOCK_OVERRIDE
client.register_mock_override(
"checkout-agent",
lambda ctx: (
{"label": "refund"}
if ctx.node.span_name == "Classifier"
else NO_MOCK_OVERRIDE
)
)
```
```ruby Ruby theme={null}
client.register_mock_override("checkout-agent", lambda do |ctx|
if ctx[:node][:span_name] == "Classifier"
{label: "refund"}
else
Bitfab::NO_MOCK_OVERRIDE
end
end)
```
The resolver form also works directly in one replay's `mockOverride` / `mock_override` option. Precedence for each span: a per-call override wins, then a registered override, then the base `mock` strategy. The sentinel declines only the current resolver, so lower-priority overrides still get a chance before the base strategy.
In the TypeScript SDK, a **synchronous** wrapped function cannot wait for any Promise-returning resolver, including one that eventually resolves to `NO_MOCK_OVERRIDE`. For a global resolver used across mixed sync/async spans, use a non-`async` routing function: return `NO_MOCK_OVERRIDE` synchronously for sync keys, and return a Promise only for async keys. `getOriginalOutput()` also fetches asynchronously, so a synchronous span cannot use it; make the span `async`, or use `mock: "all"`. A flat `value` or synchronous function works on synchronous spans. Python, Ruby, and Go read the recorded output synchronously, so this restriction does not apply there; Go's fetch blocks only that replay worker.
## Parameterize registry mocks
Keep reusable mock construction in the replay registry instead of adding application-specific flags to the SDK command. Supply JSON values with `--params scenario.json` and override individual values with repeatable `--param name=value`. Values that parse as JSON retain their type; other values remain strings.
```typescript TypeScript theme={null}
optionsFactory: ({ params }) => ({
mockOverride: createIntentMock(params.forcedLabel),
})
```
```python Python theme={null}
options_factory=lambda ctx: {
"mock_override": create_intent_mock(ctx.params.get("forcedLabel"))
}
```
```ruby Ruby theme={null}
options_factory: ->(ctx) {
{mock_override: IntentMocks.build(ctx.params["forcedLabel"])}
}
```
```bash theme={null}
bitfab-replay --registry scripts/replayRegistry.ts agent \
--params scenarios/refund.json \
--param 'forcedLabel="refund"' \
--param confidence=0.95
```
The parameters are configuration for the registry's option factory only. The command never appends them to the historical inputs passed into the production function.
## Common patterns
### Keep expensive model calls fixed
Mark a paid classification or extraction span with `mockOnReplay` / `mock_on_replay`, then iterate on the decision logic that consumes its output.
### Stabilize infrastructure while changing prompts
Mock data loading, retrieval, or external tools, then keep the prompt or formatter real. This isolates prompt changes from local setup problems.
### Pair with database branching
Replay mocking is about replacing child function outputs. [Database branching](/db-branching) is about replaying against the database state that existed when the trace was captured. Use both when a replay needs production-like state and selected child calls should still be short-circuited.
## Mocking a seeded trace
A [seeded trace](/typescript-sdk#seeding-traces) is one written from a case you already held rather than captured from production. What mocking can do with it depends on how it was seeded.
| Seeded how | What replay mocking can substitute |
| ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
| By running the function once (`seedTrace(key, fn, ...)`, `seed_trace(key, fn, ...)`) | Everything a captured trace offers. The run recorded a full first-party subtree, so marked children, `mock: "all"`, and overrides all behave normally |
| From a case without running (TypeScript `seedTrace(key, case)`) | Nothing recorded. The trace has only a root span, so `mock: "marked"` and `mock: "all"` find no recorded child output to return |
For a case-seeded trace, use [overrides](#inject-custom-values-overrides) to supply values for calls that must not run for real. An override injects a value you write, so it does not need a recorded output to work from.
Neither kind of seeded trace carries a database pin, so `dbBranch` / `db_branch` refuses it rather than silently pinning the wrong moment.
## SDK references
* [TypeScript replay options](/typescript-sdk#mocking-child-spans-during-replay)
* [Python replay options](/python-sdk#mocking-child-spans-during-replay)
* [Ruby replay options](/ruby-sdk#mock-child-spans-during-replay)
* [Go replay options](/go-sdk#mock-child-spans)
# Ruby SDK
Source: https://docs.bitfab.ai/ruby-sdk
Instrument Ruby AI workflows as a span tree with bitfab_span, then replay production traces against your current code
The Bitfab Ruby SDK captures your AI function calls to automatically generate evaluations. Re-run your prompts with different models, parameters, and inputs to iterate faster.
Framework-native adapters (LangGraph, OpenAI Agents, BAML, Claude Agent SDK) are not yet available for Ruby. See [Frameworks overview](/frameworks/overview) for current coverage. Instrument Ruby code manually via `Bitfab::Traceable` or `Bitfab.span`.
## Installation
```bash theme={null}
# Bundler
bundle add bitfab
# Gem
gem install bitfab
```
## Quick Start
```ruby theme={null}
require "bitfab"
Bitfab.configure(api_key: ENV.fetch("BITFAB_API_KEY"))
```
Need an API key? Get one from the [Bitfab dashboard](https://bitfab.ai/setup) or see the [API Keys guide](/api-keys) for detailed setup instructions.
Copy this prompt into your coding agent (tested with Cursor and Claude Code using Sonnet 4.5):
```text theme={null}
Modify existing Ruby code to add Bitfab tracing.
Do NOT browse or web search. Use ONLY the API described below.
Bitfab Ruby SDK (authoritative excerpt):
- Install: `gem install bitfab` or `bundle add bitfab`
- Init:
require "bitfab"
Bitfab.configure(api_key: ENV.fetch("BITFAB_API_KEY"))
- Instrumentation (ONLY allowed form):
class MyService
include Bitfab::Traceable
bitfab_function ""
bitfab_span :method_name, type: "function"
def method_name
# ...
end
end
(bitfab_span must be placed immediately ABOVE the `def` it instruments.)
- Span types: "llm", "agent", "function", "guardrail", "handoff", "custom"
- DO NOT use a block form of bitfab_span.
- DO NOT extract helper methods.
Task:
1) Ensure the bitfab gem is added and initialization exists (Gemfile + initializer).
2) Read the codebase and identify ALL AI workflows (LLM calls, agent runs, AI-driven decisions).
3) Present me with a numbered list of workflows you found. For each, describe:
- What it does
- Why it's worth instrumenting: what visibility tracing gives you into each step
4) After I choose which workflow(s) to instrument:
- Add `bitfab_span` directly ABOVE each method's `def`
- Produce a SPAN TREE, not one span. The workflow root gets a span, and so does every step inside it:
each model call, each read of external state (DB query, HTTP GET, storage, vector search, cache),
each transform of the model output (parsing, validation, ranking, formatting), each retry or loop
iteration, and each external write. A single span around the outer function records one input and
one output for the whole workflow, which leaves replay mocking and per-step diagnosis with nothing
to work on. Do not wrap trivial in-memory helpers or per-item work inside a large loop.
- Mark reads and writes to mock on replay so a replayed trace does not repeat side effects
- Ensure each class includes `Bitfab::Traceable` and has `bitfab_function` set
5) Do not change method signature, behavior, or return value. Minimal diff.
Output:
- First: your numbered list of workflows with why each is worth instrumenting
- After my selection: minimal diffs for Gemfile, initializer, and the method changes
```
## Basic Configuration
```ruby theme={null}
Bitfab.configure(api_key: "...")
# Omit api_key entirely: the SDK reads ENV["BITFAB_API_KEY"]
Bitfab.configure
# Disable tracing (methods still execute, but no spans are sent)
Bitfab.configure(api_key: "...", enabled: false)
```
**Missing API key doesn't crash.** If the API key is missing, empty, or whitespace-only, the SDK automatically disables tracing and logs a one-time warning at first use. All instrumented methods still execute normally: no spans are sent, no errors are thrown. You don't need any conditional logic around the API key.
### API key resolution
The key is resolved **lazily, the first time a span runs**, not when `configure` is called. Resolution order is the configured value (a `Proc` is called while still unresolved), then a fallback read of `ENV["BITFAB_API_KEY"]`.
```ruby theme={null}
# Pass a Proc to defer resolution explicitly (resolved at first use):
Bitfab.configure(api_key: -> { ENV["BITFAB_API_KEY"] })
```
For standalone scripts where a run that emits no traces should be treated as a failure rather than silently skipped, set `strict`:
```ruby theme={null}
# Raises on the first traced call if no key resolves, instead of disabling quietly
Bitfab.configure(api_key: ENV["BITFAB_API_KEY"], strict: true)
```
## Tracing
**Trace the whole workflow, not just its entrypoint.** A single `bitfab_span` around the outer function records one input and one output for everything inside it, which leaves replay mocking, per-step diagnosis, and prompt iteration with nothing to work on. Spans exist only where you create them: nesting is automatic, but only between spans that exist.
Give a step its own span when any of these is true:
* **It calls a model.** Always. This is the span you iterate on, compare across experiments, and attach graders to.
* **It reads external mutable state** (DB query, HTTP `GET`, object storage, vector search, cache). These are the spans you will want to mock on replay.
* **It writes external state** (DB write, queue publish, email, charge, file write). Mark these to mock on replay so a replayed trace does not repeat the side effect.
* **It transforms the model output** (parsing, validation, ranking, formatting), so a quality regression points at the model or at your post-processing.
* **It retries or loops**, one span per attempt or iteration, so a trace shows how many attempts it really took.
Skip trivial in-memory helpers, per-item work inside a large loop (wrap the loop or the batch), and internals a [framework integration](/frameworks/overview) already captures.
Worked examples, replay-mocking decisions, and common pitfalls: [Instrumentation](/instrumentation).
### Custom (Recommended)
#### Using `Bitfab::Traceable` to Link Spans
Include `Bitfab::Traceable` in a class, declare the trace function key once with `bitfab_function`, then use `bitfab_span` to wrap methods. Three declaration styles are supported:
```ruby theme={null}
class OrderService
include Bitfab::Traceable
bitfab_function "order-processing"
# The root calls the traced methods, so they record as its children
bitfab_span :process_order, type: "agent"
def process_order(order_id)
order = load_order(order_id)
classification = classify_order(order)
validate_order(classification)
end
# Style 1: Before-def (recommended): declare bitfab_span above the method
bitfab_span :load_order, mock_on_replay: true
def load_order(order_id)
Order.find(order_id)
end
# Style 2: Inline: wrap the def directly
bitfab_span def classify_order(order)
client.responses.parse(model: MODEL, input: build_prompt(order))
end, type: "llm"
# Style 3: After-def: declare bitfab_span after the method definition
def validate_order(classification)
{ valid: classification.confidence > 0.8 }
end
bitfab_span :validate_order, type: "guardrail"
end
```
Calling `process_order(id)` records one trace with four spans:
```
process_order
├── load_order
├── classify_order
└── validate_order
```
Tracing only `process_order` would record the same work as a single node, with the model call, the database read, and the validation collapsed into the root's input and output.
All three styles are equivalent. The before-def style is recommended for readability.
#### Experimental: Trace a Whole Call Tree
`bitfab_trace` records one root method plus every first-party Ruby method it calls, without adding a declaration to each descendant:
```ruby theme={null}
class OrderService
include Bitfab::Traceable
bitfab_function "order-processing"
bitfab_trace :process_order, type: "agent"
def process_order(order_id)
order = load_order(order_id)
summarize(order)
end
def load_order(order_id)
Order.find(order_id)
end
def summarize(order)
{ id: order.id, total: order.total }
end
end
```
Ruby's `TracePoint` hook is enabled only during active capture. It records project methods across files, recursion, modules, inherited methods, `define_method`, and same-thread child Fibers; aliases use the name that was invoked. It skips Ruby/runtime code, dependency gems, Bitfab SDK internals, blocks, lambdas, forwarding decorator methods, and child-thread calls, and continues through lazy `Enumerator` consumption. If the traced root belongs to an installed application gem, that gem is the first-party boundary; neighboring gems remain excluded. Use `max_depth:` (default `30`), `max_spans:` (default `500`), `exclude:`, and `include_wrappers:` to control capture.
Child Fibers created during capture inherit node policies, trace identity, parenting, and depth limits while keeping separate call stacks. Pre-existing Fibers and `Fiber.new(storage: nil)` do not inherit subtree context. When an inner capture ends, child Fibers retain any still-active outer capture; inherited context expires once all enclosing captures end. While a root remains active, hitting `max_spans` stops new capture but allows already-open spans in every Fiber to finish. Independent roots in interleaved Fibers remain isolated; untraced scheduler calls between them are not captured.
When capture ends, already-open automatic calls in suspended Fibers are emitted once with a nil output and `Subtree capture ended before the call returned` as the error, unless a real error was already observed. Their inputs and parenting are retained. The SDK does not resume Fibers, and later resumption cannot add spans to the ended capture. If hot reload moves the root to an unresolved source, the last valid first-party boundary and its exclusions remain in use.
Returning an `Enumerator` keeps capture pending until iteration finishes or raises, including calls already paused in child Fibers. Capture is inactive between the initial return and consumption. Automatic spans record lazy return values as placeholders without iterating them; the traced root records the yielded values.
**Nesting traced roots.** A nested `bitfab_trace` with a different key starts an independent trace while its complete subtree also remains in every outer `bitfab_trace` capture. Each trace has distinct trace and span IDs, including during replay, and the same tree it would record alone. Two active roots therefore record two copies of their shared region; use this only when both boundaries need to stand alone as trace functions. Recursive calls and nested annotations using the same key stay in the active trace rather than starting more roots.
**Configure important calls.** `bitfab_node` changes how a method is captured only when it runs beneath `bitfab_trace`. Standalone calls remain completely untraced, and the enclosing trace owns the node's function key, parent, limits, and lifecycle.
```ruby theme={null}
bitfab_node :summarize,
name: "Model call",
type: "llm",
mock_on_replay: true,
finalize: ->(response) { {text: response.text} }
def summarize(order)
model.generate(order)
end
```
Use `name:` and `type:` to identify the call, `capture: false` to omit it while transparently reparenting captured descendants, `test_run_id:` for attribution, `mock_on_replay: true` to reuse recorded output under marked replay, and `finalize:` to record a serializable view without changing the caller's return value. Finalizer failures are recorded on the node and do not crash the caller. The before-`def`, inline, and after-`def` forms preserve visibility. Configure an external method for one client with `client.node(MyClass, :method)`.
**Performance.** The SDK prepares source boundaries at declaration, refreshes a boundary only if hot reload moves the root definition, caches canonical paths, and shares one `TracePoint` dispatcher across active roots on a thread. After `max_spans` is exhausted, it stops collecting metadata and values immediately, then stops processing events for that session once its captured calls finish. It must still inspect every Ruby method event while capture is active and serialize one span per captured call. Keep `bitfab_trace` for discovery, set conservative limits, and prefer explicit `bitfab_span` declarations on production hot paths.
```ruby theme={null}
bitfab_trace :build_order_detail, trace_function_key: "order-detail"
def build_order_detail(order)
summarize(order)
end
bitfab_trace :process_order, trace_function_key: "order-processing"
def process_order(order_id)
build_order_detail(load_order(order_id))
end
```
This API is experimental. Automatically captured calls cannot be replay-mocked, so declare a call with `bitfab_span` when replay must substitute it. The explicit span records once, and automatically captured calls beneath it keep that span as their parent. When different-key subtree roots overlap, each trace still retains its own complete copy of the shared region.
#### Multi-File Projects
For projects with instrumented methods spread across multiple files, create an initializer that configures Bitfab, then include `Bitfab::Traceable` in any class that needs tracing.
```ruby theme={null}
# config/initializers/bitfab.rb (single source of truth)
require "bitfab"
Bitfab.configure(api_key: ENV.fetch("BITFAB_API_KEY"))
```
```ruby theme={null}
# app/services/process_order_service.rb
class ProcessOrderService
include Bitfab::Traceable
bitfab_function "order-processing"
bitfab_span :process_order, type: "function"
def process_order(order_id)
ValidateOrderService.new.validate_order(order_id)
{ order_id: }
end
end
```
```ruby theme={null}
# app/services/validate_order_service.rb
class ValidateOrderService
include Bitfab::Traceable
bitfab_function "order-processing"
bitfab_span :validate_order, type: "guardrail"
def validate_order(order_id)
{ valid: true }
end
end
```
Classes sharing the same `bitfab_function` key are grouped together. Spans from different classes are automatically linked as parent-child when one instrumented method calls another.
#### Using `bitfab_span` with Explicit Key
For a single span with an explicit trace function key:
```ruby theme={null}
class StandaloneService
include Bitfab::Traceable
bitfab_span :standalone_task, trace_function_key: "one-off-operation"
def standalone_task
"done"
end
end
```
#### Automatic Nesting
Spans nest automatically based on call stack:
```ruby theme={null}
class Pipeline
include Bitfab::Traceable
bitfab_function "pipeline"
bitfab_span :outer, type: "agent"
def outer
inner # Becomes a child of "outer"
end
bitfab_span :inner, type: "function"
def inner
# ...
end
end
```
For reusable helpers that should appear only inside an existing trace, set `capture_when: "nested"`:
```ruby theme={null}
class Pipeline
include Bitfab::Traceable
bitfab_function "pipeline"
bitfab_span :helper, type: "function", capture_when: "nested"
def helper
# ...
end
end
Pipeline.new.helper # Runs normally without creating a trace when no parent is active
```
#### Span Options
**Parameters:**
* `method_name` (required): Symbol of the method to wrap
* `trace_function_key` (optional): Override class-level `bitfab_function`
* `name` (optional): Display name. Defaults to method name
* `type` (optional): Span type. Defaults to `"custom"`. A label only, used to organize and filter spans in the dashboard; it does not change how the span is traced, replayed, or evaluated
* `capture_when` (optional): `"always"` / `:always` (default) or `"nested"` / `:nested`. Nested-only spans are captured under an active parent and run untraced when called standalone. Unknown values warn once and default to `"always"`
**Span Types:**
```ruby theme={null}
SPAN_TYPES = %w[
llm # LLM calls
agent # Agent workflows
function # Function calls
guardrail # Safety checks
handoff # Human handoffs
custom # Default
]
```
**Examples:**
```ruby theme={null}
class SafetyService
include Bitfab::Traceable
bitfab_function "safety-service"
# Method name is automatically captured as span name
bitfab_span :check_safety, type: "guardrail"
def check_safety(content)
{ safe: !content.include?("unsafe") }
end
# Override with name option
bitfab_span :validate_input, name: "InputValidator", type: "guardrail"
def validate_input(input)
{ valid: !input.empty? }
end
end
```
#### Span Context
Use `Bitfab.current_span` to get a handle to the active span, then call `.add_context()` to attach contextual key-value pairs from inside a traced method, useful for runtime values like request IDs, computed scores, or dynamic context:
```ruby theme={null}
class OrderService
include Bitfab::Traceable
bitfab_function "order-processing"
bitfab_span :process_order, type: "function"
def process_order(order_id)
user_id = current_user_id
Bitfab.current_span.add_context("user_id" => user_id, "order_id" => order_id)
{ order_id: order_id, status: "completed" }
end
end
```
Each `add_context` call pushes the entire hash as one entry. Multiple calls accumulate entries:
```ruby theme={null}
Bitfab.current_span.add_context("user_id" => "u-123")
Bitfab.current_span.add_context("request_id" => "req-789")
# Result: contexts: [{ "user_id" => "u-123" }, { "request_id" => "req-789" }]
```
You can also access the canonical Bitfab span and trace IDs via `Bitfab.current_span.id` and `Bitfab.current_span.trace_id` (both return an empty string outside a span):
```ruby theme={null}
span_id = Bitfab.current_span.id
trace_id = Bitfab.current_span.trace_id
```
#### Span Prompt
Use `Bitfab.current_span` to set the prompt string on the current span. This is stored in `span_data.prompt` and is useful for capturing the exact prompt text sent to an LLM:
```ruby theme={null}
class ClassificationService
include Bitfab::Traceable
bitfab_function "classification"
bitfab_span :classify_text, type: "llm"
def classify_text(text)
prompt = "Classify the following text: #{text}"
Bitfab.current_span.set_prompt(prompt)
llm.complete(prompt)
end
end
```
The prompt is metadata only. It records the prompt text for display and reference in the dashboard; it does not send the prompt to any model or change what the span executes. The last `set_prompt` call wins -- it overwrites any previously set prompt on the span. Calling `set_prompt` outside a span context is a no-op (it never crashes).
#### Trace Context
Use `Bitfab.current_trace` to set context that applies to the entire trace (all spans within a single execution). This is useful for grouping traces by session or attaching trace-level metadata:
```ruby theme={null}
class OrderService
include Bitfab::Traceable
bitfab_function "order-processing"
bitfab_span :process_order, type: "function"
def process_order(order_id)
trace = Bitfab.current_trace
# Set session ID (stored as database column, filterable in dashboard)
trace.set_session_id("session-123")
# Name the trace (its title in Bitfab, searchable and filterable)
trace.set_name("Order #{order_id}")
# Set trace metadata (stored in raw trace data)
trace.set_metadata("region" => "us-west-2", "environment" => "production")
# Add context entries (stored as key-value pairs, accumulates across calls)
trace.add_context("workflow" => "checkout-flow", "batch_id" => "batch-2024-01")
{ order_id: order_id, status: "completed" }
end
end
```
* `set_session_id(id)`: Groups traces by user session. Stored as a database column for efficient filtering.
* `set_name(name)`: The trace's title in Bitfab, and a field you can search and filter on. Use it for the case, ticket, or record the run is about. Unset, the trace is titled by its trace function key.
* `set_metadata(hash)`: Arbitrary key-value metadata on the trace. Merges with existing metadata.
* `add_context(hash)`: Key-value context entries. Accumulates across multiple calls.
#### Read One Persisted Span
Fetch one span without loading the full trace. Repeated name matches default to the last span.
```ruby theme={null}
span = Bitfab.client.get_trace_span(trace_id, name: "GenerateAnswer")
first = Bitfab.client.get_trace_span(trace_id, name: "GenerateAnswer", occurrence: "first")
exact = Bitfab.client.get_trace_span(trace_id, id: span_id)
```
Both IDs are canonical Bitfab IDs; ingestion source IDs are not accepted. `occurrence` also accepts a zero-based integer. A missing trace or span returns `nil`.
#### Dropping a Trace
Call `.drop` on the current-trace handle to discard the in-flight trace. Once flagged, spans that complete afterward are not uploaded at all, and the flag rides out on the completion payload, so when the trace completes the server scrubs any payloads that already raced out (the trace, its external trace, and sibling spans), deletes the archived S3 objects, and marks it `dropped` instead of `completed`, keeping only a skeleton audit row. Use it to discard runs you never want stored (health checks, test traffic) or a run you know carries sensitive data.
```ruby theme={null}
class OrderService
include Bitfab::Traceable
bitfab_function "order-processing"
bitfab_span :process_order, type: "function"
def process_order(order_id)
Bitfab.current_trace.drop if health_check?(order_id)
{ order_id: order_id, status: "completed" }
end
end
```
* Safe to call outside a trace (a no-op), and never raises into your application.
#### Error Handling
Errors are captured in the span and re-raised:
```ruby theme={null}
class RiskyService
include Bitfab::Traceable
bitfab_function "risky-service"
bitfab_span :risky
def risky
raise "error"
end
end
begin
RiskyService.new.risky
rescue => e
# Span records error and timing
end
```
Each error is classified by source. Errors raised by your code are recorded with `error_source: "code"`. SDK-internal errors are recorded with `source: "sdk"`. Both appear in the span's `errors` array in the Bitfab dashboard.
#### Flushing Traces
```ruby theme={null}
raise "Bitfab traces were not delivered before the deadline" unless Bitfab.flush_traces(timeout: 30)
```
The return value is `false` when an export fails or the deadline expires.
Traces flush automatically on process exit via `at_exit` hook.
#### OpenTelemetry Transport
| Variable | Default | Purpose |
| -------------------------------- | --------- | --------------------------------------------- |
| `BITFAB_OTEL_EXPORT_CONCURRENCY` | `32` | Concurrent direct requests, `1` through `64`. |
| `BITFAB_OTEL_MAX_REQUEST_BYTES` | `3000000` | Request-size target. Can only be lowered. |
The Ruby SDK lazily creates one private OpenTelemetry provider and bounded
`BatchSpanProcessor` per client; it does not replace your application's global
OTel provider, and an unused client starts no OTel worker. Traced methods
submit the same replay-safe Bitfab payloads through a transport interface. Batches
are sent to Bitfab as OTLP/JSON. Each carrier is encoded once and the request
body is assembled from those encodings, so a batch is never re-encoded to
measure its size.
Live and replay traces share the same OTel pipeline. Before completing a replay
test run, the SDK flushes OTel and uses delivery acknowledgments to confirm
every carrier that reached Bitfab. If any delivery is uncertain, it polls
Bitfab until every submitted replay trace completion and expected span count is
persisted.
The SDK partitions count-based OTel exports into requests of at most about
3 MB. Each request contains at most eight carriers and up to 32 run
concurrently. Set `BITFAB_OTEL_MAX_REQUEST_BYTES` (a positive integer no greater
than `3000000`) for a stricter proxy, and `BITFAB_OTEL_EXPORT_CONCURRENCY` (`1`
through `64`) to tune request concurrency. Invalid values warn once and fall
back to the defaults.
A single span may use up to 7,800,000 carrier bytes when its dedicated request
gzips below the 3,000,000-byte wire target and remains below the 8,000,000-byte
decompressed ingress limit. The carrier is the payload re-escaped into the OTLP
attribute. If it does not compress enough, compression is unavailable, or it
exceeds the raw ceiling, the SDK replaces its largest fields with
`` placeholders until it fits the
2,800,000-byte fallback budget. The trim is recorded on the span's `errors` so
the trace is flagged as incomplete.
The queue holds 8,192 carriers (a span and its trace completion are two), and a
burst that queues faster than the exporter drains loses the excess rather than
growing your process's memory. `Bitfab.flush_traces` reports on what reached
the exporter, so it returns `true` even when the queue had to drop carriers;
keep bursts inside that capacity when every span matters.
A long-running process that builds transient clients should release each
client's batch worker when it is done with it; a shared client is closed at
process exit.
```ruby theme={null}
client = Bitfab::Client.new(api_key: ENV.fetch("BITFAB_API_KEY"))
begin
# ... traced work ...
ensure
client.close(timeout: 30)
end
```
For the full ownership model, carrier format, live and replay flows, batching
limits, lifecycle, and failure semantics, see
[OpenTelemetry Transport Architecture](/otel-architecture).
#### Wrapping Third-Party Methods
Use `Bitfab::Traceable.wrap` to trace methods on external classes:
```ruby theme={null}
require "openai"
Bitfab::Traceable.wrap(
OpenAI::Client, :chat,
trace_function_key: "openai",
name: "Chat",
type: "llm"
)
# Now all calls to client.chat are traced
client = OpenAI::Client.new(access_token: ENV["OPENAI_API_KEY"])
client.chat(parameters: { model: "gpt-4", messages: [...] })
```
For experimental whole-subtree capture on an external class, use `Bitfab::Traceable.trace` with the same `klass`, `method_name`, `trace_function_key:`, `name:`, and `type:` arguments plus the subtree limit options.
### Replay
A trace is replayable when its root span has serializable inputs (the recorded inputs must round-trip through `to_json`). Framework handlers are not yet available for Ruby, so serializable root inputs are the only path to a replayable trace; instrument the outer workflow method so its inputs serialize.
Replay historical traces through a method to create test runs. This re-runs past inputs through your updated code and compares the results.
```ruby theme={null}
client = Bitfab.client
# Pass an instance for instance methods
service = OrderService.new(api_key: "...", db: db)
result = client.replay(
service, :process_order,
trace_function_key: "order-processing",
limit: 5
)
# Or pass a Class for class methods
result = client.replay(
OrderService, :process_order,
trace_function_key: "order-processing",
limit: 5
)
puts "Test Run URL: #{result[:test_run_url]}"
puts "Test Run ID: #{result[:test_run_id]}"
result[:items].each do |item|
puts "Input: #{item[:input]}, Result: #{item[:result]}, Error: #{item[:error]}"
puts "Duration (ms): #{item[:original_duration_ms]}"
puts "Tokens: #{item[:tokens]}" # { input:, output:, cached:, total: } or nil
puts "Model: #{item[:model]}"
puts "Trace ID: #{item[:trace_id]}" # Server trace ID for the replayed execution
end
```
Replay waits for each item's trace (spans + completion) to be persisted server-side before completing the test run, so `:trace_id` is a real server trace ID for completed items. If NO completed item's trace persisted (uploads wholesale failed, or the replayed method isn't traced), `replay` raises a `RuntimeError` instead of silently returning `nil` trace IDs. If only SOME items' traces are missing (a transient per-item upload failure), those items get `nil` trace IDs with a loud warning and the rest of the run is returned intact. `:trace_id` is also `nil` for errored (unreplayable) items, and for all items when the server predates the trace-ID mapping (a warning explains which).
Anything describing the trace being replayed carries the `original_` prefix: `:original_duration_ms`, `:original_model`, and `:original_tokens`. Unprefixed fields are the replay's own: `:duration_ms` is how long this run's call took, and `:tokens` is the **replayed run's** usage (the same numbers Studio's experiments view shows). Comparing `:tokens[:total]` against `:original_tokens[:total]` tells you how your change moved cost. Each field is `nil` when it wasn't captured.
The same rule names the two trace outlines: `:original_trace_outline` is the original trace's span tree and `:trace_outline` is the replayed trace's. An outline carries each span's name, type, nesting, order, duration, tokens, model, errors, and whether it was mocked, and no inputs or outputs, so it is small enough to keep on every item. Both are filled in at completion (they are `nil` in `on_item_finish` and against older servers) so a grader can compare the path the replay took against the original's, not only its output. The shape is documented in the [Ruby reference](/reference/ruby).
`:model` remains as a deprecated alias for `:original_model`. Note that `:duration_ms` changed meaning: it used to report the original trace's duration and now reports the replay's.
**Parameters:**
* `receiver` (required): An instance for instance methods, or a Class for class methods
* `method_name` (required): Symbol of the method to replay
* `trace_function_key` (required): The trace function key
* `limit` (optional): Max recent traces to replay (default: 5; maximum: 5,000). Ignored when `trace_ids`, `dataset_id` or `dataset_ids` is passed, since an explicit ID list or a dataset already determines how many traces replay. To replay part of a dataset selection, name the members to run in `trace_ids`.
* `trace_ids` (optional): Array of specific trace IDs to replay (max 100). The ID count determines how many traces replay, and `limit` is ignored when both are passed. Passed alongside a dataset selector it pins which members of that selection replay, and the server rejects any ID none of those datasets contains.
* `name` (optional): Display name for the resulting experiment/test run.
* `max_concurrency` (optional): Max threads for parallel replay (default: 10)
* `code_change_description` (optional): Rationale for the code change being tested in this replay (stored on the experiment); when supplied alone, it is preserved while files are captured automatically
* `code_change_files` (optional): Array of edited files, each as `{ path:, before:, after: }` (use `""` for newly created or deleted files); omit to capture automatically or pass `nil` to suppress capture
* `mock` (optional): Mock strategy for child spans during replay. One of `"marked"` (default, only spans tagged with `mock_on_replay: true` return historical output), `"none"` (every child runs real code), or `"all"` (every matched recorded child returns its historical output; a missing occurrence fails the item closed)
* `mock_override` (optional): One `{ match:, value: }` hash, or an array of them, that substitutes a supplied output for matched spans. Per-call overrides take precedence over registered overrides and the base `mock` strategy.
* `experiment_group_id` (optional): UUID string that groups multiple replay runs into a single experiment batch. Pass the same ID across successive `replay()` calls to link them together in the dashboard.
* `dataset_id` (optional): UUID of the dataset this replay runs against, durably attributing the experiment to that dataset. Pass `trace_ids` alongside it to replay only those members.
* `dataset_ids` (optional): UUIDs of the datasets this replay runs against, for benchmarking one method against several corpora in a single run. Replays the union of their traces, graded by the union of their graders, and attributes the experiment to every one of them.
* `grader_ids` (optional): Array of grader UUIDs (max 100) attached directly to this replay run, independent of the dataset's own graders. The resulting experiment is graded by the union of these and the dataset's runnable graders. Use it to grade a single run with a check you don't want to add to the dataset permanently. Each must be an active grader in the same organization and trace function, or the replay is rejected with a 400. A replay with no dataset can still carry graders this way.
* `adapt_inputs` (optional): A callable `->(args, kwargs, ctx)` that reshapes recorded inputs onto the method's current signature when its shape changed after the traces were captured. See **Adapting inputs after a signature change** below.
* `on_item_start` (optional): A callable fired when a worker begins processing an item, before replay setup and customer code run. Pair it with `on_item_finish` to distinguish queued items from in-flight items whose callback has not returned. A raising callback never crashes the run.
* `on_item_finish` (optional): A callable `->(progress)` fired exactly once per item as it finishes, always with that item plus running totals (original/source trace id, the server replay `:trace_id` read back per trace off the OTLP ingest response and surfaced as each item finishes (its trace is flushed on finish, so the id is in hand at the callback, not only after the whole run), input, result, original output, error, duration, tokens/model metadata). It never emits a whole-run completion event. Use it to render live progress or start evaluating completed items while replay runs. A raising callback never crashes the run. The deprecated `on_progress` callback receives the same per-item events plus its legacy item-less terminal `complete` event, and is ignored when both are supplied. The installed `bitfab-replay` command passes `Bitfab.method(:report_replay_progress)` as both `on_item_start` and `on_item_finish`; it writes lifecycle events to stderr, which the plugin uses to identify in-flight traces, report finished items, and write per-item result files while stdout remains available for direct-run `ReplayResult` JSON.
* `db_branch` (optional): `true` or a `Hash`. `db_branch: true` requests a DB branch per replay item with the mirror's own sizing; pass a hash to tune it, and `false` or `nil` leaves branching off. Each replay worker resolves its branch from the source trace's captured snapshot reference, so `max_concurrency` also bounds live branches. `Bitfab.current_replay_branch` hands the branch to you inside the replayed method, and the SDK releases it after the item. The reader returns `nil` when no branch was resolved (e.g. the trace predates snapshot capture, or DB branching isn't configured), so `branch ? branch.database_url : ENV["DATABASE_URL"]` falls back to your live database. The keys tune the branch, symbol or string: `min_cu` and `max_cu` are the compute's autoscaling floor and ceiling (0.25 to 56). Equal values pin a fixed size, allowed up to 56; an autoscaling range may not span more than 8 CU or exceed 16 CU; setting them equal pins the size, so one item can't post a better number purely because it ran against an already-scaled endpoint. `warmup_sql` is appended to the branch's readiness check, so the cache is warm before your method sees the branch and the warm-up is never charged to the replayed call. Omit them and the branch keeps the mirror's own defaults.
**Notes:**
* `receiver` + `method_name` must resolve to the method that carries the `traceable` decoration. Passing a plain wrapper around it will not resolve the trace function key.
* **`trace_function_key` must match the method's declared key.** It is read from the `bitfab_span` / `Bitfab::Traceable.wrap` declaration on the method you point at. If the key you pass contradicts that declared key, `replay` raises an `ArgumentError`: it would otherwise fetch one function's historical traces but record the replay under the method's own key, producing an incoherent test run. (Ruby has no plain-callable replay form, so there is no reason to pass a non-matching key.)
* `trace_function_key` is passed explicitly, so instance methods and class methods are disambiguated by the `receiver`.
* **Use a single `Bitfab::Client` across instrumentation and replay.** If your instrumented module constructs a client at load and your replay registry constructs another, they do not share registered trace functions; import the client from the instrumented module (or a shared singleton) rather than constructing a new one in the registry.
**Replay specific traces:**
```ruby theme={null}
service = OrderService.new(api_key: "...")
result = client.replay(
service, :process_order,
trace_function_key: "order-processing",
trace_ids: ["trace-id-1", "trace-id-2"]
)
```
**Attaching a code change:**
Each replay creates an experiment (test run). When you're iterating on a method and replaying after every edit, attach the change so the dashboard can show *exactly what was edited* alongside the results. Read each file before editing, edit, then read it again: the two strings go straight into `code_change_files`. There's no diff format to construct.
```ruby theme={null}
before = File.read("lib/order_service.rb")
# ...edit lib/order_service.rb...
after = File.read("lib/order_service.rb")
result = client.replay(
service, :process_order,
trace_function_key: "order-processing",
code_change_description: "fix off-by-one in retry logic",
code_change_files: [
{path: "lib/order_service.rb", before:, after:}
]
)
```
Both options are optional and independent: pass just `code_change_description` for a quick rationale-only annotation, or just `code_change_files` to record the literal edits.
If you omit `code_change_files`, `replay` falls back to capturing your working-tree diff against the trunk merge-base (best-effort, only inside a git repo), so an experiment still shows a diff. This fallback uses Git rename detection: a renamed-and-edited file is compared once under its destination path, while an unchanged rename adds no content diff. A supplied `code_change_description` is preserved while the files are captured. Passing `code_change_files` explicitly always wins and is the way to record a precise per-edit before/after. To opt out for one replay run, pass `code_change_files: nil` (and optionally `code_change_description: nil` if you also want no description). Set `BITFAB_DISABLE_CODE_CHANGE_CAPTURE` to turn the fallback off for every replay in the process.
#### Mock child spans during replay
For the workflow-level guide, see [Replay Mocking](/replay-mocking).
By default replay uses `"marked"`: child spans tagged with `mock_on_replay: true` return their historical outputs, while every other child runs real code. Three mock strategies control this behavior:
Traced methods that return an `Enumerator` preserve their parent span and replay context inside same-thread `Enumerator.new` / `enum_for` source fibers, so marked descendants remain mockable while the stream is consumed. Child threads and processes do not inherit replay context.
```ruby theme={null}
# "marked" (default): only spans declared with mock_on_replay: true return historical output
client.replay(service, :process_order, trace_function_key: "order-processing")
# "none": every child span runs real code
client.replay(service, :process_order, trace_function_key: "order-processing", mock: "none")
# "all": every matched recorded child returns its historical output; only the root runs real code
client.replay(service, :process_order, trace_function_key: "order-processing", mock: "all")
```
Tag the child spans you want mocked at definition time:
```ruby theme={null}
class OrderService
include Bitfab::Traceable
bitfab_function "order-processing"
# Paid LLM call: skip during replay by default
bitfab_span :classify_intent, type: "llm", mock_on_replay: true
def classify_intent(prompt); end
# Cheap, deterministic: keep running real
bitfab_span :persist, type: "function"
def persist(order); end
bitfab_span :process_order, type: "agent"
def process_order(order)
classify_intent(order.description)
persist(order)
end
end
```
Use the default `mock: "marked"` behavior when you want to iterate on `process_order`'s logic without paying for the LLM call on each replay. Use `mock: "all"` when the goal is the cheapest possible replay (every matched recorded child span returns its recorded output; only the root function executes real code).
Calls are matched by trace function key and span name. Repeated calls with the same key and name are distinguished by call order, so their first, second, and third invocations receive the corresponding historical outputs. Calls that share a trace function key but have different span names are tracked independently.
If a strategy selects a call for mocking but its recorded occurrence is missing or exhausted, the item errors and the real method does not execute.
#### Injecting custom values with overrides
A **mock override** substitutes a value you supply for a matched span, so downstream real code runs against it -- for "what if this step returned X" experiments without editing the traced code. An override is a `{ match:, value: }` hash: `match` is a callable selecting spans by structural metadata (`node[:trace_function_key]`, `node[:span_name]`, `node[:type]`, `node[:original_span_id]`); `value` is a flat value injected as-is, or a callable that returns one.
The `trace_function_key:` passed to `replay` selects the workflow's historical root traces; it does not by itself identify the descendant to override. Because `bitfab_function "order-processing"` binds every method in this example to the same key, match `classify_intent` by its span name.
```ruby theme={null}
result = client.replay(
service, :process_order,
trace_function_key: "order-processing",
mock: "none", # run everything real...
mock_override: {
# ...except this span, which gets the value you supply
match: ->(node) { node[:span_name] == "classify_intent" },
value: { label: "refund" } # flat value
}
)
```
A callable `value` receives a context hash with the live positional `:inputs`, the live keyword `:kwargs` (empty when the call used none), and `:get_original_output` (synchronous in Ruby; its first call may fetch the recorded output lazily) to tweak the recorded output instead of replacing it:
```ruby theme={null}
value: ->(ctx) { ctx[:get_original_output].call.merge(score: 1) }
```
Register overrides on the client to apply them to every replay (keyword or positional form), and reset with `clear_mock_overrides`:
```ruby theme={null}
client.register_mock_override(
match: ->(node) { node[:type] == "llm" },
value: { label: "refund" }
)
# Positional form (equivalent): client.register_mock_override(match, value)
client.clear_mock_overrides
```
Use the keyed form when one registration belongs to a known trace function. The second argument can be a resolver or `{ match:, value: }`; for an override hash, both the registered trace function key and its `match` predicate must match:
```ruby theme={null}
client.register_mock_override(
"classify-intent",
->(ctx) { {label: ctx[:inputs].first.to_s} }
)
client.register_mock_override(
"shared-workflow",
{match: ->(node) { node[:span_name] == "Classifier" }, value: {label: "refund"}}
)
```
Pass one callable directly for a client-wide resolver that routes by trace function key. Return `Bitfab::NO_MOCK_OVERRIDE` to decline a span; `nil` remains a valid mocked output:
```ruby theme={null}
client.register_mock_override(lambda do |ctx|
if ctx[:node][:trace_function_key] == "classify-intent"
{label: ctx[:inputs].first.to_s}
else
Bitfab::NO_MOCK_OVERRIDE
end
end)
```
Precedence per span: per-call `mock_override:`, then registered overrides, then the base `mock:` strategy. `Bitfab::NO_MOCK_OVERRIDE` continues at the next override, then falls back to that base strategy. Pass a single override hash, resolver, or array.
### Fluent API: `client.get_function`
Bind a `trace_function_key` once and wrap multiple classes or methods against it. Mirrors `client.get_function` in the Python SDK and `client.getFunction` in TypeScript.
```ruby theme={null}
fn = Bitfab.client.get_function("openai")
fn.wrap(OpenAI::Client, :chat, name: "Chat", type: "llm")
fn.wrap(OpenAI::Client, :embeddings, name: "Embed", type: "llm")
```
`#wrap` accepts the same options as `Bitfab::Traceable.wrap` (`name`, `type`, `mock_on_replay`), but the `trace_function_key` is fixed to the one bound on the returned `Bitfab::BitfabFunction`.
The experimental `#trace` method binds the same key while capturing an external method's first-party call subtree:
```ruby theme={null}
fn.trace(OrderService, :process_order, type: "agent", max_spans: 200)
```
#### Adapting inputs after a signature change
Replay deserializes each trace's inputs exactly as they were captured against the method's signature **at trace time**, then calls the current method with them. If the signature drifted since capture (an argument renamed, reordered, folded into a hash, or a new required argument added), the call no longer lines up and raises. The `adapt_inputs` hook reshapes the recorded inputs onto the current signature so replay can still run:
```ruby theme={null}
# Recorded as positional (user_id, limit); current signature takes a hash.
adapt = ->(args, _kwargs, _ctx) { [[{user_id: args[0], limit: args[1]}], {}] }
result = client.replay(
service, :process_order,
trace_function_key: "order-processing",
adapt_inputs: adapt
)
```
The hook receives the deserialized `(args, kwargs)` plus a per-trace `ctx` (`{ original_trace_id:, original_span_id: }`, with deprecated `source_*` aliases) and returns `[new_args, new_kwargs]`. The returned `args` is what `item[:input]` reports. It runs once per item, **inside the same rescue as the method**: if it raises, that item's `:error` is set and the run continues, so one unmappable trace never crashes the batch.
`ctx[:original_trace_id]` (the original Bitfab trace ID) lets a table-driven adapter look up a per-trace transform. That's the escape hatch for reshapes that need judgement rather than mechanical rearrangement: compute the adapted inputs per trace up front, then have the hook look them up by `original_trace_id`, keeping replay deterministic instead of calling a model mid-replay.
When the new signature has a genuinely new **required** argument with no analog in the recorded trace, don't fabricate one. There's nothing faithful to map it to, so leave those traces unmapped (let them raise) rather than inventing test inputs.
For anything beyond a one-liner, keep the adapter in its own file next to the replay registry and require it:
```ruby theme={null}
# scripts/replay_adapters/extraction.rb
EXTRACTION_ADAPTER = ->(args, _kwargs, _ctx) do
user_id, limit = args
[[{user_id:, limit:}], {}]
end
```
```ruby theme={null}
# scripts/replay.rb
require_relative "replay_adapters/extraction"
client.replay(service, :extract_memories, trace_function_key: "my-function", adapt_inputs: EXTRACTION_ADAPTER)
```
That keeps the transform versioned and reviewable alongside the method it adapts, and you add the require only when a drift actually needs it.
#### Replay Output Contract
Replay results are typically consumed by automation (CI logs, code reviewers, and coding agents). When `BITFAB_REPLAY_RESULT_PATH` is set, `Bitfab.replay` automatically writes the full replay result JSON to that file. For direct/manual runs, **emit the full replay result hash as a single stdout JSON block** so a consumer can `JSON.parse` it and reason about every field, including the new per-item `:duration_ms`, `:tokens`, and `:model`. Never print only lengths, counts, hashes, or truncated previews, and never replace the JSON block with ad-hoc per-field log lines.
Recommended script tail:
```ruby theme={null}
result = Bitfab.client.replay(service, :process_order, trace_function_key: "order-processing", limit:)
# Human-readable summary goes to stderr, so stdout stays pure JSON.
warn "Test run: #{result[:test_run_url]}"
warn "Items: #{result[:items].length}"
# Full structured dump to stdout, ready for JSON.parse. The SDK serializer
# retains useful fields from trace_error and replay_error exception objects.
puts Bitfab.serialize_replay_result(result)
```
The dumped object includes every item's `:input`, `:result`, `:original_output`, `:error`, structured `:trace_error` and `:replay_error`, `:duration_ms`, `:tokens`, `:model`, and `:trace_id`, plus `:test_run_id` and `:test_run_url`. Use `Bitfab.serialize_replay_result`; `JSON.pretty_generate(result)` raises when a result contains an exception object. When the Bitfab plugin runs this script, it sets `BITFAB_REPLAY_RESULT_PATH`; the SDK writes the same structured JSON there, and the plugin reads that file into the replay run's `.bitfab/replays//events.jsonl` while writing large per-item payloads under `.bitfab/replays//items/`.
**Per-item errors are part of the contract.** If the wrapped method raises while executing the replayed trace, `Bitfab.replay` retains the actual exception in `item[:trace_error]`, copies its message to `item[:error]`, leaves `item[:result]` as `nil`, and continues. If replay setup fails before the method starts (for example database warmup or input loading), the actual exception is instead in `item[:replay_error]`. A database branch resolution failure is a `Bitfab::DbBranchReplayError`; inspect its `code`, message, and `original_trace_id` to distinguish failures such as `branch_create_failed`, `snapshot_from_replaced_origin`, and `invalid_snapshot_ref` without parsing `item[:error]`. A lease-endpoint HTTP, timeout, or network failure uses `lease_request_failed` and retains the original client exception as `cause`; unexpected resolver failures use `internal_error`. Treat either error kind as **unreplayable**, not as a failing output. If the whole run later raises, `Bitfab::ReplayError#items` still contains every collected item and `cause` retains the whole-run exception.
**Don't swallow per-item errors in the script.** A custom `begin/rescue` that returns a placeholder turns infra failures into fake successes. Let the SDK record them. The only allowed top-level `rescue` is a fatal handler around `main` that exits non-zero, so callers can tell a whole-replay crash from a clean run with some unreplayable items.
**Input serialization caveat.** Replay deserializes historical span inputs and passes them back to your method. This works for strings, numbers, and plain hashes. If your span wraps a method that takes hydrated domain objects (ActiveRecord models, class instances, DB records), they won't round-trip through serialization; move the span to where inputs are IDs or plain data and let the method fetch objects internally, or reshape arguments in the wrapper.
#### Replay Registry
Create a small registry module. Your project owns only the requires and registrations; installing the gem also installs the standard `bitfab-replay` executable, which owns command-line flags, lifecycle progress, code-change loading, result serialization, and summary output.
```ruby theme={null}
require "dotenv/load"
require_relative "../config/initializers/bitfab"
require_relative "replay_mocks"
require_relative "../services/extraction_pipeline"
require_relative "../services/search_pipeline"
REGISTRY = Bitfab::ReplayRegistry.new
REGISTRY.register("extraction", ExtractionPipeline.new, :extract_memories)
REGISTRY.register(
"search",
SearchPipeline.new,
:search_documents,
# Plain handler methods need the key explicitly. A bitfab_span method carries
# its key, so trace_function_key: can be omitted.
trace_function_key: "my-search-pipeline",
mock: "marked",
options_factory: lambda { |ctx|
{mock_override: ReplayMocks.search_mock(ctx.params["scenario"])}
}
)
```
Keep non-trivial executable configuration in a sibling module and require it from the registry. For example, `replay_mocks.rb` can define a mock factory whose `{ match:, value: }` result uses live replay inputs and caller-supplied parameters. `options_factory:` receives JSON values loaded from `--params ` and repeated `--param name=value`; direct parameters override file values.
Run the executable with `bundle exec bitfab-replay --registry scripts/replay_registry.rb `. The registry module must define `REGISTRY`.
| Flag | Value | Effect |
| ------------------------------------ | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `--limit` | `N` | Most traces to select. It bounds a `--trace-ids` or `--dataset-ids` selection instead of replacing it |
| `--trace-ids` | `id1,id2` | Replay exactly these traces. Mutually exclusive with `--dataset-ids` |
| `--dataset-ids` | `uuid1,uuid2` | Replay the membership of one or more datasets, as their deduped union. Mutually exclusive with `--trace-ids`. `--dataset-id` is the same flag |
| `--name` | `NAME` | Title for the resulting experiment |
| `--concurrency`, `--max-concurrency` | `N` | Items in flight at once |
| `--experiment-group-id` | `UUID` | Add this run to an existing experiment group |
| `--grader-ids` | `id1,id2` | Attach graders to this run only |
| `--code-change` | `PATH` | Load a code-change description from a file |
| `--no-code-change` | | Record no code change, overriding a registry default |
| `--mock` | `none\|all\|marked` | Which recorded child spans return their historical output |
| `--db-branch`, `--no-db-branch` | | Turn per-item database branching on or off |
| `--params` | `PATH` | JSON file of values passed to `options_factory:` |
| `--param` | `name=value` | One value passed to `options_factory:`. Repeatable, and overrides `--params` |
| `-h`, `--help` | | Print usage |
Pass static function-specific behavior such as `adapt_inputs:`, `mock_override:`, or `db_branch:` to `register`; build parameterized behavior with `options_factory:`. Command-line values override overlapping scalar defaults without removing unrelated executable options. Unknown registry option names fail when the module loads instead of reaching replay as misspelled keywords.
## Datasets
`client.datasets` creates, reads, and modifies datasets programmatically, with the same operations your coding agent reaches through the Bitfab MCP tools. A dataset is a named bucket of traces under one trace function. Experiments replay against it and its graders score its members.
```ruby theme={null}
client = Bitfab.client
saved = client.datasets.save(
trace_function_key: "checkout-agent",
name: "Refund failures",
description: "Checkout runs where the refund was declined"
)
dataset_id = saved["dataset"]["id"]
added = client.datasets.add_traces(dataset_id, [trace_id])
warn "not in this trace function: #{added["skippedTraceIds"]}" if added["skippedTraceIds"].any?
trace_ids = client.datasets.list_traces(dataset_id)["traceIds"]
client.datasets.add_graders(dataset_id, [grader_id])
run = client.datasets.rerun_graders(dataset_id)["run"]
puts run["status"], run["result"]
```
`save` is an upsert on the dataset name within its trace function, so re-running a script does not accumulate duplicates. Membership and grader calls accept up to 100 ids and report ids they skipped rather than failing the whole call. `remove_traces` only drops membership. Traces are never deleted. `rerun_graders` waits for the run by default (90 seconds, configurable) and returns whatever state it last saw. Pass `wait: false` to return immediately and poll with `get_grader_rerun`. See the [reference](/reference/ruby#datasets) for every method and result shape.
# TypeScript SDK
Source: https://docs.bitfab.ai/typescript-sdk
Instrument TypeScript AI workflows with withSpan or optional method decorators, then replay production traces against your current code
The Bitfab TypeScript SDK captures your AI function calls to automatically generate evaluations. Re-run your prompts with different models, parameters, and inputs to iterate faster.
## Installation
```bash theme={null}
# npm
npm install @bitfab/sdk
# pnpm
pnpm add @bitfab/sdk
# yarn
yarn add @bitfab/sdk
```
## Quick Start
```typescript theme={null}
import { Bitfab } from "@bitfab/sdk"
const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
```
### Choose an instrumentation style
Two decisions, in order. First **opt-out or opt-in**: `trace()` records a root and every first-party call beneath it, while `withSpan()` records exactly what you wrap. Opt-out is the recommended starting point and is still experimental; see [Instrumentation](/instrumentation) for the comparison and [Experimental subtree tracing](#experimental-subtree-tracing) below for setup.
Then, within opt-in, **wrapper or decorator**:
`withSpan` works with every supported TypeScript version and with class methods, standalone functions, class fields, accessors, and functions from other libraries.
```typescript theme={null}
const pipeline = bitfab.getFunction("document-pipeline")
class DocumentService {
async process(text: string): Promise {
return text.trim().toLowerCase()
}
}
DocumentService.prototype.process = pipeline.withSpan(
{ type: "agent" },
DocumentService.prototype.process,
)
```
Standard ECMAScript decorators are an optional shorthand for class methods.
```typescript theme={null}
const pipeline = bitfab.getFunction("document-pipeline")
class DocumentService {
@pipeline.span({ type: "agent" })
async process(text: string): Promise {
return text.trim().toLowerCase()
}
}
```
The stable `span()` decorator form requires TypeScript 5.0 or newer and a build pipeline that supports the standard ECMAScript decorator transform. Leave `experimentalDecorators` and `emitDecoratorMetadata` disabled or omitted. Use `withSpan()` in legacy-decorator projects. The experimental subtree transform accepts both standard `@bitfab.trace` / `@bitfab.node` output and the legacy output used by frameworks such as NestJS.
Need an API key? Get one from the [Bitfab dashboard](https://bitfab.ai/setup) or see the [API Keys guide](/api-keys) for detailed setup instructions.
Copy this prompt into your coding agent (tested with Cursor and Claude Code using Sonnet 4.5):
```text theme={null}
Modify existing TypeScript code to add Bitfab tracing.
Do NOT browse or web search. Use ONLY the API described below.
Bitfab TypeScript SDK (authoritative excerpt):
- Install: `npm install @bitfab/sdk` or `pnpm add @bitfab/sdk`
- Init:
import { Bitfab } from "@bitfab/sdk"
const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
- Framework integrations:
If the codebase uses LangGraph or LangChain (`@langchain/langgraph`, `@langchain/core`, or other `@langchain/*` packages), use the callback handler instead of manually wrapping graph nodes, tools, retrievers, or model calls:
const handler = bitfab.getLangGraphCallbackHandler("")
await graph.invoke(input, { callbacks: [handler] })
For plain LangChain chains, `getLangChainCallbackHandler("")` is an identical alias. The handler records a replayable root from the framework input, so no outer `withSpan` root is needed when the workflow is just the graph/chain invocation. Add a same-key outer root only for meaningful surrounding application work. Callbacks tagged `langsmith:hidden` remain local for parent resolution and are not submitted as Bitfab spans.
If replay must return recorded `ToolNode` results instead of executing live tools, use the Experimental (alpha) first-class integration:
const integration = bitfab.getLangGraphIntegration("")
const tools = integration.wrapTools(originalTools)
// Pass `tools` to both model.bindTools() and new ToolNode().
const runGraph = integration.createInvoker(graph)
Every wrapped tool is marked for recorded-output mocking by default. Use `mockToolsOnReplay: ["toolName"]` to select tools. Callback-only tool spans remain observable but cannot be short-circuited.
- Manual instrumentation (when no framework handler applies, or for meaningful work around a framework call):
// Declare trace function key once
const myService = bitfab.getFunction("")
// Recommended on every supported TypeScript version
const tracedFn = myService.withSpan({ name: "DisplayName", type: "function" }, originalFunction)
// Optional for class methods on TypeScript 5+ using standard decorators
// @myService.span({ type: "function" })
// Span types: "llm", "agent", "function", "guardrail", "handoff", "custom"
- DO NOT modify the original function.
- DO NOT extract helper methods.
Task:
1) Ensure bitfab is installed and initialization exists.
2) Read the codebase and identify ALL AI workflows (LLM calls, agent runs, AI-driven decisions). Check for LangGraph/LangChain before planning manual instrumentation.
3) Present me with a numbered list of workflows you found. For each, describe:
- What it does
- Why it's worth instrumenting -- what visibility tracing gives you into each step
4) After I choose which workflow(s) to instrument:
- If it uses LangGraph/LangChain, add the Bitfab callback handler to the framework invoke config instead of wrapping framework-managed internals. Use `getFunction("").getLangGraphCallbackHandler()` only when a same-key outer `withSpan` root is needed for surrounding application work.
- For non-framework workflows, create a key-bound handle with `bitfab.getFunction("")`
- Default to `myService.withSpan(originalFunction)` on every TypeScript version.
- If the user explicitly prefers decorators and the project uses TypeScript 5+ with the standard decorator transform, `@myService.span(options)` is available for class methods. Do not use it when `experimentalDecorators` or `emitDecoratorMetadata` is enabled.
- Produce a SPAN TREE, not one span. The workflow root gets a span, and so does every step inside it:
each model call, each read of external state (DB query, HTTP GET, storage, vector search, cache),
each transform of the model output (parsing, validation, ranking, formatting), each retry or loop
iteration, and each external write. A single span around the outer function records one input and
one output for the whole workflow, which leaves replay mocking and per-step diagnosis with nothing
to work on. Do not wrap trivial in-memory helpers or per-item work inside a large loop.
- Mark reads and writes to mock on replay so a replayed trace does not repeat side effects
- If the workflow calls the Vercel AI SDK, also wrap the model with `wrapLanguageModel({ model, middleware: bitfab.getVercelAiMiddleware("") })`. A `withSpan` around the calling function does NOT create a span for the model call.
- Replace usages of manually wrapped non-framework functions with the traced versions
5) Do not change function signature, behavior, or return value. Minimal diff.
Output:
- First: your numbered list of workflows with why each is worth instrumenting
- After my selection: minimal diffs for dependencies, initialization, and the function wrapping
```
## Basic Configuration
```typescript theme={null}
new Bitfab({ apiKey: string })
// Omit apiKey entirely: the SDK reads BITFAB_API_KEY from the environment
new Bitfab({})
// Turn capture off (functions still execute, but no spans are sent)
new Bitfab({ apiKey: string, captureEnabled: false })
```
**Capture off is not tracing removed.** With `captureEnabled: false` the wrappers stay in place and keep their trace function keys, so `replay` and `seedTrace` still work against a capture-off client. Replay records inside each item, and `seedTrace` records the one call it runs. Everything else runs untraced and sends nothing. `enabled` is a deprecated alias that warns once.
**Missing API key doesn't crash.** If the API key is missing, empty, or whitespace-only, the SDK automatically disables tracing and logs a one-time warning at first use. All wrapped functions still execute normally -- no spans are sent, no errors are thrown. You don't need any conditional logic around the API key.
### API key resolution
The key is resolved **lazily, the first time a span runs**, not when the client is constructed. This matters in scripts: with ES modules, an `import` that constructs the client is evaluated *before* the importing file's body runs `dotenv.config()`, so a key read at construction would be empty even though it is set moments later. Resolving at first use reads the key after env loading has happened.
```typescript theme={null}
// Pass a function to defer resolution explicitly (resolved at first use):
new Bitfab({ apiKey: () => process.env.BITFAB_API_KEY })
```
When no key is passed (or it resolves empty), the SDK falls back to reading `BITFAB_API_KEY` from the environment, again at first use. For standalone scripts where a run that emits no traces should be treated as a failure rather than silently skipped, set `strict`:
```typescript theme={null}
// Throws on the first traced call if no key resolves, instead of disabling quietly
new Bitfab({ apiKey: process.env.BITFAB_API_KEY, strict: true })
```
If you load env with dotenv in a script, prefer loading it before the module graph is evaluated, for example `node --env-file=.env script.ts`, so every module-level read sees the key.
## Tracing
### Experimental subtree tracing
**Experimental.** `trace()` and `withTrace()` subtree expansion may change in
a future release. `withSpan()` is the stable explicit instrumentation API.
Opt-out is the recommended starting point despite that, because the trace
shape you get by default is the one you want; see [Instrumentation](/instrumentation).
For server-side TypeScript, Bitfab can record one root plus the first-party function calls beneath it, at any depth, without wrapping every function. Install the experimental build package and add one adapter at the point where your server code is first compiled:
```bash theme={null}
pnpm add -D @bitfab/transform
```
If you previously installed `bitfab-transform`, replace that dependency with
`@bitfab/transform`. Update imports and build configuration from
`bitfab-transform/` to `@bitfab/transform/`.
| Build path | Integration |
| ---------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Next.js 16 with Turbopack or webpack | Wrap `next.config` with `@bitfab/transform/next` |
| NestJS with the webpack builder | Set `builder` to `webpack` and use `@bitfab/transform/nest` as the custom webpack config |
| Vite, Astro, Nuxt's Vite builder, React Router | Add `@bitfab/transform/vite` to the server-side Vite plugins |
| webpack, Rspack, Rsbuild | Add `@bitfab/transform/webpack`, `/rspack`, or `/rsbuild` to `plugins` |
| Rollup or Rolldown | Add `@bitfab/transform/rollup` or `/rolldown` to `plugins` |
| esbuild or Bun | Add `@bitfab/transform/esbuild` or `/bun` to `plugins` |
| Babel | Add `@bitfab/transform/babel` before decorator-lowering plugins |
| SWC | Use `transformWithBitfabSwc` from `@bitfab/transform/swc` before SWC lowers decorators |
| TypeScript compiler API | Add `createBitfabTypeScriptTransformer` from `@bitfab/transform/typescript` to `transformers.before` |
| Direct Node TypeScript, including `tsx` | Start Node with `--import @bitfab/transform/register` |
Next.js needs only a config wrapper. It instruments Node server modules in both Turbopack and webpack builds, while leaving client and Edge bundles untouched:
```typescript next.config.ts theme={null}
import type { NextConfig } from "next"
import withBitfabTrace from "@bitfab/transform/next"
const nextConfig: NextConfig = {}
export default withBitfabTrace(nextConfig)
```
Vite and its server-side meta-frameworks use the ordinary Vite plugin shape:
```typescript vite.config.ts theme={null}
import bitfabTrace from "@bitfab/transform/vite"
import { defineConfig } from "vite"
export default defineConfig({ plugins: [bitfabTrace()] })
```
For a direct TypeScript process, the registration hook both instruments and compiles repository modules. With `tsx`, register `tsx` first:
```bash theme={null}
node --import @bitfab/transform/register src/server.ts
node --import tsx --import @bitfab/transform/register src/server.ts
```
In a monorepo, run from the repository root to include shared packages in
capture. The loader resolves its runtime from the SDK available to the transform
installation, even when a shared package has no SDK dependency. To select a
different application's SDK, set `BITFAB_SDK_RESOLVE_FROM` to its `package.json`:
```bash theme={null}
BITFAB_SDK_RESOLVE_FROM=./packages/actor/package.json node --import ./packages/actor/node_modules/@bitfab/transform/dist/register.js packages/actor/src/server.ts
```
This selects the SDK installation; the working directory still determines
capture scope and relative function IDs. It works for both ESM and CommonJS.
Nest has a dedicated config wrapper. Select its webpack builder in `nest-cli.json`:
```json nest-cli.json theme={null}
{
"compilerOptions": {
"builder": "webpack",
"webpackConfigPath": "webpack.config.cjs"
}
}
```
Then append the transform to Nest's generated webpack configuration:
```javascript webpack.config.cjs theme={null}
const bitfabNestTrace = require("@bitfab/transform/nest").default
module.exports = bitfabNestTrace()
```
A Nest application using the `tsc` or SWC builder still records the rich root but does not gain automatic descendants from this adapter. Plain `tsconfig.json` cannot load a TypeScript transformer. Complete runnable transform configs, direct-Node and Next.js fixtures, and a real Nest server fixture with OTLP delivery assertions live in `bitfab-typescript-example/auto-trace/`.
The root APIs are ordinary runtime wrappers; the transform does not rewrite their arguments or change their behavior. Without a transform, they still record one normal rich root span and run the code unchanged. They cannot discover calls beneath the root unless a compatible transform instruments those functions.
For a class method, use the normal method decorator:
```typescript theme={null}
import { Bitfab } from "@bitfab/sdk"
const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
class SupportAgent {
@bitfab.trace("support-agent", {
name: "support-agent-root",
type: "agent",
mockOnReplayDefault: true,
maxDepth: 30,
maxSpans: 500,
exclude: ["debugPayload"],
})
async run(message: string) {
const request = await this.prepare(message)
return this.generate(request)
}
@bitfab.node({ mockOnReplay: false })
private async prepare(message: string) {
return { message: message.trim() }
}
@bitfab.node({ name: "Model call", type: "llm" })
private async generate(request: { message: string }) {
return callModel(request)
}
}
```
For function-oriented code, use `withTrace`:
```typescript theme={null}
export const runSupportAgent = bitfab.withTrace(
"support-agent",
{
type: "agent",
mockOnReplayDefault: true,
maxDepth: 30,
maxSpans: 500,
},
async function runSupportAgent(message: string) {
const request = await prepare(message)
return generate(request)
},
)
const generate = bitfab.withNode(
{ name: "Model call", type: "llm" },
async function generate(request: { message: string }) {
return callModel(request)
},
)
```
Both root forms create the same rich root span and call-scoped subtree context. Most calls require no Bitfab wrappers. Use `node()` for a transformed method or `withNode()` for a transformed standalone function only when one discovered call needs explicit naming, typing, capture, finalization, or replay-mocking policy. A configured call outside the enclosing subtree trace runs normally and never creates a span or trace. The build adapter rewrites eligible first-party functions because imports, callbacks, dependency injection, and virtual dispatch make the runtime call tree impossible to predict statically. Each rewritten function checks the call-scoped context first. Outside the traced invocation, it runs its original body directly without emitting a span or constructing span metadata, captured inputs, or an invocation closure. A capture-off client or a client without a resolvable API key also runs the root directly: it does not request capture policy or activate automatic trace context.
The options follow the same subtree model as Python:
* `name` and `type` control the rich root span. Unconfigured descendants are `function` spans.
* `mockOnReplayDefault: true` makes replay mocking the default for nodes under the trace's default `mock: "marked"` replay strategy. A configured node with `mockOnReplay: false` overrides it. The option is off by default.
* `maxDepth` (default 30) and `maxSpans` (default 500) bound each invocation. Hitting either limit warns once and marks the trace metadata as truncated because the emitted tree is incomplete.
* `exclude` accepts simple or qualified function names. Skipping is transparent: calls below an excluded function still parent to the nearest recorded caller.
* Anonymous callbacks are omitted. Rest-argument wrapper functions are omitted unless `includeWrappers: true` is set.
Each invoked function in a transformed server module becomes a real nested span. Automatic child spans capture their full inputs, outputs, and thrown errors by default, with no `node()` annotation or Studio setup. Inputs are the original call arguments supplied at the function boundary, including the complete objects and arrays used by destructured parameters, not reconstructed bindings or placeholders. Each span also carries timing, parentage, its qualified name, and a stable source-derived function ID.
`node()` and `withNode()` apply trace-owned configuration to one discovered call. `capture: true` (the default) makes that call rich and applies `name`, `type` (default `custom`), `testRunId`, `mockOnReplay`, and `finalize`. `capture: false` omits the call and transparently attaches its captured descendants to the nearest captured parent. Combining `capture: false` with `mockOnReplay: true` throws because an omitted call has no recorded output.
The runtime protocol is transform-agnostic: every adapter emits the same calls to `@bitfab/sdk/auto`, so application code and versioned, lexical function IDs do not change when the build tool changes. Re-running a transform is safe: generated modules carry an idempotence marker and are not instrumented twice.
Bitfab's installed SDK implementation is a dependency and is never transformed. Functions passed directly or by reference to `withSpan()` and `withTrace()` are also left alone because those wrappers already own their span. This prevents an explicit span from gaining an automatic duplicate. Functions passed to `withNode()` remain eligible because the enclosing subtree trace still owns their span. Named repository functions called inside an explicit span remain part of the automatic tree.
**Nested `withTrace()` roots.** A `withTrace()` or `trace()` root entered beneath another one starts a separate trace while the outer root keeps recording. The nested root's function and every call beneath it appear in both traces with separate span IDs, and each trace has the same shape it would record alone, so you can test either boundary of an agent on its own. Two overlapping roots therefore double span volume in the region they share, and each root applies its own limits, exclusions, and capture policy to its copy. A `node()` or `withNode()` configuration applies in every root's copy, including `testRunId` and the finalized output, and `finalize` runs once per call; framework spans (the OpenAI Agents processor, the Vercel AI SDK middleware, the LangGraph integration) attach inside the innermost trace only. The outer trace's span for the nested root carries `nested_trace_id`, `nested_trace_function_key`, and `nested_root_span_id`, and the nested root span carries `enclosing_trace_id`, `enclosing_span_id`, and `enclosing_trace_function_key`, so the two traces point at each other. In the trace viewer the enclosing trace's span for the nested root shows a lip naming the nested trace function. It opens that trace at its root span. The nested trace's root shows a lip back to the enclosing trace function that opens the span that started it.
```typescript theme={null}
const buildTicketDetail = bitfab.withTrace("ticket-detail", (ticket: Ticket) =>
enrichTicket(ticket),
)
const processTicket = bitfab.withTrace("ticket-workflow", (ticket: Ticket) =>
buildTicketDetail(ticket),
)
await processTicket(ticket)
// Records two traces:
// ticket-workflow -> buildTicketDetail -> enrichTicket
// ticket-detail -> enrichTicket
```
Inside `replay()` or `seedTrace()` a nested root starts no trace of its own. The item's trace records it as an ordinary descendant, so an experiment never gains traces under a second trace function key.
#### Mixing opt-in and opt-out tracing
`withSpan()` is the opt-in surface: it records exactly the functions you wrap. `withTrace()`, `trace()`, `node()`, and `withNode()` are the opt-out surface: they record a root plus the first-party calls beneath it. Pick one per workflow. Entering one beneath the other throws `MixedTracingError` naming both surfaces:
```
MixedTracingError: opt-in and opt-out tracing can't be mixed in one call stack: withSpan() (opt-in)
for "fetch-context" was entered inside a withTrace() call (opt-out). Inside a withTrace/trace subtree,
configure a discovered call with node()/withNode() instead, or trace this workflow with withSpan() only.
```
A blended stack produces a trace whose shape misrepresents how the code is instrumented, and the replay boundaries stop being predictable, so the SDK stops rather than recording it. To resolve it, pick one surface per workflow. Inside a subtree trace, reach for `node()` or `withNode()` when a discovered call needs its own name, type, capture, finalization, or replay-mocking policy: they configure the call the trace already owns rather than opening a second surface. Going the other way, wrap the caller with `withTrace()` too, or drop back to `withSpan()` throughout.
Wanting a `withSpan()` root with a subtree lower down is really just a subtree with a smaller root. Move `withTrace()` to the function you wanted the subtree under.
`node()` and `withNode()` follow the same rule. Beneath a `withSpan()` with no enclosing trace there is nothing for them to configure, so they throw rather than run as a silent no-op. With no tracing active at all they still run the function untouched.
The root span that `replay()` and `seedTrace()` wrap around an undecorated callable is the one exception. It belongs to neither surface, so a `withTrace()` root called from that callable nests beneath it instead of throwing, matching Python.
The Python SDK raises `MixedTracingError` in the same places. Framework adapters are unaffected. The LangGraph, OpenAI Agents, and Vercel AI SDK wrappers open their spans on the surrounding surface rather than declaring one, and the callback-based handlers emit spans directly, so an adapter instrumenting part of your stack never trips the check.
Studio configuration is optional. In a trace's header, choose **Configure capture** only when you want later traces to keep rich content for a narrower set of discovered functions. Running SDK processes refresh a confirmed policy in the background within about one minute; a trace snapshots its selection when it begins, so an in-flight trace never changes capture behavior halfway through. The policy request never blocks application code. If no confirmed policy exists, or the first policy request fails before any policy has loaded, full capture remains the default. Once a policy has loaded, refresh failures preserve it and retry later. Source-level `exclude` and `capture: false` opt-outs remain authoritative.
Without a compatible transform, both root APIs still emit their normal rich root and run unchanged; automatic descendants are absent. Compatibility requires the adapter to see your original server-side TypeScript or JavaScript before decorators are lowered. Generator function definitions, JavaScript `#private` methods, getters/setters, dependency modules, declarations, `"use client"` modules, and Edge bundles are currently skipped. Async-generator roots preserve automatic context while their results are consumed, so eligible regular functions called from the generator body still appear.
**Trace the whole workflow, not just its entrypoint.** A single decorator or `withSpan` wrapper on the outer function records one input and one output for everything inside it, which leaves replay mocking, per-step diagnosis, and prompt iteration with nothing to work on. Spans exist only where you create them: nesting is automatic, but only between spans that exist.
Give a step its own span when any of these is true:
* **It calls a model.** Always. This is the span you iterate on, compare across experiments, and attach graders to.
* **It reads external mutable state** (DB query, HTTP `GET`, object storage, vector search, cache). These are the spans you will want to mock on replay.
* **It writes external state** (DB write, queue publish, email, charge, file write). Mark these to mock on replay so a replayed trace does not repeat the side effect.
* **It transforms the model output** (parsing, validation, ranking, formatting), so a quality regression points at the model or at your post-processing.
* **It retries or loops**, one span per attempt or iteration, so a trace shows how many attempts it really took.
Skip trivial in-memory helpers, per-item work inside a large loop (wrap the loop or the batch), and internals a [framework integration](/frameworks/overview) already captures.
Worked examples, replay-mocking decisions, and common pitfalls: [Instrumentation](/instrumentation).
### Declaring the trace function key
#### Using `getFunction()` to Link Spans
Declare the trace function key once and wrap multiple functions. This is the recommended form on every supported TypeScript version:
```typescript theme={null}
const orderService = bitfab.getFunction("order-processing")
async function loadOrder(orderId: string) {
return db.orders.findById(orderId)
}
const tracedLoadOrder = orderService.withSpan({ mockOnReplay: true }, loadOrder)
async function classifyOrder(order: Order) {
return generateObject({ model, schema, prompt: buildPrompt(order) })
}
const tracedClassifyOrder = orderService.withSpan({ type: "llm" }, classifyOrder)
async function validateOrder(classification: Classification) {
return { valid: classification.confidence > 0.8 }
}
const tracedValidateOrder = orderService.withSpan(validateOrder)
async function processOrder(orderId: string) {
const order = await tracedLoadOrder(orderId)
const classification = await tracedClassifyOrder(order)
return tracedValidateOrder(classification)
}
const tracedProcessOrder = orderService.withSpan({ type: "agent" }, processOrder)
```
Calling `tracedProcessOrder(id)` records one trace with four spans:
```
processOrder
├── loadOrder
├── classifyOrder
└── validateOrder
```
Wrapping only `processOrder` would record the same work as a single node, with the model call, the database read, and the validation collapsed into the root's input and output.
#### Multi-File Projects
For projects with instrumented functions spread across multiple files, create a dedicated file that initializes Bitfab and exports the function. Import it wherever you need to instrument.
```typescript theme={null}
// lib/bitfab.ts -- single source of truth
import { Bitfab } from "@bitfab/sdk"
const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
export const orderService = bitfab.getFunction("order-processing")
```
```typescript theme={null}
// services/validateOrder.ts
import { orderService } from "../lib/bitfab"
async function validateOrder(orderId: string) {
return { valid: true }
}
export const tracedValidateOrder = orderService.withSpan(validateOrder)
```
```typescript theme={null}
// services/processOrder.ts
import { orderService } from "../lib/bitfab"
import { tracedValidateOrder } from "./validateOrder"
async function processOrder(orderId: string) {
await tracedValidateOrder(orderId)
return { orderId }
}
export const tracedProcessOrder = orderService.withSpan(processOrder)
```
Spans from different files are automatically linked as parent-child when one wrapped function calls another.
#### Wrapping Existing Functions Inline
When wrapping a function you didn't define (e.g. an SDK or library call), pass it directly to `withSpan` and call the result immediately. This ensures the arguments are captured as span input.
```typescript theme={null}
// ✅ GOOD -- pass function directly, arguments are captured as span input
const result = await orderService.withSpan(
{ name: "ProcessOrder", type: "function" },
processOrder,
)(orderId)
// ❌ BAD -- anonymous wrapper loses all input capture (span has no input)
const result = await orderService.withSpan(
{ name: "ProcessOrder", type: "function" },
async () => processOrder(orderId),
)()
```
Never wrap functions in an anonymous function like `async () => fn(args)`. The SDK captures the wrapper function's arguments as span input -- an anonymous wrapper has no arguments, so the span records nothing.
#### Using `withSpan()` Directly
For a single span without linking to a function group:
```typescript theme={null}
const standaloneTask = bitfab.withSpan("one-off-operation", () => {
return "done"
})
```
#### Automatic Nesting
Spans nest automatically based on call stack:
```typescript theme={null}
const outer = bitfab.withSpan("outer", { type: "agent" }, async () => {
await inner() // Becomes a child of "outer"
})
const inner = bitfab.withSpan("inner", { type: "function" }, async () => {
// ...
})
```
For reusable helpers that should appear only inside an existing trace, use `captureWhen: "nested"`:
```typescript theme={null}
const helper = bitfab.withSpan(
"workflow",
{ type: "function", captureWhen: "nested" },
async (input: string) => transform(input),
)
const workflow = bitfab.withSpan("workflow", async (input: string) => {
return helper(input)
})
await helper("standalone") // Runs normally without creating a trace
await workflow("nested") // Captures helper as a child
```
#### Method Decorators (TypeScript 5+)
Decorators are an optional shorthand for class methods. `withSpan()` remains the recommended default because it works with every supported TypeScript version and every kind of callable.
**Requirements**
Use decorators only when all of these requirements are met:
* The project compiles with TypeScript 5.0 or newer
* Its compiler or transpiler supports the standard ECMAScript decorator transform
* `experimentalDecorators` is disabled or omitted
* `emitDecoratorMetadata` is disabled or omitted
The stable `span()` decorator does not support TypeScript's older legacy decorator transform. Projects that use legacy decorators or emitted decorator metadata should continue using `withSpan()`. The experimental subtree `trace()` and `node()` decorators are separate APIs and support legacy method decorator output when `@bitfab/transform` runs first.
**Bind methods to one trace function**
Use `getFunction()` when several methods belong to the same traced workflow. Calls nest automatically when one decorated method calls another:
```typescript theme={null}
const pipeline = bitfab.getFunction("document-pipeline")
class DocumentService {
@pipeline.span({ type: "agent" })
async process(text: string): Promise {
return this.#normalize(text)
}
@pipeline.span({ type: "function", captureWhen: "nested" })
async #normalize(text: string): Promise {
return text.trim().toLowerCase()
}
}
```
Calling `new DocumentService().process(text)` records one trace with `process` as the root and `#normalize` as a nested call.
**Decorate a method directly**
Use `@bitfab.span(traceFunctionKey, options)` when a class has only one method to trace or you do not need a key-bound `getFunction()` handle:
```typescript theme={null}
class DocumentClassifier {
@bitfab.span("document-classification", { type: "llm" })
static async classify(text: string): Promise {
return classifyDocument(text)
}
}
```
**Supported methods and limitations**
Decorators work on instance, static, and private methods. Arguments, return values, errors, nesting, span options, and `this` are handled exactly as they are with `withSpan()`.
Continue using `withSpan()` for:
* Standalone functions
* Class fields and accessors
* Functions from other libraries
* TypeScript 4.x projects
* Projects using the legacy decorator transform
Projects using TypeScript 4.x can continue using every non-decorator API, including `withSpan()`, `getFunction()`, framework handlers, and replay. The published SDK declarations do not reference TypeScript 5-only global types. Only the `@...span()` syntax requires TypeScript 5.0 or newer.
#### Span Options
**Parameters:**
* `traceFunctionKey` (required): String identifier for grouping spans
* `name` (optional): Display name. Defaults to the function's qualified name (`Order.process` for a method, `process` for a plain function), then the trace function key
* `type` (optional): Span type. Defaults to `"custom"`. A label only, used to organize and filter spans in the dashboard; it does not change how the span is traced, replayed, or evaluated
* `captureWhen` (optional): `"always"` (default) or `"nested"`. Nested-only spans are captured under an active parent and run untraced when called standalone. Unknown values warn once and default to `"always"`
* `testRunId` (optional): Link the span and, when it is the root, its trace to a test run
* `mockOnReplay` (optional): Return this call's recorded output under the default `mock: "marked"` replay strategy
* `finalize` (optional): `(result) => serializableView`. Record a serializable view of a non-serializable result (a live stream). See [Tracing streaming functions](#tracing-streaming-functions)
**Span Types:**
```typescript theme={null}
type SpanType =
| "llm" // LLM calls
| "agent" // Agent workflows
| "function" // Function calls
| "guardrail" // Safety checks
| "handoff" // Human handoffs
| "custom" // Default
```
**Examples:**
```typescript theme={null}
// Function name is automatically captured as span name
async function processOrder(orderId: string) {
return { orderId }
}
const traced = bitfab.withSpan("order-processing", processOrder)
// Span name: "processOrder"
// Override with name option
const traced = bitfab.withSpan(
"order-processing",
{ name: "OrderProcessor" },
processOrder
)
// Span name: "OrderProcessor"
// Set span type
const checkSafety = bitfab.withSpan(
"safety-check",
{ type: "guardrail" },
async (content: string) => ({ safe: true })
)
// With getFunction()
const service = bitfab.getFunction("order-processing")
const traced = service.withSpan({ name: "CustomName", type: "function" }, processOrder)
```
#### Tracing Streaming Functions
A streaming function returns a live stream object that the caller consumes directly (an SSE response, a UI message stream). That object isn't serializable as a trace output, and awaiting it to completion before returning would break streaming and first-byte latency. The `finalize` option solves this: `withSpan` hands the live stream back to the caller **unchanged**, but records `await finalize(result)` as the span output, a drained, serializable, replayable value such as `{ text, usage, toolCalls }`.
For the Vercel AI SDK, use the prebuilt `finalizers.aiSdk` helper. Reading the result's `text` / `totalUsage` promises does not consume the live stream (the AI SDK tees internally), so your own streaming is unaffected. (Use both together: the [middleware](/frameworks/vercel-ai-sdk) records the model call as a child span, `finalizers.aiSdk` records the streamed output on the root. The middleware alone, with no root, records one `llm` span per call and no workflow around it.)
```typescript theme={null}
import { openai } from "@ai-sdk/openai"
import { streamText, wrapLanguageModel } from "ai"
import { finalizers } from "@bitfab/sdk"
// Wrap the model too, or the root is the only span in the trace
const model = wrapLanguageModel({
model: openai("gpt-4o"),
middleware: bitfab.getVercelAiMiddleware("chat-turn"),
})
const runChatTurn = bitfab.withSpan(
"chat-turn",
{ type: "agent", finalize: finalizers.aiSdk },
() => streamText({ model, messages }),
)
// In your route handler:
const result = runChatTurn() // live StreamTextResult
return result.toUIMessageStreamResponse() // stream to the user as usual
// The span records { text, usage, finishReason, toolCalls } in the background.
```
The root and the model call share the `chat-turn` key, so the model span nests beneath the root:
```
chat-turn
└── chat-turn (llm)
```
Without `wrapLanguageModel`, `runChatTurn` records a single node: the model call happens inside the span but is not one, so there is no recorded prompt, no per-call token usage, and nothing to mock on replay.
Provide your own `finalize` to record a specific shape:
```typescript theme={null}
const runChatTurn = bitfab.withSpan(
"chat-turn",
{
type: "agent",
finalize: async (r) => ({ answer: await r.text, tokens: await r.totalUsage }),
},
() => streamText({ model, messages }),
)
```
For a raw `ReadableStream`, use `finalizers.readableStream`, which `tee()`s the stream and collects its chunks; the caller must use the live branch it hands back:
```typescript theme={null}
import { finalizers } from "@bitfab/sdk"
let live: ReadableStream
const traced = bitfab.withSpan(
"render",
{ finalize: (r) => finalizers.readableStream(r, (s) => { live = s }) },
() => makeReadableStream(),
)
traced()
return new Response(live!)
```
`finalize` runs in the background and never affects the caller's value; a `finalize` that throws records an error on the span instead of crashing the host. It is ignored for async-generator results, which are captured automatically. Inputs to the wrapped function must still be serializable for the trace to replay.
An async generator has two async chains: the service produces values and the controller consumes them. To include spans from both chains in one trace, make the controller the outer root and trace the generator as its child:
```typescript theme={null}
const chatTurn = bitfab.getFunction("chat-turn")
const streamFromService = chatTurn.withSpan({ name: "stream-service" }, async function* () {
yield "first"
yield "second"
})
const handleChunk = chatTurn.withSpan({ name: "handle-chunk" }, async (chunk: string) => chunk)
const runController = chatTurn.withSpan({ name: "controller", type: "agent" }, async () => {
for await (const chunk of streamFromService()) {
await handleChunk(chunk)
}
})
await runController()
```
#### Span Context
Use `getCurrentSpan()` to get a handle to the active span, then call `.addContext()` to attach contextual key-value pairs from inside a traced function -- useful for runtime values like request IDs, computed scores, or dynamic context:
```typescript theme={null}
import { getCurrentSpan } from "@bitfab/sdk"
async function processOrder(orderId: string) {
const userId = await getCurrentUser()
getCurrentSpan()?.addContext({ user_id: userId, order_id: orderId })
return { orderId, status: "completed" }
}
const traced = bitfab.withSpan("order-processing", { type: "function" }, processOrder)
```
Each `addContext` call pushes the entire object as one entry. Multiple calls accumulate entries:
```typescript theme={null}
getCurrentSpan()?.addContext({ user_id: "u-123" })
getCurrentSpan()?.addContext({ request_id: "req-789" })
// Result: contexts: [{ user_id: "u-123" }, { request_id: "req-789" }]
```
#### Span IDs
Access the canonical Bitfab span and trace IDs from `getCurrentSpan().id` and `getCurrentSpan().traceId`. These are useful for persisted lookups, replay, or logging:
```typescript theme={null}
import { getCurrentSpan } from "@bitfab/sdk"
const traced = bitfab.withSpan("my-function", async () => {
const { id, traceId } = getCurrentSpan()
console.log("Current span:", id, "trace:", traceId)
return { id, traceId }
})
```
Outside a span context, both IDs are empty strings.
#### Span Prompt
Use `getCurrentSpan()` to set the prompt string on the current span. This is stored in `span_data.prompt` and is useful for capturing the exact prompt text sent to an LLM:
```typescript theme={null}
import { getCurrentSpan } from "@bitfab/sdk"
async function classifyText(text: string) {
const prompt = `Classify the following text: ${text}`
getCurrentSpan()?.setPrompt(prompt)
const result = await llm.complete(prompt)
return result
}
const traced = bitfab.withSpan("classification", { type: "llm" }, classifyText)
```
The prompt is metadata only. It records the prompt text for display and reference in the dashboard; it does not send the prompt to any model or change what the span executes. The last `setPrompt` call wins -- it overwrites any previously set prompt on the span. Calling `setPrompt` outside a span context is a no-op (it never crashes).
#### Framework Integrations
Bitfab provides automatic tracing for popular AI frameworks. See the dedicated guides for full API references:
Callback tracing plus Experimental (alpha) `ToolNode` output mocking for replay
Trace processor for agent runs
Auto-capture prompts and LLM metadata
Capture LLM turns, tool calls, and subagents
Language model middleware for every generateText / streamText call
#### Trace Context
Use `getCurrentTrace()` to set context that applies to the entire trace (all spans within a single execution). This is useful for grouping traces by session or attaching trace-level metadata:
```typescript theme={null}
import { getCurrentTrace } from "@bitfab/sdk"
const traced = bitfab.withSpan("order-processing", { type: "function" }, async () => {
const trace = getCurrentTrace()
// Set session ID (stored as database column, filterable in dashboard)
trace?.setSessionId("session-123")
// Name the trace (its title in Bitfab, searchable and filterable)
trace?.setName("Order 8f21")
// Set trace metadata (stored in raw trace data)
trace?.setMetadata({ region: "us-west-2", environment: "production" })
// Add context entries (stored as key-value pairs, accumulates across calls)
trace?.addContext({ workflow: "checkout-flow", batch_id: "batch-2024-01" })
return { status: "completed" }
})
```
* `setSessionId(id)` -- Groups traces by user session. Stored as a database column for efficient filtering.
* `setName(name)` -- The trace's title in Bitfab, and a field you can search and filter on. Use it for the case, ticket, or record the run is about. Unset, the trace is titled by its trace function key.
* `setMetadata(obj)` -- Arbitrary key-value metadata on the trace. Merges with existing metadata.
* `addContext(obj)` -- Key-value context entries. Accumulates across multiple calls.
#### Dropping a Trace
Call `.drop()` on the current-trace handle to discard the in-flight trace. Once flagged, spans that complete afterward are not uploaded at all, and the flag rides out on the completion payload, so when the trace completes the server scrubs any payloads that already raced out (the trace, its external trace, and sibling spans), deletes the archived S3 objects, and marks it `dropped` instead of `completed`, keeping only a skeleton audit row. Use it to discard runs you never want stored (health checks, test traffic) or a run you know carries sensitive data.
```typescript theme={null}
import { getCurrentTrace } from "@bitfab/sdk"
const traced = bitfab.withSpan("order-processing", { type: "function" }, async () => {
if (isHealthCheck) {
getCurrentTrace()?.drop()
}
return { status: "completed" }
})
```
* Safe to call outside a trace (a no-op), and never throws into your application.
#### Detached Trace
Use `client.getTrace(traceId)` to get a handle to a trace that has already closed. This lets you add context, merge metadata, or set the session ID from any process, thread, or agent that knows the trace ID, with no shared in-memory state.
```typescript theme={null}
const trace = client.getTrace(traceId)
await trace.addContext({ refund_status: "approved" })
await trace.setMetadata({ region: "us-west" })
await trace.setSessionId("session_xyz")
```
The `traceId` is Bitfab's canonical trace ID, the same UUID exposed by `getCurrentSpan().traceId` for native SDK traces and used in Bitfab trace URLs. All methods block: each resolves once the server has applied the change and rejects if the server refused it, the same way `getTraceSpan` behaves. If you were ignoring the returned promise, start awaiting it - a rejected update (for example one naming a trace ID that does not exist) now surfaces to you instead of being logged and dropped.
* `addContext(context)` -- Appends a context entry. Existing entries are preserved.
* `setMetadata(metadata)` -- Shallow-merges new keys into existing metadata.
* `setSessionId(sessionId)` -- Replaces any existing session ID.
#### Read One Persisted Span
Use `getTraceSpan` to fetch one span without loading the full trace. Both the trace ID and exact span ID are canonical Bitfab IDs; ingestion source IDs are not accepted. Repeated name matches default to the last span.
```typescript theme={null}
const span = await client.getTraceSpan(traceId, { name: "GenerateAnswer" })
const first = await client.getTraceSpan(traceId, {
name: "GenerateAnswer",
occurrence: "first",
})
const exact = await client.getTraceSpan(traceId, { id: spanId })
```
`occurrence` also accepts a zero-based integer. A missing trace or span returns `null`.
#### Error Handling
Errors are captured in the span and re-raised:
```typescript theme={null}
const risky = bitfab.withSpan("risky-service", () => {
throw new Error("error")
})
try {
risky()
} catch (e) {
// Span records error and timing
}
```
Each error is classified by source. Errors thrown by your code are recorded with `error_source: "code"`. SDK-internal errors (e.g. serialization failures) are recorded with `source: "sdk"`. Both appear in the span's `errors` array in the Bitfab dashboard.
#### Span Delivery
Spans are batched and delivered in the background, so tracing never sits on
your request path. To make sure everything reached Bitfab before a script or
test exits:
```typescript theme={null}
import { flushTraces } from "@bitfab/sdk"
if (!(await flushTraces(30_000))) {
throw new Error("Bitfab traces were not delivered before the deadline")
}
```
The return value is `false` when an export fails or the deadline expires.
Traces also flush automatically on process exit via a `beforeExit` hook.
#### OpenTelemetry Transport
| Variable | Default | Purpose |
| -------------------------------- | --------- | --------------------------------------------- |
| `BITFAB_OTEL_EXPORT_CONCURRENCY` | `32` | Concurrent direct requests, `1` through `64`. |
| `BITFAB_OTEL_MAX_REQUEST_BYTES` | `3000000` | Request-size target. Can only be lowered. |
The SDK lazily creates one private OpenTelemetry provider and bounded
`BatchSpanProcessor` per client. It does not replace your application's global
OTel provider, and an unused client starts no OTel worker. `withSpan` and every
framework handler submit the same replay-safe Bitfab payloads through one
transport interface. Batches are sent to Bitfab as OTLP/JSON. Each carrier is encoded once and the
request body is assembled from those encodings, so a batch is never re-encoded
to measure its size.
Live and replay traces share the same OTel pipeline. Before completing a replay
test run, the SDK flushes OTel and uses delivery acknowledgments to confirm
every carrier that reached Bitfab. If any delivery is uncertain, it polls
Bitfab until every submitted replay trace completion and expected span count is
persisted.
For the full ownership model, carrier format, live and replay flows, batching
limits, lifecycle, and failure semantics, see
[OpenTelemetry Transport Architecture](/otel-architecture).
The SDK partitions count-based OTel exports into requests of at most about
3 MB. Each request contains at most 128 carriers and is packed using their
exact encoded size, and up to 32 are sent concurrently
(`BITFAB_OTEL_EXPORT_CONCURRENCY`, 1-64). Set
`BITFAB_OTEL_MAX_REQUEST_BYTES` to a positive integer no greater than `3000000`
to use a smaller target for a stricter proxy; unsafe values warn
and fall back to `3000000`. An oversized carrier gets its own request and uses
the compression and trimming fallback described below. If it still cannot fit
or ingress rejects it with HTTP 413, the SDK reports an export failure.
If Bitfab rejects malformed carriers from an
otherwise valid direct batch, the standard OTLP `partialSuccess` response is
logged with the rejected-span count and reason.
A single span may use up to 7,800,000 carrier bytes when its dedicated request
gzips below the 3,000,000-byte wire target and remains below the 8,000,000-byte
decompressed ingress limit. The carrier is the payload re-escaped into the OTLP
attribute. If it does not compress enough, compression is unavailable, or it
exceeds the raw ceiling, the SDK replaces its largest fields with
`` placeholders until it fits the
2,800,000-byte fallback budget. The trim is recorded on the span's `errors` so
the trace is flagged as incomplete.
For transient clients in long-running processes, call `client.close(30_000)`
when finished. Closing is idempotent: it flushes and shuts down only that
client's OTel worker, which is also reused by framework handlers created from
that client. A shared application client can remain open and will still shut
down automatically at process exit. Handlers you construct directly, without a
`Bitfab` client, own their worker and expose their own `close()`.
### Content capture from the sim plan
The client reads your organization's sim plan in the background and applies it
to every span it sends. A span whose content is turned off in the plan still
records its name, type, timing, errors, contexts, and links to other traces,
but not its inputs and outputs, and it carries
`content_off_by_simulation_plan: true` so Bitfab knows why the content is
missing. The plan is matched by the trace's root trace function key and the
span's name. The read starts as soon as you wrap a function or call
`getFunction`, is given a five second timeout, and it never blocks a traced
call. A span sent before the first read has succeeded is held back and sent
once the plan arrives, with the plan applied, so no span ever leaves the
process with content the plan turned off, not even the first one; held spans
go out on `flushTraces()`, on `close()`, and at exit, as soon as the plan has
loaded, and while records are held those paths wait up to the five second
read timeout for the plan before giving up on them. At most 1,000 records are
held per client, and past that the oldest
are dropped with a one-time warning. A trace's completion waits behind
whichever of that trace's spans are held, and goes out right away when none
are, so a trace made only of framework spans is never held back. A failed read
is retried every ten seconds while a record is held, and otherwise on the next
span: before the first success that keeps spans held, after it the last plan
stays in effect. The plan is refreshed about once a minute while spans flow,
so a change takes effect within one refresh. Spans recorded by a framework
integration (the OpenAI Agents processor, the LangGraph handler and
integration, the Claude Agent SDK handler, the Vercel AI SDK middleware)
always keep their content, and the plan cannot turn those off; every span
carries a `span_origin` record (the SDK name and version, and
`instrumentation.name` saying what recorded it). A server that has no sim plan
feature answers the read with 404, which counts as an empty plan loaded:
nothing is held and nothing is stripped. Pass `simulationPlan: false` to the
client to turn the plan off entirely (no read, nothing held back, nothing
stripped), the same as setting `BITFAB_DISABLE_SIM_PLAN` to any value that is
not empty or whitespace. A browser deployment must proxy
`GET /api/sdk/sim-plan` alongside the OTLP route, or pass
`simulationPlan: false`.
### Advanced Configuration
```typescript theme={null}
new Bitfab({
apiKey: string, // Required
serviceUrl?: string, // Default: https://bitfab.ai
timeout?: number, // Request timeout in ms (default: 120000)
envVars?: { OPENAI_API_KEY: string }, // For native function execution
enabled?: boolean, // Default: true
simulationPlan?: boolean, // Read and apply the sim plan (default: true)
bamlClient?: unknown // Generated BAML client (for wrapBAML)
})
```
* `timeout`: Request timeout in milliseconds for API calls. Defaults to 120000 (2 minutes).
* `envVars`: Pass LLM provider API keys for native function execution via `call()`.
* `enabled`: When `false`, all tracing is disabled. Wrapped functions still execute normally but no spans are sent.
* `simulationPlan`: When `false`, the sim plan is never read and content capture is never narrowed. See [Content capture from the sim plan](#content-capture-from-the-sim-plan).
* `bamlClient`: The generated BAML client instance (e.g., `b` from `@baml`). See [BAML framework guide](/frameworks/baml) for full usage.
### Replay
A trace is replayable when its root span has serializable inputs, or when the workflow is instrumented through a [framework handler](#replaying-handler-instrumented-functions) (whose recorded root input is itself serializable). One of these must hold for replay to work.
Replay historical traces through an updated function version to compare outputs:
```typescript theme={null}
const pipeline = bitfab.getFunction("my-function")
const updatedFn = pipeline.withSpan(
{ name: "UpdatedPipeline", type: "function" },
async (input: string) => {
// New implementation to test
return { result: input.toUpperCase() }
},
)
// Replay all traces (up to limit)
const result = await bitfab.replay("my-function", updatedFn, {
name: "Uppercase candidate",
limit: 10,
maxConcurrency: 5, // Default: 10
attempts: 3, // Default: 1
})
// Replay specific traces by ID
const result2 = await bitfab.replay("my-function", updatedFn, {
traceIds: ["trace-id-1", "trace-id-2"],
})
// Result structure
console.log(result.testRunId) // Test run identifier
console.log(result.testRunUrl) // Dashboard URL
for (const item of result.items) {
console.log(item.input) // Original input
console.log(item.result) // New output
console.log(item.originalOutput) // Original output
console.log(item.error) // Error if any
console.log(item.durationMs) // How long THIS run took, in ms (or null)
console.log(item.originalDurationMs) // The replayed trace's duration in ms (or null)
console.log(item.tokens) // { input, output, cached, total } or null
console.log(item.model) // Original model name, or null
console.log(item.traceId) // Server trace ID for the replayed execution
}
```
Pass `replay()` **either** an already-`withSpan`-wrapped function (it carries its trace function key, so `replay()` runs it as-is) **or** a plain callable (which `replay()` wraps under the key for you). Do not wrap an already-instrumented function in a fresh closure: a plain arrow like `(input) => myWrappedFn(input)` carries no trace function key, so `replay()` adds its own root span around it while `myWrappedFn` records its own span underneath, nesting a duplicate. If your root is already wrapped, pass it directly: `bitfab.replay("my-function", myWrappedFn, ...)`.
Replay waits for each item's trace (spans + completion) to be persisted server-side before completing the test run, so `item.traceId` is a real server trace ID for completed items. Plain callables are wrapped in `withSpan` internally, so every replayed invocation records a trace. If NO completed item's trace persisted (uploads wholesale failed), `replay()` throws a `BitfabError` instead of silently returning `null` trace IDs. If only SOME items' traces are missing (a transient per-item upload failure), those items get `null` trace IDs with a loud `console.error` and the rest of the run is returned intact. `item.traceId` is also `null` for errored (unreplayable) items, and for all items when the server predates the trace-ID mapping (a console warning explains which).
Anything describing the trace being replayed carries the `original` prefix: `originalDurationMs`, `originalModel`, and `originalTokens`. Unprefixed fields are the replay's own: `durationMs` is how long this run's call took, and `tokens` is the **replayed run's** usage (the same numbers Studio's experiments view shows). Comparing `tokens.total` against `originalTokens.total` tells you how your change moved cost. Each field is `null` when it wasn't captured.
The same rule names the two trace outlines: `originalTraceOutline` is the original trace's span tree and `traceOutline` is the replayed trace's. An outline carries each span's name, type, nesting, order, duration, tokens, model, errors, and whether it was mocked, and no inputs or outputs, so it is small enough to keep on every item. Both are filled in at completion (they are `null` in `onItemFinish` and against older servers) so a grader can compare the path the replay took against the original's, not only its output. See `TraceOutline` in the [TypeScript reference](/reference/typescript).
`model` remains as a deprecated alias for `originalModel`. Note that `durationMs` changed meaning: it used to report the original trace's duration and now reports the replay's.
**Options:**
* `limit` -- Maximum number of recent traces to replay (default: 5; maximum: 5,000). Ignored when `traceIds`, `datasetId` or `datasetIds` is passed, since an explicit ID list or a dataset already determines how many traces replay. To replay part of a dataset selection, name the members to run in `traceIds`.
* `traceIds` -- Specific trace IDs to replay (max 100). The ID count determines how many traces replay, and `limit` is ignored when both are passed. Passed alongside a dataset selector it pins which members of that selection replay, and the server rejects any ID none of those datasets contains.
* `name` -- Optional display name for the resulting experiment/test run.
* `maxConcurrency` -- Number of traces to replay in parallel (default: 10)
* `attempts` -- How many times to replay each trace (1 to 100). Every attempt is its own replay trace under the same experiment, so the experiment reports per-attempt pass rates and flags traces whose attempts disagree. Default: `1`
* `codeChangeDescription` -- Optional rationale for the code change being tested in this replay (stored on the experiment); when supplied alone, it is preserved while files are captured automatically
* `codeChangeFiles` -- Optional list of edited files, each as `{ path, before, after }` (use `""` for newly created or deleted files); omit to capture automatically or pass `null` to suppress capture
* `mock` -- Mock strategy for child spans during replay: `"marked"` (default, only return historical output for child spans declared with `mockOnReplay: true`), `"none"` (run real code for every child), or `"all"` (return historical output for every matched recorded child; a missing occurrence fails the item closed). See **Mocking child spans during replay** below.
* `mockOverride` -- One `{ match, value }` pair, or an array of them, that injects a custom value into matched spans (first matcher wins). Takes precedence over `registerMockOverride` and the base `mock` strategy. See **Injecting custom values with overrides** below.
* `experimentGroupId` -- Optional UUID string that groups multiple replay runs into a single experiment batch. Pass the same ID across successive `replay()` calls to link them together in the dashboard.
* `datasetId` -- Optional dataset UUID. Replays that dataset's traces and durably attributes the resulting experiment to it, and `limit` is ignored because the dataset determines the item count. Pass `traceIds` alongside it to replay only those members.
* `datasetIds` -- Optional dataset UUIDs, for benchmarking one function against several corpora in a single run. Replays the union of their traces, graded by the union of their graders, and attributes the experiment to every one of them. Pass one dataset through `datasetId` and several through this.
* `graderIds` -- Optional array of grader UUIDs (max 100) attached directly to this replay run, independent of the dataset's own graders. The resulting experiment is graded by the union of these and the dataset's runnable graders. Use it to grade a single run with a check you don't want to add to the dataset permanently. Each id must be an active grader in the same organization and trace function, or the replay is rejected with a 400. A replay with no dataset can still carry graders this way.
* `adaptInputs` -- Optional hook to reshape recorded inputs onto the function's current signature when its shape changed after the traces were captured. See **Adapting inputs after a signature change** below.
* `onItemStart` -- Optional callback fired when a worker begins processing an item, before replay setup and customer code run. Pair it with `onItemFinish` to distinguish queued items from in-flight items whose callback has not returned. A throwing callback never crashes the run. The installed `bitfab-replay` command uses `reportReplayProgress` for both callbacks so liveness heartbeats identify the historical traces currently in flight.
* `onItemFinish` -- Optional callback fired exactly once per item as it finishes, always with that item plus running totals (original/source trace id, the server replay `traceId` read back per trace off the OTLP ingest response and surfaced as each item finishes (its trace is flushed on finish, so the id is in hand at the callback, not only after the whole run), input, result, original output, error, duration, tokens/model metadata). It never emits a whole-run completion event. Use it to render live progress or start evaluating completed items while replay runs. A throwing callback never crashes the run. The deprecated `onProgress` callback receives the same per-item events plus its legacy item-less terminal `complete` event, and is ignored when both are supplied. The installed `bitfab-replay` command uses the SDK's ready-made `reportReplayProgress` callback for both lifecycle hooks; it writes events to stderr, which the plugin uses to identify in-flight traces, report finished items, and write per-item result files (stdout remains available for direct-run `ReplayResult` JSON).
* `dbBranch` -- Optional `boolean | DbBranchOptions`. `dbBranch: true` requests a DB branch per replay item with the mirror's own sizing; pass an object to tune it, and `false` or omission leaves branching off. Each replay worker resolves its branch from the source trace's captured snapshot reference, so `maxConcurrency` also bounds live branches. `getCurrentReplayBranch()` hands the branch to you inside the replayed function, and the SDK releases it after the item. The accessor returns `null` when no branch was resolved (e.g. the trace predates snapshot capture, or DB branching isn't configured), so `branch?.databaseUrl ?? process.env.DATABASE_URL` falls back to your live database. The fields tune the branch: `minCu` and `maxCu` are the compute's autoscaling floor and ceiling (0.25 to 56). Equal values pin a fixed size, allowed up to 56; an autoscaling range may not span more than 8 CU or exceed 16 CU; setting them equal pins the size, so one item can't post a better number purely because it ran against an already-scaled endpoint. `warmupSql` is appended to the branch's readiness check, so the cache is warm before your function sees the branch and the warm-up is never charged to the replayed call. Omit them and the branch keeps the mirror's own defaults.
#### Replaying handler-instrumented functions
Workflows instrumented through a framework handler (`getLangGraphCallbackHandler`, `getLangChainCallbackHandler`, `getClaudeAgentHandler`, `getOpenAiAgentHandler`) have no `withSpan`-wrapped root in the application code: the handler (or run wrapper) records the framework invocation itself as the root span, with the framework's own input (a LangGraph initial state, an agent prompt, the run input) as the recorded root input. LangGraph/LangChain roots are registered as pending traces when the root callback starts and completed when it ends, so long-running runs can appear before final output is available. **These traces are fully replayable.** Pass the handler's trace function key plus any plain callable that re-invokes the framework entrypoint:
The OpenAI Agents SDK uses `getOpenAiAgentHandler(key).wrapRun(agent, input)` (a drop-in for `run`) for the replayable root; the bare `getOpenAiTracingProcessor` captures internals only and records an empty-input root. The Claude Agent SDK handler needs a hint: the prompt is not present in the message stream, so pass it explicitly, `wrapQuery(stream, { input: prompt })` (or `wrapResponse(stream, { input })`), for the handler to record a replayable root.
```typescript theme={null}
// scripts/replay.ts
import { graph } from "../src/agent" // the compiled LangGraph graph
import { bitfab } from "../src/bitfabClient" // same client as instrumentation
const handler = bitfab.getLangGraphCallbackHandler("my-agent") // same key
const replayMyAgent = async (state: AgentState) =>
graph.invoke(state, {
callbacks: [handler],
configurable: buildReplayConfig(),
})
const result = await bitfab.replay("my-agent", replayMyAgent, { limit: 10 })
```
How it fits together:
* `replay()` fetches the handler-recorded production traces by the key string, and wraps a plain callable in `withSpan` under that key internally so each replayed invocation records a trace tied to the test run. The key is the only link; it does not matter that production traces were written by the handler and the callable was written today.
* Passing an already-`withSpan`-wrapped function under the same key also works (older SDKs require this form); a wrapped function whose key contradicts the replay key throws.
* The recorded root input is whatever the handler captured at the framework boundary (a LangGraph state object arrives as a single argument).
* Attaching the handler inside the callable makes the replayed graph's node/LLM/tool spans nest under the replay span, so replay traces have the same tree as production ones.
* The callable rebuilds runtime wiring the trace never captured: framework `configurable`, dependency objects, API keys. Put every unsafe call made by that wiring behind a replay-mockable marked span. Use a no-op value only for a replay-only callback slot with no recorded call to mock.
#### Mocking child spans during replay
For the workflow-level guide, see [Replay Mocking](/replay-mocking).
When iterating on a root function, child spans sometimes fail in your local environment for reasons unrelated to the code under test: a paid API key is missing, an external service is flaky, or a production-only DB row isn't seeded locally. The `mock` option lets the child return its recorded output so the root function can still run.
Three strategies on `replay()`:
* **`"marked"`** (default): only descendants declared with `mockOnReplay: true` are short-circuited; everything else runs real. This is the iteration-friendly mode.
* **`"none"`**: every child span runs real code. Use only when you intentionally want every dependency real and have verified that is safe.
* **`"all"`**: every matched recorded descendant `withSpan` returns its historical output. The root function still runs real; a missing or exhausted child occurrence fails the item closed. Useful for a quick sanity-check against recorded data; not the recommended iteration strategy because changes to matched descendants won't execute.
Per-span opt-in via `SpanOptions.mockOnReplay`:
```typescript theme={null}
const articlePipeline = bitfab.getFunction("process-article")
const fetchArticle = articlePipeline.withSpan(
{ name: "fetch-article-from-db", mockOnReplay: true },
async (id: string) => db.articles.findById(id),
)
const summarize = articlePipeline.withSpan(
{ name: "summarize-article" },
async (article: Article) => summarizeWithNewPrompt(article),
)
const processArticle = articlePipeline.withSpan(
{ name: "process-article" },
async (id: string) => summarize(await fetchArticle(id)),
)
// During replay, fetch-article-from-db returns its recorded output;
// summarize-article runs real so you can iterate on it.
const result = await bitfab.replay("process-article", processArticle, {
limit: 10,
})
```
`mockOnReplay` is a per-span tag at definition time -- it has no effect outside replay, and it's read by the default `mock: "marked"` strategy. The root function always runs real code; only descendants can be mocked.
When a strategy selects a child for mocking but no historical occurrence is available (for example, the recorded trace did not reach that branch or the replay called it more times), the item errors and the real child does not execute.
#### Injecting custom values with overrides
Marking a span replays its *recorded* output. A **mock override** substitutes a value you supply for a matched span, so downstream real code runs against it -- for "what if this step returned X" experiments without editing the traced code. An override is a `{ match, value }` pair: `match` selects spans by structural metadata (`spanName`, `type`, `traceFunctionKey`, `originalSpanId`); `value` is the substitution (full replacement) -- a flat value, or a function of the context.
The trace function key passed to `replay()` selects the workflow's historical root traces; it does not by itself identify the descendant to override. Because the example above binds every call to `process-article`, match the descendant by its span name. If your calls use separate trace function keys, matching `node.traceFunctionKey` is also valid.
```typescript theme={null}
const result = await bitfab.replay("process-article", processArticle, {
mock: "none", // run everything real...
mockOverride: {
// ...except this span, which gets the value you supply
match: (node) => node.spanName === "fetch-article-from-db",
value: { id: "fixed", title: "Fixed title" }, // a flat value
},
})
```
`value` can also be a function receiving the span's **live** replay `inputs` and a `getOriginalOutput()` that lazily fetches the recorded output (memoized per trace) when you want to tweak it rather than replace it:
```typescript theme={null}
value: async ({ inputs, getOriginalOutput }) => ({
...(await getOriginalOutput()),
score: 1,
})
```
A flat or synthetic `value` (one that never calls `getOriginalOutput`) fetches no recorded outputs at all. Register overrides on the client to apply them to every replay:
```typescript theme={null}
bitfab.registerMockOverride({
match: (node) => node.type === "llm",
value: { label: "refund" },
})
// or the ordered form: registerMockOverride(match, value)
bitfab.clearMockOverrides() // reset
```
Use the keyed form when one registration belongs to a known trace function. The second argument can be either a resolver or another `{ match, value }` pair; in the latter case both the key and `match` must match:
```typescript theme={null}
bitfab.registerMockOverride("classify-intent", ({ inputs }) => ({
label: String(inputs[0]),
}))
bitfab.registerMockOverride("shared-workflow", {
match: (node) => node.spanName === "Classifier",
value: { label: "refund" },
})
```
For one client-wide resolver, pass a function directly. It runs for every child span, so it can route by trace function key. Return `NO_MOCK_OVERRIDE` to decline a span without using `undefined` (which remains a valid mocked output):
```typescript theme={null}
import { NO_MOCK_OVERRIDE } from "@bitfab/sdk"
bitfab.registerMockOverride(({ node, inputs }) => {
if (node.traceFunctionKey === "classify-intent") {
return { label: String(inputs[0]) }
}
return NO_MOCK_OVERRIDE
})
```
Precedence per span: per-call `mockOverride`, then registered overrides, then the base `mock` strategy. `NO_MOCK_OVERRIDE` continues at the next override, then falls back to that base strategy. A **synchronous** wrapped function cannot wait for any Promise-returning resolver, including one that eventually resolves to `NO_MOCK_OVERRIDE`. For mixed sync/async trees, use a non-`async` routing function that returns `NO_MOCK_OVERRIDE` synchronously for sync keys and returns a Promise only for async keys. Likewise, because `getOriginalOutput()` is async, a synchronous span cannot use it; make the span `async`, or use `mock: "all"`. A flat value or synchronous function works on synchronous spans.
#### Adapting inputs after a signature change
Replay deserializes each trace's inputs exactly as they were captured against the function's signature **at trace time**, then spreads them into the current function. If the signature drifted since capture (a param renamed, reordered, collapsed into an options object, or a new required arg added), that spread no longer lines up and the call throws. The `adaptInputs` hook reshapes the recorded inputs onto the current signature so replay can still run:
```typescript theme={null}
const result = await bitfab.replay("my-function", updatedFn, {
// Recorded as (userId, limit); current signature is ({ userId, limit }).
adaptInputs: (inputs, ctx) => {
const [userId, limit] = inputs as [string, number]
return [{ userId, limit }]
},
})
```
The hook runs once per item, **inside the same error boundary as the function**: if it throws, that item's `error` is set and the run continues, so a single unmappable trace never crashes the batch. The array it returns is what gets spread into the function and what `item.input` reports.
`ctx` carries `{ originalTraceId, originalSpanId }` (with deprecated `sourceTraceId`/`sourceSpanId` aliases) so a table-driven adapter can look up a per-trace transform (`ctx.originalTraceId` is the original Bitfab trace ID). This is the escape hatch for reshapes that need judgement rather than mechanical rearrangement: compute the adapted inputs per trace up front, then have the hook look them up by `sourceTraceId`, keeping replay deterministic instead of calling a model mid-replay.
When the new signature has a genuinely new **required** input with no analog in the recorded trace, don't fabricate one -- there's nothing faithful to map it to. Leave those traces unmapped (let them error) rather than inventing test inputs.
For anything beyond a one-liner, keep the adapter in its own file next to the replay registry and import it -- the `AdaptInputsFn` type is exported for this:
```typescript theme={null}
// scripts/replay-adapters/extraction.ts
import type { AdaptInputsFn } from "@bitfab/sdk"
export const adaptInputs: AdaptInputsFn = (inputs, ctx) => {
const [userId, limit] = inputs as [string, number]
return [{ userId, limit }]
}
```
```typescript theme={null}
// scripts/replay.ts
import { adaptInputs } from "./replay-adapters/extraction"
await bitfab.replay("my-function", updatedFn, { limit, adaptInputs })
```
That keeps the transform versioned and reviewable alongside the function it adapts, and you add the import only when a drift actually needs it.
#### Attaching a Code Change
Each replay creates an experiment (test run). When you're iterating on a function and replaying after every edit, attach the change so the dashboard can show *exactly what was edited* alongside the results. The agent reads each file before editing, edits, then reads it again -- the two strings go straight into `codeChangeFiles`. There's no diff format to construct.
```typescript theme={null}
import { readFileSync } from "node:fs"
const before = readFileSync("src/foo.ts", "utf8")
// ...edit src/foo.ts...
const after = readFileSync("src/foo.ts", "utf8")
const result = await bitfab.replay("my-function", updatedFn, {
name: "Retry fix candidate",
codeChangeDescription: "fix off-by-one in retry logic",
codeChangeFiles: [{ path: "src/foo.ts", before, after }],
})
```
Both options are optional and independent -- you can pass just `codeChangeDescription` for a quick rationale-only annotation, or just `codeChangeFiles` to record the literal edits.
If you omit `codeChangeFiles`, `replay()` falls back to capturing your working-tree diff against the trunk merge-base (best-effort, only inside a git repo), so an experiment still shows a diff. This fallback uses Git rename detection: a renamed-and-edited file is compared once under its destination path, while an unchanged rename adds no content diff. A supplied `codeChangeDescription` is preserved while the files are captured. Passing `codeChangeFiles` explicitly always wins and is the way to record a precise per-edit before/after. To opt out for one replay run, pass `codeChangeFiles: null` (and optionally `codeChangeDescription: null` if you also want no description). Set `BITFAB_DISABLE_CODE_CHANGE_CAPTURE` to turn the fallback off for every replay in the process.
**Notes:**
* **Use a single `Bitfab` client across instrumentation and replay.** If your instrumented module constructs `new Bitfab()` at import and your replay registry constructs another, they do not share registered trace functions -- import the client from the instrumented module (or a shared singleton) rather than constructing a new one in the registry.
#### Replay Output Contract
Replay results are typically consumed by automation (CI logs, code reviewers, and coding agents). When `BITFAB_REPLAY_RESULT_PATH` is set, `bitfab.replay()` automatically writes the full `ReplayResult` JSON to that file. For direct/manual runs, **emit the full `ReplayResult` as a single stdout JSON block** so a consumer can `JSON.parse` it and reason about every field, including the per-item `originalDurationMs`, `originalTokens`, `originalModel`, `tokens`, `originalTraceOutline`, and `traceOutline`. Never print only lengths, counts, hashes, or truncated previews, and never replace the JSON block with ad-hoc per-field log lines.
Recommended script tail (TypeScript):
```typescript theme={null}
const result = await bitfab.replay("my-function", updatedFn, { limit })
// Human-readable summary goes to stderr, so stdout stays pure JSON.
console.error(`Test run: ${result.testRunUrl}`)
console.error(`Items: ${result.items.length}`)
// Full structured dump to stdout, ready for JSON.parse. The SDK serializer
// retains useful fields from traceError and replayError exception objects.
console.log(serializeReplayResult(result))
```
The dumped object includes every item's `input`, `result`, `originalOutput`, `error`, structured `traceError` and `replayError`, `durationMs`, `originalDurationMs`, `originalTokens`, `originalModel`, `tokens`, `model`, `dbBranchTimings`, `traceOutline`, `originalTraceOutline`, and `traceId`, plus `testRunId` and `testRunUrl`. Import `serializeReplayResult` from `@bitfab/sdk`; raw `JSON.stringify` drops the useful fields on JavaScript `Error` objects. When the Bitfab plugin runs this script, it sets `BITFAB_REPLAY_RESULT_PATH`; the SDK writes the same structured JSON there, and the plugin reads that file into the replay run's `.bitfab/replays//events.jsonl` while writing large per-item payloads under `.bitfab/replays//items/`.
**Per-item errors are part of the contract.** If the wrapped function throws while executing the replayed trace, `bitfab.replay` retains the actual exception in `item.traceError`, copies its message to `item.error`, leaves `item.result` undefined, and continues. If replay setup fails before the function starts (for example database warmup or input loading), the actual exception is instead in `item.replayError`. A database branch resolution failure is a `DbBranchReplayError`; inspect its `code`, `message`, and `originalTraceId` to distinguish failures such as `branch_create_failed`, `snapshot_from_replaced_origin`, and `invalid_snapshot_ref` without parsing `item.error`. A lease-endpoint HTTP, timeout, or network failure uses `lease_request_failed` and retains the original client exception as `cause`; unexpected resolver failures use `internal_error`. Treat either error kind as **unreplayable**, not as a failing output. If the whole run later throws, `ReplayError.items` still contains every collected item and `ReplayError.cause` retains the whole-run exception.
**Don't swallow per-item errors in the script.** A custom try/catch that returns a placeholder turns infra failures into fake successes. Let the SDK record them. The only allowed top-level catch is a fatal handler around `main()` that exits non-zero, so callers can tell a whole-replay crash from a clean run with some unreplayable items.
**Input serialization caveat.** Replay deserializes historical span inputs and passes them back to your function. This works for strings, numbers, and plain objects. If your span wraps a function that takes hydrated domain objects (ORM models, class instances, DB records), they won't round-trip through serialization -- move the span to where inputs are IDs or plain data and let the function fetch objects internally, or reshape arguments in the wrapper.
#### Replay Registry
Create a small registry module. Your project owns only the imports and the mapping from command names to the exact traced functions production calls. The SDK installs the standard `bitfab-replay` executable, so upgrading `@bitfab/sdk` updates argument parsing, progress events, code-change loading, result serialization, and summary output without regenerating project code.
```typescript theme={null}
import "dotenv/config"
import { defineReplayRegistry } from "@bitfab/sdk"
import { bitfab } from "../lib/bitfab"
import { createSearchMock } from "./replayMocks"
import { extractMemories } from "../services/extraction"
import { searchDocuments } from "../services/search"
export default defineReplayRegistry({
extraction: { client: bitfab, fn: extractMemories },
search: {
client: bitfab,
fn: searchDocuments,
// Plain handler functions need the key explicitly. A withSpan-wrapped
// function carries its key, so `traceFunctionKey` can be omitted.
traceFunctionKey: "my-search-pipeline",
options: { mock: "marked" },
optionsFactory: ({ params }) => ({
mockOverride: createSearchMock(params.scenario),
}),
},
})
```
Keep non-trivial executable configuration in a sibling module and import it into the registry. For example, `replayMocks.ts` can export a mock factory whose `{ match, value }` result uses live replay inputs and caller-supplied parameters. `optionsFactory` receives JSON values loaded from `--params ` and repeated `--param name=value`; direct parameters override file values.
Run the executable with `bitfab-replay --registry scripts/replayRegistry.ts `. It loads TypeScript registry modules directly.
| Flag | Value | Effect |
| ------------------------------------ | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `--limit` | `N` | Most traces to select. It bounds a `--trace-ids` or `--dataset-ids` selection instead of replacing it |
| `--trace-ids` | `id1,id2` | Replay exactly these traces. Mutually exclusive with `--dataset-ids` |
| `--dataset-ids` | `uuid1,uuid2` | Replay the membership of one or more datasets, as their deduped union. Mutually exclusive with `--trace-ids`. `--dataset-id` is the same flag |
| `--name` | `NAME` | Title for the resulting experiment |
| `--attempts` | `N` | Replay each selected trace N times in one run (1 to 100, default 1) |
| `--concurrency`, `--max-concurrency` | `N` | Items in flight at once |
| `--dry-run` | | Resolve every item's inputs and stop without calling the function |
| `--experiment-group-id` | `UUID` | Add this run to an existing experiment group |
| `--grader-ids` | `id1,id2` | Attach graders to this run only |
| `--only-with-assertions` | | Replay only the selected traces that carry an assertion |
| `--code-change` | `PATH` | Load a code-change description from a file |
| `--no-code-change` | | Record no code change, overriding a registry default |
| `--mock` | `none\|all\|marked` | Which recorded child spans return their historical output |
| `--db-branch`, `--no-db-branch` | | Turn per-item database branching on or off |
| `--seed` | `cases.jsonl` | Seed the file's cases instead of replaying. See [Seeding traces](#seeding-traces) |
| `--run` | | With `--seed`, run each case once and record the execution |
| `--params` | `PATH` | JSON file of values passed to `optionsFactory` |
| `--param` | `name=value` | One value passed to `optionsFactory`. Repeatable, and overrides `--params` |
| `-h`, `--help` | | Print usage |
Put static function-specific behavior such as `adaptInputs`, `mockOverride`, or `dbBranch` in `options`; build parameterized behavior in `optionsFactory`. Command-line values override overlapping scalar defaults without removing unrelated executable options.
### Seeding traces
Replay needs a trace to replay. Until production has produced one, there is nothing to select, so a corpus you already hold (a dataset export, a spreadsheet, hand-written cases) cannot be run. `seedTrace` writes that corpus into Bitfab as replayable traces and returns each trace ID.
It has two forms, chosen by the second argument. Pass a **function** to run it once and record the execution. Pass a **case** to write a trace without running anything.
#### Seed by running the function once
```typescript theme={null}
import { Bitfab } from "@bitfab/sdk"
import { runTicket } from "../services/agent"
const bitfab = new Bitfab({ captureEnabled: false })
const traceId = await bitfab.seedTrace("agent-turn", runTicket, {
args: ["T-1"],
metadata: { caseUid: "c-1", suite: "smoke" },
name: "T-1",
})
```
This runs `runTicket("T-1")` once and records the execution as an **original** trace: root span, first-party subtree, the real input, and whatever the run produced as the output. Capture stays off for everything else, so a seeding script does not have to run with tracing on for the rest of the process.
The trace lands under `agent-turn` with `ingestion_type: seeded`, `replay` selects it like any captured trace, and every replay of it links back as `originalTraceId`. Because it has a full recorded subtree, replay mocking works on it exactly as it does on a captured trace.
`fn` resolves the same way it does for `replay`. A `withSpan`-wrapped function records under its own key, which must match the key you pass, and a plain callable is wrapped under the key here. An exception is recorded on the root span, the trace still persists, and the exception is re-thrown. A call that records nothing (no API key resolved) rejects rather than handing back an ID replay could never find.
#### Seed from a case without running
```typescript theme={null}
const traceId = bitfab.seedTrace("support-agent", {
input: [{ ticketId: "T-1", body: "Where is my refund?" }],
expected: { intent: "refund_status" },
fn: handleTicket,
name: "T-1",
})
```
The recorded root span carries `input` as its input and `expected` as its output, so a later replay reports the item against the value you expected rather than against a previous run. Passing `fn` checks the case against the function's required argument count, so a case that could never run fails while you seed instead of during replay. Omit it when the seeding script cannot import the callable.
A case-seeded trace has **no child spans**, so replay mocking has nothing recorded to substitute. Supply `mockOverride` at replay time for calls that must not run for real. Neither seeding form pins a database, so `dbBranch` refuses a seeded trace.
#### Options both forms share
`name`, `metadata`, and `sessionId` apply to either form. `name` is the trace's title and a searchable, filterable field. Put the case's own label there (a ticket ID, a dataset row name) so the seeded trace can be found by it. `metadata` is stored on the trace and handed to a later replay's `adaptInputs` hook as `ctx.metadata`, so a case's provenance rides with the trace instead of through the recorded inputs. In the run form, if the function also emits its own trace through an integration that exports trace metadata, the caller's metadata is merged onto that export and wins on any shared key.
The case form additionally takes `spanName` and `spanType`, which label and type the root span it writes. The run form has no equivalent, since the executed function names its own root span.
#### Re-seeding a trace
A trace whose recorded run is wrong (it errored, or the world it ran against has moved on) can be re-seeded. `reseedTrace` reads the trace's recorded inputs, name, session, and metadata, runs the function once the way `seedTrace` does, and asks Bitfab to adopt that run under the same trace id.
```typescript theme={null}
const { traceId, previousRunTraceId } = await bitfab.reseedTrace(
"agent-turn",
runTicket,
{ traceId: "3f2a..." },
)
```
The trace keeps its id, labels, assertions, dataset membership, name, and metadata, so anything you stored against it still names the same case. The previous run is kept as its own trace, `previousRunTraceId`, with `reseedOfTraceId` pointing back at the case. Nothing is mocked and no experiment is created; a re-seed is a seed, not a replay. A run that throws is recorded but never adopted, so the trace is untouched, and Bitfab rejects a run that comes from another function or already belongs to a dataset. Graders on the datasets holding the trace re-run afterwards, and default replay selection skips previous runs.
From the shell, `bitfab-seed --from-trace [,...]` does the same through the replay registry:
```bash theme={null}
bitfab-seed --registry scripts/replay.registry.ts extraction --from-trace 3f2a...
```
A re-seed runs the function exactly as production does, side effects included, so treat it like running the code, not like a replay.
#### Seeding a whole cases file
`seedFromRegistry` seeds through an already-registered pipeline, reusing its client, callable, and trace function key, so every case is written against the exact function the later replay selects. The installed command does the same from the shell:
```bash theme={null}
# Write each case directly, no execution.
bitfab-replay --registry scripts/replayRegistry.ts extraction --seed cases.jsonl
# Run each case once through the registered function and record it.
bitfab-replay --registry scripts/replayRegistry.ts extraction --seed cases.jsonl --run
```
Each line of `cases.jsonl` is a JSON object with an `input` array plus optional `expected`, `metadata`, and `sessionId`. A JSON array of those objects works too. With `--run`, the output is what the run produced, so a case carrying `expected` is rejected. The registration's `adaptInputs` is a replay hook and is not applied at seed time, so a seeded trace is never adapted twice.
#### Replaying seeded traces
`replay` selects seeded traces the same way it selects captured ones, and each item reports its source's `ingestionType` (a source with none reads as captured). The `bitfab-replay` summary counts a seeded item as **matched** or **missed** against its expected value rather than same or changed against a previous run, because a seeded trace's recorded output is an assertion and not a prior run's result. One run can replay both kinds, so both pairs of counts print when both are present.
See the [reference](/reference/typescript#seedtrace) for full signatures.
## Datasets
`client.datasets` creates, reads, and modifies datasets programmatically, with the same operations your coding agent reaches through the Bitfab MCP tools. A dataset is a named bucket of traces under one trace function. Experiments replay against it and its graders score its members.
```typescript theme={null}
const { dataset, created } = await bitfab.datasets.save({
traceFunctionKey: "checkout-agent",
name: "Refund failures",
description: "Checkout runs where the refund was declined",
})
const added = await bitfab.datasets.addTraces(dataset.id, [traceId])
if (added.skippedTraceIds.length > 0) {
console.warn("not in this trace function:", added.skippedTraceIds)
}
const { traceIds } = await bitfab.datasets.listTraces(dataset.id)
await bitfab.datasets.addGraders(dataset.id, [graderId])
const { run } = await bitfab.datasets.rerunGraders(dataset.id)
console.log(run.status, run.result)
```
`save` is an upsert on the dataset name within its trace function, so re-running a script does not accumulate duplicates. Membership and grader calls accept up to 100 ids and report ids they skipped rather than failing the whole call. `removeTraces` only drops membership. Traces are never deleted. `rerunGraders` waits for the run by default (90 seconds, configurable) and returns whatever state it last saw. Pass `wait: false` to return immediately and poll with `getGraderRerun`. See the [reference](/reference/typescript#datasets) for every method and result type.
## Labels
`client.labels` writes pass/fail verdicts and reads them back, the same operations your coding agent reaches through the `save_agent_labels`, `save_human_labels`, and `get_trace_labels` MCP tools. A verdict says how a run that already happened turned out. Written per assertion, it is stored and read back per assertion, so a judge inside a replay process can score each assertion on its own and verify what landed without opening Studio.
```typescript theme={null}
const { assertions } = await bitfab.traces.getAssertions(item.originalTraceId)
await bitfab.labels.saveAll(
assertions.map((assertion) => {
const assessment = {
assertion: assertion.assertion,
passCriteria: assertion.passCriteria,
failCriteria: assertion.failCriteria,
targetOnEvaluatedTrace: assertion.targetOnEvaluatedTrace,
}
return {
originalTraceId: item.originalTraceId,
attempt: item.attempt,
assertionId: assertion.id,
label: judge(assessment, output),
annotation: explain(assessment, output),
}
}),
testRunId,
)
const [labels] = await bitfab.labels.getAll([item.traceId])
for (const verdict of labels.assertions) {
console.log(verdict.assertion, verdict.label, verdict.annotation)
}
const graded = await bitfab.graders.getLabels({ traceIds: [item.traceId] })
```
`humanNote` is returned with each assertion so your tooling can display people-only context. It is written through MCP or Studio, not SDK saves. Do not include it in the object sent to an LLM judge; verdict evidence is the assertion, its pass/fail criteria, its target, and the evaluated trace.
`save` and `saveAll` write the agent's verdicts, which start unapproved until a person approves them in Studio. `saveHuman` and `saveHumanAll` write verdicts that are validated on write, for cases a person has already decided, such as a production bug captured as a regression test. Both batches are all-or-nothing: a trace outside the organization, a repeated target, or an assertion that is not active on its trace rejects the call and writes nothing. `get` and `getAll` return each trace's effective verdict plus one row per scored assertion. `graders.getLabels` is the per-grader breakdown the effective verdict folds together. Approving a verdict is not on this surface, or on MCP, by design. See the [reference](/reference/typescript#labels) for every method and type.
# Labeling
Source: https://docs.bitfab.ai/web-portal/labeling
Label traces to build evaluation datasets and track quality in the Bitfab web portal
## Overview
Labeling lets you review traces and mark them with pass/fail outcomes or tags. Use it to:
* Build evaluation datasets from real production data
* Train automated graders by providing labeled examples
* Track quality trends over time
* Identify patterns in failures
## Viewing Traces
Navigate to a trace function and open the **Labeling** view to see all captured traces.
### Trace List
The list shows:
| Column | Description |
| ------------------ | ----------------------------------------- |
| **Status** | Success or error |
| **Duration** | Execution time |
| **Grader Results** | Pass/fail outcomes from automated graders |
| **Labels** | Human-applied labels |
| **Created** | When the trace was recorded |
### Filtering
Filter traces by:
* **Status**: Success or error
* **Date Range**: Filter by time period
* **Grader Results**: Pass or fail on specific graders
* **Labels**: Filter by human-applied labels
* **Tags**: Filter by assigned tags
### Searching
Use the search bar to find traces by:
* Input content
* Output content
* Error messages
## Trace Details
Click on a trace to view details:
### Input
The exact input passed to the function:
```json theme={null}
{
"order_id": "order-123",
"items": ["item1", "item2"]
}
```
### Output
The output returned:
```json theme={null}
{
"order_id": "order-123",
"total": 20
}
```
### Span Tree
When functions call other wrapped functions, you see a hierarchical view:
```
Trace: agent-workflow
├── Span: validate-input (guardrail)
├── Span: fetch-data (function)
└── Span: generate-response (llm)
```
Click on individual spans to see their inputs, outputs, context, and timing.
## Applying Labels
Label traces to indicate quality:
1. Open a trace
2. Review the input, output, and span tree
3. Apply a pass/fail label or add tags
Labels feed into grader training - the more labeled examples you provide, the better automated graders perform. Labeling a trace as a pass when a grader failed it also records a correction on that trace, which is the signal to tighten the grader's criteria. See the [Graders guide](/primitives/graders).
## Tagging
Organize traces with tags for filtering and dataset building:
### Adding Tags
1. Open a trace
2. Click **Add Tag**
3. Select an existing tag or create a new one
### Managing Tags
Navigate to **Tags** in the user menu to:
* Create new tags
* Edit tag names and colors
* Archive unused tags
## Building Evaluation Datasets
Use labeled traces to build evaluation datasets:
1. Filter to traces you want to include
2. Label traces with pass/fail outcomes
3. Apply tags to organize by feature, environment, or use case
4. Use these labeled traces to train and validate graders
## Best Practices
* **Label regularly**: Consistent labeling improves grader accuracy
* **Tag strategically**: Use tags to organize by feature, environment, or failure mode
* **Label both passes and failures**: Graders need examples of both to learn effectively
* **Start with the most impactful function**: Focus labeling effort on the trace function with the highest volume or most critical failures
# Web Portal Overview
Source: https://docs.bitfab.ai/web-portal/overview
Navigate trace functions, inspect traces, and label results in the Bitfab web portal
## Overview
The Bitfab web portal at [bitfab.ai](https://bitfab.ai) is where you view your traced AI workflows, inspect individual traces, label results, and manage automated graders.
## Trace Functions
When you instrument your code with the SDK, each `getFunction()` or `get_function()` call registers a **trace function** - a named group of related spans. The main navigation lists all your trace functions with summary stats:
* **Total traces** captured
* **Evaluation status** - how traces are performing against your graders
* **Recent activity** - latest traces and their outcomes
Click a trace function to see all of its traces, graders, and evaluation results.
## Traces
Each execution of an instrumented function produces a **trace** - a tree of spans capturing the full call stack with inputs, outputs, timing, and errors.
### Trace List
The trace list for a function shows:
| Column | Description |
| ------------------ | -------------------------------------------------------------------------------------------------------------------- |
| **Status** | Success or error, and whether the trace is a run of your code, a replay, or [seeded](/typescript-sdk#seeding-traces) |
| **Duration** | Execution time |
| **Grader Results** | Pass/fail outcomes from automated graders |
| **Created** | When the trace was recorded |
A trace written by `seedTrace` / `seed_trace` shows a **seeded** marker once it has settled cleanly, so a corpus you loaded on purpose is distinguishable from traffic your app produced. While it is still running, or if it threw, it shows the running or error state instead, like any other trace. A replay of a seeded trace is its own run and shows the replay states.
### Trace Detail
Click a trace to see the full span tree:
```
Trace: order-processing
├── Span: validate-input (guardrail)
├── Span: fetch-data (function)
└── Span: generate-response (llm)
```
Each span shows its input, output, errors, context, timing, and prompt (if set). Click individual spans to drill into nested calls.
### Searching and Filtering
Filter traces by:
* **Status**: Success or error
* **Date range**: Filter by time period
* **Grader results**: Pass or fail on specific graders
* **Full-text search**: Search across inputs, outputs, and error messages
* **Tags**: Filter by assigned tags
## Labeling
Label traces to build evaluation datasets and track quality. See the [Labeling guide](/web-portal/labeling) for details.
## Datasets, Graders, and Experiments
Beyond browsing and labeling traces, the portal is where you manage Bitfab's other primitives: [datasets](/primitives/datasets) of curated traces, [graders](/primitives/graders) that score them automatically, and [experiments](/primitives/experiments) that replay a dataset against your changed code. See the [Primitives overview](/primitives/overview) for how they fit together.
These are created from your coding agent rather than from a form in the portal. The **New** buttons on each list show the prompt to use. Scripts and CI jobs that need to build or maintain a dataset without an agent in the loop can use the `client.datasets` namespace in any of the four SDKs instead. See [Datasets from the SDK](/primitives/datasets#from-the-sdk).
## Settings
Additional configuration is available in the user menu:
Manage API keys for SDK authentication
Configure external service connections
Organize traces with tags
## Organizations
Bitfab uses organizations to group resources:
* Each user belongs to one or more organizations
* Trace functions, traces, and API keys are scoped to an organization
* Team members can be invited to collaborate
### Switching Organizations
If you belong to multiple organizations, use the organization switcher in the header to change between them.
# Settings
Source: https://docs.bitfab.ai/web-portal/settings
Configure your Bitfab organization settings
## Overview
Bitfab settings are accessible from the user menu in the top-right corner:
* **API Keys**: Authenticate your applications
* **Integrations**: Configure LLM provider API keys
* **Tags**: Organize your captured function calls
## API Keys
Manage API keys for authenticating your SDK applications.
### Creating an API Key
1. Click your profile icon and select **API Keys**
2. Click **Create API Key**
3. Enter a descriptive name
4. Copy the generated key immediately
API keys are only shown once. Store them securely and never commit them to version control.
### Managing API Keys
| Action | Description |
| ---------- | ------------------------------------------ |
| **View** | See key name, creation date, and last used |
| **Revoke** | Permanently disable a key |
## Integrations
Configure API keys for LLM providers used by your BAML functions.
### Configuring Providers
1. Click your profile icon and select **Integrations**
2. Find the provider (OpenAI, Anthropic, etc.)
3. Enter your API key
4. Click **Save**
### Supported Providers
| Provider | Configuration |
| ---------------- | ------------------ |
| **OpenAI** | API key |
| **Anthropic** | API key |
| **Google AI** | API key |
| **Azure OpenAI** | Endpoint + API key |
Provider API keys are encrypted at rest and never logged.
## Tags
Manage tags for organizing captured function calls.
### Creating Tags
1. Click your profile icon and select **Tags**
2. Click **Create Tag**
3. Enter a name and select a color
4. Click **Save**
### Tag Colors
Available colors:
* Slate, Red, Orange, Amber
* Green, Blue, Purple, Pink
### Archiving Tags
Archive tags you no longer need:
1. Find the tag in the list
2. Click the menu icon
3. Select **Archive**
Archived tags are hidden but captures with those tags are preserved.
## Profile
Manage your personal profile from the user menu:
* **Name**: Your display name
* **Email**: Your login email
* **Avatar**: Your profile picture