DashboardPluginsCLITypeScript SDKPython SDKRuby SDKGo SDK
Plugins v0.12.68, CLI v0.2.455, TypeScript SDK v0.65.0, Python SDK v0.65.0, Ruby SDK v0.65.0, Go SDK v0.65.0
File scenarios under categories
You can now file scenarios under the same categories your assertions use. On a scenario’s page, add or remove categories from the header, and the scenario list shows each scenario’s categories. Coding agents can set them withsave_scenario and read them back with get_scenario.Assertion categories are now categories
Assertion categories are renamed to categories across the dashboard, the MCP tools (save_category, get_category, list_categories, delete_category), and the SDKs (categories, with category_id on assertion saves). The old names keep working for now and are deprecated.Dashboard
Correct costs for prompts that hit the cache
The cost shown for each trace in an experiment comparison no longer overcounts cached prompt tokens. Before, the cached part of a prompt was charged at the full input rate and again at the cached rate, so runs that reused a cached prompt looked more expensive than they were. Cached tokens are now charged once, at the cached rate, so comparing the cost of two runs reflects what each one actually spent.PluginsCLITypeScript SDKPython SDKRuby SDKGo SDK
Plugins v0.12.67, CLI v0.2.454, TypeScript SDK v0.64.30, Python SDK v0.64.27, Ruby SDK v0.64.25, Go SDK v0.64.26
Cloud replay works with any checkout action
bitfab-replay --cloud-update now adds fetch-depth: 2 to your replay workflow’s checkout step even when it uses a checkout action other than actions/checkout, such as useblacksmith/checkout. With it set, a cloud replay records your branch, the commit you started from, and your uncommitted changes, the same as a local replay.DashboardPluginsCLITypeScript SDKPython SDKRuby SDKGo SDK
Plugins v0.12.66, CLI v0.2.453, TypeScript SDK v0.64.29, Python SDK v0.64.26, Ruby SDK v0.64.24, Go SDK v0.64.25
The assertion judge abstains when it cannot find the evidence
When the evidence Bitfab reads from a replay does not show the part of the run an assertion checks, the judge now abstains instead of guessing. Before, an assertion that said the run does not do something passed because nothing showed the run doing it, and one that said the run does something failed because nothing showed it, even when the message or call in question was simply not in what the judge read. An abstention still counts as a fail in every pass rate, since anything short of an evidenced pass is a fail, but it is stored and shown apart from a fail: the labeling panel and the experiment comparison mark it “Abstained”, the experiment breakdown says how many of the failing assertion labels were abstentions, andget_trace_labels reads it back as ABSTAINED (counted as FAIL) with an abstained count beside passed and failed.Your own judge can abstain too. Pass abstain: true to save_agent_labels, or call labels.abstain / labels.abstain / labels.abstain / Labels.Abstain in the TypeScript, Python, Ruby and Go SDKs, with an optional annotation saying what the evidence would have had to show. Reads carry a new verdict field of pass, fail, or abstained on each assertion verdict, with label still false for an abstention, an abstained count on the trace, and abstained on every experiment rollup tally. An abstention is not a skip: a skip is for a check that never ran and counts as nothing.DashboardPluginsCLITypeScript SDKPython SDKRuby SDKGo SDK
Plugins v0.12.65, CLI v0.2.452, TypeScript SDK v0.64.28, Python SDK v0.64.25, Ruby SDK v0.64.23, Go SDK v0.64.24
Name an experiment group when you launch it
Runs you want to compare can join one experiment group from the start, already named. Pick a new group id, pass it to every replay, and add a name withexperimentGroupName in any SDK or --experiment-group-name on bitfab-replay. The first replay to start creates the group with that name, and a group that already has a name keeps it.Dashboard
Sim plan loads for large traced functions
The sim plan page now loads for traced functions with a long history and many nodes whose content capture is turned off. Before, these pages could stay on “Building the code map…” without ever finishing. Size estimates for nodes with content turned off now come from the traces recorded just before capture was switched off.DashboardPluginsCLITypeScript SDKPython SDKRuby SDKGo SDK
Plugins v0.12.64, CLI v0.2.451, TypeScript SDK v0.64.27, Python SDK v0.64.24, Ruby SDK v0.64.22, Go SDK v0.64.23
Compare experiments by time and tokens
You can now list one experiment’s traces from code and see how long each one took and how many tokens it used. Pass an experiment id tosearch on the traces client in any SDK, and each result now carries durationMs and tokens (input, output, cached, and total), with null when a value was not recorded.list_experiment_traces MCP tool also shows each replay’s duration next to its token usage.Dashboard
Assertion categories in scenario review
Each assertion in a scenario’s review panel now shows its category under the assertion text, so you can see what kind of expectation you are approving without opening the row. Hover or focus the category to read its description.Dashboard
Filter experiments by dataset
You can now narrow the experiments page to a single dataset. Pick one from the Dataset menu next to Group by to see only the runs that replayed it, or choose All datasets to see every run again. The filter is kept in the page address, so you can share the filtered view.Dashboard
Remove traces from a dataset
You can now take a trace out of a dataset from the dataset page. Each row has a remove button next to Details, and removing asks you to confirm first. The trace itself is kept, so you can add it back later, and the dataset’s trace count updates right away.DashboardPluginsCLITypeScript SDKPython SDKRuby SDKGo SDK
Plugins v0.12.63, CLI v0.2.450, TypeScript SDK v0.64.26, Python SDK v0.64.23, Ruby SDK v0.64.21, Go SDK v0.64.22
Rename a dataset and edit its description
You can now rename a dataset or change its description without losing its traces, graders, or link. On the dataset page, click the pencil next to the name or the description to edit it in place. A name another dataset on the same traced function already uses is refused, and an empty description clears it.From code, pass adatasetId to the save_dataset MCP tool, or call update on the datasets client in any SDK:save still finds a dataset by name, so passing it a new name creates a second dataset rather than renaming the first.Automatic tracing no longer installs the TypeScript compiler
The TypeScript SDK no longer depends on the TypeScript compiler, so it is much smaller to install. Every automatic tracing adapter, including the@bitfab/sdk/transform/register hook for running TypeScript directly in Node, now compiles your code with Babel and needs Node 22.18 or later. The @bitfab/sdk/transform/typescript entry is removed, so if you used createBitfabTypeScriptTransformer, switch to the adapter for your build tool.More accurate automatic traces
A function defined inside another function with destructured parameters is now recorded once instead of twice, and functions defined inside a constructor or getter keep the same name they had before. Decorated classes compiled to CommonJS now load under automatic tracing,.tsx files work through the Node register hook, and a TypeScript this parameter is no longer recorded as an argument. The register hook also transforms each file much faster.Dashboard
Switching between scenarios is immediate
Moving from one scenario to the next on a function’s Scenarios page no longer pauses on a blank frame while the whole page comes back from the server. The page keeps the scenarios on either side of the one you are viewing ready before you move to them, so clicking a row or pressing J or K shows the next scenario at once. A scenario that is not ready yet shows a placeholder in place of its detail while it loads, and saving a label refreshes the detail in place.DashboardPluginsCLITypeScript SDKPython SDKRuby SDKGo SDK
Plugins v0.12.61, CLI v0.2.448, TypeScript SDK v0.64.24, Python SDK v0.64.22, Ruby SDK v0.64.20, Go SDK v0.64.21
Record what a scenario’s run actually did
A scenario can now hold what its run actually did: a short summary and the actions it took, in order, with any that did not go through marked. Send it asactualOutcome with the save_scenario MCP tool, or with saveScenario / save_scenario / SaveScenario in the TypeScript, Python, Ruby and Go SDKs. Sending null clears it, and leaving it out keeps the saved one. get_scenario and the SDK reads return it. The scenario page shows it below the trigger, labeled as one example of how the run went and not necessarily what should have happened, so you still decide the right behavior from the trigger and the data. Bitfab does not fill it in for you.Dashboard
Comparing experiments with the same flags stays on Differences
On an experiment comparison, the Feature flags section now opens on Differences even when every experiment ran under the same flags, and says so in place of the list. Before, it switched itself to All flags and listed every flag the runs recorded, which hid the fact that nothing differed. Pick All flags when you want the full list. The feature flag views on a single experiment and on a replay open the same way.Dashboard
Review a whole trace’s assertions at once
A function’s Assertions page has a Group by option that turns the queue into one row per trace. Open a trace and its assertions are stacked in a single panel with the trace’s context beside them, so you can read what the run did and decide every assertion on it without moving between rows. Approve all decides only the assertions the panel is showing, so a category or assignee filter you have on is respected rather than approving what it hides. An approval that stalls now gives up after 15 seconds and says it failed, instead of leaving the buttons disabled with no explanation.The queue scrolls and holds its place
The assertions queue loads more as you scroll instead of stepping through numbered pages, and deciding an assertion no longer pulls its row out of a filtered queue. The row stays where it is and its state changes, the tab counts move, and Refresh rebuilds the queue when you want it. Opening a link to an assertion far down the queue now keeps loading until it finds that assertion, where before it could quietly select a different one.See a trace’s scenario while you review it
A trace with an extracted scenario now shows its trigger and the conversation leading up to the run in the review panel, drawn the same way as the scenario page, rather than the raw input and output of the trace’s root span. A trace without a scenario says so and still shows its input and output.Dashboard
Feature flags are read from feature_flags
Bitfab now reads the feature flags a run executed under from a feature_flags object in a trace’s metadata. Before, the key was effective_flags. A trace that still sends effective_flags keeps it in its raw metadata, but Bitfab no longer records it as the trace’s flag set, so replays, experiments and the Feature flag filter will not see it. Rename the key to feature_flags to keep those working.Dashboard
Save scenario records on large traces
Saving records onto a scenario withsave_scenario no longer fails on traces whose spans repeat the same content many times. Before, matching the records to the trace’s spans could run long enough to return a server error or time out without saving anything. It now takes a fraction of a second on those traces, and each record is matched to the same span as before.Dashboard
Review scenarios as a queue
A function’s Scenarios page is now a review queue with Needs label, Needs review and Done tabs, each with a count, and Back and Next buttons (or J and K) to move between scenarios. At the bottom of each scenario, write a label saying what the run should have done, generate assertions from it, then press Approve & next to approve them and open the next scenario. If an assertion is wrong, edit the label and regenerate: drafts you haven’t approved are replaced and approved ones stay.Add a scenario to a dataset
The scenario header has an Add to dataset menu that adds the scenario’s trace to one of the function’s datasets without leaving the page.A scenario’s data comes only from you
Creating a scenario withsave_scenario now builds only its trigger from the trace and starts its data empty. The data holds only the records and data you send, so the conversation on the scenario page is your own message records in time order, and drafted assertions cite only what you sent.Dashboard
Move an experiment to another group
You can now change an experiment’s group from its page. Click the group name in the experiment header to search the groups for the same function, which are listed with the most recently run first, then pick one to move the experiment there or choose No Group to take it out of every group. The experiments list and group pages update right away.Scenarios are created before their records are saved
save_scenario now creates a scenario in its own step: send the trace with records as an empty list, and Bitfab builds the trigger and data from the trace without saving any records. Check that the returned trigger is not empty, then send the records. Records are refused for a trace that has no scenario yet and for a scenario whose trigger is empty, so a scenario’s records never land on a trigger that is missing.Dashboard
Feature flags on traces, replays and experiments
Send the feature flags a run executed under as aneffective_flags object in a trace’s metadata, and Bitfab keeps each flag’s value as sent, including true/false, numbers and lists. A replay shows how many of its flags differ from its original’s and opens a side-by-side table with the differences highlighted. An experiment shows the flag set most of its traces ran under and marks traces that ran under different flags with a “Flags differ” chip, and comparing experiments adds a Feature flags card listing the flags that differ between them.Filter traces by feature flag
The traces page has a new Feature flag filter that shows only traces whose flag matches a value. A list flag matches when it contains the value, andtrue, false and numbers match the typed value. The caller metadata filter on the same page now also applies and stays set when you reload.DashboardPluginsCLITypeScript SDKPython SDKRuby SDKGo SDK
Plugins v0.12.58, CLI v0.2.445, TypeScript SDK v0.64.23, Python SDK v0.64.21, Ruby SDK v0.64.19, Go SDK v0.64.20
Cloud replays get a key that ends with the run
A replay started withbitfab-replay --cloud now runs as you with a short-lived Bitfab API key that only its own GitHub Actions run can claim, proven by GitHub’s signed identity for that run. The key turns off when the replay ends and expires on its own, and no API key is stored as a GitHub secret anymore. Run bitfab-replay --cloud-update and commit the workflow it updates: it grants id-token: write and removes a line older versions wrote that gave each run access to every secret in the repository. Until then, replays use the workflow’s BITFAB_API_KEY secret, and each run shows a warning in GitHub Actions saying what to update.Temporary keys on the API keys page
The API keys page has a new Temporary keys section that lists the keys your cloud replays used over the last 7 days, shows whether each is waiting for its run, active, or ended, and lets you turn one off immediately. Your regular keys are listed as before.Remove key copies left by older versions
bitfab-replay --cloud-update now lists any BITFAB_REPLAY_KEY_* secrets an older version left in your repository when a replay could not delete them. Each holds a copy of the Bitfab API key of whoever started that replay. Add --remove-old-key-secrets to delete them.See how much of each scenario’s data was read
A scenario can now record how much of each kind of data was captured for its run, so a record missing from it no longer looks the same as one that never existed. Sendsources beside the records in save_scenario, or include frameAt, capturedAt, windows, and coverage in the file you pass as recordsPath.The scenario page then marks each kind of data as read in full or not fully read, and says why: cut by a time window, captured after the run instead of as of it, columns filled because the capture predates them, or not captured at all. get_scenario returns the same, and generate_scenario_assertions no longer drafts an assertion that rests on something being absent from data that was not read in full.Dashboard
Experiment bars match their pass rate
On experiments with graders, the bar next to each run’s pass rate now splits green and red by that pass rate, on the experiments list and at the top of each experiment. Before, it split by individual grader verdicts, so a run where 2 of 19 traces passed could show a half green bar. Grader errors and checks still grading keep their own part of the bar.While a run is still going, its pass rate now shows the result so far with a spinning outline, instead of a blank placeholder.Draft scenario assertions from a scenario id
generate_scenario_assertions now finds the scenario by its id as well as by a trace it was built from, the same as get_scenario and save_scenario. Create the scenario with save_scenario, check it, then pass the id it returns. Each drafted assertion is saved on the trace whose spans it cites.Dashboard
Browse your scenarios
Each trace function now has a Scenarios page, marked alpha, that lists the situations its runs were in, newest first, and loads more as you scroll. Open a scenario to read the conversation before the run, the message that started it, and what the run did not have.Open the scenario’s trace, or the exact span a message came from, in a panel over the page without losing your place.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.64.21, Python SDK v0.64.20, Ruby SDK v0.64.18, Go SDK v0.64.19, Plugins v0.12.52, CLI v0.2.439
Cloud replays show the code change they ran
An experiment started withbitfab-replay --cloud now fills its Code change panel with the uncommitted changes it replayed, and records your branch and the commit you were on, the same as a replay on your machine. Before, a cloud replay showed no code change and a temporary replay branch in place of yours.The replay workflow now checks out two commits so the runner can see the commit you started from. Run bitfab-replay --cloud-update to add fetch-depth: 2 to a workflow set up with an earlier SDK.Dashboard
Create, edit, and archive assertions with your coding agent
A function’s Assertions page now has a New assertion button, and an empty page shows a guide with a prompt you can copy into your coding agent and the MCP tools it will use. Each assertion has Edit and Archive buttons that hand your agent a prompt naming that assertion and its trace. A rewritten assertion comes back for review, and an archived one stops being checked on replays while its past verdicts stay.Manage assertion categories with your coding agent
The Categories page has a New category button, and each category has Edit and Delete buttons with a ready-to-copy prompt. Deleting a category keeps its assertions and their verdicts, filed under no category.Up-to-date MCP tool lists in every guide
The create and edit guides for datasets, graders, experiments, assertions, and categories now list the current MCP tools, includinglist_trace_assertions for the review queue, get_scenario and save_scenario, save_agent_labels, and rerun_graders_on_dataset and rerun_graders_on_experiment.Dashboard on phones
On narrow screens the top bar wraps instead of running off the edge, and the function sidebar opens from a menu button instead of taking up the page.DashboardTypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.64.20, Python SDK v0.64.19, Ruby SDK v0.64.17, Go SDK v0.64.18, Plugins v0.12.51, CLI v0.2.438
Read the first model call, and every span of a trace
getTraceSpan / get_trace_span / GetTraceSpan now accepts a span type, alone or with a name, so the first model call of a trace is one call with no span ID, including model calls from the OpenAI Agents SDK that carry no name. client.spans.list reads every span of a trace in the order it ran, with each span’s ID, parent, name, and type, a page at a time, fetching content only for the fields you name.get_traces takes spanTypes next to spanNames, and lists each trace’s span types with their counts. In an experiment, the trace drawer has a Model input view that shows the input of the replay’s first model call beside the original’s, with the differences highlighted.Dashboard
Clear errors for invalid trace and dataset ids
Starting a replay with a trace id that is not a valid id now returns a validation error that names thetraceIds field, instead of a server error. Dataset requests made with an id that is not a valid dataset id now return “not found”. Requests with valid ids work exactly as before.Open a scenario by its id
Your coding agent can now read and update a scenario by its id, not only through a trace it was built from. Theget_scenario and save_scenario MCP tools take either scenarioId or traceId; send one of the two. A history entry can cite a span on any trace linked to the scenario, and edits through the SDKs keep working after one does.Dashboard
Every credit line on the usage page
The credits table on the Usage page now lists every kind of credit Bitfab charges for, even the ones you did not use in the selected range. Unused lines show 0 credits and $0.00, so the table reads like your full invoice and it is clear what was not billed.Captured data in MB and GB
Captured data on the Usage page now shows how much data was billed, in MB or GB, next to its credits and cost, instead of a count of days. Sizes use 1,000-based units so they line up with the rate: 1 GB is 200 credits, or $2.Dashboard
Compare experiments side by side
You can now compare experiments against each other instead of opening them one at a time. On a function’s Experiments page, tick two or more finished experiments and click Compare. The first one you pick is the baseline that every other experiment is measured against, and experiments that share no traces with your picks, or are still running, can’t be picked and say why.See what changed, down to each trace
The comparison lines experiments up in columns: pass rates, assertion verdicts, jitter, cost and tokens per trace, and latency split into time in LLM calls and everything else. Assertion categories and graders are ranked by biggest change, and clicking one filters the traces below to the ones that moved. Open a trace to see which assertion verdicts changed from the baseline, where the time went, each replay’s output, and a link to the full replay.PluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
TypeScript SDK v0.64.18, Python SDK v0.64.18, Ruby SDK v0.64.16, Go SDK v0.64.17, Plugins v0.12.47, CLI v0.2.434
Per-replay key lands in the replay step
bitfab-replay --cloud-update and --cloud-init now ignore comments in your workflow when they look for the replay step. Before, a comment mentioning --cloud-execute above the replay step could send the per-replay Bitfab API key to the step before it, and cloud replays quietly ran with the shared key instead. If you ran --cloud-update on a workflow with such a comment, check that BITFAB_REPLAY_BITFAB_API_KEY sits in the replay step’s env:, and move it there by hand if it does not. See Cloud replay.PluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
TypeScript SDK v0.64.17, Python SDK v0.64.17, Ruby SDK v0.64.15, Go SDK v0.64.16, Plugins v0.12.46, CLI v0.2.433
Cloud replays run as the person who started them
A replay started withbitfab-replay --cloud now runs on GitHub Actions with the Bitfab API key it would use on your machine, so the experiment shows you as its author instead of whoever owns the workflow’s shared key. The key travels as a GitHub secret made for that one replay and deleted as soon as the replay starts, never in the snapshot or the run’s inputs. It needs the GitHub CLI and permission to manage the repository’s Actions secrets; without them, the run uses the shared key as before and says why. For a workflow written by an older SDK, run bitfab-replay --cloud-update (the same command as --cloud-init): it adds what the workflow is missing, including the run name that keeps several --cloud-ref replays of one branch apart, and leaves the rest alone. See Cloud replay.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
TypeScript SDK v0.64.16, Python SDK v0.64.16, Ruby SDK v0.64.14, Go SDK v0.64.15, Plugins v0.12.45, CLI v0.2.432
Read and update scenarios
A scenario describes the situation a traced run was in: the trigger that started it and the data it had when it arrived. The newget_scenario and save_scenario MCP tools read and update the scenario built from a trace, and every SDK can do the same with getScenario and saveScenario (get_scenario and save_scenario in Python and Ruby, GetScenario and SaveScenario in Go).Add to a scenario without resending it
PassappendData (append_data in Python and Ruby) to add earlier messages to a scenario’s history or note data the run did not have. New entries go after the saved ones, and appends made at the same time are all kept. A history entry can name the span its text came from, or leave it out.PluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
TypeScript SDK v0.64.15, Python SDK v0.64.15, Ruby SDK v0.64.13, Go SDK v0.64.14, Plugins v0.12.44, CLI v0.2.431
Cloud replay checks your options before uploading
Adding--cloud to a replay now works like the local command: your SDK checks every replay option before anything is uploaded, so a typo or an unsupported option fails right away with the same error you would see locally. A cloud replay with no trace or dataset selection now replays the datasets your registry names, as it does locally.Safer snapshots and cleanup
Credential-like files, such as.env or .pem files, are refused when the snapshot would be the first to put them on GitHub, including files in commits you have not pushed. When one is safe to upload, such as a test certificate, name it with --cloud-allow-file on a private repository. --cloud-full-output is refused on a public repository, where anyone can read the job log.More reliable runs
When GitHub refuses a dispatch, the snapshot branch is removed and the command stops waiting instead of watching a run that never started.--cloud-cleanup refuses while a run is still going and tells you to stop it with --cloud-cancel first. Workflows from --cloud-init now pick the pnpm and Ruby versions your project uses, and name each run after its execution, so several --cloud-ref replays of one branch no longer get mixed up.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
TypeScript SDK v0.64.14, Python SDK v0.64.14, Ruby SDK v0.64.12, Go SDK v0.64.13, Plugins v0.12.43, CLI v0.2.430
Read a span from its span ID alone
The newget_span MCP tool takes just a span ID, either Bitfab’s or the one your SDK sent, and returns the span’s output and the trace it belongs to. You no longer need the trace ID first. In the dashboard, /spans/<span ID> opens the span on its trace, and every SDK can do the same read with getSpan(spanId) (get_span in Python and Ruby, GetSpan in Go).”Not found” says which organization was searched
Whenget_span, get_traces, or get_span_field cannot find what you asked for, the message names the organization it searched and how to switch, so a span or trace recorded in another organization is no longer a dead end.Dashboard
Skipped assertions stay skipped when Bitfab judges a replay
When you skip an assertion on a replay, for example because the original trace was never labeled, Bitfab’s assertion judging no longer replaces the skip with a pass or fail. The skip and the reason you gave stay on the assertion, and no credit is charged for the verdict that was set aside. Assertions still waiting for a verdict are judged as before.Dashboard
See which experiments ran in cloud replay
Experiments that ran in cloud replay now show a cloud icon beside the run’s author and branch, on the experiment list and on each experiment’s page. Hover it to see where the run happened: in cloud replay (started by a GitHub Action), locally, or in CI. Experiments from SDK versions that did not report where the replay ran show no icon.PluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
TypeScript SDK v0.64.13, Python SDK v0.64.13, Ruby SDK v0.64.11, Go SDK v0.64.12, Plugins v0.12.41, CLI v0.2.428
Replays keep running when span delivery can’t be confirmed
When Bitfab can’t confirm that a replay’s spans arrived in time, every SDK now prints one warning naming that replay and lets the run finish. Before, the TypeScript, Python, Ruby, and Go SDKs could stop the whole run or mark the item as failed, depending on how the replay was run. The run now completes in every mode, no item is marked as failed, and--fail-on-error in Go no longer exits with an error for this case. Bitfab shows the affected replay as capture incomplete.DashboardPluginsTypeScript SDKPython SDKGo SDKCLI
TypeScript SDK v0.64.12, Python SDK v0.64.12, Go SDK v0.64.11, Plugins v0.12.40, CLI v0.2.427
Experiment totals type includes judging
The experiment totals type in the TypeScript, Python, and Go SDKs now has judging, the number of traces in the run, replays or originals, that still have an assertion verdict being judged. The server already returned it, but the SDK types left it out, so you had to read it as an untyped key.Faster rollups for large experiments
Experiment pages andget_rollup now return quickly for runs with hundreds of replays. Before, the time grew with the square of the run’s size, and a run with 550 replays took over two seconds.Dashboard
The usage page loads for high-volume organizations
The usage page now loads for organizations that capture a large volume of spans. Before, its chart, per-function chart, and summary cards could keep loading for minutes, because adding up captured data looked through far more than your organization’s own spans. Now they read only your organization’s spans, and the numbers they show are unchanged.Dashboard
Experiment pass rates show as soon as the run finishes
When an experiment run finishes with a replay that never received all of its spans, the pass rate now shows straight away instead of spinning for up to an hour. Those replays count as errored and are left out of the pass rate, and their trace page says the run finished before every span arrived. Replays in runs that are still going are unchanged.PluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
TypeScript SDK v0.64.11, Python SDK v0.64.11, Ruby SDK v0.64.10, Go SDK v0.64.10, Plugins v0.12.39, CLI v0.2.426
No sim plan requests without an API key
When no API key is set, the TypeScript, Python, Ruby, and Go SDKs no longer ask Bitfab for your sim plan. Before, a client without a key sent a request that Bitfab rejected, and sent it again every 10 seconds while spans flowed. Now the client logs a single warning that no API key is set and makes no request. If a key becomes available later, for example throughBITFAB_API_KEY or a key function, the plan loads on the next try.DashboardPluginsTypeScript SDKPython SDKGo SDKCLI
TypeScript SDK v0.64.10, Python SDK v0.64.10, Go SDK v0.64.9, Plugins v0.12.38, CLI v0.2.425
Experiment rollups report the run’s pass and fail totals
In an experiment rollup,passing and failing now count every trace, assertion, or check that passed or failed, including ones that were also compared against the original as fixed, still passing, regressed, or still failing. Before, they counted only results with nothing to compare against, so passing / (passing + failing) did not match the pass rate on the experiment page; now it does. If you added fixed and stillPassing to passing yourself, read passing alone instead.Rollup totals also include traces, the number of distinct traces the run replayed, alongside total, which counts one replay per trace per attempt.Dashboard
Live updates during large labeling runs
When many traces in one experiment are labeled at once, such as during a grader run or a bulk label save, the experiment’s page now applies every live update. Before, a large burst could leave some of the experiment’s results out of date until you reloaded the page.Dashboard
Credit usage on the Usage page
The Usage page now shows how many credits your organization used in the selected range and what they cost. A row for each thing Bitfab charged for (replays, grader verdicts, assertion verdicts, and captured data) shows how many there were, the credits, and the cost. Credits cover every function in your organization, so they stay the same when you filter the page to one function.Usage days are UTC days
Every day on the Usage page is now a UTC day, the same days credits are charged on, so the totals for a date range are the same wherever you view them from.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.64.9, Python SDK v0.64.9, Ruby SDK v0.64.9, Go SDK v0.64.8, Plugins v0.12.37, CLI v0.2.424
Settings for one cloud replay
You can now change an environment variable for a single cloud replay without editing the workflow. Add--cloud-env NAME=VALUE to bitfab-replay --cloud, once per variable, such as --cloud-env LOG_LEVEL=debug, or pass --cloud-env NAME to copy the value from your shell. Because the values are stored with the replay on GitHub, names that look like secrets and values that contain a password are refused; keep secrets in the workflow and set them with --cloud-secrets.Interrupted cloud replays can also be picked up where they stopped: run the same command with --resume <experiment ID> and --cloud, and only the traces that had not finished are replayed.DashboardTypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.64.8, Python SDK v0.64.8, Ruby SDK v0.64.8, Go SDK v0.64.7, Plugins v0.12.36, CLI v0.2.423
Labels are recorded on assertions
Every label now scores one of a trace’s approved assertions instead of the trace as a whole.save_agent_labels, save_human_labels, and the SDK label writes (save, saveAll, skip, saveHuman, saveHumanAll) refuse an entry without an assertionId; archive may still omit it to clear an older label. Existing labels are unchanged and still readable.No more label approval
Agent verdicts now count as soon as they are written, with no approval step. The dataset page no longer has an “awaiting your approval” section or Approve buttons; to correct an agent’s verdict, save your own on that assertion. Labels approved in the past still count as validated.Labeling in the dashboard is per assertion
The trace labeling panel records a pass or fail on each assertion. A trace with no assertions shows an empty state instead of the whole-trace Pass, Fail, and Skip buttons.The assistant labels assertions only
/bitfab:assistant writes its verdicts on approved assertions only. Traces without assertions are left for you to label, and replays of them are judged in the chat report.DashboardTypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.64.7, Python SDK v0.64.7, Ruby SDK v0.64.7, Go SDK v0.64.6, Plugins v0.12.35, CLI v0.2.422
Credit-based pricing
Bitfab pricing is now published in credits, where one credit is $0.01. The pricing page lists the Lab, Forge, and Foundry plans with monthly and annual prices, the credits each plan includes every month, and how many credits replays, grader verdicts, assertion verdicts, and captured data use. Every plan button opens a booking link to talk with the team.Replays report how they were started
When a replay starts, the SDKs now tell Bitfab which replay options were set, the SDK language and version, and whether the replay ran locally, in CI, or in cloud replay. Options whose names contain key, token, secret, or password are never sent, and metadata and parameters send only their names. If this information cannot be collected, the replay starts without it.Dashboard
Experiments tab opens without waiting for the list
Clicking the Experiments tab, for one function or for all functions, now switches to the Experiments page without waiting for the list to load. The page shows its header and placeholder rows until the experiments arrive, instead of leaving the previous page on screen.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.64.6, Python SDK v0.64.6, Ruby SDK v0.64.6, Go SDK v0.64.5, Plugins v0.12.34, CLI v0.2.421
Cloud replay runs fail when the replay fails
The GitHub Actions job that runs a cloud replay now exits with the replay’s exit code, so a workflow that dispatchesbitfab-replay.yml directly shows a red run when the replay fails. A run fails on bad arguments, when no traces match, when the replay cannot finish, or when a trace errors under --fail-on-error; changed outputs never fail it. bitfab-replay --cloud output and exit codes are unchanged, and continue-on-error: true on the replay step keeps a run green. See Cloud replay.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.64.5, Python SDK v0.64.5, Ruby SDK v0.64.5, Go SDK v0.64.4, Plugins v0.12.33, CLI v0.2.420
Scheduled cloud replays of a branch
A CI job can now replay a branch exactly as it is on GitHub, with no snapshot, and the experiment records that branch, so a nightly replay ofdev counts as a dev experiment again. Add --cloud-ref dev to a bitfab-replay --cloud command, or have another workflow dispatch bitfab-replay.yml on the branch with the replay as JSON in its request input. Only args is required in the request, and requests from earlier SDKs are still accepted. The workflow needs the optional request input, which --cloud-init now writes; see Cloud replay for the format.Dashboard
Open experiments and groups from the list
On the Experiments page, clicking an experiment now opens its own page, and clicking an experiment group’s header opens the group’s page, instead of expanding the row in place. A group’s page shows every experiment in it with its results and traces. Experiment and group pages now keep their layout while they load instead of jumping as results arrive.Review draft assertions from your coding agent
Your coding agent can now read assertions that are still waiting for review, not only approved ones. The newlist_trace_assertions tool finds them across traces by review state, optionally narrowed to one traced function or category, so the agent can work through the review queue page by page and approve or reject each one with save_trace_assertions. get_trace_assertions also takes approvalStates to read pending or rejected assertions on traces you already have. It still returns only approved assertions by default, since only those are checked on a replay.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.64.4, Python SDK v0.64.4, Ruby SDK v0.64.4, Go SDK v0.64.3, Plugins v0.12.31, CLI v0.2.418
Large cloud replays finish cleanly
bitfab-replay --cloud now prints a short summary of the replay’s result instead of every item: the experiment ID and URL, how many traces were replayed, came out the same, changed, and errored, each errored trace with its error, and the exit code. Runs with more than a few dozen traces no longer fail at the end because the result outgrew the GitHub job log. Every trace’s input and output are saved in the experiment in Bitfab, and --cloud-full-output prints the whole result as before.Dashboard
Faster dataset pages
Dataset pages and the datasets list now load much faster, especially for datasets whose traces have been replayed many times. Each dataset now lists only the experiments launched on it, so an experiment launched on a different dataset, or on no dataset, no longer appears there just because it replayed some of the same traces. Every experiment is still listed on the Experiments page.Dashboard
Each experiment shows its own dataset
On the experiments list, every experiment row now links to the dataset it ran against. Experiment group headers, and the header of an experiment group’s page, no longer show a single dataset, since the experiments in one group can run against different datasets.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.64.3, Python SDK v0.64.3, Ruby SDK v0.64.3, Go SDK v0.64.2, Plugins v0.12.30, CLI v0.2.417
Cloud replay runs stay green when the replay fails
A cloud replay’s GitHub Actions run now succeeds once the replay finishes and hands back its result, so a replay that finds nothing to run, or exits 1 under--fail-on-error, no longer leaves a failed run in your Actions tab. The exit code shows as a warning on the run, and bitfab-replay --cloud still exits with it. The run still fails when something stops it before the replay finishes, such as an install step or an empty secret.Larger cloud replays
Apps that log heavily to stdout can now run large cloud replays: when the replay prints more than 16 MiB, the runner keeps the replay’s result instead of stopping the replay partway.--trace-ids also accepts longer lists, up to the 16,000-character limit on all replay options together.JSON secrets no longer hide output
--cloud-secrets now stores a multi-line JSON value, such as a service account key, on one line. GitHub masks each line of a secret on its own, so a line holding only a brace hid every brace in the replay’s output. The runner now warns when a secret has a line that short.TypeScript SDKPython SDKRuby SDKPluginsCLI
TypeScript SDK v0.64.2, Python SDK v0.64.2, Ruby SDK v0.64.2, Plugins v0.12.29, CLI v0.2.416
Replay registries that match your app
When the Bitfab plugin sets up replay, it now puts the replay registry where your app builds the traced function’s call. That is often an app package rather than the library that defines the function. The registry rebuilds tools, clients, and other arguments a trace cannot record by calling your app’s own code, so a replay runs with the same wiring as production instead of a smaller copy. The plugin asks before moving any of your code, and Go projects now get a compiled registry command. Each SDK guide has a new section with examples for this setup.Clearer replay registry errors
When a registry cannot load one of its imports,bitfab-replay now keeps the original error and explains where the registry belongs and how to run it. In TypeScript, a decorated method registered as instance.method.bind(instance) keeps its trace function key, so replay records it as the root instead of nesting it under a second one. A misplaced registry option, such as adaptInputs outside options, now logs a warning instead of being silently ignored.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.64.1, Python SDK v0.64.1, Ruby SDK v0.64.1, Go SDK v0.64.1, Plugins v0.12.28, CLI v0.2.415
Cloud replay reads results from apps with coloured logs
Cloud replay failed to fetch results for apps with coloured logs when GitHub CLI 2.101 or later was installed. The run finished on GitHub Actions, butbitfab-replay --cloud and --cloud-watch ended with “gh api failed (exit 1); check authentication and permissions”, because that GitHub CLI release refuses to print a log that contains terminal colour codes. Cloud replay now takes the GitHub CLI’s login and talks to GitHub directly, so the same runs print their output and exit with the replay’s exit code. Update your SDK; no setup changes are needed.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.64.0, Python SDK v0.64.0, Ruby SDK v0.64.0, Go SDK v0.64.0, Plugins v0.12.27, CLI v0.2.414
Cloud replay behaves like your local replay
Adding--cloud to a bitfab-replay command now prints the same output the replay prints locally and exits with the same exit code, so scripts and CI read a cloud run exactly like a local one. --dry-run and --fail-on-error work the same way. While a run is queued or running, the command shows the GitHub Actions link and each step as it starts. If a run fails before the replay starts, such as a failed install step, the log up to the error is printed.New files are included automatically
The snapshot a cloud replay runs on now includes new files that git does not ignore, just like your local working tree, so--cloud-include is no longer needed. Ignored files stay out, and credential-like files such as .env are refused. Use --cloud-preview to see what a snapshot would carry without pushing anything.Simpler setup
bitfab-replay --cloud-init writes a smaller workflow file and ends with a checklist of what is left to do, including the exact secrets to create and where. Go projects no longer need --run: it finds the program that calls RunCloudReplayCLI. --cloud-secrets now creates a missing GitHub Environment, and without the GitHub CLI it lists the secrets to create by hand instead of failing.Breaking changes
--cloud-dry-run is now --cloud-preview, --cloud-check is now --dry-run, and --cloud-include is gone; each old flag stops with a message naming its replacement. A cloud replay where every trace errored now exits 0 like a local replay, so add --fail-on-error to exit 1. In Go, RunCloudReplayCLI returns only an error. A setup that still has .bitfab/cloud.json needs that file and its workflow deleted, then --cloud-init run again. Workflows made by the previous release keep working.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.63.3, Python SDK v0.63.3, Ruby SDK v0.63.3, Go SDK v0.63.4, Plugins v0.12.26, CLI v0.2.413
Successful cloud replays report success
A cloud replay that finished on GitHub Actions no longer fails at the very end. Before this fix,bitfab-replay with --cloud exited 1 after the replay had already succeeded, and the result never came back to your machine, so the run looked failed. Passing --cloud --dry-run checks had the same problem. Update your SDK and the same runs now exit 0 and report their result locally.Dashboard
Assertions judged on the part of the trace they check
An assertion that checks the trace output is now judged on the trace’s final output alone, instead of on every step in the trace. Text the run was given as input, or that appears only in an intermediate step, no longer counts against an assertion about the output, so these verdicts are more accurate and more consistent from run to run.An assertion that checks one named span is judged on that span. If the span is not on the trace, the assertion is marked skipped rather than passed.Dashboard
Assertion counts on the datasets list
The datasets list now shows how many assertions each dataset’s traces carry. Hover the count to see how many are approved, not reviewed, and rejected, and click the Assertions header to sort datasets by it.Latest experiment on the datasets list
Each dataset now shows the most recent experiment run on it: that experiment’s pass rate, its name, and how long ago it ran. The pass rate matches the one on the experiment’s own card. This replaces the per-grader pass rate pills, so every dataset reads as one number in one column.Generated assertions state the answer
When Bitfab generates assertions from a trace, each draft is now one self-contained claim that states the settled outcome with its real values, such as names, dates, times, and attendees. Drafts no longer open on a condition like “if the meeting is booked” or point at a value like “the times the user approved”, which left replays undecided. They also no longer pin a choice the original run happened to make when other choices would be just as correct, such as which of several open slots it offered.Generated drafts no longer carry pass or fail criteria, because the claim alone is the check. Assertions you already have keep theirs. Your coding agent gets the same guidance when it writes assertions withsave_trace_assertions.Dashboard
Graders and assertions shown apart
An experiment’s breakdown now shows a Graders row and an Assertions row under Labels, so you can see which kind of check passed and which failed. Each grader gets its own tile next to your assertion categories, with its pass rate and how many traces it fixed or regressed. On a trace, the Labels panel lists assertion verdicts under the graders, andget_experiment reports grader and assertion results separately.Experiments wait for assertion judging
An experiment no longer reads as done while its assertions are still being judged. Each assertion shows as Judging until its verdict lands, and a judge that fails marks the assertion skipped instead of waiting forever. Replays that crashed without output are no longer sent to graders or the assertion judge.Fixed and regressed compare like with like
When an experiment judges assertions, the original trace’s approved assertions are judged too, so each assertion compares with its own verdict on the original. A grader or assertion the original has no verdict for counts as passing or failing, never as fixed or regressed.Dashboard
One pass rule for graders and assertions
Every place that shows whether a trace passed now uses the same rule: a trace passes when at least 75% of its verdicts passed, counting automated grader verdicts, assertion verdicts, and a whole-trace verdict together. The trace list, the experiment totals and breakdown, and the experiment trace filters now agree on the same trace, where before one could say passed while another said failed. A whole-trace verdict counts as one verdict among the rest rather than deciding the trace on its own.Trace pass ratios count every verdict
The pass ratio on each trace now includes every grader, assertion, and whole-trace verdict on it, whether a person or a grader wrote it, so a trace with a failing grader no longer reads as a clean pass. Verdicts from archived graders are no longer counted, and the Labels panel counts automated grader verdicts toward how many graders are labeled.Large traces stay readable in your coding agent
Traces from long-running agents can hold tens of thousands of spans, andget_traces used to return all of them at once, far more than a coding agent can read. It now returns spans a page at a time (50 per trace by default) and keeps every response small enough to read inline. When a trace has more spans than one page, the response lists each span name with its count and the spanOffset for the next page.Narrow a read with spanNames and spanFields. Span fields over 10,000 characters are left out of the default view and shown as their size, and get_span_field now reads a field of any length in parts, using the offset each response gives. Paging and filtering need the latest plugin. Older plugins still get the smaller responses and are told to run bitfab:update.TypeScript SDKPython SDKRuby SDKGo SDKDashboard
TypeScript SDK v0.63.2, Python SDK v0.63.2, Ruby SDK v0.63.2, Go SDK v0.63.2, Plugins v0.12.22, CLI v0.2.409
Dry runs no longer create experiments
A replay dry run (dryRun in TypeScript, dry_run in Python and Ruby, DryRun in Go, or --dry-run on bitfab-replay) still resolves every trace’s inputs without running your function, and it no longer adds an empty row to the experiments page. Dataset, grader, and experiment group checks still run, so a dry run fails where the real replay would.With no experiment behind it, the result’s experiment ID and URL are null (None in Python, nil in Ruby, empty strings in Go), and the experiment-start callback does not fire. In TypeScript, experimentId and experimentUrl on the replay result are now typed string | null, so code that reads them may need a null check.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.63.1, Python SDK v0.63.1, Ruby SDK v0.63.1, Go SDK v0.63.1, Plugins v0.12.21, CLI v0.2.408
Cloud replay is your normal replay plus —cloud
Take the replay command you already run and add--cloud. Every option works as it does locally, in any order, and the runner starts the same bitfab-replay from the same directory with the same arguments. --dry-run with --cloud checks the runner without replaying anything.One file in your repository
bitfab-replay --cloud-init --secret OPENAI_API_KEY writes .github/workflows/bitfab-replay.yml and nothing else, with the runtime, package manager, and install command detected for Node, Python, and Ruby projects. The secrets the replay reads are listed once, in that workflow. A setup made with .bitfab/cloud.json needs bitfab-replay --cloud-init run once after upgrading: until then cloud replay stops and says so. That run moves the settings into the workflow and deletes the file.No GitHub CLI required
Cloud replay reaches GitHub through the GitHub CLI when it is logged in, otherwise withGH_TOKEN or the github.com credential git already stores for pushing. Only copying secrets from local files needs the GitHub CLI, and without it that command says which secrets to create by hand and where.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.63.0, Python SDK v0.63.0, Ruby SDK v0.63.0, Go SDK v0.63.0, Plugins v0.12.20, CLI v0.2.407
Replays judge assertions by default
Every replay now judges its approved assertions as it finishes and saves each verdict as that assertion’s agent label, with no flag needed. Judging costs model calls, so turn it off with--skip-assertion-judging on bitfab-replay, or the replay option:skipAssertionJudging in TypeScript and SkipAssertionJudging in Go. The old --judge-assertions flag and judgeAssertions option still work for now and print a deprecation warning.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.62.8, Python SDK v0.62.7, Ruby SDK v0.62.7, Go SDK v0.62.7, Plugins v0.12.19, CLI v0.2.406
Experiment diffs show the change you are trying
When you replay without passing a code change, the SDK now attaches your uncommitted changes against your last commit, instead of everything your branch changed sincemain. Each experiment’s diff shows only the edit being tried. To diff your whole branch again, set BITFAB_CODE_CHANGE_BASE to a ref such as origin/main.PluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.12.18, TypeScript SDK v0.62.7, Python SDK v0.62.6, Ruby SDK v0.62.6, Go SDK v0.62.6, CLI v0.2.405
Cloud replay explains what went wrong
When a replayed trace errors on GitHub Actions, the job log and summary now list each failing trace with its error, so a permissions problem shows up on the first run. If the replay command itself fails, the end of its output is printed too. The runner also refuses to start when a secret named in.bitfab/cloud.json is empty, and names each missing one, and setup errors say which check failed, such as a workflow GitHub has not registered.Check the runner before you replay
bitfab-replay --cloud <pipeline> --trace-ids <id> --cloud-check starts a run that replays nothing. It confirms every configured secret has a value, runs an optional checkCommand from .bitfab/cloud.json (such as reading one object from your bucket), and loads your registry to resolve the traces.Use your usual replay options in the cloud
Replay options after the pipeline, such as--dataset-ids, --name, --attempts, or --resume, now pass through to the replay on GitHub Actions. A workflow GitHub has registered can be dispatched even when your default branch holds an older copy, since each replay runs the workflow in its own snapshot.Exit with an error when items fail
bitfab-replay --fail-on-error exits 1 when any replayed item errored, locally and with --cloud, so scripts and CI can stop on a failing run. Without the flag the command still exits 0.Renamed secrets and repository variables in cloud.json
Anenv field in .bitfab/cloud.json maps a replay variable to a differently named GitHub secret or to a repository variable. --cloud-init writes it into the workflow and --cloud-secrets sets the right secret. Running bitfab:setup cloud now also explains that the runner signs in only with its own secrets, recommends a separate Bitfab key and a GitHub Environment with required reviewers, and writes credential files outside the checked-out repository.Dashboard
Clearer pass and fail bars on experiments
The Traces and Labels bars on an experiment page now use four colors in a fixed order: held, fixed, regressed, and still failing. Traces with no earlier run to compare against are counted in held when they pass and in still failing when they fail, instead of getting their own pale segments that were hard to tell apart. The filters above the trace list count them the same way, so the counts there match the bars.Dashboard
Every judged assertion gets a pass or a fail
When a replay judges assertions, each one now comes back as a pass or a fail. Before, the judge set aside cases where it could not find the deciding detail, which left them out of the pass rate. Now an action the run never took counts as a fail, and an assertion that the run does not do something passes unless the run did it.Judging reads the whole run
The judge now reads the full trace, up to a size limit, instead of a handful of spans picked in advance, and it tells the run’s own actions apart from earlier conversation it was handed. Past messages quoted in a prompt no longer count as something the run did. When a trace is too large to read in full, the verdict’s note says how much was left unread.PluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.12.17, TypeScript SDK v0.62.6, Python SDK v0.62.5, Ruby SDK v0.62.5, Go SDK v0.62.5, CLI v0.2.404
Cloud replay workflows hold only your settings
The cloud replay workflow in your repository now holds only your own settings, so every fix to cloud replay reaches you when you upgrade the SDK, with no workflow edits. Runningbitfab:setup cloud or bitfab-replay --cloud-init again on an existing setup updates it in place and keeps your install steps, secrets, runner, and environment. When your setup is already current, it changes nothing.Cloud replays can run for hours
Cloud replays no longer stop at 25 minutes. A replay can now run up to GitHub’s 6-hour job limit, and you can set a shorter limit for one replay with--cloud-timeout <minutes>. When a replay stops early, the command still reports its test run, and the traces that finished are saved in Bitfab.PluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.12.15, TypeScript SDK v0.62.4, Python SDK v0.62.3, Ruby SDK v0.62.3, Go SDK v0.62.3, CLI v0.2.402
Cloud replay updates with your SDK
The code that runs cloud replay now ships inside each Bitfab SDK, so upgrading the SDK brings its fixes to your GitHub Actions runs.bitfab:setup cloud now keeps only your settings in the repository, .bitfab/cloud.json and the workflow, and runs setup through your SDK’s bitfab-replay command with --cloud-init and --cloud-secrets. If an earlier setup left .bitfab/cloudReplay.py in your repository, it keeps working, and running bitfab:setup cloud again moves the workflow to the SDK command.Cloud replay reports traces that errored
A cloud replay now reports how many traces it replayed and how many errored. When every trace errored, so nothing actually ran, the command says so and exits with an error instead of reporting success.More reliable cloud replay submissions
Submitting a cloud replay and waiting for it no longer stops early right after the job starts on GitHub, and it works in terminals that force colored output.TypeScript SDKPython SDKRuby SDKGo SDK
TypeScript SDK v0.62.5, Python SDK v0.62.4, Ruby SDK v0.62.4, Go SDK v0.62.4, Plugins v0.12.16, CLI v0.2.403
Read experiment results from every SDK
Each SDK client now has anexperiments namespace. list pages your experiments newest first with optional filters for dataset, traced function, git branch, experiment group, status, metadata, and a creation window. get reads one experiment with who ran it, its git state, and when it completed. getRollup and getRollupAll return the pass/fail rollup the dashboard shows: totals, per-trace and per-assertion tallies, and one row per assertion category, for up to 100 experiments in one request. A dashboard outside Bitfab can now chart pass rate by category over time from a few calls.Tag an experiment with metadata
Passmetadata on replay, or --metadata key=value on the replay command, to save your own string tags on the experiment. They come back on every read and experiments.list filters on them, so a scheduled run is easy to tell apart from an ad hoc one on the same branch. Experiments also record completedAt, the moment the replay reported completion, for trends that key off when a run finished.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.12.14, TypeScript SDK v0.62.3, Python SDK v0.62.2, Ruby SDK v0.62.2, Go SDK v0.62.2, CLI v0.2.401
External graders
Bring the checks you already run in your own eval harness, CI, or review queue into Bitfab. An external grader is a grader Bitfab never runs: you post its pass or fail verdict for each trace, and those verdicts count in dataset and experiment pass rates and show in each trace’s grader panel like any other grader’s. They count as already decided, so they never wait for approval.Ask your coding agent to create one: the plugin’ssave_grader tool takes external: true, and save_grader_labels posts verdicts. From code, every SDK can post them too, one at a time or up to 200 at once with saveLabelAll (save_label_all in Python and Ruby, SaveLabelAll in Go).Sign-in no longer depends on Clerk
Bitfab no longer uses Clerk for sign-in or account data. Signing in with Google now refreshes your name and profile photo each time, so your account stays current across the dashboard. The Trust Center subprocessor list and the Data Processing Agreement, now version 2.1, no longer list Clerk.Cloud replay covers your whole registry
bitfab:setup cloud now sets up GitHub Actions replay for every function in your replay registry, so one setup can replay any of them. To limit cloud replay to a single function, name it in the pipeline field of .bitfab/cloud.json.Cloud replay setup checks more before it installs
Setup now checks that your installed SDK supports cloud replay and offers to update it first. It recommends database branching, so the runner needs no database secret, and checks that your code reads the replay branch URL on each call. It also handles a Bitfab key your code reads under a different name, checks that the GitHub Environment you pick exists, and a dry run now names any setup file you have not committed yet.Dashboard
Rename experiments and experiment groups
You can now rename an experiment or an experiment group from its page in the dashboard. Click the pencil next to the title, type the new name, and click Save. A group’s name also appears on its row in the experiments list and in the page breadcrumb, which makes a set of runs launched together easier to find again.PluginsTypeScript SDKRuby SDKGo SDKCLI
Plugins v0.12.11, TypeScript SDK v0.62.2, Ruby SDK v0.62.1, Go SDK v0.62.1, CLI v0.2.398
Judge assertions from the replay command in every SDK
The replay command in the TypeScript, Ruby, and Go SDKs now accepts--judge-assertions, as the Python SDK already does. It judges every approved assertion on each replayed trace as it finishes and saves the verdict as that assertion’s agent label. Leaving the flag off keeps a value set on the replay registry entry, so a registry can make judging the default. Judging stays off unless asked, because each judgement costs model calls.--skip-assertion-judging to turn it off.DashboardPluginsTypeScript SDKPython SDKCLI
Plugins v0.12.10, TypeScript SDK v0.62.1, Python SDK v0.62.1, CLI v0.2.397
Resumed replays keep their settings
When a replay stops partway, theResume with: line that bitfab-replay prints now carries the --mock, --params, and --param flags the run started with, so the remaining items run exactly like the first ones instead of falling back to the defaults. Paths with spaces are quoted so the line can be pasted as is. The coding agent plugins also resume correctly when replay runs through node or python -m, and warn when a command cannot be resumed instead of silently starting over.Stopped replays resume cleanly
Pressing Ctrl-C during a TypeScript replay now stops new items from starting, even when your program keeps running or runs two replays at once. A trace that was cut off mid-run is replayed again when you resume, even hours later, while traces that genuinely errored are kept as they were. Resuming a selective replay is now refused rather than finishing it as a full replay, and experiments over very large datasets start without error.Dashboard
Environment filter remembered per organization
The environment you pick in the dashboard is now remembered separately for each organization you belong to. Before, a pick made in one organization carried over to the others, and in an organization that never sends that environment it could hide every trace with no way to clear it. Each teammate’s pick also stays their own.Cloud replay secrets are easier to set up safely
Setting up replay in your own GitHub Actions no longer means hand-copying every credential. Setup now names each runner secret with aBITFAB_CLOUD_ prefix, so replay credentials never collide with secrets your repository already uses, while the replay still reads each one under its own name. With your approval it can also copy the values from your local environment files, handling quoted, multi-line, and JSON values and passing each one to GitHub without printing it anywhere.Dashboard
Experiments load quickly with large code changes
The Experiments page no longer stalls when experiments carry large code changes. Each experiment row shows its added and removed line counts and changed file count right away, and the changed files themselves load only when you open the code change view.Dashboard
Organization invitations report failed emails
When Better Auth is enabled and an organization invitation email can’t be sent, the invite now fails with a message asking you to try again instead of appearing to succeed, and the unsent invitation is cancelled. Inviting the same address again replaces any earlier pending invitation. On the invitation page, someone signed in with the wrong account can now choose “Sign in with a different account” and come back to the invitation after signing in.Experiments say why a replay can’t be compared
When a replayed trace has no before and after result, experiments now say why instead of calling it unpaired. A replay whose original trace was never labeled is counted apart from a replay with no original at all, so a run on a dataset of unlabeled traces no longer reports every trace as unpaired. Replays still waiting for a label, and replays that didn’t finish capturing, are named as such in the experiment results and in what your coding agent reads back.Dashboard
The experiments page stays responsive while a run streams in
Watching an experiment fill in live no longer slows the page down. The dash that circles a pill while its traces are still running used to make the browser restyle the entire page on every frame, so scrolling and clicking lagged during a busy run, and it got worse the more results were on screen. The dashes now animate on their own, and they still circle in step with each other.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.62.0, Python SDK v0.62.0, Ruby SDK v0.62.0, Go SDK v0.62.0, Plugins v0.12.7, CLI v0.2.394
Replay items no longer carry trace outlines
Replay items no longer carry the original and replayed trace outlines. On runs with big traces they made the response that finishes a replay too large, and finishing the replay failed. To compare how two runs executed, open the trace in Bitfab.TypeScript SDKPython SDKPluginsCLI
TypeScript SDK v0.61.6, Python SDK v0.61.6, Plugins v0.12.6, CLI v0.2.393
Replayed traces keep their id when Bitfab confirms them late
A replayed trace now comes back with its Bitfab trace id even when the upload took a while to be confirmed. The Python and TypeScript SDKs read the id from whichever confirmation arrives first, the upload response or the status check that follows it, instead of only the first one.This shows up most underprimitive="process" in the Python SDK, where an on_item_finish_in_child_process hook grades each item inside the process that replayed it. Such a hook used to receive no trace id for those items, so it could not look up that trace’s labels or generate label evidence for it. The Ruby and Go SDKs already worked this way.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.61.5, Python SDK v0.61.5, Ruby SDK v0.61.2, Go SDK v0.61.2, Plugins v0.12.5, CLI v0.2.392
Replay experiments take notes, and names say what they test
bitfab-replay now accepts --notes, and every SDK’s replay call takes a matching notes option. Use it for run conditions Bitfab cannot see on its own, such as an environment override or a forced feature flag. The notes are saved on the experiment next to its name.The --name help and the replay name option now explain that a name should say what the run is testing, such as baseline or shorter system prompt. Bitfab already records the commit, branch, datasets, and who ran it with every experiment, so they no longer need to go in the name. The MCP save_experiment tool describes its name and notes fields the same way.Dashboard
Span links open inside experiments
A link to a span inside an experiment or an experiment group now opens the trace with that span selected, whether you refresh the page, open it in a new tab, or share it with a teammate. These links used to land on a not-found page.DashboardTypeScript SDKPython SDKPluginsCLI
TypeScript SDK v0.61.4, Python SDK v0.61.4, Plugins v0.12.4, CLI v0.2.391
Stopped experiments show as interrupted
An experiment whose replay was stopped with Ctrl-C, crashed, or was killed now shows as Interrupted on the experiments page, with how many of its traces finished and the command to resume it. It used to sit there with a spinner that never stopped.Press Mark finished on the experiment to grade the traces that already landed, or resume it to run the rest. A replay sends a heartbeat while it runs, so an experiment is marked interrupted only once it has gone quiet for 15 minutes, andbitfab-replay prints the resume command when it stops.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
TypeScript SDK v0.61.3, Python SDK v0.61.3, Ruby SDK v0.61.1, Go SDK v0.61.1, Plugins v0.12.3, CLI v0.2.390
Experimental selective replay
Choose which calls must run during a replay withexperimentalSelectiveReplay in TypeScript or its equivalent in Python, Ruby, and Go. Eligible calls around them can reuse recorded outputs when their inputs match. Calls needed by assertions stay live, while marked safety mocks cannot execute. Each replay item reports execution coverage separately from assertion verdicts.TypeScript SDKPython SDKDashboardPluginsCLI
TypeScript SDK v0.61.2, Python SDK v0.61.2, Plugins v0.12.2, CLI v0.2.389
Resume a stopped replay experiment
When a replay crashes or you stop it, you can now pick it back up by its experiment ID instead of starting over. Runbitfab-replay --registry <path> <pipeline> --resume <experimentId>, or pass resume to replay in the TypeScript and Python SDKs. Only the traces that never finished are replayed, and the ones that already finished come back marked as carried over.--force resumes on your current code instead. After a crash, the Bitfab assistant now resumes a dataset replay rather than starting it over.TypeScript opt-out tracing now needs one package
@bitfab/sdk now includes every automatic subtree transform adapter. New projects install only the SDK, then wire the adapter for their build from @bitfab/sdk/transform/next, /vite, /esbuild, /register, or another supported subpath. The separate @bitfab/transform installation is no longer required.Existing @bitfab/transform imports remain compatible while projects move their build configuration to the SDK subpaths.Setup preserves existing transform configurations
Bitfab setup checks now recognize both@bitfab/sdk/transform/ and the legacy @bitfab/transform/ adapter paths as valid subtree tracing configurations. Existing projects keep their current adapter instead of receiving a duplicate SDK adapter during setup.PluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.12.0, TypeScript SDK v0.61.0, Python SDK v0.61.0, Ruby SDK v0.61.0, Go SDK v0.61.0, CLI v0.2.387
Experimental replay on your own GitHub Actions
You can now try running replay in your own GitHub Actions environment with the opt-in cloud pilot. Runbitfab:setup cloud in your coding agent to generate the repository-owned setup files. Configure the requested GitHub secrets before landing the workflow on your default branch. Live customer replay verification remains pending.Preview the source snapshot before submitting a replay with your configured pipeline and a real trace UUID.--cloud-status with the printed execution UUID to check progress. Temporary branches are retained when dispatch status is uncertain. See the cloud replay guide for setup requirements and pilot limits.Dashboard
Faster, recoverable sign-in
Better Auth sign-in now keeps OAuth cancellations and callback failures inside Bitfab, with retry actions that preserve whether you were signing in or creating an account and return you to the page you originally opened. Authentication pages and organization access also perform fewer database round trips, improving response times when Better Auth is enabled.Warnings when old plugins ignore new options
When a newer Bitfab workflow sends options that the installed plugin version does not support, Bitfab now completes the supported part of the call and warns which options it ignored. Plugin warnings identify the receiving version so you can tell when an update is needed, and direct backend tool calls now surface the same compatibility warning.Dashboard
More reliable live updates
Live trace, dataset, experiment, and labeling views now stay responsive during large bursts of updates without overwhelming database connections. Lightweight updates continue concurrently while heavier refreshes are paced in the background.Dashboard
Assertion judges get more time and run in parallel
When a replay judges assertions, each judgement now has up to 800 seconds to reach a verdict instead of 180. Assertions on large traces that used to be recorded as undecided with “The judge ran out of time before reaching a verdict” now get a real pass or fail. A trace’s assertions are also judged at the same time rather than one after another, so verdicts for a replay land sooner after it finishes.Judge assertions from the Python client and the replay command
Bitfab.replay() now accepts judge_assertions=True. The replay judges every approved assertion on each replayed trace as it finishes and saves the verdict as that assertion’s agent label with its cited evidence. Before this release only the module-level bitfab.replay() accepted the flag, and passing it to the client raised a TypeError.skip_assertion_judging=True to turn it off.The bitfab-replay command gains --judge-assertions, and a replay registry entry can set judge_assertions=True so every run from that entry judges by default. Judging stays off unless asked, because each judgement costs model calls.Dashboard
Signing in no longer depends on the hourly profile refresh
Bitfab refreshes your name, email, and avatar from your identity provider about once an hour. Until now, if that refresh could not be saved, for example while the database was unavailable for writes or the identity provider was unreachable, every request you made failed, including pages that only read data. The refresh is now best-effort. When it fails, Bitfab serves your stored profile and you stay signed in, and your details catch up on the next successful refresh.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.60.0, Python SDK v0.60.0, Ruby SDK v0.60.0, Go SDK v0.60.0, Plugins v0.11.59, CLI v0.2.383
Assertion human notes are writable from every SDK
humanNote carries people-only context on an assertion, such as who wrote it or what a reviewer should compare it against. Until now only the coding agent tools and the Bitfab UI could write one. An SDK save accepted the field, returned success, and stored nothing. Pass it on a saveAssertions entry and it is saved. Omit it on an edit to keep what is stored, and pass null to clear it. It is still never used as evidence when judging a replay.The detached trace handle writes even with capture off
client.getTrace(id) names a trace that already exists, so setMetadata, setSessionId, setName, and addContext on it no longer check whether capture is on. Turning capture off stops new runs being recorded. It no longer discards a stamp you put on an earlier trace. A client running with captureEnabled: false can now tag a captured trace with its ticket number and find it again by that tag.Dashboard
The Sim plan shows only the nodes your recent traces called
The Sim plan page is a picture of your most recent traces and their replays, but it also listed every node you had ever turned capture off for, including ones those traces never ran. Each of those rows read 0 calls with no type and no payload size, which made a node your code no longer reaches look like a live one.The page now lists only the nodes the sampled traces actually called. Turning capture off for a node that later stops running still keeps that setting, and the node comes back with it the next time it runs.Bitfab captures your Amp sessions
Amp joins Claude Code, Cursor, and Codex. Once you allow session logs, the Amp sessions where you use Bitfab are captured like the rest, and you can turn that on or off withbitfab:setup session-logs. Amp now also tells you at the start of a session when you are signed out or a newer plugin or SDK is available, and npx bitfab-cli analyze-repo --editor amp honors --upload-logs and --no-upload-logs for a single run.Capture starts only when you actually use Bitfab
A session is captured only once something shows you used Bitfab in it, either one of our tools or one of our skills. A tool from somewhere else that happens to share a name with ours, such asget_traces, no longer starts capture. Codex gets a fix here too: typing $bitfab:setup now counts as using Bitfab, which it previously did not.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
TypeScript SDK v0.59.1, Python SDK v0.59.1, Ruby SDK v0.59.1, Go SDK v0.59.1, Plugins v0.11.57, CLI v0.2.381
Every replay tells you its experiment ID up front
bitfab-replay now prints [replay] Experiment <id>: <url> the moment the experiment is created, and prints it again if the run fails, so a crashed or killed replay never loses track of which experiment it was writing to.Your own code can hold on to it too. Pass onExperimentStart to replay() (on_experiment_start in Python and Ruby, OnExperimentStart in Go) and it is called once with the experiment ID and link, before the first trace runs.One name for an experiment’s ID
Everything you touch now calls it the experiment ID: replay results and errors (experimentId and experimentUrl), progress events, the labels methods, the span option, and every experiment tool your coding agent calls. The old testRunId names still work for one more release, carry the same value, and are marked deprecated.Links to the experiments page now select experiments with ?experimentIds=. Links that still use ?testRunIds= keep opening the same experiments.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.59.0, Python SDK v0.59.0, Ruby SDK v0.59.0, Go SDK v0.59.0, Plugins v0.11.56, CLI v0.2.380
Generate assertions from your own code
The TypeScript and Python SDKs can now turn one original trace into assertion drafts withgenerateAssertions / generate_assertions, the same generation your coding agent reaches over MCP. Pass guidance to say what the assertions should be about, such as a root cause analysis of why the trace went wrong, and categoryIds to limit which of your assertion categories Bitfab draws from. Both are now on the MCP tool generate_trace_assertions too, where guidance replaces rootCause.You choose how to wait for it. generate_assertions blocks until the drafts are ready and returns them. generate_assertions_async is the same call as a coroutine, so you can hand it to asyncio.create_task or asyncio.gather and generate for many traces at once. In TypeScript, hold the promise and await it when you need the drafts.getAssertions / get_assertions now reports it in a generation field (status, error, and the ids of the drafts it saved), so another process can check on it. Over MCP, generate_trace_assertions waits up to four minutes and then says it is still running, and calling it again checks on it.Drafts are saved awaiting review and are not checked on replays until a person approves them.Re-running a dataset’s graders now works the same way in every SDK. rerunGraders no longer takes wait or a polling interval. It returns once the run completes, raises GraderRerunError when the run errored, and raises GraderRerunTimeoutError when it is still going after the timeout, while the run keeps going on Bitfab. Python adds rerun_graders_async. If you passed wait: false before, hold the promise (or the coroutine) and await it later instead.Dashboard
Experiment history loads without waiting on detailed results
Experiment history now loads its trace totals first, so growing histories no longer make the whole page wait for label, grader, category, and consistency breakdowns. Open an experiment to load those detailed results on demand, and expanded results stay current as new labels and grader results arrive.TypeScript tells you when subtree tracing is not wired up
TypeScript opt-out tracing needs the separate@bitfab/transform package and one build adapter. Without them, trace() and withTrace() still record the root but nothing beneath it, which looks like working tracing until you open a trace and find a single node. The SDK now prints one warning the first time a root finishes that way, naming the root and linking the install and adapter steps.Setup inspect checks that subtree capture is actually wired
/bitfab:setup inspect now reports whether @bitfab/transform is installed and whether one of its adapters is wired into the build behind your server entrypoint, alongside the authentication, SDK, and replay checks it already ran. It previously gave a clean bill of health to a project whose every trace was a single node. It can install and wire the adapter for you, showing the workspace and the build file it would change before it touches anything, and it flags a Python runtime below 3.12 the same way.Dashboard
Keep reviewing assertions while filters update
The Assertions page now keeps the current queue and detail panel visible while a filter or page change loads. You can keep opening assertions and reading the selected assertion instead of watching the page go blank until the refreshed results arrive.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.11.54, TypeScript SDK v0.58.4, Python SDK v0.58.5, Ruby SDK v0.58.4, Go SDK v0.58.4, CLI v0.2.378
Calls under a node with capture off nest where your code put them
When the Sim plan turns capture off for a node, its spans are not sent, so the calls beneath them used to be placed by timing. Bitfab nested each one under the innermost span that happened to run around it. On a concurrent agent that is often the wrong span. Four critic lanes started together would end up nested inside whichever lane ran longest, and unrelated calls would be swept in with them.Each SDK now works out, while your code runs, the closest call above a span whose capture is on, and sends it alongside the recorded parent. Bitfab nests the span there instead of guessing. On one agent turn that moves 46 of 115 spans, putting every critic lane back as a sibling under the call that launched them.Your recorded call chain is untouched, so nothing about your trace changes other than where these spans appear. Traces captured before this keep being placed by timing.Turn a trace into review-ready assertions
Your coding agent can now turn one original trace into grounded assertion drafts withgenerate_trace_assertions. It selects applicable existing categories, builds each assertion from cited trace evidence, and saves drafts awaiting human review, so they do not affect replays until approved.Pass guidance when you already know why the trace was wrong to produce smaller, more focused assertions. Repeated calls reuse the prior result; set regenerate to true to archive the old generated drafts and replace them without touching human-authored assertions.Dashboard
Grounded explanations for existing assertions
Existing assertions can now be backfilled with missing assertion and category explanations from the trace they were written on. Each explanation cites the exact spans used as evidence, while any explanation already present is left unchanged.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.11.52, TypeScript SDK v0.58.3, Python SDK v0.58.4, Ruby SDK v0.58.3, Go SDK v0.58.3, CLI v0.2.376
Have a replay judge its own assertions
A replay can now decide whether each of its assertions held, instead of leaving every verdict for you to enter by hand. PassjudgeAssertions when you start a replay and each replayed trace is judged as it finishes: one verdict per approved assertion, saved as that assertion’s label with the exact spans it rested on, ready for you to approve or overturn.skipAssertionJudging: true to turn it off.It is off unless you ask, because each verdict costs a model call. A verdict is pass, fail, or undetermined, and undetermined is its own answer: when the run never produced the thing an assertion is about, you get an honest “cannot tell” rather than a failure that sends you hunting for a bug that is not there.Evidence you can check, quoted from the run
Suggested label evidence is now copied word for word from the trace rather than paraphrased, and each item says which part of the span it came from: what the step produced, or what it was handed. Anything the server cannot find in the span it names is dropped, so an item can no longer cite something the run never said.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.11.51, TypeScript SDK v0.58.2, Python SDK v0.58.3, Ruby SDK v0.58.2, Go SDK v0.58.2, CLI v0.2.375
Assign an assertion to whoever should review it
Every assertion can now name the person on the hook for reviewing it. Pick an assignee from the assertion’s detail pane on the Assertions page, and filter the queue by assignee to see only the ones waiting on you or on a teammate, with a count beside each name.Assigning routes an assertion without deciding anything, so it leaves the approval state alone. An assertion nobody is on reads as unassigned, and the filter keeps an entry for those too.Route assertions to a reviewer from your coding agent
Your agent can assign while it writes. PassassigneeEmail on a save_trace_assertions entry, or on an assertion save in any of the four SDKs, and the assertion lands in that person’s queue.list_organization_members, or organizationMembers.list in the SDKs, to see which addresses are available.Your coding agent can approve an assertion
An assertion starts being checked on replays only once someone approves it, and until now that had to happen in the app. Ask your agent to approve or reject one and it records the decision itself, on the same call it already uses to write assertions, so it can read a trace’s assertions and decide them in one pass.Deciding an assertion takes nothing but its id and the decision, so your agent never has to retype the assertion to change its state. Filing it under a category in the same breath counts as approving that category too, the way the trace page treats them together. Rewording an assertion later returns it to not reviewed, so an approval never covers text nobody has read.A decision made this way records the owner of the API key as the approver, which means approving is now something your agent can do rather than something only a person can do. Since approval is what decides which assertions your replays are judged against, ask to see the reasoning behind an assertion before you ask for the decision on it.Dashboard
Approve assertions without leaving the trace
A trace’s Assertions panel now lets you approve or reject each assertion, rather than only reading them there and deciding in the assertions review queue. Every assertion shows whether it is approved, rejected, or not reviewed, who decided and when, and the spans its justification cites. You can file it under a category from the same row, which returns it to not reviewed, because the category assignment is part of what gets approved.See which traces are still waiting on a review
A trace row now carries a count of the assertions on it that nobody has reviewed yet. It shows up wherever traces are listed, including a dataset’s traces, so you can tell at a glance which cases need attention, and the count drops as you decide them.Nested call spans work in a Next.js app
Adding the Next.js adapter to a Next 16 project failed the build instead of recording nested calls. The adapter told Turbopack its loader output was JavaScript, which renamed every module it touched, so imports no longer resolved and client components broke. It also compiled the build output of other packages in the same repository, against those packages’ own compiler settings.The adapter no longer renames anything, and it instruments only the directory Next runs in. Traced server code in a Next app records its nested calls, in every file type, with no change to your code.DashboardTypeScript SDKPython SDKRuby SDKGo SDK
A withheld verdict says why it was withheld
Withholding a verdict now records the reason alongside it.labels.skip takes an annotation, save_agent_labels accepts one beside skip: true, and both reads return it, so a grader that could not judge an attempt (a crashed run, a check whose target never appeared) is told apart from a trace nobody looked at. The reason was accepted and dropped before, which left every withheld verdict reading as unjudged.A skip sent without a reason now clears the note the previous verdict carried, instead of leaving that verdict’s reasoning sitting under the withheld one.Narrow a list of experiments to the ones you mean
Asking your coding agent for experiments no longer means naming a traced function first.list_experiments now takes a git branch, a dataset, and a traced function, in any combination, and with none of them it lists your organization’s most recent experiments. Ask for the experiments run from a branch, or the ones run against a dataset, and get back only those.Each experiment in the list now names its traced function and its datasets, so a list spanning several functions still reads clearly.Dashboard
Replays resolve the right recorded output on traces with capture off
Replaying a trace that has capture off for one of its nodes could substitute the wrong recorded output for a call. Calls that ran inside a capture-off node were missing from the tree a replay matches against, so every later call sharing a name resolved one recorded output too early. Those calls are matched correctly now, and replays of traces with no capture-off nodes are unchanged.Dashboard
Faster trace lists
Trace lists now load much faster, especially replay-filtered views with large histories. Opening a function’s traces and paging newest-first no longer slows down as sharply as its history grows.Suggested label evidence is checked before you get it
labels.generate_label_evidence now verifies each suggested evidence item before handing it back. An item that arrives without the span it points at, or without any other field the return type promises, raises AssertionEvidenceError naming the trace and the assertion, rather than reaching your code as a value that breaks the first time you read it.Dashboard
Traces keep their shape when a node has capture off
Turning capture off for a node used to flatten everything beneath it. The calls under that node arrived with no parent to attach to, so they collected at the top of the span tree instead of nesting where they ran. One 547 span trace opened as 313 separate top-level calls.A trace now nests each of those calls under the innermost call that ran around it, so the shape of the run survives with capture off. Replay outlines, grader prompts, and label evidence read from the same tree, so a grader is no longer handed a flat list that hides which call ran inside which.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.58.1, Python SDK v0.58.1, Ruby SDK v0.58.1, Go SDK v0.58.1, Plugins v0.11.46, CLI v0.2.371
Up to 256 spans per request
Every SDK now packs up to 256 spans into each request to Bitfab, up from 128. A trace with tens of thousands of spans reaches Bitfab in half the requests it took before. Each request is still capped at 3 MB, so traces with large inputs and outputs are split exactly as before.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.58.0, Python SDK v0.58.0, Ruby SDK v0.58.0, Go SDK v0.58.0, Plugins v0.11.45, CLI v0.2.370
Spans with capture off are no longer sent
When the Sim plan turns capture off for a node, the SDKs now leave its spans out of the trace entirely instead of sending them empty. The calls beneath them are still sent and move up to the nearest visible parent, so the trace still completes. A trace’s root span, a span recorded by a framework integration, and a span that recorded an error are still sent without their inputs and outputs. Because capture-off spans no longer arrive, that node’s span samples and metrics on the Sim plan page stop receiving new occurrences.maxSpans / max_spans / MaxSpans is removed, so there is no cap on the total number of spans a trace root sends. Only discovered calls sent with their inputs and outputs are capped, by maxCapturedSubtreeSpans / max_captured_subtree_spans / MaxCapturedSubtreeSpans (default 500). Delete the option from your code when you upgrade, since passing it is now an error.A node you declare in code (node(), withNode(), @bitfab.node, bitfab_node, or Go’s Node) and a nested trace inside another trace now always keep their inputs and outputs, whatever the Sim plan says, even when the plan cannot be read.PluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.11.44, TypeScript SDK v0.57.8, Python SDK v0.57.9, Ruby SDK v0.57.8, Go SDK v0.57.8, CLI v0.2.369
Larger default span limit per trace
Every SDK now sends up to 10,000 spans per trace root by default, up from 2,048, so long agent runs are no longer cut off. The limit on calls the SDK discovers by itself and sends with their inputs and outputs is now 500 by default, down from 512. Both can still be set per trace (maxSpans / max_spans / MaxSpans and maxCapturedSubtreeSpans / max_captured_subtree_spans / MaxCapturedSubtreeSpans). Each SDK now holds up to 16,384 spans waiting to upload, up from 8,192, so a full-size trace is not dropped before it is sent.Longer span delivery for replay child processes
In the Python, Ruby, and Go SDKs, process-based replay concurrency now takes a child delivery timeout: how long each child waits for its spans to reach Bitfab (default 30 seconds). A child also waits for its remaining spans before it exits instead of giving them a few seconds, so a large trace is no longer cut short. Failed uploads inside a child now appear in the replay command’s own output.Dashboard
Faster simulation plan loading
Simulation plans now load faster for functions with large trace histories. Historical size estimates use recent captured spans more efficiently while preserving the same generated plan and support for older traces.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.11.43, TypeScript SDK v0.57.7, Python SDK v0.57.8, Ruby SDK v0.57.7, Go SDK v0.57.7, CLI v0.2.368
Faster span uploads in Python, Ruby, and Go
The Python, Ruby, and Go SDKs now keep connections to Bitfab open and reuse them, instead of opening a new connection for every upload. Busy services avoid repeated DNS lookups and connection setup, and the Sim plan read uses the same connections.Capture off also withholds prompts
When the Sim plan turns capture off for a node, its spans are now sent without their prompt as well as without inputs and outputs. This applies in the TypeScript, Python, Ruby, and Go SDKs.Span limits keep the earliest spans and label truncated traces
When a trace reachesmax_spans or max_captured_subtree_spans, every SDK now keeps the earliest calls, counts framework integration spans in the order they start, and still completes the trace. Truncated traces carry bitfab.truncated_by and bitfab.dropped_spans metadata, so you can filter for them.Capture off and span limits for older SDKs
Spans sent by SDK versions before 0.57.0 now have Sim plan capture off applied and are limited to 2,048 per trace by Bitfab itself. Upgrade to apply both before spans leave your process.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.11.42, TypeScript SDK v0.57.6, Python SDK v0.57.7, Ruby SDK v0.57.6, Go SDK v0.57.6, CLI v0.2.367
Replay-aware label evidence
Bitfab now generates suggested label evidence from the exact original or replay trace you are judging, so a replay’s candidates come from its own spans while inherited assertions still apply. Use MCP’sgenerate_label_evidence or the labels client in TypeScript, Python, Ruby, and Go.Dashboard
Capture-off nodes stay out of replay outlines and grading
A node with capture off on the Sim plan page was already left out everywhere you read a trace. It is now also left out of the outline a replay hands back to your SDK, and out of the trace outline a grader reads, so a grader no longer spends tokens on rows whose inputs and outputs were never recorded.Children still move up to the nearest visible parent, and a span that recorded an error is still shown.test_run_id on bound span decorators
get_function(key).span() now accepts test_run_id, the option its client-level counterpart client.span(key, ...) already took. Attributing a span to an experiment no longer forces you to drop the bound handle and repeat the trace function key inline.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.11.39, TypeScript SDK v0.57.4, Python SDK v0.57.4, Ruby SDK v0.57.4, Go SDK v0.57.4, CLI v0.2.364
Label evidence across MCP and SDKs
Label verdicts can now include span-by-span evidence, so agents and SDK integrations can preserve the exact trace text behind whole-trace and assertion decisions. MCP tools and the TypeScript, Python, Ruby, and Go SDKs accept{spanId, text} evidence on agent and human label writes and return it through normal label reads.The new generate_label_evidence discovery method is also available across MCP and every SDK. It currently returns an empty list while automatic evidence suggestions remain stubbed out.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.11.38, TypeScript SDK v0.57.3, Python SDK v0.57.3, Ruby SDK v0.57.3, Go SDK v0.57.3, CLI v0.2.363
One decision for assertion review
Assertion review now uses one decision covering both whether an assertion applies to its trace and whether it belongs in its current category. Reviewers still see separate cited evidence for the assertion and its category assignment, while reusable categories themselves no longer require approval.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.11.37, TypeScript SDK v0.57.2, Python SDK v0.57.2, Ruby SDK v0.57.2, Go SDK v0.57.2, CLI v0.2.362
Assertions are checked once a person approves them
An assertion you write is now saved awaiting review, and only an approved assertion is checked when a trace is replayed. Approval governs every place an assertion is evaluated:onlyWithAssertions (--only-with-assertions on bitfab-replay) selects traces carrying an approved assertion across all four SDKs, a verdict cannot name an assertion still in review, and an experiment’s pass rates, overall and per category, count approved assertions only.Existing assertions are all awaiting review, so assertion-narrowed replays select nothing and assertion pass rates read zero until you work through them. Nothing is lost: approving an assertion restores its numbers. Every assertion stays visible while you review it, and SDK reads report each one’s approvalState.Dashboard
Pages for experiment groups and runs
Every experiment and experiment group now has a page of its own. In the experiments list, the arrow next to a run or a group opens it, while clicking the row or the group band still expands it in place as before. A run’s page carries its pass rate and progress, who ran it and on what branch, and links back to the group and dataset it came from. A group’s page lists its runs already expanded, and an open trace survives a refresh on both pages.The list also names each run’s group when you turn grouping off, so a flat list still tells you which group a run belongs to.Faster trace lists
Trace lists now load faster, especially when a view contains only original traces. Lists that include replay lineage also resolve that history in parallel. Replay records must now belong to a test run before they can enter the dashboard.Labeling opens only when a trace has assertions
Opening a trace now shows the labeling column only when that trace has assertions, and it opens on the Assertions tab. A trace with no assertions opens with the column closed, so the trace itself gets the full width. Once you pick Labels, Assertions, or closed, that choice carries to the next trace for the rest of your session. A replay follows the assertions of the trace it replays.The trace viewer inside the template preview is now the same one the rest of the dashboard uses, so it picks up trace viewer improvements automatically.Faster OTLP trace ingestion
Bitfab now processes spans from the same OTLP trace together, reducing upload overhead for large traces. Benchmarks with 500-span traces completed 3.19× faster while preserving every span and the trace’s final status.PluginsPython SDKRuby SDKGo SDKCLI
Plugins v0.11.36, Python SDK v0.57.1, Ruby SDK v0.57.1, Go SDK v0.57.1, CLI v0.2.361
Process replays hold back only under real memory pressure
Replays in process mode now wait to start another child only when the operating system reports critical memory pressure, instead of whenever swap looks nearly full. A Mac with plenty of free memory and busy swap now runs replays in parallel rather than one at a time. This applies to the Python, Ruby, and Go SDKs on macOS and Linux, including containers, and the free memory check is unchanged.Reliable TypeScript replay registry loading
The installedbitfab-replay command now loads TypeScript replay registries without attaching private loader metadata to their imports. Replay registries can import Node built-ins while @bitfab/transform/register is active.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.57.0, Python SDK v0.57.0, Ruby SDK v0.57.0, Go SDK v0.57.0, Plugins v0.11.34, CLI v0.2.359
Spans with capture off cost less to record
When a Sim plan turns capture off for a node, the SDKs now skip serializing that span’s inputs and outputs altogether instead of building them and removing them before upload. The span itself, with its name, timing, and errors, is still sent. Large arguments and results in a node you turned off no longer slow your application or inflate what it sends.Spans with capture off also no longer use up the limit on discovered calls (maxCapturedSubtreeSpans, max_captured_subtree_spans, MaxCapturedSubtreeSpans), so turning capture off for a busy node never pushes the calls you kept out of the trace. They still count toward the total span limit.New default span limits
Every SDK now bounds a trace with two limits.maxSpans / max_spans / MaxSpans caps every span a trace sends at 2,048 by default, counting the root span, calls you declared as nodes, and spans with capture on or off, so a trace never goes past it. maxCapturedSubtreeSpans / max_captured_subtree_spans / MaxCapturedSubtreeSpans, now available in every SDK, caps the calls with capture on that the SDK discovers by itself at 512 by default and is never higher than the span limit. A call you declare as a node keeps recording past 512, up to 2,048. The span limit must be at least 1, and a lower value is now rejected with an error. The discovery limit is renamed from max_subtree_spans to max_captured_subtree_spans in Python and Ruby, maxCapturedSubtreeSpans in TypeScript, and MaxCapturedSubtreeSpans in Go, and Python’s BITFAB_MAX_SUBTREE_SPANS environment variable is now BITFAB_MAX_CAPTURED_SUBTREE_SPANS, so update any code or configuration that sets the old names.Previously TypeScript, Go, and Ruby used a single limit of 500 for every span, and Python capped discovered calls at 500 with no limit on the rest. In Python, setting max_spans on its own no longer raises the discovery limit.The Sim plan applies from the first trace
The SDKs now read your Sim plan as soon as the client is created, and a trace that starts before that read finishes waits for it, at most 5 seconds for the whole process, so capture settings apply from the very first trace. Synchronous TypeScript functions cannot wait and keep the previous behavior. If the Sim plan cannot be read, spans are sent without their inputs and outputs until a later read succeeds, so nothing you turned off is stored while Bitfab is unreachable. The trace’s root span always keeps its inputs and outputs.Fewer network requests from the Python, Go, and Ruby SDKs
The Python, Go, and Ruby SDKs now pack up to 128 spans into each request to Bitfab, up from 8, matching the TypeScript SDK. A busy application sends a fraction of the requests it did before, which means fewer new connections and fewer DNS lookups. Each request is still capped at 3 MB.Dashboard
Reliable trace recording when replays finish
Traces now record reliably when a replay experiment finishes while its traces are still being sent. Previously the two could collide and one would fail, leaving an experiment with missing or incomplete traces.Faster Sim plan loading
Sim plans now load quickly even for functions with very large trace histories and only a few replays. Opening the Sim plan page no longer waits while Bitfab searches through unrelated live traces.Faster large Sim plans
Sim plans now load substantially faster when recent traces contain hundreds of nodes or hundreds of thousands of spans. The same code map, size estimates, and capture decisions remain intact while the page opens in much less time.PluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.11.33, TypeScript SDK v0.54.14, Python SDK v0.56.6, Ruby SDK v0.51.10, Go SDK v0.51.9, CLI v0.2.358
Complete tracing and evaluation workflows in Ruby and Go
Ruby and Go integrations can now use the same portable tracing and evaluation workflow available in the TypeScript and Python SDKs. Both SDKs now cover declared nodes and subtree capture, capture policies, detached traces, simulation plans, replay and seeding workflows, datasets, labels, graders, assertions, assertion categories, and BAML. Go also supports optional source instrumentation, while Ruby carries trace context across application threads.Dashboard
Faster span ingestion within a trace
Trace ingestion now handles concurrent spans in the same trace without repeatedly updating its token totals. Bitfab calculates aggregate token usage and model metadata when the trace completes, reducing contention during capture while keeping the final totals accurate.Capture every subtree span by default
trace() and withTrace() now record every eligible first-party call beneath the root by default, so new spans appear without waiting for a server-selected inclusion policy. Use exclude or node({ capture: false }) to omit a call explicitly. Turning content off in a Sim plan still keeps the span in the trace while leaving out its inputs and output.Dashboard
Faster OTLP trace ingestion
Large OTLP trace batches now avoid repeating the same completion work for every span. Bitfab checks each affected trace once after a batch is stored, reducing database pressure while keeping trace completion and older SDK ingestion behavior intact.Replay all supported Rivet workflow effects
Rivet workflows can now replay every public workflow primitive and the fixed database, KV, queue, schedule, cron, persistence, broadcast, sleep, and destroy operations. Choose which effects should return their recorded outputs instead of running against a live actor:step and loop do not invoke their callbacks during replay, preventing their durable side effects from running again. Replay also restores workflow identity and cancellation state, while unsupported live capabilities fail closed instead of touching the actor.Dashboard
Faster trace environment filters
Trace environment filters now stay fast as an organization’s history grows. Bitfab now derives available environments from maintained trace counts. Opening trace filters no longer scans every trace.Dashboard
Faster pages while large traces are capturing
Capturing a trace with thousands of spans no longer slows down the rest of Bitfab. Large traces used to hold database connections open while they uploaded, which delayed unrelated pages, so the trace list could take tens of seconds to load during heavy capture. Ingestion now releases those connections as it goes, and your pages stay responsive while a large trace is still arriving.Dashboard
Review and approve assertions
The assertions your coding agent writes now have a review queue of their own. The new Assertions section lists them beside the one you are reading, and you approve or reject each without leaving the list. Filter by whether an assertion has been reviewed, by category, or by the ones nothing has been filed under yet, so a long queue splits into sections one person can work through.See the evidence behind an assertion
Each assertion shows the spans it cites: why it applies to its trace, and why it belongs in the category it is filed under. Click a cited span and the trace opens with that span already selected, so you can check the claim against what actually happened. The assertion you are reading is part of the address, so a link takes a teammate to exactly the one you mean.Dashboard
Pass rates for each assertion category
Experiment results now show a pass rate for every assertion category, not just the category names. Expand an experiment and the breakdown panel lists each category with its pass rate, its change against the baseline, and how many labels sit behind it, so you can see at a glance which kind of check is failing. The section sits below the existing Traces and Labels rates and collapses when you do not need it.Dashboard
More reliable OTLP trace ingestion
Trace ingestion is now more reliable when spans and completion records for the same trace arrive at the same time. Bitfab sequences work for each trace to prevent valid telemetry from failing during high-volume bursts. Different traces continue processing concurrently.Dashboard
Capture-incomplete traces recover promptly
Capture-incomplete traces now resume reliably when fresh trace or span activity arrives, even for teams with large trace histories. Background timeout checks stay focused on unresolved captures, reducing contention with ongoing ingestion.Tell apart calls that share a node name
When one function runs several times at once in a trace, such as a check that runs once per category on a thread pool, give itsnode() a variant computed from the call’s arguments. The span tree shows each call’s variant after the node name, so you can spot each call at a glance. The node keeps one name on the Sim plan page.PluginsTypeScript SDKPython SDKCLI
Plugins v0.11.29, TypeScript SDK v0.54.11, Python SDK v0.56.4, CLI v0.2.354
Code inside OpenAI Agents SDK tools nests under the tool
Inside atrace() or withTrace() subtree, your own functions that run while an OpenAI Agents SDK span is open, such as a function tool’s body, now appear under that tool span instead of in a separate branch beside the agent run. The trace tree now matches the Agents SDK’s own nesting in both the Python and TypeScript SDKs, including when a Python tool hops threads through asyncio.to_thread or sync_to_async. A node() inside a tool nests the same way, and span() or withSpan() there still raises MixedTracingError.Dashboard
Turn capture off for spans that were once a trace root
On the Sim plan page, a span now counts as a root only when it has no parent in its most recent trace. A span that was the root of an older trace but runs under a parent today can have its capture turned off again, instead of staying locked on. Each row also lists the other roles and types that span had in recent traces as “also seen as”, and a span whose capture has to stay on is no longer hidden from the trace view by an earlier capture-off decision.Python spans holding NamedTuples are no longer dropped
Spans whose inputs or output contain aNamedTuple, or a list subclass whose constructor needs extra arguments, now reach Bitfab intact. Before this release the Python SDK discarded those spans without any message, so their child spans appeared at the top of the trace with no parent. A NamedTuple returned from a span also keeps its type, and a span that still cannot be sent now logs a one-time bitfab: warning with its name.Dashboard
Replay diffs leave out spans with content off
Spans you turned off on the Sim plan page no longer show up as “Not called” rows when you compare a replay against its original. Until now, an original recorded before you turned those nodes off still listed every one of those calls, so the diff filled with removed rows and inflated removed counts for calls you had already decided you did not need. The diff now hides them on both sides, the same way the trace view does, and a span that recorded an error still shows.DashboardPluginsTypeScript SDKPython SDKCLI
Plugins v0.11.27, TypeScript SDK v0.54.10, Python SDK v0.56.2, CLI v0.2.352
Replay Rivet actor turns without a live actor
Rivet apps can now replay finite actor turns without starting an actor or touching its live state. Bind the actor context to an existingwithTrace() root, and Bitfab restores captured pre-turn state in memory while rerunning your application code.mockOnReplay is explicit and defaults to no effect paths: choose individual database, transaction, KV, queue, schedule, cron, persistence, broadcast, sleep, or destroy paths for replay mocking, while missing recordings and unsupported live capabilities fail closed. TypeScript and Python function handles now also expose withTrace, so framework integrations can live inside one existing traced function while keeping the fluent API.DashboardPluginsTypeScript SDKPython SDKCLI
Plugins v0.11.26, TypeScript SDK v0.54.9, Python SDK v0.56.1, CLI v0.2.351
Assertions carry their evidence, and a person signs off on them
An assertion now records why it is right for the trace it sits on, as a list of the spans it rests on with a reason on each one, so a reviewer can open the cited span and check the claim instead of taking the assertion on faith. An assertion assigned to a category carries a second one saying why that category fits. Reusable categories are organizational metadata and do not carry evidence or an approval of their own.Whoever writes the assertion writes the evidence. Passjustification and categoryJustification to save_trace_assertions, or the same fields through the SDKs.get_trace_assertions prints it. Only a person sets it, so nothing that writes an assertion can approve its own work. Editing an assertion or its category sends the decision back for review; editing a reusable category resets affected assertions, never an approval on the category itself.Label filters no longer count assertions as labels
Filtering traces by a human label, or by validated labels, no longer matches a trace whose only human-written row is an assertion rather than a verdict. This affected the trace list filters and thesearch_traces tool, where such a trace could appear as labeled without anyone having judged its output.Sim plan includes your replays
The sim plan now builds its code map from your function’s most recent traces and its most recent replays, so a call that only runs under replay shows up on the map. The page says how many of the overlaid traces were replays, and your coding agent sees the same count through theget_sim_plan tool.Metrics for a sim plan node
Clicking a node on the Sim plan page now opens a panel with two views. Span samples lists the node’s recorded calls, now including replays. Metrics breaks the node down by the environment its traces ran in, with replays on their own row, showing calls, traces, average payload, and captured content.Escape closes detail panels
Pressing Escape now closes the detail panel on the traces list, dataset review, experiments, and the Sim plan page.Subtree limits bound what Bitfab discovers, not what you declared
A function you marked withnode(), and a nested trace(), are now always recorded. Until this release the subtree limits applied to everything a trace captured, so functions the SDK found on its own could use up the budget first and a function you had explicitly decorated would go missing, with no span and nothing in the trace to say why.Limits now bound automatic discovery only. The new max_captured_subtree_spans (default 500) caps the functions the SDK finds by itself. max_depth and max_spans are unbounded by default, and max_spans is a ceiling on every span in the subtree, declared ones included. A span whose capture the sim plan turned off no longer uses any budget, since it is left out of the span tree anyway.Bitfab(max_captured_subtree_spans=...) or the BITFAB_MAX_CAPTURED_SUBTREE_SPANS environment variable, and override it per decorator. Raising max_spans on its own still lifts the discovery cap, so existing code records at least as many spans as it did before.Thread tracing keeps working alongside Sentry
trace_across_threads=True now behaves correctly in Python apps that also initialize the Sentry SDK. Before this release the worker thread died with a Thread.run() argument error, so any span opened inside a threading.Thread was lost instead of nesting under its parent trace.Dashboard
More reliable trace redirect links
Trace redirect links can now includetrace_function_key to narrow a lookup to one traced workflow, and successful matches open in that function’s scoped trace page. The docs now show the recommended pattern: attach a unique ID from your app as caller metadata during the traced run, then use the same metadata key and value in the redirect link. A redirect with no matching trace now opens the standard not-found page, while ambiguous matches still ask for another filter.Dashboard
Open traces with your own IDs
Build a trace redirect URL with IDs from your own system, such as a thread ID or block ID. Bitfab finds the matching trace and opens its canonical trace or selected-span URL.Each URL can combine exact filters from trace and span fields when one ID is not enough. Missing or ambiguous matches are shown instead of opening the wrong trace.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
Plugins v0.11.22, TypeScript SDK v0.54.8, Python SDK v0.55.7, Ruby SDK v0.51.9, Go SDK v0.51.8, CLI v0.2.347
Turn capture on or off from the environment
BITFAB_CAPTURE_ENABLED=0 turns capture off and BITFAB_CAPTURE_ENABLED=1 turns it on, in all four SDKs, so you can silence tracing in one environment without editing code or shipping a new build. It applies to any client built without the capture option, and whatever your code passes still wins.Dashboard
Diff view on skipped replays
A replay you have marked Skipped now keeps its comparison header, so you can still open the Diff view and compare it against the original trace. Previously the verdict chip and the Diff, Original, and Replayed toggle all disappeared on a skipped replay, leaving a single-trace view even though a paired original was there to compare against.Faster single-span reads
Reading one field from a span no longer loads the whole trace first.get_span_field and the SDK’s span lookup now fetch just the span you asked for, so drilling into a large trace returns much faster.Clearing assertion text works the same everywhere
Sending an empty pass criteria, fail criteria, or human note now clears that field whether you write it from the SDK or from a coding agent. Previously an empty string was stored as blank text over the SDK and cleared over the agent surface, so the same input produced two different results.Dropped traces no longer return partial spans
Reading a span from a dropped trace now reports the trace as gone instead of returning an empty shell. Dropped traces are scrubbed of their content, so the partial data was never usable.PluginsTypeScript SDKPython SDKCLI
Plugins v0.11.20, TypeScript SDK v0.54.7, Python SDK v0.55.6, CLI v0.2.345
Capture opt-out now covers framework integrations
Client-level capture controls now apply consistently across TypeScript and Python framework integrations, including OpenAI Agents, LangGraph, Claude Agent SDK, Vercel AI SDK, and local BAML calls. SetcaptureEnabled: false or capture_enabled=False to keep application and model calls running without sending traces or spans; explicit replay and seed recording still work.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.11.19, TypeScript SDK v0.54.6, Python SDK v0.55.5, Ruby SDK v0.51.8, Go SDK v0.51.7, CLI v0.2.344
Find a trace by its name
The title you give a trace withsetName is now something you can search for. Match it by its words, or by any fragment when you only remember part of it. Both are case insensitive, and both work from the trace list, the API, and your coding agent.name matches the words you give it in order, so searching for “Order ABC” also finds “Order ABC 123”. nameContains matches any run of characters, even inside a word, and needs at least 3 of them. The same method is available in Python, Ruby, and Go, and a name filter combines with a metadata filter when you want both.Metadata is still the right home for a lookup key you generate, like a ticket or tenant id, because it matches exactly. Searching by name is for the times you are looking for a run the way a person remembers it.Dashboard
Spans with content off stay out of the trace view
Turning content capture off for a node on the Sim plan page now hides that node’s spans everywhere you read the trace, including traces captured before you turned it off. Until now only spans recorded after the change were hidden, so older traces kept showing rows for calls you had already decided you did not need. Their children move up to the nearest visible parent, and a span that recorded an error still shows. Your coding agent reads the same trace you see, so a node you turn off stops filling up its reads too.Dashboard
Link straight to a span
Selecting a span in a trace now updates the address bar, so you can copy the link and send someone to the exact span you are looking at. Opening that link reopens the trace with that span already selected, and your browser’s Back button steps back through the spans you visited. Links you shared before this change keep working.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.11.18, TypeScript SDK v0.54.5, Python SDK v0.55.4, Ruby SDK v0.51.7, Go SDK v0.51.6, CLI v0.2.343
Search traces by the metadata you set
Trace metadata is now searchable. Attach a ticket ID, tenant, or any other key to a trace withsetMetadata, then filter for it from the trace list, the API, or your coding agent. Matching is exact and case insensitive, and you can combine up to 10 key and value pairs.Dashboard
Dark mode holds up across the app
Surfaces that were pinned to white now follow the theme, so cards, menus, panels, and the prompt editor all read correctly in dark mode. The prompt editor takes its syntax colours from the active theme rather than staying dark in both.Span markers read by what they do
Replay, external reads, side effects, generated calls, and untraced spans each carry their own colour, and none of them borrow the brand amber any more. Amber now means the logo and the primary action on a page, so a marker no longer competes with the thing you are meant to click.Browse assertion categories
You can now see every assertion category in your organization from the new Categories section in the dashboard sidebar. Each category shows its description and how many active assertions belong to it. On a trace function’s pages, the counts cover only that function’s traces.Assertion categories on experiments
When you expand an experiment, a new Assertion categories section above its results lists every category its replayed traces were checked against. Hover a category to read its description. The section appears only when at least one of the experiment’s traces has an assertion in a category.Safer opt-out setup migrations
Setup now migrates existing manual spans toward opt-out tracing instead of treatingMixedTracingError as a reason to stop. Before converting a shared helper, it checks every production caller so existing opt-in callers keep a compatible tracing surface. TypeScript conversions also use named functions as required by withNode. A missing dedicated organization or environment key keeps client capture off, preventing fallback to a different organization’s BITFAB_API_KEY.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.11.16, TypeScript SDK v0.54.4, Python SDK v0.55.3, Ruby SDK v0.51.6, Go SDK v0.51.5, CLI v0.2.341
Traces complete after every span arrives
Traces now remain open until execution has finished and every expected span has reached Bitfab, so grading starts with the complete trace. Updated TypeScript, Python, Ruby, and Go SDKs report the expected span total automatically, while older SDK versions retain their existing completion behavior.DashboardPluginsTypeScript SDKPython SDKRuby SDKCLI
Plugins v0.11.15, TypeScript SDK v0.54.3, CLI v0.2.340
Plugins open regular Bitfab pages
Plugin commands now return links to the regular experiments and dataset pages instead of opening a separate Studio session. Login opens the sign-in flow in the OS-default browser and returns after saving credentials, while template previews keep their focused plugin page.Opt-out tracing by default during setup
Bitfab setup now starts supported TypeScript and Python workflows withwithTrace or trace, using withNode or node to configure discovered calls. Setup also installs the required transform adapter and registers production entry points for the SDK-provided bitfab-replay --registry command.DashboardPluginsTypeScript SDKPython SDKCLI
Plugins v0.11.14, TypeScript SDK v0.54.2, Python SDK v0.55.2, CLI v0.2.339
Keep people-only context with assertions
Add a human note to an assertion when reviewers need context that should not change how a replay is judged. Notes are editable from the trace page and through MCP. Coding agents can preserve them without treating them as pass or fail criteria.TypeScript and Python assertion reads now return the note. SDK assertion saves remain focused on assessment criteria.Dashboard
See who ran an experiment and what code it ran
The experiments list now shows who launched each run and the branch it ran from, at every window width. Hover the marks at the end of a row for the person, their git email, the branch, and the commit. Runs launched with a shared CI key show who was at the keyboard instead of appearing anonymous.Experiments say when the code was uncommitted
If a run executed code that was never committed, the row marks it and the detail explains it in plain words, with the exactgit diff command to see what changed. Asking your coding agent about an experiment returns the same thing: it now names the person who ran it and says whether the tree was clean or had uncommitted changes, rather than leaving two commit hashes to compare.Dashboard
More reliable search for large traces
Search indexing now handles traces with tens of thousands of spans more reliably, preserving searchable summaries and generated embeddings across the full trace. Experiment summaries also load with less contention during concurrent activity.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.11.13, TypeScript SDK v0.54.1, Python SDK v0.55.1, Ruby SDK v0.51.5, Go SDK v0.51.4, CLI v0.2.338
Every experiment records who ran it and which code ran
An experiment now stores the person who launched it and the exact state of the code behind it: the branch, the commit, the tree at that commit, and the tree that actually ran. Nothing to opt into, sincereplay() records it itself, and every read is local with no network call.The last two are the useful pair. A commit alone lies whenever the working tree is dirty, so the experiment tree is captured separately and covers uncommitted and untracked files. Two runs sharing an experiment tree ran byte-identical code, and the difference between the base and the experiment tree is the change under test:BITFAB_DISABLE_GIT_STATE opts a process out.DashboardPluginsTypeScript SDKPython SDKCLI
Plugins v0.11.12, TypeScript SDK v0.54.0, Python SDK v0.55.0, CLI v0.2.337
Assertion categories
Group assertions into reusable categories with a title and description. Categories appear in trace and labeling views, where assignments, renames, and removals update without refreshing the page. Manage them through MCP,client.assertionCategories in TypeScript, or client.assertion_categories in Python. Deleting a category keeps its assertions and verdicts.Dashboard
Clear status for incomplete trace captures
Unfinished traces now show “Capture incomplete” after 60 minutes without activity, preserving the spans already received. Trace lists and details distinguish these unknown outcomes from failed evaluations and regressions. New activity restores the capture status on the next background check, and a recorded completion clears the warning immediately.Dashboard
Live experiment totals stay current
The experiments list no longer briefly switches back to older totals when a refresh overlaps a live update. Later refreshes can still correct the displayed totals.Dashboard
Search handles circular trace data
Trace search indexing now handles inputs and outputs with circular references, such as child objects pointing back to their parent. Searchable content is preserved while circular links are replaced with markers.More resilient trace ingestion
Trace and span uploads now recover automatically from brief database connection interruptions, reducing dropped telemetry during transient service disruptions. This applies to SDK ingestion across supported trace sources without requiring any SDK changes.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.11.11, TypeScript SDK v0.53.5, Python SDK v0.54.1, Ruby SDK v0.51.4, Go SDK v0.51.3, CLI v0.2.336
Faster dataset browsing
Dataset experiment pages and dataset lists load with less work as collections grow. Dataset-filtered trace search now applies your filters before choosing its top matches.Page through graders
Grader lists now offer Previous and Next controls for browsing larger collections. Sorting applies across the full list, including archived graders.Automatic dataset paging in every SDK
TypeScript, Python, Ruby, and Go now fetch dataset lists and dataset trace IDs in pages automatically. Existing calls still return the complete result without requiring cursor handling.Dashboard
Smoother scrolling through long experiment lists
The experiments list no longer flickers back to a loading state or jumps to the top while you scroll, and it keeps loading older runs instead of stopping partway down. Lists with many runs also settle faster after a label changes.Trace Ruby workflows from one root
Ruby 3.4+ can now record a workflow’s project methods from one experimentalbitfab_trace declaration. Use bitfab_node to configure individual calls without tracing them on their own.bitfab_span declarations on production hot paths.Keep nested and lazy work in context
Nested roots with different keys keep independent traces while outer traces continue recording shared calls. Explicit spans appear once in their owning trace. Returned Enumerators are traced during consumption without forcing lazy work to run early.Dashboard
Parent spans stay in view while you scroll
Scrolling a deep span tree no longer loses track of where you are. Each parent span now stays fixed at the top of the tree for as long as any span beneath it is on screen, up to five levels, and the sim plan’s code map pins its parent nodes the same way. Selecting a span still scrolls it fully clear of the pinned rows.Replay diffs ignore spans whose capture the sim plan turned off
Turning capture off for a sim plan node strips a span’s inputs and outputs while still recording the call itself. If the plan changed between a trace and its replay, the Diff view compared a recorded side against a blank one and reported those spans as “Not called”, filling the span tree with rows for code that ran normally. Spans with capture off on either side now report no diff, and a call the replay genuinely skipped still shows as not called.Tracing a span brings its callers with it
Turning on tracing for a span in a trace plan now turns on every span above it, so the plan always records one unbroken path from the entry point down to the span you picked. Untracing a span turns off everything beneath it, which is what the plan already claimed it did. Both hold whether you click in Studio or your coding agent adjusts the plan for you.Assertions stay on the original trace
Saving or archiving an assertion through a replay’s trace id is now refused, and the error names the original trace to retry with. An assertion describes the case rather than one run, so a replay reads its original’s assertions instead of carrying a copy. Writing onto the replay used to succeed and stop that inheritance, which left later replays of the same case with nothing to be judged against.Assertions are called assertions everywhere
The tools your coding agent calls now say assertion in their titles and in everything they print back, matching what the SDKs and the dashboard already called them. Nothing you call by name changed, so existing scripts and prompts keep working.Replay no longer runs your machine out of memory
Replaying withprimitive="process" now starts each child only once the machine has room for it, so a large run on a busy laptop finishes with fewer replays at a time instead of being killed partway through. Bitfab checks free memory and swap before starting a child, counts the room every running child still needs rather than only what it has used so far, and learns the real figure from children that have already finished. It never goes above the max_concurrency you set and never holds back the last remaining child, so a run still finishes on a machine that stays under pressure.It is on by default in process mode. Set BITFAB_REPLAY_MEMORY_THROTTLE=off to turn it off for a single run, or turn it off for a registration:BITFAB_REPLAY_CHILD_MEMORY_MB and BITFAB_REPLAY_MEMORY_FLOOR_MB tune how much memory a child is expected to need and how much is left free. Update to Python SDK v0.54.0.Dashboard
Blank spans no longer show phantom changes
Spans that recorded no input or output no longer report a change count in an experiment’s Diff view. Comparing an empty value against one that was never recorded counted as a removed line, so a blank span displayed a red removal for a difference that did not exist. The control that hides empty spans now works inside the Diff view as well, where those rows were previously stuck on screen and left out of the hidden count.Tracing works with proxies and computed getters
Automatic tracing now handles code that runs a function on property access, such as a database client wrapped in a proxy or an object with computed getters. Previously the SDK read those properties while it was still recording, which ran your code again and could exhaust the call stack, leaving the run with no trace at all. Update to TypeScript SDK v0.53.4.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.11.5, TypeScript SDK v0.53.3, Go SDK v0.51.2, CLI v0.2.330
One reference for SDK setup
SDK setup now uses the public documentation for language and framework details. The docs also clarify Go replay support, filtering dataset replays, and tracing shared packages in TypeScript monorepos.Install the TypeScript transform under the Bitfab scope
The TypeScript build adapters are now available as@bitfab/transform. If you used bitfab-transform, replace the dependency and update your imports and build configuration to the scoped package. The Next.js adapter now resolves its loader from @bitfab/transform/loader, and plugin setup instructions use the new package name.Keep partial output when model streams stop
Vercel AI SDK tracing now records partial output and the error or cancellation when a model stream stops early. Once the stream ends or is cancelled,flushTraces() can finish delivering its span without waiting on an unfinished stream finalizer.Capture shared monorepo packages
The Node transform now resolves its injected runtime from the SDK installation, so shared packages can be captured even when only the application depends on the SDK. UseBITFAB_SDK_RESOLVE_FROM to select the application’s package location when needed; ESM and CommonJS are both supported.Turning a node off turns off the nodes beneath it
When you turn content capture off for a node on the Sim plan page, every node beneath it in the map turns off too, the same way untracing a span in a trace plan drops the spans under it. Nodes that always keep their content (recorded by a framework integration, the root of the workflow, imported from another platform, or named by Bitfab) stay on. Turning a node back on leaves the nodes beneath it as they are, so switch those on individually.Thesave_sim_plan tool in the Claude Code, Cursor, Codex, and Amp plugins follows the same rule, and a node you name in the same call keeps the value you give it.DashboardPluginsTypeScript SDKPython SDKCLI
Plugins v0.11.1, TypeScript SDK v0.53.1, Python SDK v0.53.1, CLI v0.2.326
Turn content off per node from the Sim plan
The TypeScript and Python SDKs now read your Sim plan and stop sending inputs and outputs for any node whose content you turned off. The span still arrives with its name, type, timing, errors, and links, marked as content off, so the Sim plan page keeps counting it while the payload never leaves your process. Turn a node off on the Sim plan page or with thesave_sim_plan tool, and new traces pick the change up within a minute. Spans are held until the plan has loaded, so short-lived processes honor it too.See which SDK and integration recorded a span
Every span now carries its origin: the SDK name and version, and whether it came fromwithSpan or @span, withTrace or @trace, or a framework integration (Vercel AI SDK, Claude Agent SDK, OpenAI Agents, LangGraph). The trace header shows it as “Recorded by”. Spans recorded before this release resolve their instrumentation from the trace source instead, and show no SDK name or version.Roots and framework spans always keep their content
The Sim plan refuses to turn content off for a node that is the root of its traces, a node recorded by a framework integration, an imported node, or a node Bitfab named from its model or type. The switch shows on and disabled with the reason, and thesave_sim_plan tool returns an error that says why.Browse a node’s recent occurrences from the Sim plan
Click any row on the Sim plan page to open that node’s recent occurrences, newest first, each with its size and whether content was captured. Open one to see the span inside its full trace.DashboardPluginsTypeScript SDKPython SDKCLI
Plugins v0.11.0, TypeScript SDK v0.53.0, Python SDK v0.53.0, CLI v0.2.325
Decide per call what a Sim plan records
Each call on a trace function’s Sim plan now has a Capture switch. Turning it off marks that call as one whose inputs and outputs should stop being recorded, while the call itself, with its name, type, timing, and errors, keeps being captured so the map stays complete. The decision is stored with the plan and shows who made it. The SDKs will start honoring it in an upcoming release. Sim plan is an alpha switched on per organization.See what each call costs before you decide
Every call on the Sim plan shows its average recorded payload and an estimate of what it costs per month at $2 per GB, based on how often the function ran in the last 30 days. Click the payload to open recent examples of that call in the standard trace view, inputs and outputs included, and step through them. The plan also flags a call reached from more than one parent, since a switch there affects every one of them.Edit the Sim plan from your coding agent
Two new MCP tools let a coding agent read and change a plan.get_sim_plan lists a trace function’s calls with their type, call counts, payload size, monthly estimate, and capture state, and save_sim_plan turns capture off or on for the calls you name. The Sim plan page’s Edit plan button shows the prompt to use, and a trace function’s traces page now links to its plan.Method spans are named after their class
A span on a class method now defaults toClass.method in both SDKs, so two methods that share a name are two distinct spans rather than one. Plain functions, closures, and explicit names are unchanged, and the raw function name still travels separately as function_name.Dashboard
Jump between a nested trace and the trace that started it
When a trace function runs inside another traced function, the span tree now shows the connection. In the outer trace, the span for the nested call carries a small link naming the nested trace function, and clicking it opens that trace at its root span. In the nested trace, the root span carries a link back to the outer trace and opens it at the span that made the call. Traces recorded before this release show no link.TypeScript SDKPython SDKPluginsCLI
TypeScript SDK v0.52.6, Python SDK v0.52.7, Plugins v0.10.12, CLI v0.2.324
Nested trace roots record both traces
When atrace() root runs beneath another one, both functions now get a complete trace: the outer keeps recording every call beneath it, and the nested root records its own trace under its own key, so you can label, collect, and replay one part of an agent on its own. This is new in TypeScript, where a nested withTrace() or trace() root used to take over the outer’s capture; Python already recorded both traces and now links them. Each root applies its own limits, exclusions, and capture policy to its copy, and a node() inside the nested region keeps its name, type, test run, and finalized output in every copy.Nested traces link to each other
The outer trace’s span for the nested root now carries the nested trace’s id, trace function key, and root span id, and the nested root’s span carries the enclosing trace’s id and span, so the two traces can be followed from either side. Inside a replay or while seeding a trace, a nested root is absorbed into the item’s trace instead of starting a second one, so an experiment never gains traces under another key.TypeScript SDKPython SDKRuby SDKGo SDKDashboard
TypeScript SDK v0.52.5, Python SDK v0.52.6, Ruby SDK v0.51.2, Go SDK v0.51.1
Every trace records its commit
Every trace now carries the commit its code was running at, in all four SDKs, so you can tie a captured run to a point in your repository’s history rather than only to a timestamp. The SDK reads it from your deploy platform’s build variables on Vercel, GitHub Actions, Railway, Render, Heroku, Cloudflare Pages, GitLab CI, Azure Pipelines, and CircleCI, falls back togit in a plain checkout, and never slows the traced call. In a container built without either, set BITFAB_COMMIT_SHA at build time, and set BITFAB_DISABLE_COMMIT_REF to turn the capture off. The ref records the commit, the branch, whether the tree had uncommitted changes, and the repository as host/owner/repo with any credentials removed.Dashboard
An error partway through no longer erases a trace’s verdict
A trace that hit an error somewhere in its span tree but still ran to completion now keeps its verdict and counts toward your experiment pass rate. Until now a single error anywhere in the tree pulled the whole trace out of the results, out of the pass rate, and out of the before and after comparison, even when graders had already scored it. Only a run that crashed outright, where the traced function threw all the way out and produced no result, is still set aside as ungradable.Errors read as their own signal, beside the verdict
Trace rows now carry an error badge showing how many spans threw, next to the pass rate rather than in place of it. A run that crashed at the root reads Crashed and keeps its red marker, while a run that recovered shows its error count alongside its normal verdict, so you can tell a broken run from a noisy one at a glance.Experiment breakdowns keep a label’s baseline
The Labels and Attempts rows of an experiment breakdown no longer report work as having no baseline when the Traces row already had something to compare it against. An original labelled as a whole, or scored by different graders than its replay, carried a baseline at the trace level but not at the label level, so its labels fell into the no-baseline segment of the bar and sat outside the change in points. All three rows now read the original’s verdict the same way, so that segment is left for originals that really were never labelled.Sim plan, an alpha code map of each trace function
A trace function now has a Sim plan: a code map of every call its traces have recorded, built from its 20 most recent runs and shown in the sidebar below Graders. Each call appears once, with its span type, how many times it ran, the share of recent runs it appeared in, and when it was last seen, so a branch that stopped running fades while the rest of the map stays put. Each call also carries the judgments the plan will drive, whether its inputs and outputs are captured and whether a reviewer needs to see it, and the root of every span tree links to its function’s plan. Sim plan is an alpha and is switched on per organization, so ask us if you would like it enabled.Dark mode
Bitfab now has a dark theme. Flip it with the sun and moon toggle in the header, or choose Light, Dark, or System from the account menu, where System follows your operating system. Your choice is remembered, and it carries across the trace, dataset, experiment, and grader views as well as Studio and the plugin pages.Dashboard
See how an experiment moved, at three zoom levels
Opening an experiment now shows a breakdown beneath it that reads the same run three ways: by trace (a trace passes when at least 75% of its labels pass), by label (each assertion or grader rolled up across its attempts, with at least 75% of attempts passing counting as a pass), and by every label on every attempt. Each row shows the pass rate, the change against the original traces in points, and a bar splitting the run into held, fixed, regressed, and still failing, so a run that looks unchanged by trace can show where it improved underneath. Runs with several attempts per trace get a Jitter row counting the attempts that disagreed with their trace’s verdict. When there is nothing to compare against, the rows show plain passing and failing instead of an empty change, and a run that mixes compared traces with ones whose originals were never labelled counts both, drawing the uncompared ones in a lighter shade.Filter an experiment’s traces by how they changed
The trace list under an experiment now filters to fixed, regressed, still passing, or still failing traces, with an Unstable filter on multi-attempt runs for traces whose attempts disagreed. The list scrolls on its own, so the breakdown above it stays in view.Grader labels count the same as your labels
An experiment’s pass rate, raw count, change buckets, and filters now read one verdict per trace that includes grader labels alongside labels written by people or agents. A grader’s label counts whenever it is on the trace, whether or not that grader is attached to the experiment. Two things follow. A grader attached to an experiment that has not yet labelled a trace no longer holds that trace’s verdict back, and a run’s headline pass rate counts traces rather than individual grader checks, so grader-scored experiments will read differently than before. The graders panel keeps its own per-grader counts.Experiment rows show the count and the change
Each experiment’s row now carries the passed-over-scored trace count and the change against the originals in points beside its pass rate, and its bar is a single fill at the pass rate. All four read the trace rollup, so a five-attempt run of 50 traces reads 36/50 rather than 180/250. The four-way split moved into the breakdown, where it has the room to be read.Dashboard
Label badges count every check a trace is held to
A trace’s label badge now counts what the trace is judged against, which is its assertions when it has any and the trace as a whole when it has none. Nothing counts as passed except a label you marked pass, so a trace whose assertions nobody has labelled reads as unlabelled instead of showing a full pass. Anything still unlabelled counts on neither side of the badge, and a trace where one assertion passed and another is still open now shows the pass instead of reading as unlabelled.Deselect a label by clicking it again
Clicking the Pass, Fail, or Skip you already chose now clears it and returns the trace or assertion to having no label. This works the same way on trace labels and on grader labels. Clearing your own label leaves any agent or automated suggestion untouched, so the suggestion is still there when you come back to it.TypeScript SDKPython SDKPluginsCLI
TypeScript SDK v0.52.4, Python SDK v0.52.4, Plugins v0.10.9, CLI v0.2.321
Re-seed a trace in place
When a captured trace errored, or its recorded run is stale, you can now re-seed it: run the function once more on the trace’s recorded inputs and record the result under the same trace id. The trace keeps its id, labels, assertions, dataset membership, name, and metadata, so anything you stored against it still points at the same case. The previous run is kept as its own trace, linked back withreseedOfTraceId, so nothing is lost. Nothing is mocked and no experiment is created; a re-seed is a seed, not a replay.bitfab-seed is the new home of seeding: --from-trace re-seeds, and --cases cases.jsonl does what bitfab-replay --seed did (that spelling still works). From code, bitfab.reseedTrace(key, fn, { traceId }) / reseed_trace(key, fn, trace_id=) does the same. The Bitfab assistant’s replay mode takes this path when you ask for a re-seed or the trace errored, and warns first when the function has side effects, since a re-seed runs them for real. Graders on the trace’s datasets re-run afterwards, and default replay selection skips previous runs.Dashboard
Empty spans hidden in the trace view
The span tree now hides spans that recorded no input and no output, lifting their children up a level, so every row you click has something to show. A toggle at the top of the span panel says how many are hidden and brings them back, and your choice is remembered. Spans that recorded an error, and spans that differ in a replay comparison, always stay visible.TypeScript SDKPython SDKPluginsCLI
TypeScript SDK v0.52.3, Python SDK v0.52.3, Plugins v0.10.8, CLI v0.2.320
Seeded metadata survives a streamed agent run
Metadata you record withseedTrace / seed_trace or setMetadata / set_metadata now stays on the trace even when an agent framework exports its own trace after your traced function has returned. A streamed OpenAI Agents run ends its trace when the stream drains, which happens after the function returns, so its export used to take the trace back and a later replay read the wrong values in adaptInputs / adapt_inputs. Requires TypeScript SDK v0.52.3 or Python SDK v0.52.3.Clashing metadata keys warn once per trace
When you and an integration both set the same metadata key to different values, your value is the one kept on the trace and the SDK warns. That warning now fires once per trace and names the trace id instead of once per process, so a large replay run shows every affected trace rather than only the first.TypeScript SDKPython SDKRuby SDKPluginsCLI
TypeScript SDK v0.52.2, Python SDK v0.52.2, Ruby SDK v0.51.1, Plugins v0.10.7, CLI v0.2.319
Replay part of a dataset with --limit
--limit now narrows a dataset or trace ID selection instead of being ignored. bitfab-replay --dataset-ids <id> --limit 10 replays ten of that dataset’s traces rather than all of them, so you can sample an expensive corpus while iterating and pay for the full run only when you want it. The same bound applies to an explicit --trace-ids list, and to a dataset your registry entry already declares.A bounded dataset run selects the same traces every time, so two runs at the same limit stay comparable. Available in the TypeScript, Python, and Ruby SDKs.TypeScript SDKPython SDKPluginsCLI
TypeScript SDK v0.52.1, Python SDK v0.52.1, Plugins v0.10.6, CLI v0.2.318
Seeded trace metadata survives an agent framework’s own trace
Metadata you pass toseed_trace / seedTrace, or set with set_metadata / setMetadata, now always reaches a replay’s adapt_inputs / adaptInputs hook. Tracing a function that also opened an OpenAI Agents trace previously replaced your metadata with the framework’s own, so a seeded case’s provenance went missing exactly where the replay needed it. Both sets of keys are kept now, and yours win on a conflict.Save assertions across many traces in one call
Both SDKs can now write assertions for many traces in a single request instead of one request per trace.saveAssertionsAll in TypeScript and save_assertions_all in Python take one entry per trace, so a script attaching assertions to hundreds of traces makes one call rather than hundreds.Dashboard
Traces are identified by name
The trace list now shows the name your code gave each trace instead of an eight-character id. Names come fromsetName in the SDK, from a seeded trace, or from the trace’s name on a platform you imported from, so a run reads as something you recognize rather than a fragment of a uuid. A trace whose name only repeats its function name keeps showing the id, so no row gains a label that tells you nothing new.Assertions show on dataset and experiment lists
Dataset and experiment trace lists now name who authored a trace’s assertions, which until now only the main traces list did. Hovering says how many assertions the trace carries, and on an experiment row it adds that they were written on the original trace rather than on the replay in front of you.Your coding agent can name the trace to open
Trace reads through the Bitfab MCP server now include the trace name, so an agent can tell you which trace to look at by name instead of quoting a uuid. Reading traces, searching them, and listing an experiment’s traces all report it.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
TypeScript SDK v0.51.0, Python SDK v0.51.0, Ruby SDK v0.51.0, Go SDK v0.51.0, Plugins v0.10.4, CLI v0.2.316
Replay several datasets in one experiment
A replay could only point at one dataset, so measuring a function against several corpora meant running each of them separately and comparing the experiments by hand. You can now name as many datasets as you like. The run replays their combined traces once, scores them with the graders from every dataset you named, and files the experiment under each one, so it shows up on all of their experiments pages.bitfab-replay <pipeline> --dataset-ids ds-a,ds-b. The existing --dataset-id flag keeps working and now takes a list too, so the replay scripts you already have need no changes. All four SDKs have it, as datasetIds, dataset_ids, and DatasetIDs.DashboardPluginsTypeScript SDKPython SDKCLI
TypeScript SDK v0.50.0, Python SDK v0.50.0, Plugins v0.10.3, CLI v0.2.315
Read verdicts back per assertion
A verdict written against one assertion could only be read back as a passed and failed count, so an agent that scored six assertions on a trace could not say which one failed without opening Studio.get_trace_labels now lists each scored assertion on its own line under the trace, with that assertion’s verdict, annotation, confidence, and author, keyed by the same assertion id the write used.Labels and grader verdicts from the SDKs
The TypeScript and Python SDKs can now read verdicts, write human-validated verdicts, and read the individual verdicts each grader recorded, the same operations available through the Bitfab MCP tools. Reading returns each trace’s effective verdict plus one row per scored assertion, so a judge running inside a replay can verify what it wrote without leaving the process.labels.saveHuman writes a verdict that is validated on write, for cases a person has already decided. Human and agent batches are all-or-nothing. A trace outside your organization or an assertion that is no longer active rejects the whole call and writes nothing. Python exposes the same methods as labels.get_all, labels.save_human, and graders.get_labels.Retire an assertion from your coding agent
archive_trace_assertions retires one or more assertions on a trace so later replays stop checking them. Archiving keeps the row for audit and leaves any verdicts already recorded against it in place. The call is all-or-nothing, so one unknown or already-archived id fails it and archives nothing.DashboardPluginsTypeScript SDKPython SDKCLI
TypeScript SDK v0.48.0, Python SDK v0.49.0, Plugins v0.10.2, CLI v0.2.314
Replay only the traces you have assertions for
Replaying a dataset re-ran every trace in it, including the ones nobody had written an assertion for. Those runs cost a full re-execution and come back with nothing to grade. PassonlyWithAssertions to narrow a replay down to the traces carrying at least one assertion.bitfab-replay the flag is --only-with-assertions, and it narrows --limit, --trace-ids, and --dataset-id alike. Paired with --limit it picks the N most recent traces that have assertions, instead of taking the N most recent and leaving you with however many of those turn out to be gradeable.Available in the TypeScript and Python SDKs.Dashboard
Experiment totals now match the verdicts on your traces
When a trace carried a label on each of its assertions, its experiment totals could count it as passed even though the trace list showed it as failed. Totals and trace verdicts now follow the same rule, so any failing assertion fails the trace. Grader results count toward that verdict as well, so an experiment’s pass rate reflects every label written against a trace.Dashboard
Replay diffs stop flagging repeated calls that just ran in a different order
When a replay called the same function many times, the diff matched those calls by position, so a set of identical calls that came back in a different order was reported as a wall of changes. Calls are now matched by what each one was given, so only real differences surface. On one replay with 31 repeated calls, the reported changes dropped from 30 spans to 2, and the change that mattered (a verdict moving from approve to request changes) went from buried to second in the tree.The Diff view has a new Diff algorithm control. Smart is the default and matches calls by content. In order keeps the old positional matching, for when the sequence itself is what you are testing.Threads in the span tree
A trace that ran work on more than one thread now marks each span with the thread it ran on, as a smallT1 or T2 chip on the row with the full thread name in the row tooltip. Traces that used a single thread are unchanged.Grade a replay item inside the process that ran it
ReplayConcurrency now takes on_item_finish_in_child_process, a hook that runs in the child interpreter that replayed the item rather than in the process that owns the run. Under primitive="process" that child is the only place the replayed run’s own state still exists, so a judge can read what the run left in memory before the process exits. The item arrives with its replay trace ID already resolved, so a verdict can be keyed by trace_id directly instead of by lineage.on_item_finish is unchanged and still runs in the process that owns the run, so both can be set at once. The child grades what only the child can see, and the parent keeps the running totals and still reports an item whose child died before finishing.DashboardTypeScript SDKPython SDKPluginsCLI
TypeScript SDK v0.47.0, Python SDK v0.47.0, Plugins v0.10.0, CLI v0.2.312
One verdict per assertion
A trace’s verdict now comes from its assertions instead of a single pass or fail. Each assertion carries its own verdict and its own reasoning, so a run that meets four of six assertions reads as 4/6 rather than one failure with the detail buried in a note. The trace verdict is derived from those labels, and any failing assertion fails the trace.Traces with no assertions are unchanged. They keep their single whole-trace verdict, and every verdict recorded before this release reads exactly as it did.Labels and assertions are separate panels
The trace drawer now has a Labels panel and an Assertions panel behind a toggle, and either one collapses to give the full width back to the trace. Labels is where you record verdicts, one row per assertion with its own pass, fail, and note. Assertions stays read only and shows what the trace is expected to do when it is replayed.Labels are counted rather than named. A badge reads passing over total, so a single verdict shows as 1/1 and six assertions with one passing show as 1/6. Traces with nothing recorded keep the dashed unlabeled pill.Score one assertion from the SDKs
The labels namespace takes an optional assertion id, so a replay can grade one assertion at a time. Omit it to write the trace’s whole-trace verdict, which behaves exactly as before.assertion_id on save, save_all, skip, and archive.Dashboard
Live in-progress trace indicator
Trace detail pages now make it clear when a trace or replay is still running, with a banner that appears while it’s in flight and disappears once results are in. The page keeps checking for updates in the background and jumps to the latest span when you open a trace that’s still going.Diff view always highlights a span
The experiment diff view now always opens with a span selected, even when every difference between the original and replayed run is a span that simply didn’t run this time. Previously the view could open with nothing highlighted.Python SDKTypeScript SDKPluginsCLI
TypeScript SDK v0.46.0, Python SDK v0.46.0, Plugins v0.9.105, CLI v0.2.311
Grade each replay attempt as it finishes
A replay registry entry can now carry anon_item_finish callback, and bitfab-replay runs it after its own progress reporter instead of replacing it. That is the earliest point a verdict can be written, because a replay verdict is keyed by the original trace plus the attempt plus the test run, and that row only exists once the attempt’s own trace has been flushed.--dry-run, since nothing ran, and one that raises is reported on stderr with the trace and attempt instead of failing the run. Python for now, with Ruby and TypeScript to follow.One verdict with save, many with save_all
save_one is gone. save is now the single-verdict call and takes the same target arguments skip and archive take, and save_all takes a list. TypeScript is save and saveAll.This is a breaking change. Update any call that used save_one or saveOne, and any call that passed a list to save.Dashboard
Faster loading for large traces
Traces with large payloads now open in about a second instead of the fifteen to twenty seconds they could take before. The trace page used to download the full input and output of every span before it could show you anything, which made big agent traces slow to open. Span content now loads when you select a span, so the span tree and timings appear right away.Dashboard
Automatic capture policy is in Alpha
The capture control on a TypeScript SDK function trace is now labeled Policy with an Alpha badge. It still opens the Studio capture policy for that function, but the page is read-only for now: you can see which functions include content on future traces, and you cannot change or save the selection while the feature is in Alpha.TypeScript SDKPython SDKPluginsDashboardCLI
TypeScript SDK v0.45.0, Python SDK v0.45.0, Plugins v0.9.104, CLI v0.2.310
Say what a trace should do the next time it runs
A grader checks something that must hold for every trace of a function. An assertion is the other half: what the right answer is for one specific input. Write it against the trace you are looking at, and it is checked against the replay of that trace rather than against the run you wrote it on.Your coding agent writes them through two new MCP tools,save_trace_assertions and get_trace_assertions, and both SDKs read and write them directly:client.labels is new too. Verdicts from a replay process used to be reachable only through MCP, which meant a judge running inside your own replay script had nowhere to write. It now writes the same rows, keyed the same way, and skip withholds a verdict on an attempt that crashed or an assertion whose target could not be found, so neither one is recorded as a behavior regression.Assertions show up read-only on the trace page and in the labeling panel. Fill them in from your agent or the SDK for now.An assertion takes the same assertion, passCriteria, and failCriteria a grader takes, so one that proves out across many traces is promoted into a grader by copying its fields.TypeScript SDKPython SDKRuby SDKGo SDKPluginsDashboardCLI
TypeScript SDK v0.44.1, Python SDK v0.44.1, Ruby SDK v0.43.1, Go SDK v0.41.1, Plugins v0.9.103, CLI v0.2.309
Trace outlines on every replay item
Every replay result item now carries two trace outlines:originalTraceOutline for the trace that was replayed and traceOutline for the trace the replay produced. An outline is the span tree without any inputs or outputs, so it stays small: each span’s name, type, order, duration, tokens, model, errors, and whether it was served from a recording. Use them to grade how a replay reached its output and not only what it produced, such as whether it called the same tools in the same order or leaned on a mocked span that used to run real code.trace_outline and original_trace_outline, and Go as TraceOutline and OriginalTraceOutline. Both are filled in when the replay completes, so they are null on the per-item finish callback and against older servers.The assistant grades replays by their path
The Bitfab plugin now writes both outlines into each replay run’s per-item files and compares the two span trees when it judges a replay, so a replay that produced the right text by skipping a required tool call, erroring in a child span, or leaning on a mocked span no longer passes on output alone. Items judged while the replay is still running are re-checked against their outlines once the run completes.Replay each trace in its own process
Python replay can now give every work item its own process, which makes it possible to replay code that keeps its world in process-global state: a settings module read once at import, a database provisioned per item, or a module-level registry. Previously all items shared one process, so anything that could not be set up twice in the same interpreter only ever replayed its first trace.ReplayConcurrency also carries attempts and max_concurrency, so a run’s repeat count and its parallelism are set in one place and cannot disagree. The fan-out unit is the work item rather than the attempt, so 40 traces at 3 attempts is 120 processes drawn from one queue, bounded by max_concurrency (4 by default in process mode). The default primitive stays "async", so existing replays are unchanged. Process mode runs through the bitfab-replay --registry command, which is what re-runs your pipeline once per item.The Amp plugin, at parity
Amp support graduates from the Alpha skill pack to a native plugin. It registers the same three skills the other hosts get,bitfab:setup, bitfab:assistant, and bitfab:update, and every Bitfab tool, so Amp runs the full workflow itself instead of handing off to a second agent in your terminal.bitfab:update sdk, and Bitfab tools you can call directly. Ask Amp to search your traces and it can. The tools are registered by the plugin under their bare names (search_traces, get_traces, save_trace_plan) rather than through an MCP server.bitfab-cli installs the plugin into ~/.config/amp/plugins/bitfab and confirms Amp loaded it. analyze-repo --editor amp runs headless through amp -x.Still to come on Amp: session log capture and the auth and update banner at session start. Both ship through Amp’s agent.start and agent.end events in a follow-up. Details are in the Amp plugin docs.Bitfab in Amp, in Alpha
Amp is now a supported host, in Alpha.bitfab init installs Bitfab for Amp the same way it does for the other editors:--editor amp works for plugin-install, setup, and analyze-repo too, and the CLI offers Amp automatically when it is the editor on your PATH.Three skills. bitfab-setup instruments workflows and covers modify, inspect, replay, database snapshots, and templates. bitfab-analyze-repo scans the repository and uploads draft trace plans without prompts or code edits. bitfab-account handles sign-in, organization switching, and health checks.Alpha means the improvement loop is not there yet. Datasets, graders, experiments, trace labeling, and Studio review all need the native Amp plugin, which is planned. Amp also registers no Bitfab MCP tools of its own, so you cannot ask Amp to query your traces directly, though the setup flow itself has the full tool set. Run the improvement loop in Claude Code, Cursor, or Codex for now. Full limits are in the Amp plugin docs.Mixing the two tracing styles now fails fast
withSpan records exactly the functions you wrap. withTrace, trace, and node record a root plus the first-party calls beneath it. Mixing them in one call stack used to record both surfaces and log a warning. It now throws MixedTracingError, matching the Python SDK. A blended stack produced a trace whose shape misrepresented how your code was instrumented and left replay boundaries unpredictable, so it is better caught at the call that made the mistake.node() or withNode() when a discovered call needs its own name, type, or replay-mocking policy. Two things are deliberately exempt: framework integrations open their spans on whichever surface surrounds them, and the root that replay() and seedTrace() wrap around an undecorated function still accepts a subtree trace inside it.Dashboard
A clearer span tree in the trace view
Span icons are now filled in their type color instead of drawn as outlines, so the kind of each span reads at a glance. Rows are tighter and every row is the same height, which fits more of a trace on screen before you scroll. The detail pane sits on a light grey ground against the white span list, so the two halves of the page no longer blend into one sheet.The collapsed sidebar opens when you click a function
Clicking a function name or All Functions while the sidebar is collapsed now expands it, instead of only navigating. The section links under a function name still navigate without expanding, so you can jump straight to Traces or Datasets and keep the sidebar narrow.Seed a replayable trace by running your code once
seedTrace can now run a function once and record that execution as a replayable original. Pass the function instead of a case, and the SDK records the real input, the real output, and the first-party subtree beneath it, while capture stays off for everything else. A corpus you hold outside Bitfab becomes traces that replay can select, without waiting for production to produce them.adaptInputs hook now receives the seeded trace’s stored metadata on ctx.metadata, so a case keeps its provenance instead of smuggling it through the recorded inputs. The replay CLI gained --run for seeding a whole cases file through the registered function.captureEnabled replaces enabled
The client option that turns capture off is nowcaptureEnabled. The old enabled still works and warns once. Turning capture off no longer removes the tracing wiring, so wrapped functions keep their trace function key and both replay and seedTrace still record against a capture-off client. If you relied on enabled: false handing back your original function, you now get a wrapper that runs it untraced.A warning when both tracing styles meet
withSpan records exactly the functions you wrap. withTrace, trace, and node record a root plus the calls beneath it. Both still work when they meet in one call stack, and the spans nest where they were called, but the SDK now warns once per trace function key so a mixed setup is visible rather than silent.TypeScript subtree tracing is ready to install
Marking one function withtrace() or withTrace() records every first-party call beneath it, at any depth, with no wrapper on each one. The build adapters that make those nested calls visible now ship as bitfab-transform. Install it next to the SDK and add one adapter where your server code is compiled.tsx. The August 25 entry called this package @bitfab/transform. It is published as bitfab-transform, and the docs now match.Dashboard
The Diff view collapses to the spans that changed
A replay’s Diff view now folds the spans that match the original trace out of the span tree. Any run of two or more neighboring spans with no change, their children included, collapses into one row reading how many spans it hides, which opens and closes on click. A span whose child changed stays on screen, so nothing that moved is ever hidden behind a fold.Because the fold follows the diff, loops no longer group the tree here: a loop with one changed iteration reads as that iteration between two folded runs rather than as a single loop row. Each changed span’s+N −M line counts now sit at the right edge of its row, so they line up down the tree.Archive trace functions you are done with
You can now archive a trace function from the sidebar, so a workflow whose code is gone stops crowding the function list. Archiving deletes nothing. Its traces, datasets, experiments, and graders all stay, and the function comes back on its own the next time a trace arrives for it. Archived functions collect in an Archived section at the foot of the sidebar, where you can restore one at any time.See whether a traced function still exists in your code
Clicking a function in the sidebar now shows when it last traced, when an agent last confirmed its trace plan, and how likely it is that the code producing it still exists. That likelihood reads as a plain label rather than a number, because it is inferred from those two signals rather than measured.LangGraph tracing works inside trace() subtrees
A LangGraph graph wrapped with get_langgraph_integration() can now run inside a Python trace() subtree. Previously the integration’s tool and invoke spans raised MixedTracingError and stopped the run, so opt-out tracing and the LangGraph tool replay hooks could not be used together. Put @trace on the function that calls the graph and everything beneath it is recorded, with tool spans attaching under the call that ran them, counting toward the trace’s span budget, and following its mock_on_replay_default policy. Outside a trace() subtree the integration behaves exactly as before.Framework spans nest where they were called
Inside a Pythontrace() subtree, spans from the OpenAI Agents tracing processor, the Claude Agent SDK handler, and the LangGraph callback handler now attach to the function that invoked them instead of collapsing onto the trace root. An agent run started deep inside a workflow now appears at that depth in the span tree, so you can see which step made the call. Span types are unchanged.DashboardTypeScript SDKPython SDKRuby SDKPluginsCLI
TypeScript SDK v0.42.0, Python SDK v0.42.0, Ruby SDK v0.42.0, Plugins v0.9.92, CLI v0.2.299
Replay every trace K times in one experiment
Replays can now run each trace several times inside a single experiment, so a flaky result reads as flaky instead of as a regression. Passattempts to replay in the TypeScript or Python SDK, or --attempts N to bitfab-replay, and every trace replays that many times, each attempt with its own verdict, tokens, and database branch.attempt alongside originalTraceId in save_agent_labels.Dashboard
Loops fold up in the span tree
When an agent or batch run repeats the same call, the trace viewer now folds those repeats into one row showing the iteration count, how many iterations failed, and the loop’s total duration. Expand it to see the first, last, and failed iterations, and reveal the rest ten at a time. Repeating cycles fold too, such as an LLM call followed by a tool call, and arrowing into a folded loop opens it on the span you land on.A cleaner span tree
Each span type now has its own icon color, so LLM calls, agents, and tool calls read at a glance, and connector lines fold into each span instead of running as bare borders. The span panel in the trace view can be dragged wider or narrower, and folds away behind a “Spans” button in the header when you want the full width for the span you are reading.TypeScript SDKPython SDKRuby SDKGo SDKDashboardPluginsCLI
TypeScript SDK v0.41.0, Python SDK v0.41.0, Ruby SDK v0.41.0, Go SDK v0.41.0, Plugins v0.9.91, CLI v0.2.298
Name a trace by the record it is about
Every trace now has aname: the title Bitfab shows for it, and a field you can search and filter on. Set it from inside any traced function with getCurrentTrace().setName(...) (set_name in Python and Ruby, SetName in Go), pass name when seeding a trace, or set it later through the detached trace handle. Give it the ticket, order, or dataset row the run handled, and the trace list filter and search_traces will find that run by its label.workflow_name field the SDKs used to send is now name; older SDKs and direct HTTP integrations that still send workflow_name keep working, and an explicit name always wins over it. The CSV export’s workflowName column is now name.Seed originals by running a function once with capture off
seed_trace in the Python SDK now runs your function once and records that run as an original trace, with capture still off. The recorded trace carries the root span, the full first-party subtree, and the real inputs and output, so replay selects it like any captured trace and every replay of it links back as original_trace_id. Pass metadata to store a case’s provenance on the trace.seed_trace from synchronous code. With trace_across_threads=True, spans from worker threads nest under the seeded root, and an exception is recorded on the root span and re-raised. bitfab-replay --seed cases.jsonl runs each case the same way: a case’s input and kwargs are the call itself, and expected is no longer accepted because the output is what the run produced. This replaces the earlier Python seed_trace, which wrote a root-only trace from a case without running it. The TypeScript seedTrace keeps that behavior for now.Original trace metadata in the replay input adapter
Theadapt_inputs hook’s context now includes metadata, the original trace’s stored metadata, so a table-driven adapter can key off a case id or suite without smuggling it through the recorded inputs. It is fetched only when an adapter is registered. In the trace list, a seeded trace that is still running or threw now shows its running or error state instead of the seeded icon.TypeScript SDKPython SDKRuby SDKGo SDKDashboardPluginsCLI
TypeScript SDK v0.40.0, Python SDK v0.40.0, Ruby SDK v0.40.0, Go SDK v0.40.0, Plugins v0.9.89, CLI v0.2.296
Manage datasets from the SDKs
You can now create, read, and modify datasets programmatically, without a coding agent in the loop. Every SDK exposes a datasets namespace (client.datasets in TypeScript, Python, and Ruby, client.Datasets in Go) with the same operations the Bitfab agent tools offer: save a dataset, list datasets, fetch one, list its trace ids, add or remove traces, assign or unassign graders, and re-run graders over the dataset./api/sdk/datasets for anything not using an SDK.Opt-in and opt-out tracing never mix
The Python SDK has two tracing surfaces.@span is opt-in: only the functions you decorate are recorded. @trace with @node is opt-out: one root records every first-party function beneath it, and @node configures a discovered call without creating a boundary. They were never meant to share a call stack, and a @trace root called beneath a @span used to detach into a trace of its own, or, inside a replay item, into a second parentless root. The SDK now raises MixedTracingError at the boundary in either direction, naming which decorator was entered inside which.@trace subtree, configure a step with @node. The root span that replay("key", fn) wraps around an undecorated callable belongs to neither surface, so an entrypoint that calls a @trace root now nests that root beneath the item root instead of leaving it parentless. Nested @trace roots still record independent traces.Replay with capture off in the Python SDK
Bitfab(enabled=False) used to hand your functions back undecorated, so a client with tracing off could not replay them: replay(fn) and the replay registry had no trace function key to find. The flag is now capture_enabled, and it only controls capture. Decorated functions keep their wrapper, a replay item always records its trace, and seed_trace always writes, so one client serves both the environments where you do not want capture and the replay runs you start on purpose.enabled still works as a deprecated alias and logs a one-time warning; move to capture_enabled when you upgrade to Python SDK v0.39.1. TypeScript, Ruby, and Go are unchanged for now.DashboardPluginsTypeScript SDKPython SDKCLI
TypeScript SDK v0.39.0, Python SDK v0.39.0, Plugins v0.9.86, CLI v0.2.293
Seed replayable traces from cases you already have
Replay used to require a captured trace, so a corpus of test cases could not be replayed until the code had run against every one of them.seedTrace / seed_trace writes a replayable trace directly from a case, recording its arguments as the input and the value you expect as the output, with no execution at all.fn checks the case against the function’s real signature, so a case that could never run is rejected while you seed rather than failing later during replay. The replay registry seeds too, with bitfab-replay --registry <path> <pipeline> --seed cases.jsonl, which binds every case to the exact function that replay will select.A seeded trace records no inner calls, so replay mocking has nothing to substitute and database snapshots are refused rather than silently pinned to the wrong moment. Replays of seeded traces report each item as matched or missed against its expected value instead of same or changed against a previous run, and the trace list marks them with their own icon.Keyword arguments now replay as they were called
In the Python SDK, a traced function called with keyword arguments recorded them correctly but replayed them as a single trailing positional argument, so a keyword-only signature raised aTypeError during replay. Whether it broke depended on the values, not the signature: a datetime or UUID anywhere in the call happened to preserve the shape, and plain JSON did not.Recorded inputs now keep the positional and keyword split whenever keyword arguments are present. If you worked around this with an adapt_inputs hook that re-splits a trailing dict, remove it when upgrading to Python SDK v0.39.0, as replay now hands your function the arguments it was originally called with.Preview what a replay will run
--dry-run resolves every selected trace’s inputs, applies any input adapter, and prints the exact arguments your function would receive without calling it. It is the quick way to confirm that recorded inputs still fit a signature you have changed since capture, and no database branches are provisioned for a run that executes nothing.A replay whose selection matched no traces now exits non-zero instead of reporting a clean run of zero items.Setup now surfaces subtree tracing
Setup treated spans as the only instrumentation primitive, so it reached forwithSpan and @span and never surfaced the newer subtree API. It now names all three primitives when it reads the SDK reference, so trace (a root plus every first-party call beneath it, with no decorators on those calls) and node (naming, typing, capture, and replay-mock policy for a single call inside that subtree) are both on the table while your code is instrumented. The Reference overview in the docs gained a table showing which primitives each SDK supports.Setup entirely from the CLI
Bitfab’s experimental setup can now plan, instrument, and verify AI workflows without leaving your terminal. Runbitfab init --v2 for onboarding or bitfab setup --v2 [mode] for a specific setup workflow; repository edits, commands, and setup decisions stay behind terminal approval. Use --diagram to print the flow’s state diagram without starting the agent.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.9.83, TypeScript SDK v0.38.10, Python SDK v0.38.7, Ruby SDK v0.38.3, Go SDK v0.37.0, CLI v0.2.290
Replay production traces from Go
Go SDK v0.37.0 can now run a trace’s historical inputs through your current Go code with typed arguments and results.GetFunction(...).Replay(...) supports replay lifecycle callbacks, replay mocking, mock overrides, input adapters, code-change capture, and database snapshots, while waiting for each replayed trace to reach Bitfab before finishing the experiment.Stable call ordering across every SDK
TypeScript, Python, Ruby, and Go now record strictly increasing microsecond timestamps, preserving call order even when several calls start within the same millisecond. Go replay also uses delivery acknowledgments from the SDK transport before finalizing an experiment, matching the other SDKs; server polling remains a fallback when delivery is ambiguous.Opt a traced subtree into default replay mocking
@trace(..., { mockOnReplayDefault: true }) / @trace(..., mock_on_replay_default=True) now establishes replay mocking as the default for nodes under that trace. Use @node({ mockOnReplay: false }) / @node(mock_on_replay=False) to override the default for code that should keep running live.This behavior is opt-in per trace. Existing traces and replay strategies are unchanged, including mock: "all", which still mocks every matched recorded descendant. TypeScript transformed descendants inherit the trace policy automatically; Python requires @node() on descendants that need replay boundaries.More reliable large trace uploads
Large span payloads now go directly to durable storage as they arrive, reducing duplicate work during trace ingestion. If that upload fails, Bitfab temporarily retains the payload and retries automatically so the span is not lost. TypeScript SDK v0.38.8 also sends OpenAI Agents span data only when the span is complete, giving Bitfab one authoritative snapshot with its output, error, and timing.DashboardPluginsTypeScript SDKPython SDKCLI
Plugins v0.9.80, TypeScript SDK v0.38.7, Python SDK v0.38.5, CLI v0.2.287
Replay LangGraph tools without running them again
LangGraphToolNode calls can now use output mocks during replay. Selected tools return their recorded or overridden output without executing, so unsafe side-effects such as sending an email or charging a card are not triggered again.getLangGraphIntegration() in TypeScript wraps the tool array passed to ToolNode. get_langgraph_integration() in Python exposes the native wrap_tool_call and awrap_tool_call hooks accepted by ToolNode. createInvoker() / create_invoker() then returns the normal graph entry point with the callback handler and replayable root already connected. Both integrations preserve native ToolMessage and Command results and refuse to execute a live tool when replay expected a recorded result but none exists.This API is experimental (alpha). Calls are currently matched by tool name and occurrence order, so start with deterministic tool flows and test replay against non-production dependencies before relying on it for unsafe side-effects.Dashboard
Live experiment grading stays responsive
Experiments now apply grader results smoothly as they arrive, without freezing the page or repeatedly refreshing summaries and statistics. Run headers and grader statistics reconcile once grading finishes, so final results remain accurate even when checks complete together.Dashboard
Faster, more reliable trace ingestion
Trace ingestion now completes without waiting for search indexing, reducing timeouts when applications upload large or highly concurrent traces. Newly completed traces may take a brief moment to appear in dashboard search while their lexical and semantic indexes finish in the background.DashboardPluginsTypeScript SDKPython SDKRuby SDKCLI
Plugins v0.9.78, TypeScript SDK v0.38.5, Python SDK v0.38.4, Ruby SDK v0.38.2, CLI v0.2.285
Route replay mocks by traced function
You can now register replay mock overrides once on a TypeScript, Python, or Ruby client and scope them to a trace function key, so large replay suites can share mocks without re-declaring them on every call. Resolvers can inspect each span, return a replacement, or returnNO_MOCK_OVERRIDE (Bitfab::NO_MOCK_OVERRIDE in Ruby) to fall through to lower-priority overrides and the replay’s base mock strategy; null and undefined remain valid mocked outputs.Capture complete TypeScript trace subtrees
@bitfab.trace and withTrace now capture every discovered call’s full inputs, output, and thrown error by default. No node() annotation or Configure capture step is required. Source exclusions and capture: false still omit calls, and an optional Studio policy can narrow later traces to selected function IDs.Use @bitfab.node(options) on transformed methods or bitfab.withNode(options, fn) around named standalone functions only when one call needs an explicit name, type, finalizer, omission, or replay policy. Outside an enclosing trace they execute normally without emitting a span. Omitted nodes keep captured descendants attached to the nearest captured parent, and replay-mocked nodes return their recorded outputs.Dashboard
Experiment updates keep your place
Experiment pages now keep open modals and expanded run details in place while new experiment results arrive. Live updates refresh the list and its statistics without flashing the page or closing what you had open.DashboardPluginsTypeScript SDKPython SDKRuby SDKCLI
Plugins v0.9.76, TypeScript SDK v0.38.3, Python SDK v0.38.3, Ruby SDK v0.38.1, CLI v0.2.283
Replay with one SDK-owned command
TypeScript SDK v0.38.3, Python SDK v0.38.3, and Ruby SDK v0.38.1 now installbitfab-replay, so your project owns only a replay registry mapping short names to the exact traced functions production calls. Pass the registry path and registered name to the command; common flags, progress reporting, and result output stay current when you update the SDK.Parameterize custom replay behavior
Registry entries can combine defaults with an options factory that receives values from--param or --params, so a replay can configure mock overrides and input adapters without editing the registry. Command-line values override overlapping defaults, while source-conflict validation catches incompatible trace and dataset selections before running.Dashboard
Know which function every row came from
The All Functions views for traces, datasets, graders, and experiments pool work from every traced function, and until now nothing on a row said which one produced it. Every row now names its function. Traces, datasets, and graders give it a column of its own, so the names line up and you can scan down them to see where one function ends and the next begins. Lists already scoped to a single function are unchanged, since repeating the same name on every row tells you nothing. Dataset names also stopped truncating early, because the pass rate column was holding on to width it never used.Trace complete TypeScript call paths from one root
Mark one TypeScript workflow with@bitfab.trace(key, options) or bitfab.withTrace(key, options, fn) to record its nested repository calls as lightweight spans. Add bitfab-transform through your build tool, then use Configure capture in the trace view to choose which functions include inputs, outputs, and errors on future traces. This experimental release supports Babel, the TypeScript compiler, SWC, common bundlers, direct Node execution, Next.js, and NestJS.Dashboard
Labels update everywhere without a refresh
Labeling a trace now updates its trace list row, open trace, dataset bucket, and related experiments together without refreshing the page or refetching whole lists. Trace-function counts also update immediately, while older sessions and uncertain filter matches still reconcile safely.Dashboard
An interactive tour of how Bitfab works
There is a new tutorial page at/tutorial that walks through Bitfab in two steps: a small traced workflow you can read in TypeScript, Python, or Ruby, and a live replay of it. Every call in the replay plan has a Re-run or Mock toggle, and flipping one restarts the animation, so you can watch what a replay actually does: which calls execute against your current code, which answer from the recording, and which are skipped because they sit inside a mocked call. Hovering a call links it to the lines of code that created it, and back.Dashboard
Runs nobody labels no longer look unfinished
An experiment run with no graders attached now reads as finished once its replays land. Until now the pass rate kept circling and Done never showed the run as complete, because a replay waiting on a person or an agent to label it counted as work still in flight. Runs that do have graders attached now circle until the last grader check reports, instead of reading as done the moment their replays finish.Dashboard
See how much each span changed
Comparing a replay against its original trace now shows a +N −M line count on every span in the list, the way a diff counts changed lines. The Diff view already marked which spans differed, but working out which one moved the most meant opening each span in turn. Spans whose content is identical, and spans that ran on only one side of the comparison, show no count.Dashboard
Filters stay put when you open a trace
Opening a trace from the traces, experiments, or dataset lists now slides the detail panel in directly beneath the page header, with the filter bar behind it left where it is. You can change Group by, switch between original and replay traces, or switch organizations without first closing the trace you are reading. Regrouping experiments keeps the open trace open and leaves expanded runs expanded, instead of resetting the list underneath you.Links to an experiment trace open that trace
A link to a trace inside an experiment now opens straight to it, and reloading the page keeps it open. The run holding that trace opens with it, so the arrows in the panel still step to the next and previous traces in the run.Configure traced Python calls without another span
Python SDK v0.38.2 adds@client.node to configure a function only when it runs inside an enclosing @client.trace, without turning that function into a standalone span. Set capture=False to omit the node while keeping its captured descendants connected to the nearest captured parent, or set mock_on_replay=True on a captured node to return its recorded output during replay.Name what to instrument instead of picking from a list
/bitfab:setup now takes a file, function, or directory directly when you already know what you want traced. Type it at the prompt and setup reads only that location, instead of scanning your codebase and handing you a list of candidates. If you would rather it find the workflows for you, that option is still there and still the recommended default.Setup stops asking questions you already answered
When your project already has instrumentation, setup asks once what you want to do and carries that answer forward. Naming a workflow at the first prompt survives the SDK install, so you are not asked the same thing again a few steps later, and naming an existing trace function key takes you straight to changing what it captures. Setup now also tells you before it installs the SDK and writes your API key, rather than doing it silently.Dashboard
Reliable experiment startup
New experiment pages now open into a preparation state while the first replay starts, rather than failing because the experiment group has not been created yet. If setup takes longer, the page switches to a neutral waiting message; genuine request and authorization failures remain distinct and provide diagnostics the Bitfab team can investigate.Dashboard
Experiment history appears sooner
Experiment history now shows its first page with the initial page load, so organization-wide and function-specific lists no longer wait for an extra browser request before appearing. Statistics still fill in asynchronously, and pagination and live updates continue working as before.Dashboard
Faster, more reliable live updates
Live trace, label, dataset, and experiment updates now arrive with less duplicate work across open Bitfab dashboard tabs. Experiment summaries update immediately when Bitfab has the complete result, while targeted refreshes keep moved runs and labels on replayed or original traces accurate.Dashboard
Closely timed calls keep their captured order
Trace views and replays now preserve the original order of calls captured within the same millisecond. Bitfab keeps the full timestamp precision from supported SDKs and trace imports, so closely timed calls no longer swap positions between rendering and replay.Dashboard
Faster experiment history
Experiment history now shows each run as soon as its metadata arrives, then fills in verdicts, trace counts, and token costs without blocking the page. Loading rows keep their final size while those statistics arrive, so the list stays stable as it becomes interactive.Setup restores a disabled Bitfab MCP server
Runningbitfab init now re-enables the Bitfab MCP server if it had been switched off for that project. Coding agents remember that setting per project and it survives reinstalling the plugin, so the Bitfab skills would come back while every Bitfab tool stayed missing.Clearer recovery when the Bitfab tools are missing
When setup cannot reach the Bitfab tools, it now points you at your editor’s MCP settings and stops, rather than continuing into steps that depend on those tools. In Claude Code, enable Bitfab from/mcp and re-run setup: the tools load into the running session, and restarting does not help because the setting is stored per project.Nested Python traces capture complete trees
Nestedtrace() roots in Python now each record a complete independent trace, matching the result you would get from running either root alone. Shared work appears in both traces with distinct span IDs, while span()-decorated functions remain in their active trace and record only once.Async-generator traces also release capture between pulls and still emit their span if closing raises, preventing unrelated work from leaking into a suspended generator’s trace. Nested trace regions intentionally record each session separately, so their shared region produces twice the span volume.Dashboard
Re-run graders on a single trace
You can now re-run graders on one trace instead of re-grading a whole dataset. Open a trace and use the Graders panel: each grader row has its own re-run control, and the panel header opens a picker for running several at once. A re-run also takes over a grader that never finished, so a trace left mid-grade can be restarted rather than waited out.Grading progress reads as a live pass rate
While a trace is being graded, the score counts passes over the graders that have answered so far, so “3/4” means three of the four back so far passed. A trace whose graders were all just queued shows a grading indicator instead of a zero score.DashboardPluginsTypeScript SDKPython SDKRuby SDKCLI
TypeScript SDK v0.38.0, Python SDK v0.38.0, Ruby SDK v0.38.0, Plugins v0.9.70, CLI v0.2.277
Replay items report their own duration, and name the original’s fields
Each replay result item now reports how long that replay took underdurationMs / duration_ms, measured around the replayed call. It previously carried the duration of the original trace, which meant a live progress line could show a number from a run that happened weeks earlier.Everything describing the trace being replayed now carries an original prefix: originalDurationMs, originalModel, and the new originalTokens, which was already computed server-side but never surfaced. With tokens reporting the replayed run, one item now holds both halves of a cost or latency comparison. model / :model stays as a deprecated alias for the original’s model, since there is no replay equivalent.Replay branch provisioning is also measured now: dbBranchTimings / db_branch_timings breaks down how long resolving the project, creating the branch, connecting the compute, proving it serves, and running your warm-up SQL each took, on success and on failure alike.If you read durationMs today, it now means the replay’s duration. Switch to originalDurationMs to keep the previous value.DashboardPluginsTypeScript SDKPython SDKRuby SDKCLI
Plugins v0.9.69, TypeScript SDK v0.36.13, Python SDK v0.37.12, Ruby SDK v0.36.11, CLI v0.2.276
Replays stop before an unsafe mocked call can run
Selected replay mocks now fail closed across TypeScript, Python, and Ruby. If a replay cannot load the historical span tree, match the recorded occurrence, or retrieve its output, the item returns an error instead of falling through to the real child call. The Claude, Cursor, and Codex plugins also audit unsafe actions and execution-context limitations before running a replay, and database-snapshot checks target the exact captured trace.Trace a whole call tree from one decorator
The Python SDK can now record an entire call tree from a single annotation.trace() captures the function you decorate plus every function in your own code that it calls, at any depth, with none of them decorated.max_depth and max_spans bound each subtree and warn once if they truncate it.Nested trace() roots produce complete independent traces, each with its own span IDs; two roots double span volume in their shared region. A span()-decorated call behaves differently: it stays in the active trace, records once, and parents the automatically captured calls beneath it. Async-generator capture is released between yielded items, even when the caller stops consuming without closing the generator.Experimental, and requires Python 3.12 or newer. On older versions the decorated function still records its own span exactly as span() does.DashboardPluginsTypeScript SDKPython SDKRuby SDKCLI
Plugins v0.9.67, TypeScript SDK v0.36.12, Python SDK v0.37.10, Ruby SDK v0.36.10, CLI v0.2.274
Replay diffs show every line
Comparing a replay against its original trace now shows the input and output in full. Unchanged lines that used to sit behind an “N unchanged lines” toggle are always visible, so you read a change in the context it happened in without expanding anything.See how each replayed span was produced
The diff view now carries the same header as the span view, so a span keeps its name and any error while you are reading its diff, alongside how the replay produced it. A mocked span says whether it returned the output recorded in the original trace or a value your replay code supplied, and a replay that ran against a database snapshot says which database it branched from and when the snapshot was pinned.Reporting the kind of mock needs the TypeScript, Python, or Ruby SDK version above. Spans captured by earlier versions keep the general “Mocked” label.Reach every organization in the switcher
The organization switcher scrolls now. Long organization lists were cut off at the bottom with no way to scroll, which put the organizations latest in the alphabet out of reach.Dashboard
More resilient concurrent Studio sessions
Studio is more resilient when multiple trace sessions are active at once. Live event polling now uses backend capacity more efficiently, reducing interruptions while traces are updating.DashboardPluginsTypeScript SDKPython SDKRuby SDKCLI
Plugins v0.9.66, TypeScript SDK v0.36.11, Python SDK v0.37.9, Ruby SDK v0.36.9, CLI v0.2.273
Replay diffs recognize renamed files
Replay code diffs now recognize files you renamed, so experiments show the edits inside a moved file instead of counting the whole file as deleted and added. Automatic code-change capture applies this behavior across the TypeScript, Python, and Ruby SDKs and ignores moves with no content changes.Replay surfaces each trace’s server ID as it finishes
Replay now gives you the servertraceId for each item the moment that item finishes, instead of only after the whole run completes. The id arrives on both the returned ReplayItem and the onItemFinish callback, so you can link straight to the trace in Bitfab or start per-item work while the rest of the run is still going. Available across the TypeScript, Python, and Ruby SDKs.Dashboard
Function counts arrive with the sidebar
Function names and their exact trace counts now appear together when the Bitfab dashboard loads. The function sidebar no longer fills in totals after navigation is already available, while counts still stay current as traces arrive.Trace ingestion survives count update failures
Trace ingestion now completes even if Bitfab encounters a temporary problem updating dashboard function counts. Any resulting count drift is repaired automatically, so newly ingested traces remain available without sacrificing accurate sidebar totals.Graders stay on the model you chose
Graders using Gemini on Vertex now retry temporary connection failures without switching away from the model you selected. If a run still fails, Bitfab records safer diagnostic context for faster investigation while keeping credentials out of error reports.TypeScript replays finish reliably
TypeScript replays now keep the process alive until their final traces are safely stored or the persistence deadline is reached. This prevents a locally completed replay from leaving its experiment unfinished while the SDK is still waiting for confirmation.Dashboard
Faster trace function counts
Trace counts in the function sidebar now load with the page instead of waiting for Bitfab to scan your trace history. Counts stay current as new traces arrive and when traces move between functions, so navigating large projects remains fast without showing stale totals.Dashboard
Grader results stay current during re-runs
Re-running graders now shows grading as soon as work is queued, then shows the latest verdict when each check finishes. Past automated results remain available in history, while counts and verdict pills use only the newest result and any human label stays authoritative. Dataset and experiment views update across open browser windows throughout the re-run.Dashboard
Live traces keep up with busy runs
The Traces page now stays responsive and up to date when many traces arrive at once. High-volume arrivals refresh in batches, and newly created traces remain visible instead of being lost among follow-up updates.One trace plan per function, kept up to date
Bitfab’s coding plugins now revise the trace plan you are already looking at instead of posting a new one beside it. Asking for a change while the plan is open in Studio updates that page in place, so the plan you are reviewing is the plan that gets saved, and the window stays where you left it.The same rule now covers the plans an earlier repo analysis drafted. A draft whose code has moved or been renamed since it was written is rebuilt against your current code and saved back onto the same plan, its trace boundary included, rather than being set aside for a fresh scan to duplicate. A draft for work that is already instrumented, or for a workflow that no longer exists, is retired so it stops being offered every time. Re-analyzing a repository refreshes the drafts it made before instead of adding a second draft per function, and leaves alone any function whose plan you are in the middle of working on.Every path that would post a plan now checks whether the function already has one, so instrumenting or modifying a function picks up the plan waiting for it instead of starting a rival. Viewing a function’s plan shows an unconfirmed one where before it reported nothing to view.Dashboard
Trace function pages open instead of erroring
Opening a trace function’s traces, or one of its versions, now loads rather than showing an error page. Those two pages worked out which function you meant differently from every other page in that group; they all share one answer now.A color means one thing
Language and agent chips, status badges and cautions are now easy to tell apart at a glance. A Python chip used to render in exactly the color that means “passed” and a Go chip in the color that means “mocked”, so a language could read as a result. Cautions shared the brand color, so a warning looked like an ordinary highlight. Each of these now has a color of its own, and hover labels that had gone missing from the session timeline are back.Dashboard
Graders that error or are still running are now visible
When a grader throws while scoring a trace, or is still working on it, that state now shows on the trace verdict pill, the experiment bars, and the labeling panel. Previously a broken grader looked the same as one that had never run. Neither state counts toward the pass rate, so a grader that failed reads as failed rather than as a lower score, and you can still label a trace by hand while its graders are mid-run.One failing grader no longer stops the others
If a grader errors while scoring a trace, the remaining graders now finish instead of being skipped. Re-running graders also picks up the ones that errored, rather than treating them as already scored.Dashboard
Database snapshots replay from their original source
Replays now restore database snapshots from the exact source recorded when each trace was captured, even when both a direct Neon connection and a mirrored connection are enabled. If that source has since been replaced or disconnected, replay stops safely instead of restoring from a different database.Replay completion stays fast under heavy span traffic
Python and Ruby replays now keep their fast completion path even when applications submit spans faster than the local export queue can drain. A dropped span no longer leaves acknowledgment state behind, and sustained traffic no longer displaces acknowledgments for spans still waiting to be sent, so completed replays avoid falling back to slower server polling.Dashboard
Faster trace function navigation
The trace functions sidebar now loads function names before waiting for exact trace counts, so you can navigate as soon as the sidebar appears. Trace totals fill in shortly afterward and remain scoped to your selected environment, without showing placeholder zeros while data is still loading.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLI
TypeScript SDK v0.36.8, Python SDK v0.37.6, Ruby SDK v0.36.6, Go SDK v0.36.2, Plugins v0.9.61, CLI v0.2.268
Replays finish as soon as their traces are safely stored
Replays no longer wait on a fixed delay before checking whether their traces were saved. Each SDK now tracks the acknowledgment the server already returns when it accepts a batch, so a replay finishes the moment its traces are durable instead of pausing and then polling. Runs that used to sit waiting now complete as quickly as the data lands.Steadier span delivery when ingestion is busy
All four SDKs now retry span delivery the way the OpenTelemetry protocol prescribes. When the server asks for a pause, that pause is respected exactly rather than cut short, and the whole client holds off rather than only the one request that was turned away. Without such a request, retries back off with jitter so many clients do not return in lockstep. Spans are also retried in more of the cases worth retrying, so a brief hiccup during ingestion is less likely to cost you data.Dashboard
More reliable live trace updates
Live trace updates in the Bitfab dashboard now stay connected when trace details cannot be loaded, so a temporary database problem no longer interrupts the rest of the stream. The dashboard also stops pending trace loads when you disconnect and fails stalled database connections promptly instead of leaving the stream hanging.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.9.60, TypeScript SDK v0.36.7, Python SDK v0.37.5, Ruby SDK v0.36.5, Go SDK v0.36.1, CLI v0.2.267
Large spans stay complete
Bitfab SDKs now preserve highly compressible span payloads up to 7.8 MB instead of trimming them at the normal 2.8 MB request budget, keeping large documents, message histories, and agent state available for replay and evaluation. The TypeScript, Python, Ruby, and Go SDKs send an oversized single span intact when it fits within the 3 MB wire target; payloads above the 8 MB ceiling or those that cannot fit still use the visible trimming fallback.DashboardPluginsTypeScript SDKPython SDKRuby SDKCLI
Plugins v0.9.59, TypeScript SDK v0.36.6, Python SDK v0.37.4, Ruby SDK v0.36.4, CLI v0.2.266
Know when every replay item starts and finishes
Replay lifecycle callbacks now identify exactly when each item starts and finishes, so integrations can distinguish queued work from in-flight work and handle each result as it arrives. UseonItemStart and onItemFinish in TypeScript, or on_item_start and on_item_finish in Python and Ruby; every callback includes the replay item, including finishes with an error. The deprecated onProgress and on_progress callbacks remain compatible for existing integrations.Trace class methods with TypeScript decorators
TypeScript 5+ projects can now trace class methods with standard ECMAScript decorators. Decorators are an optional shorthand;withSpan remains the recommended default for TypeScript 4.x, standalone functions, class fields, accessors, and legacy decorator projects.Dashboard
Replay counts on the usage page
The usage page now reports replays alongside the traces you send. A Replays card sits next to Total traces, the chart adds a Replays metric that works in both the total and by-function views, and the usage-by-period table and CSV export each carry a Replays column. Replays are counted within your total traces, not on top of them.Dashboard
Move experiments between groups
You can now move an existing experiment into another experiment group, or remove it from its group, from your coding agent. Usesave_experiment with a group ID to move it or null to ungroup it; open experiment views update automatically.Dashboard
Experiment history your agent can organize
Coding agents can now name and annotate individual experiments and experiment groups, then list recent groups or retrieve one group with its member experiments. This keeps related replay runs discoverable and gives your agent the context behind each iteration when it returns to an experiment later.DashboardPluginsTypeScript SDKPython SDKRuby SDKCLI
Plugins v0.9.56, TypeScript SDK v0.36.3, Python SDK v0.37.2, Ruby SDK v0.36.3, CLI v0.2.263
Every replay failure stays visible
Replay now returns one item for every attempted trace, even when setup fails before your function runs or prevents a replay trace from being created. The TypeScript, Python, and Ruby SDKs distinguish trace errors from replay errors, preserve the original exceptions and database snapshot failure details, and retain collected items if the whole run throws. New replay-result serializers keep those structured errors intact in direct stdout and SDK-managed result files.Dashboard
More reliable background jobs
Scheduled jobs and other durable background work now keep reaching Inngest when optional authentication or request-logging services are unavailable. This prevents unrelated dependency disruptions from interrupting grading, imports, enrichment, and other workflows that run in the background.Trace plans stay current with your code
When you modify tracing for an existing function, Bitfab’s coding plugins now reread the current instrumentation before revising its trace plan. The stored plan preserves your earlier capture and replay choices, while newly added, removed, moved, renamed, or unwrapped calls are reconciled against the code so the plan does not silently omit current work. If the trace boundary itself changed, the existing plan moves to the current root instead of preserving an obsolete one. Newly discovered calls also receive samples and replay analysis before the updated plan is saved.Dashboard
Grading progress across a whole dataset
Re-running graders on a dataset or experiment now shows every trace in it as grading right away, instead of lighting up a handful of rows at a time as the work moves through them. Each row settles back to its pass rate as its own graders report, and a re-run that fails part-way no longer leaves rows stuck looking like they are still being graded.Trace rows say what they are waiting on
A trace whose graders are still scoring now counts up from zero, so a re-grade no longer shows the previous run’s pass rate as though it were current. Rows still waiting on the trace itself read “Trace” rather than “Label”, and anything in flight carries a moving marker around its pill, so you can tell at a glance which rows are still working.Dashboard
Live grading recovers from stalled checks
Live grading now stops individual grader checks that stay stuck too long, then retries a timed-out check once without consuming the retry budget for other temporary failures. The deadline covers the full check, including its own model retries and failure explanation, so one stalled grader no longer keeps an entire live-grading run open indefinitely.All Functions keeps the section you are in
Switching from a single function to All Functions in the sidebar now keeps you in the section you were already viewing. Previously it always returned you to Traces, so going from one function’s Experiments to every function’s Experiments took an extra click. This applies to Traces, Datasets, Experiments, and Graders.Simpler trace plan saves
Bitfab’s coding plugins now use onesave_trace_plan tool to create new plans and revise existing ones. Older plugin installations that call create_trace_plan or update_trace_plan remain compatible, so trace setup and re-instrumentation continue working during upgrades.Dashboard
More reliable live grading
Live grading now preserves completed grader checks when a later check needs to retry, avoiding repeated evaluations on traces with multiple graders. Longer grading runs also get more time to finish, while successful results remain authoritative if overlapping runs settle in a different order.Trace plans stay within the workflow you are tracing
When the Bitfab plugin drafts a trace plan, it no longer attaches callers from above your workflow’s root function. Those nodes counted toward the plan’s node total but were never drawn, so the count could be higher than the tree you actually saw. Plans now cover the root and the code beneath it, and the node count matches what is in front of you.DashboardPluginsTypeScript SDKPython SDKRuby SDKCLI
Plugins v0.9.51, TypeScript SDK v0.36.2, Python SDK v0.37.1, Ruby SDK v0.36.2, CLI v0.2.258
Faster replay mocking
Replays now transfer only the recorded input and output data needed to reconstruct calls, reducing payload size for traces with large captured context. Eagermock: "all" runs also avoid fetching the root output twice, while marked and selective mocking continue to load recorded outputs only when they are used. This improvement is available in the TypeScript, Python, and Ruby SDKs.Dashboard
Neon databases connect on every plan
Connecting your own Neon database now works whichever Neon plan you are on. Bitfab used to ask Neon for a custom scale-to-zero delay on the branches it creates, which Neon accepts only on its Scale plan and rejects outright on Free and Launch, so the connection check failed at the create-branch step. Bitfab now leaves that setting alone on databases you own, and the branches it creates for replay follow your own project’s scale-to-zero.Dashboard
Neon connection errors say what actually went wrong
When connecting a Neon database on the Integrations page fails, the error now quotes Neon’s own explanation instead of a bare HTTP status. A refusal that used to read “Request failed with status code 412” now names the cause, such as reaching your project’s branch limit, alongside Neon’s error code and request id so their support can trace it. Each failed step is still listed separately, so you can see exactly how far the check got.Trace plans update in place
Your coding agent now revises an existing trace plan instead of creating a new one each time it changes what gets captured. Each traced function keeps a single authoritative plan, so coming back to it later shows the current capture and replay decisions rather than a stack of competing drafts. Every revision is recorded, so the earlier version is still there to compare against.The trace plan page updates while you watch
The trace plan page in Studio now reflects your agent’s changes as they happen, with no reload. Spans it captures, uncaptures, or switches between replay and mock update in place, keeping the groups you expanded and your place on the page. The primary button reads Save once the plan differs from the one you were first shown.Codex setup launches autonomously again
Codex users can once again runbitfab init without permission prompts. The CLI’s --skip-permissions option now uses Codex’s supported autonomous mode instead of passing a removed flag that stopped the agent before setup began.Dashboard
Replay traces always open in the comparison view
Opening a replay trace now shows the same comparison view experiments use: a verdict chip and a Diff | Original | Replayed toggle that flips between a per-span diff against the original trace, the original itself, and the replayed run. This applies everywhere a replay opens, including the traces list and direct links, so replays no longer fall back to the plain trace view. Labels and a code change are optional; the view shows whatever the replay has.Fixed invisible tooltip text
Fixed an issue where tooltips across the dashboard rendered their text in a dark color on the tooltip’s dark background, making them unreadable. Tooltips on the sidebar navigation, trace status icons, and elsewhere now show their labels clearly.Span-by-span diffs in the replay comparison view
The Diff view now compares a replay to its original span by span instead of only the trace’s overall input and output. Spans whose content changed open a line diff, spans the replay never ran appear as faded red rows in the span tree, and new spans the change introduced are tinted green with their full content one click away. Spans that match the original stay selectable and render their content as unchanged lines, so you can still inspect what didn’t move.Dashboard
Span timing at a glance in the trace view
Every span row in the trace tree now carries a timing bar showing how long that span took as a share of the whole trace, offset by the time that had already elapsed when it started. Durations sit inline next to each span name, index badges are gone, and nesting is tighter, so deep traces fit in the sidebar without truncating span names as early.The trace and span views also move to a calmer, more consistent color treatment: the selected trace and span read as one highlight color everywhere, span types are identified by their icons, and error, replay, and mock states keep their own distinct colors so problems still stand out.Trace across threads in the Python SDK
Functions dispatched to worker threads (ThreadPoolExecutor, loop.run_in_executor, or threading.Thread) previously recorded their spans as separate single-span traces, and replay mocking could not serve them. Construct the client with trace_across_threads=True (or set BITFAB_TRACE_ACROSS_THREADS=1) and those spans now nest under the trace that submitted the work, with mock="marked" replay serving their recorded outputs on the worker thread. Plain asyncio code needs no flag.See where every span ran
Every Python SDK span now records aruntime block in its raw data: the thread it executed on, the thread its parent span ran on, and, for work dispatched across threads, the thread that submitted it. Open any span’s raw view in the dashboard to inspect it.Replay warns when a mock cannot be served
During replay, a span markedmock_on_replay=True with nothing to serve now logs a warning saying it ran live, and whether that is because the replayed trace recorded no matching span or because all recorded occurrences were already consumed by earlier calls.Dashboard
Faster experiment history
Experiment history now stays responsive in organization, dataset, function, and multi-run views by loading trace details only when you expand an experiment. Single experiments and experiment groups still preload their traces, so focused drill-down remains immediate.More reliable Neon database snapshot setup
Connecting a Neon project for database snapshots no longer depends on a restore point from the last minute, so valid projects are not rejected when restore history is still catching up. The Integrations flow now validates the connection against the project’s current state and clearly tells you when point-in-time restore is disabled and a restore window needs to be enabled.Cancel database snapshot setup while it runs
Setting up per-trace database snapshots can take a while, and you can now stop it partway. The Database section of Integrations shows which step is running (checking your database, preparing branching, validating branches) instead of a single spinner, and offers Cancel setup, which stops the work and removes whatever had been built.Snapshot setup failures say what actually went wrong
A failed setup no longer always points at your connection string. Bitfab now tells apart a problem with your own database, a setup it could not finish on its side, and a database that is already connected elsewhere in Bitfab, and gives each its own advice. A status Bitfab could not read is reported as unverified and keeps re-checking itself instead of being shown as a failure.Disconnecting a database removes its snapshot copy
Deactivating snapshots now deletes the snapshot copy Bitfab built for that database, so nothing of yours keeps being replicated afterwards. The confirmation spells out what that means first: existing snapshots go with it, traces captured earlier will no longer replay against a snapshot, and connecting again rebuilds from scratch.Connecting a Neon project one step at a time
The Neon connection flow now asks one question at a time and collapses each answer into a line you can change, so the key, project, and database you picked stay visible as you go. Start over is always available, and leaving the page and coming back mid-setup no longer clears what you had entered.TypeScript SDKPython SDKRuby SDKPluginsCLIDashboard
TypeScript SDK v0.36.1, Python SDK v0.36.1, Ruby SDK v0.36.1, Plugins v0.9.47, CLI v0.2.253
Replay branches tell you what they are pinned to
The database branch handed to your code during a replay now carries the moment it was pinned to, so you can confirm a replay is really reading your data as it stood when the original trace ran. It also carries the branch’s own id and the name of the environment variable your app reads. Reading the connection string is still the only thing that counts as using the branch, so inspecting these costs nothing.Traces record the region a replay branch ran in
A replayed trace now records the region its database branch lived in, and the snapshot badge on the trace names it. When one replay looks slower than another, that tells you whether the difference is your code or a database in another region paying a round trip on every query.Ruby replay branches keep the database URL out of your logs
In the Ruby SDK, turning a replay branch into JSON no longer includes its database connection string or the internal replay state. Under Rails, serializing one of these objects into a log line or an API response previously carried the connection string with it. It now serializes only the descriptive fields, while code that asks for the connection string directly still gets it.Dashboard
Experiment results count every graded replay
Experiments now show a grader verdict on every replay the graders scored, including replays that hit an error partway through. Those rows used to show no result at all and their checks were left out of the run’s totals, so the pass rate could disagree with the traces listed under it. Grader progress now reads passed, failed, and grading, and in label mode the counts add up to the number of traces in the run.Labels show what is still waiting on you
A verdict written by an agent now stays marked as unreviewed until someone approves it, while keeping its pass or fail color, so you can scan a list and see which traces still need a look. Once approved it reads as settled while still showing that a machine wrote it.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLIDashboard
TypeScript SDK v0.36.0, Python SDK v0.36.0, Ruby SDK v0.36.0, Go SDK v0.36.0, Plugins v0.9.46, CLI v0.2.252
Spans capture much more of your data
Each span now records up to 2.8 MB of its inputs and outputs, up from 512 KB. A large document, a long message history, or an agent state that used to be replaced with a placeholder is captured in full, so traces stay complete enough to read and replay. This applies to the TypeScript, Python, Ruby, and Go SDKs.Large spans arrive instead of going missing
A span carrying more than the limit now ships with its largest fields replaced by placeholders, keeping its name, timing, and remaining fields intact. Previously a span that big could be discarded on the way out, leaving a hole in the trace with nothing to explain it. When a span is trimmed the SDK warns once in your logs and marks the trace as incomplete, so a partial capture is never mistaken for a faithful one.Dashboard
Steadier trace list when opening a trace
Opening a trace from the trace list no longer shifts the list sideways. The list used to slide left and drift back as the detail panel slid in, which made it easy to lose your place while working through a run of traces. The list now stays put while the panel opens, and the span you select still scrolls into view inside the panel.Dashboard
Long code change descriptions no longer cut off the window
Opening the code change behind an experiment whose description ran several lines used to push the bottom of the window off screen, with no way to scroll back to it, leaving the file diffs underneath unreachable. The window now stays on screen, with the description pinned at the top and the diffs scrolling beneath it.Confirm your database snapshot connection
Your coding agent can now tell whether database snapshots use a direct Neon project or a Bitfab-managed Postgres mirror. Theget_database_connection_status tool also reports the pinned Neon project name and ID for direct connections, making it easy to confirm which project replay uses.TypeScript SDKPython SDKRuby SDKGo SDKPluginsCLIDashboard
TypeScript SDK v0.35.0, Python SDK v0.35.0, Ruby SDK v0.35.0, Go SDK v0.35.0, Plugins v0.9.44, CLI v0.2.250
SDKs gzip the traces they send
Every SDK now compresses request bodies of 8,192 bytes or more with gzip, which is where most of a trace’s bandwidth goes. Smaller requests are unchanged, because compressing them would cost more than the bytes it saves. Nothing to configure, and no new dependencies: each SDK uses its language’s standard library.Compression is best-effort. If it ever fails, the SDK sends the original body rather than dropping the recorded call. SetBITFAB_DISABLE_COMPRESSION to turn it off, and note that browsers without CompressionStream fall back to uncompressed automatically.Shareable links to any trace
Opening a trace now puts it in the page address everywhere you can open one: the trace list, inside a dataset, and inside an experiment. Copy that link to send someone the exact trace you are looking at, and it opens straight to it. Back and Forward now step through the traces you opened instead of leaving the view.A clearer sidebar
The sidebar runs the full height of the window and lists Traces, Datasets, Experiments and Graders under whichever function you are working on, so you can move between them directly instead of through a menu. Each function shows how many traces it has. Collapsing it leaves a strip of icons rather than hiding it, so it is always one click back.Dashboard
Reliable trace delivery
SDK trace uploads now complete reliably during concurrent ingestion instead of timing out. This keeps traces flowing into the dashboard even when many spans for the same trace arrive together.Traces with self-referencing data are readable again
Opening a trace whose recorded input or output contained a self-referencing value used to fail. Because traces are read in batches, a single affected trace could also stop the others in the same request from loading. These traces now open normally, with the repeating value shown as a placeholder, and one unreadable trace no longer affects the others in the same request. This applies to traces you have already recorded, so nothing needs to be re-run.Ruby spans containing NaN or Infinity now send
The Ruby SDK now records non-finite numbers such asNaN and Infinity as a placeholder instead of failing to encode them. Previously a single one of these values could stop an entire batch of spans from reaching Bitfab, so unrelated spans were lost alongside it. This matches how the Python SDK already handles the same values.Dashboard
Sharper cutoff for snapshots from a replaced database connection
The date Bitfab uses to decide which traces predate a replaced database connection now comes from the connection that actually holds your snapshots, rather than the oldest database connection on your account. If you connect a second snapshot provider after running on a first one, traces captured before that switch now correctly show “Snapshot on old branch” instead of replaying against a project that never held their data.Dashboard
Grader reruns recover from interrupted jobs
Dataset and experiment grader reruns no longer stay blocked forever when an earlier run is interrupted. After a stale run expires, you can start the graders again instead of remaining stuck on an in-progress state.More reliable database snapshot replay
Database snapshot metadata now fills in reliably across large trace histories, avoiding stalls that could leave older traces incomplete for replay. This improves replay consistency for organizations with extensive trace histories.Re-run graders from your coding agent
Your coding agent can now re-score existing traces with thererun_graders_on_dataset and rerun_graders_on_experiment tools. Reach for them when traces were added to a dataset before a grader existed, or after you change a grader’s criteria and want everything scored again, since attaching a grader records the assignment without grading anything on its own. Both default to every grader attached to the dataset or experiment, and report how many traces were graded once the run finishes.Grading progress on dataset and experiment rows
Traces now show that grading is under way while a re-run is in flight, instead of holding the previous run’s scores until new results arrive. Each row’s pass-rate pill switches to a pending state when its graders start and settles on the new score when they finish.Dashboard
Open Studio resources across organizations
You can now open Studio datasets, experiments, traces, graders, functions, and trace plans from any organization you belong to without switching your active organization first. When a resource belongs to a different organization, Studio shows its organization in the header so you can see which workspace you are working in.Leaner plugin skill discovery
Bitfab’s Claude, Cursor, and Codex plugins now load leaner skill descriptions, leaving more of the agent’s startup context available while preserving natural-language routing. Command usage hints are generated from the same declared modes the flows execute, so setup, update, and assistant guidance stays aligned as those modes evolve.Setup reuses trace plans you already drafted
When you instrument your app with the Bitfab plugin, setup now picks up a trace plan you drafted earlier while analyzing your repository instead of rebuilding one from scratch. If a saved draft already covers the workflow you’re instrumenting, the plugin reuses it, so the analysis work carries straight into instrumentation.Discover existing trace plans from your coding agent
A newlist_trace_plans tool lets your coding agent see the trace plans already in your workspace, filtered by status or how they were created. This is what lets setup find and reuse an earlier draft rather than starting over.Snapshots from a replaced database connection
Traces whose database snapshot was captured against a connection you have since replaced now show “Snapshot on old branch” instead of “Snapshot captured”, and replaying one is refused with an explanation. Those replays previously ran against your current database without saying so, returning results that looked valid but restored the wrong data. The check applies whether Bitfab maintains the snapshot copy for you or you connect your own Neon project.TypeScript SDKPython SDKRuby SDKGo SDK
TypeScript SDK v0.34.2, Python SDK v0.34.0, Ruby SDK v0.34.0, Go SDK v0.34.0
Faster trace export in every SDK
Each carrier is now encoded exactly once and the request body is assembled from those encodings. Packing a batch used to re-encode the whole request for every span it considered, which re-escaped the captured inputs and outputs over and over. A full export window is now roughly 10x cheaper to prepare, which in Node.js is time given back to your event loop.OpenTelemetry Collector delivery has been removed
BITFAB_OTEL_EXPORTER_ENDPOINT no longer routes SDK traces through your own OpenTelemetry Collector; setting it has no effect and SDKs deliver to Bitfab directly. Removing it drops @opentelemetry/exporter-trace-otlp-proto from the TypeScript SDK, opentelemetry-exporter-otlp-proto-http (and protobuf) from the Python SDK, and gRPC, protobuf, and grpc-gateway from the Go SDK’s dependency graph entirely.If you route telemetry through a Collector today, point its otlphttp exporter at Bitfab instead. Bitfab’s ingress accepts standard OTLP/JSON, so a Collector can still forward to us:More reliable TypeScript trace delivery
TypeScript SDK traces now use OpenTelemetry’s batching and lifecycle machinery behind the existingwithSpan wrapper and framework integrations, so no instrumentation changes are required. Large captured inputs and outputs are packed beneath request-size limits, completed requests are delivered concurrently, and flushTraces() now reports whether delivery actually succeeded instead of only that the queue drained.Route TypeScript traces through an OpenTelemetry Collector
TypeScript applications can send Bitfab traces through an OpenTelemetry Collector by settingBITFAB_OTEL_EXPORTER_ENDPOINT to the Collector’s OTLP/HTTP base URL. Collector delivery replaces direct delivery, so enabling it does not duplicate traces. The protobuf exporter ships with the SDK and is code-split, so it loads only when you set an endpoint.Replay waits for every TypeScript span
TypeScript replay now verifies that each trace is complete and every expected span has persisted before finalizing the run. A successful queue flush or Collector acknowledgment can no longer make an incomplete replay appear ready.BAML call() traces ride the same pipeline
Traces from locally executed BAML functions now travel through the same batching transport as spans and trace completions, instead of posting one request each. Nothing changes in how you call call().Detached trace updates now confirm they applied
client.getTrace(id).addContext(), .setMetadata(), and .setSessionId() now wait for the server and reject if the update is refused. Previously they resolved regardless and a rejected update, such as one naming a trace ID that does not exist, was only written to the log. If you were ignoring the returned promise, await it.Close a client when you are done with it
client.close() flushes and releases the trace transport shared by a client’s wrapped functions and framework handlers, which matters for long-running processes that create short-lived clients.See OpenTelemetry Transport Architecture for the full design.More reliable Ruby trace delivery
Ruby SDK traces now use OpenTelemetry’s batching and lifecycle machinery behind the existingbitfab_span and Bitfab::Traceable APIs, so no instrumentation changes are required. Large captured inputs and outputs are packed beneath request-size limits, completed requests are delivered concurrently, and Bitfab.flush_traces now reports whether queued traces actually landed instead of only that the queue drained. A new client.close releases a transient client’s batch worker when a long-running process builds clients it later discards.Route Ruby traces through an OpenTelemetry Collector
Ruby applications can now send Bitfab traces through an OpenTelemetry Collector by adding theopentelemetry-exporter-otlp gem and setting BITFAB_OTEL_EXPORTER_ENDPOINT to the Collector’s OTLP/HTTP base URL. Collector delivery uses the official OpenTelemetry exporter and replaces direct delivery, so enabling it does not duplicate traces. If the gem is missing, the SDK warns once and keeps delivering directly rather than dropping spans.Bring your own OpenTelemetry version
The gem acceptsopentelemetry-sdk from 1.2 up to but not including 2.0, verified across that range, so an application already pinning its own OpenTelemetry version does not have to move it to upgrade Bitfab.Replay waits for every Ruby span
Ruby replay now verifies that each trace is complete and every expected span has persisted before finalizing the run, instead of waiting on per-item upload threads. A successful queue flush or Collector acknowledgment can no longer make an incomplete replay appear ready.Faster trace export in the Python SDK
Preparing a batch of spans to send is now about 2.5 times faster, continuing the encoding work in v0.33.8. Sending 64 spans spends 4.9ms on preparation instead of 12.8ms, and the saving grows with batch size, so the busiest traced functions benefit most. Nothing changes in how you instrument your code.Go SDK spans now ship over OpenTelemetry
The Go SDK now batches and delivers spans through the OpenTelemetry trace SDK instead of one HTTP request per span. Nothing changes in how you instrument:Span, Start/End, and GetFunction still produce Bitfab spans with Bitfab trace IDs, and no OpenTelemetry type appears in the Bitfab API. Under load this means far fewer, larger requests, and a bounded queue that will not pile up unbounded work behind a slow network.Each client lazily starts one background worker on its first span, so a client that never traces starts nothing. The SDK keeps its own private tracer provider and never touches your application’s global OpenTelemetry provider, so your traces and Bitfab’s stay separate.Close a client when you are done with it
Close flushes pending spans and shuts the worker down for good. FlushTraces now also reports whether delivery actually succeeded.Send Go spans through your own collector
If you already run an OpenTelemetry collector, setBITFAB_OTEL_EXPORTER_ENDPOINT to it (for example http://localhost:4318) and Bitfab spans flow through your existing telemetry pipeline instead of going straight to us. Configure that collector to forward to Bitfab. Leave the variable unset and the SDK talks to Bitfab directly, which is the default.Go 1.25 is now required
The OpenTelemetry Go modules require Go 1.25, so the Bitfab Go SDK does too. The official OTLP/HTTP exporter is linked into every build, because Go has no dynamic import to defer it behind the collector opt-in.One note on v0.12.7’s encoding change: spans are still encoded only once, but that encode now happens on the thread that ran your traced function rather than in a per-span background sender, because queueing a span no longer starts a goroutine. Queueing is non-blocking, so the delivery itself never sits on your thread.See OpenTelemetry Transport Architecture for the full design.Lower tracing overhead in the Python and Go SDKs
Both SDKs were encoding each span twice on its way out: once to check the data could be serialized, then again to build the request that ships it. They now reuse the first encode, which roughly halves the encoding work a span costs in Python and cuts it by about a third in Go. The saving scales with how much your spans capture, so the largest inputs and outputs benefit most.The Go SDK also moves the span’s final encode into its background sender, so less work happens on the thread that ran your traced function.Plugin updates work wherever the plugin is installed
Claude Code can install the Bitfab plugin for a single project instead of your whole account, and updating a project-scoped install now works. Previouslynpx bitfab-cli init and /bitfab:update assumed the account-wide install, so they failed with a “not installed at scope user” error and quietly left you on the old version. Both now update the install that belongs to the project you are in, and if you have the plugin installed in more than one place, a failure on one of them is reported instead of being hidden behind a success message.Dashboard
Database connection changes now take effect
Updating the connection string for database snapshots now applies to the connection you already have, instead of being saved without changing anything. Rotating a password or username updates in place, with no interruption to snapshots and no loss of history.Confirmation before replacing a connected database
Pointing snapshots at a different database now asks you to confirm first. The existing snapshot connection cannot be moved across, so it has to be torn down and rebuilt, and snapshots captured before the switch are lost. Bitfab checks the new database is set up for replication before anything is torn down, so a database it cannot mirror is refused while your existing connection stays untouched.More accurate snapshot status
The database snapshot status now reflects the live state of your connection. A temporary outage on our side no longer shows a working connection as failed, and a connection that stalled partway through setup now reports a failure you can act on instead of showing as still activating.Diff view opens on the first real change
The experiments Diff view now lands directly on the first span whose input or output actually differs from the original run, instead of opening on the trace root. Spans with no differences (and the trace root itself) are dimmed and unselectable, and when a replay matches the original everywhere, the view says “No diff available” instead of showing an empty comparison.Detached trace updates now confirm they applied
In the Python SDK,client.get_trace(id).add_context(), .set_metadata(), and .set_session_id() now wait for the server and raise if the update is rejected. Previously they returned a threading.Thread you could join, and a rejected update, such as one naming a trace ID that does not exist, was only written to the log. If you were joining the returned thread you can drop that call, because the update has already been applied by the time the method returns.Dashboard
Compressed trace ingestion
Bitfab’s ingestion endpoints now accept gzip-compressed request bodies. If you send traces through an OpenTelemetry Collector, its default compression now works as-is, so there is no need to turn compression off on the exporter. Requests compressed with deflate work too, and the accepted encodings are listed in the HTTP endpoints reference.A connected Neon database now counts everywhere
If you connect a Neon project directly, Bitfab treats it as your database connection across the product: traces show the “Snapshot captured” badge, and the connection status shown on the Integrations page and reported to your coding agent reflects the Neon project you pinned. Those signals previously only recognized a connection made through the Database section, so a Neon-connected workspace could look unconnected while replay was already branching its database.A brief Neon outage no longer looks like a broken connection
When Neon cannot be reached for a moment, your connection is now marked unverified and re-checked on its own instead of being reported as broken. You are only asked to reconnect when the API key or the project has actually changed.Setup teaches the primitives before it touches your code
Setup now opens by explaining the two primitives you instrument with,withSpan and replay, including the five ways replay can change a method when you re-run a captured trace: run it normally, feed it the recorded inputs, feed it modified inputs, or skip it and return its recorded or modified outputs. Instrumenting asks you to make that choice per method, so the explanation now comes first, right after you sign in and before any part of your repository is read. Running /bitfab:setup explain on its own shows the same thing without signing in.Choose guided setup, or instrument it yourself
After signing in, setup asks whether you want to be walked through instrumenting or would rather do it yourself. Guided is the default and drives the whole thing, checking with you at each decision. Choosing to do it yourself hands over the SDK guide for your language and where to fetch your API key, then stops without scanning your codebase or changing a line.Know what setup will do before it starts
Once you pick the guided path, setup says what is about to happen before it reads anything: it analyzes your repository, instruments the AI features you choose, and writes a replay script. It also notes that it will prompt you whenever it needs input, and that the whole thing usually takes 10 to 17 minutes depending on how many features you instrument.Close or update a trace plan and keep going
The trace plan page in Studio now has a single button. It reads Close when you have not changed anything, and Update once you have toggled which spans get captured or how they replay. Either one saves the plan and hands straight back to your coding agent, which carries on writing the instrumentation. Closing the window does the same thing, so opening a plan to look at it is never a dead end.Skip the browser without losing your edits
While the plan is open, your agent asks in the terminal whether to continue instrumenting, so you can skip the review entirely and it will close the plan page for you. Answer it whenever you like: a capture set you saved in Studio always takes precedence over that prompt, so the spans you chose are the spans that get traced.Dashboard
Connect a Neon database directly
If your production database runs on Neon, you can now connect it directly from the Integrations page instead of provisioning a replicated copy. Bitfab branches your own project at each trace’s timestamp, so replay reads the database as it was when the trace was captured. Paste a Neon API key and Bitfab handles the rest, asking which project or database only when there is more than one to choose from.Before anything is saved, the key is checked against the real workflow: Bitfab creates a branch at a past point in time, reads its connection string, and deletes it again. A key that cannot do all three is rejected with the step that failed, so a broken connection surfaces during setup instead of during your first replay.DashboardPluginsTypeScript SDKPython SDKCLIVS Code
Plugins v0.9.28, TypeScript SDK v0.33.7, Python SDK v0.33.6, CLI v0.2.234
More reliable Python trace delivery
Python SDK traces now use OpenTelemetry’s batching and lifecycle machinery behind the existing Bitfab decorators and framework integrations, so no instrumentation changes are required. Large captured inputs and outputs are packed beneath request-size limits, completed requests can be delivered concurrently, and shutdown reports when queued traces could not finish instead of silently claiming success.Route Python traces through an OpenTelemetry Collector
Python applications can now send Bitfab traces through an OpenTelemetry Collector by settingBITFAB_OTEL_EXPORTER_ENDPOINT to the Collector’s OTLP/HTTP base URL. Collector delivery uses the official OpenTelemetry exporter and replaces direct delivery, so enabling it does not duplicate traces.Replay waits for every Python span
Python replay now verifies that each trace is complete and every expected span has persisted before finalizing the run. A successful queue flush or Collector acknowledgment can no longer make an incomplete replay appear ready.Dashboard
Clearer errors when replay warm-up SQL fails
When a replay provisions a database branch and your warm-up SQL fails, the error now names the actual failure instead of reporting it as a four-minute timeout. A statement the database rejects, such as one referencing a table or index that is not in the snapshot, comes back immediately asbranch_warmup_invalid, while a warm-up that genuinely runs out of time still reports as branch_warmup_failed.Faster failure when a database branch cannot start
A replay database branch that never accepts a connection now fails in about ten seconds instead of holding the request for up to four minutes.Dashboard
Collapse the labeling panel on any trace
The labeling panel can now be collapsed on every trace you open, including the traces list and dataset review, where it was previously pinned open and always held a column of the drawer. Collapsing hands that space back to the trace itself, and your choice carries as you step between traces and reopen the drawer. The toggle sits in the trace header beside the previous and next controls, and names what it opens: Graders on grader-scored traces, Label everywhere else.Two ways into every empty rail section
A dataset’s Graders and Experiments sections now stay on the page even when they’re empty, and each teaches both ways to fill it: ask your coding agent (a card with a copyable prompt and the MCP endpoints it uses) or do it yourself with the Manage modal. Editing a dataset moved into an Edit popover in the header, and Manage and Re-run live in a compact menu on the Graders section.Bulk approval back in the grader rail
When your coding agent has labeled traces that await your review, the Graders section shows “Approve all pending” with a live count, opening the same confirmation flow as before. This also fixes the Re-run action in the section menu, which previously closed without opening the grader picker.Studio keeps experiments in reach
Finishing a labeling session in Studio brings back the list of experiments run against that dataset, so the natural next step after labeling stays one click away.Clearing a filtered experiments view
Opening experiments scoped to a dataset or a single run now shows that scope as a pill in the list header, with an x to clear it. Clearing drops you back to every run for the function, so leaving a filtered view no longer means editing the URL or navigating away.Dashboard
Settings pages scroll again
Content that ran past the bottom of the window on the Integrations, API Keys, and Export Traces pages was being cut off with no way to scroll down to it. These pages scroll normally now, so the connection string form on Integrations and the full list on API Keys stay reachable whatever the height of your window.Dashboard
The MCP endpoints behind graders, datasets, and experiments
Each of these pages now lists the MCP endpoints your coding agent calls to read and manage that resource, so the surface is visible from the page itself instead of living only in your agent’s tool list. The list sits under the empty state on a function with nothing to show yet, and the “New” button on each of these pages carries the same list in a section you can expand, next to a copyable prompt scoped to the function you’re on.Graders explain how they get made
A trace function with no graders now shows what a grader is and a prompt you can copy into your coding agent to create one, matching what datasets and experiments already did. A grader’s own page adds an “Edit” button that hands you a prompt naming that grader, for when you want to tighten its criteria or rename it.Dashboard
Experiments list, grouped and one line per run
The experiments list now gives each run a single line, gathered under the experiment group that produced it, so pass rates, before and after deltas, and code changes line up in columns you can read straight down. A “Group by” control in the header switches between grouping by experiment group and a flat list, and your choice travels in the URL, so a link you share opens on the same view you were looking at.Opening a run is quicker as well. Its traces show a placeholder the moment you click, then fill in, and they stay put when you collapse the run, so reopening it is immediate.Dashboard
Framework-observed spans can’t be mocked in replay plans
Replay plans no longer let you mock spans that a framework’s instrumentation only observes, such as LangChain or LangGraph calls. Because the SDK can’t intercept these spans, a mock would silently have no effect and the call would re-run anyway, so the trace plan now locks them with a clear explanation instead of showing a warning you could override. Spans that genuinely support mocking, including Vercel AI SDK model calls, are unaffected.Turn on replay database branching with a boolean
Replaying against a database branch restored to each trace’s original state now reads as the switch it is. PassdbBranch: true in TypeScript, db_branch=True in Python, or db_branch: true in Ruby, and every replay item runs against a branch sized by your mirror project’s own defaults. Passing an options object still works and is now only for tuning the branch’s compute and warm-up, and false turns branching off explicitly.Trace helpers only when nested
Reusable helper functions can now appear as child spans inside an existing Bitfab trace without creating standalone root traces when called on their own. SetcaptureWhen: "nested" in TypeScript, capture_when="nested" in Python or Ruby, or WithCaptureWhen(CaptureWhenNested) in Go. Your helper still runs normally without a parent span, and unknown values warn once before falling back to the existing always-capture behavior.Control replay code-change capture
Replay scripts now distinguish omitted code-change files from an explicit null value, so you can keep automatic git-diff capture by default or turn it off for a specific run. Supplying only a custom code-change description preserves that description while Bitfab still captures the changed files automatically. PasscodeChangeFiles: null in TypeScript, code_change_files=None in Python, or code_change_files: nil in Ruby to opt out.Dashboard
Clearer experiment completion and grader progress
Completed experiment replays without labels now appear as unlabeled and count as complete, instead of carrying a separate awaiting-label status. While graders are still running, experiment and trace progress bars now separate finished pass/fail results from pending grader checks in purple, so replay completion and grader completion remain distinct.The usage page opens right away
The usage page now renders as soon as you open it, instead of waiting on a scan of your whole trace history before showing anything. For organizations with a lot of history that wait ran past ten seconds, long enough that the Usage link looked like it did nothing at all. The heading, controls, and layout appear immediately now, and the numbers fill in as they arrive.Loading placeholders instead of misleading empty states
While usage data is still loading, the stat cards, chart, and period table now show placeholders rather than dashes and a spinner. The period table also no longer says “No usage in this period yet” before its data has arrived, and the CSV download stays disabled until there is something to export.Consistent names across every MCP tool
Bitfab’s MCP tools now follow one naming rule:list_ for collections, get_ when you already have the id, create_ for things that are new every time, save_ for create-or-update, and add_/remove_ for attaching and detaching. Ten tools were renamed under it, including read_traces to get_traces and update_agent_labels to save_agent_labels. Nothing breaks if you are on an older plugin version: the previous names are still accepted, so your coding agent keeps working until you update.Datasets stop duplicating when a flow re-runs
save_dataset (previously create_dataset) now updates the existing dataset when you save one whose name already exists on the same traced function, instead of adding a second copy. Re-running a labeling flow, or picking one back up after your coding agent loses context, no longer leaves duplicates behind to clean up. The same name under a different traced function still creates its own dataset.Grader changes across a whole experiment group
add_graders_to_experiment_group now takes several graders in one call, and the new remove_graders_from_experiment_group detaches them again from every experiment currently in the group. A grader archived after it was assigned can still be removed, and ids that were never assigned are reported back instead of failing the call.Replay concurrency now bounds database branches
Replay now creates a historical database branch only when an item enters a worker, somaxConcurrency and max_concurrency limit both active work and live branches. Larger runs avoid provisioning every branch at startup, and older SDKs remain compatible through the eager server path. If a branch cannot be created, the item reports the resolver’s specific reason instead of silently running without historical data.Warm-up SQL gets 240 seconds
Warm-up SQL passed with a replay’sdbBranch settings now gets 240 seconds to run, rather than sharing the 10 seconds the branch readiness check allows. Warming a real working set takes far longer than proving a branch answers a query, so warm-ups that used to fail the lease now finish. If warm-up SQL does fail or outrun its budget, the error says so specifically instead of reporting the branch as never having come up.Starting a replay with database branching on now also waits up to 300 seconds rather than 180. A warm-up longer than about three minutes needs an SDK carrying that longer wait, so upgrade the TypeScript, Python, or Ruby SDK before relying on one.Dashboard
Keyboard navigation in the trace list
Arrow keys now move through traces one row at a time instead of jumping to the end of the list. Up and down (orj and k) walk the list, left and right (or h and l) step to the previous or next trace, and Enter opens the one under the cursor. With a trace open, up and down move between its spans while left and right keep moving between traces.Save & Next keeps you moving through a dataset
Saving a label from a dataset’s trace panel now opens the next trace in the list instead of closing the panel. This previously only worked for traces waiting on your approval, so labeled and unlabeled traces dropped you back to the list and made you pick the next one by hand.Database-snapshot setup wires the new replay branch
/bitfab:setup db-snapshot in Claude Code, Cursor, and Codex now wires replay using the dbBranch option and the replay-branch accessor introduced in SDK v0.33.0, in place of the removed ReplayEnvironment.Breaking: ReplayEnvironment is replaced by a replay-branch accessor
Database branching for replay is now turned on by the dbBranch replay option, and the resolved branch is read through an accessor rather than an object you construct and pass in. ReplayEnvironment is removed.The old shape asked two unrelated questions with one signal: constructing an environment object was both “give me a database branch” and “here is the thing I will read inside my function”. Since the branch was always resolved through the replay context, never from the object you passed, two different instances behaved identically and the object could not actually be used for dependency injection. Making dbBranch’s presence the switch and the branch an accessor states what was already true.Before:db_branch={"min_cu": 2} with get_current_replay_branch()) and Ruby (db_branch: {min_cu: 2} with Bitfab.current_replay_branch). Passing dbBranch at all is what enables branching, so dbBranch: {} turns it on with your mirror’s own sizing. The accessor returns null / None / nil outside a replay item and for items with no branch, which replaces the old active flag: gate on it and fall back to your normal connection string. Branch sizing and warm-up, and every size the server accepts, are unchanged.Breaking: database branch settings move onto ReplayEnvironment
The branch sizing and warm-up settings introduced in v0.31.0 are now constructor arguments on ReplayEnvironment, not a separate replay option. As a sibling of environment they could be passed on their own, where they were silently discarded: the settings only ever applied to a replay that had an environment. On the constructor that state cannot be expressed at all.Before:ReplayEnvironment(min_cu=2, max_cu=2, warmup_sql="SELECT 1;")) and Ruby (Bitfab::ReplayEnvironment.new(min_cu: 2, max_cu: 2, warmup_sql: "SELECT 1;")). The dbBranch / db_branch replay option is removed rather than deprecated. All three fields stay optional, so an environment you construct with no arguments keeps working and leaves the mirror’s own defaults in place. Everything the settings do, and the sizes the server accepts, are unchanged.Superseded by v0.33.0 above:
ReplayEnvironment is gone, and these settings are back on the dbBranch replay option.Clear replay outcomes when local capture is unavailable
When a replay command finishes successfully but the plugin cannot read its local result, Claude Code, Cursor, and Codex now report the run as unverified instead of showing a misleading background-command failure. The assistant checks the server for the actual experiment outcome before evaluating results, while genuine replay-script crashes still appear as failures.Quoted numbers no longer break read_span_field over MCP
Some MCP clients send numeric arguments as strings. When a client connected directly tohttps://bitfab.ai/mcp called read_span_field with a quoted maxChars (for example "200000"), the request failed with an invalid-arguments error, even though the identical call succeeded through the Bitfab editor plugins. The endpoint now accepts either form, so reading a full span field works the same way from any client.Size and warm the database branch your replay runs against
When you replay traces against a historical database branch, that branch used to start at whatever compute the snapshot mirror happened to be provisioned with, and it served its first query from a cold cache. Both made replay latency reflect the branch instead of your code. PassdbBranch to size the compute and warm it before your function runs:minCu equal to maxCu pins the size, so every item in a run is measured against identical compute rather than one that warms up as the run proceeds. warmupSql runs inside the branch’s readiness check, before your function sees the lease, so warm-up time is never charged to the replayed call. The same options are available as db_branch in the Python and Ruby SDKs.Replay tells you when it could not reach a database branch
When a database branch could not be provisioned, replay used to run your function against your live database and hand back a result that looked normal. It now fails that item and reports why, so a run that could not use the historical data you asked for never reports a passing result. Traces captured before snapshot support still replay against your live database as before, since no branch was requested for those.See which grader failed a trace, and why
Your coding agent can now read the individual verdict each automated grader recorded on a trace, instead of only the overall pass or fail. In Claude Code, Cursor, or Codex,read_grader_labels returns each grader’s name, its verdict, the reasoning behind it, and its failure diagnostic, so you can ask which specific check a trace failed. Query it by trace to see every grader’s verdict on those traces, or by grader to review that grader’s most recent verdicts across traces when you are working out why a grader keeps misfiring.Name experiment groups from your coding agent
You can now give experiment groups a name and rename them later, making related runs easier to identify across replay and grading workflows. In Claude Code, Cursor, or Codex,save_experiment_group creates a group from existing experiments or updates an existing group name without changing its membership.Browse workflows across every trace function
All Traces now includes Datasets, Experiments, and Graders in the same workflow dropdown used inside each trace function. The organization-wide dataset and experiment views let you browse and open work across functions from one place, with function context on dataset rows, pagination for larger histories, and live experiment progress and results.Read a single experiment’s grader results from your coding agent
The Bitfab MCP tools now includeget_experiment, which returns one experiment (replay test run) by id: its status, pass/fail totals, replay-versus-original delta, experiment group, and grader results, including a per-grader passing and failing breakdown. Use it when you already have an experiment id and want just that run’s results, instead of listing a function’s whole experiment history. The list_experiments results now also include each run’s experiment group id.Dashboard
Consistent MCP tools across local and remote connections
Coding agents now receive the same 32 Bitfab tools, descriptions, validation schemas, and metadata whether they connect through the local plugin or the remote MCP server. Trace searches now validate real calendar dates and integer limits consistently, and trace-plan creation derives initial replay mocking decisions server-side so agents cannot accidentally override them before confirmation.Attach the first grader from an experiment
You can now add or remove graders from an experiment even when none are currently attached, as long as its trace function has graders available. The Graders section remains visible in this state, so you can open Manage without needing an existing assignment first.Choose the model that judges your graders
You can now pick which model runs each LLM-as-judge grader instead of every grader using the same one. On a grader’s detail page, the new Model dropdown lets you choose Gemini 2.5 Flash (the default), GPT-5.6 Sol, GPT-5.6 Terra, Claude Opus 4.8, or Gemini 3.1 Pro, and the choice takes effect the next time that grader evaluates a trace. You can also set it from your coding agent: thesave_grader plugin tool now takes a model option, so a request like “have this grader use Opus 4.8” sticks. Graders you do not change keep running on the default.Dashboard
Arrow keys move between spans inside an open trace
With a trace open on the experiments, dataset, or traces pages, up and down (or j and k) now step through that trace’s spans instead of jumping you to a different trace. Left and right (or h and l) still move between traces, and when no trace is open, up and down keep moving the cursor through the list.Incremental summaries for resumed coding sessions
Coding sessions resumed under the same session now produce a summary of only the new work instead of repeating earlier turns. Sessions that never include both a user request and an assistant response no longer generate a summary, reducing noise from open-and-close sessions.Dashboard
Grader results in the experiment comparison view
Opening a trace in a grader-scored experiment now shows its grader pass and fail results in the before-and-after comparison, matching what the trace list already shows, instead of an “Awaiting label” placeholder. You see the grader pass rate move from the original run to the replay, or a live grading indicator while scoring is still in progress.Filter traces by original vs. replay
The traces list now has an Original / Replays / All filter so you can separate live production traces from replay traces generated by experiments and test runs. It defaults to Original, so you see real production activity first, and you can switch to Replays or All whenever you need to inspect experiment re-runs.Dashboard
Complete grader set on experiment panels
The graders panel on an experiment now lists every grader scoring that run, including the ones inherited from its dataset, so it matches the pass and fail results shown on each trace. Graders that come from the dataset are marked with a small icon you can hover to confirm, and re-runs stay limited to the experiment’s own graders.Dashboard
Reliable grader reruns
You can now see live progress when re-running graders on a dataset or experiment, and that progress resumes if you refresh or return while the run is still active. Starting the same rerun again reconnects to the existing work instead of creating a duplicate, while a different selection is blocked until the current run finishes. Completion updates recover even if a live update is missed, so results do not stay stuck in a running state.Live grading progress on trace rows
When graders re-score a dataset or experiment, each trace row now shows its grading as it happens instead of going blank. A loading ring appears the moment scoring starts and the pass rate fills in live as each grader reports, so you can watch results arrive. Finished rows keep their green or red verdict, with the pill’s border showing the pass/fail split.Dashboard
Per-grader labeling on the trace list
When you open a trace that has graders from the Traces page, the labeling panel now shows each grader’s verdict so you can review and label them one by one, the same per-grader view you get inside a dataset. Traces without graders keep the standard pass/fail labeling panel.Dashboard
Related experiments on the dataset page
Each dataset now shows the experiments run against it, right in the dataset view. See each run’s pass rate at a glance and jump straight to a single experiment, or open the full list filtered to that dataset, without leaving your workflow.Dashboard
Graders now have their own section
Every trace function has a Graders section alongside Traces, Datasets, and Experiments. It lists your graders with their pass rate and the number of datasets and experiments each one is attached to, and you can sort by status, pass rate, evaluation volume, or recency, and archive or unarchive a grader without leaving the list.Grader overview with dataset and experiment footprint
Opening a grader now shows an overview of what it does and where it runs: its overall pass rate, the criteria it evaluates, and every dataset and experiment it is attached to with that grader’s pass rate on each. Pass rates combine the grader’s automated results with your own human labels.Automatic code-change capture on replay
When you replay traces after editing your code, the SDK now attaches the diff to the resulting experiment automatically, so you can see exactly what changed alongside the results with no extra arguments. If you don’t passcodeChangeFiles to replay(), it captures your working-tree changes against your trunk and includes them; passing an explicit code change still takes precedence when you want a precise per-edit before/after.Available in the TypeScript, Python, and Ruby SDKs. Opt out with BITFAB_DISABLE_CODE_CHANGE_CAPTURE.TypeScript SDKPython SDKRuby SDKDashboard
TypeScript SDK v0.30.0, Python SDK v0.30.0, Ruby SDK v0.30.0
Attach graders to a single replay
replay() now takes a graderIds / grader_ids option so you can grade one experiment with specific graders without permanently adding them to the dataset. The run is graded by the union of the graders you pass and the dataset’s own graders, so a one-off check runs alongside your standard ones.grader_ids), and Ruby (grader_ids) SDKs.Dashboard
Accurate experiment pass rates
Completed experiments without assigned graders now show a pass percentage from their trace labels instead of--%, so the summary reflects the results already available below it. Experiment rows use grader results when graders are assigned and label results otherwise, while runs with labels still outstanding remain pending.Honest experiment comparisons
Experiment trace rows now omit the before-to-after grader badge when the original trace has no compatible grader results. The current grader score remains visible, so you can see the replay outcome without a misleading missing baseline.Graders on the experiment page
Experiments now show the graders attached to each run right beside the traces, with a per-grader pass/fail breakdown for the run. Open a grader to read its criteria, use Manage to attach or detach graders, and Re-run to re-grade the experiment’s traces. The pass and fail bars update live as grading runs.Group and grade experiments from your coding agent
You can now organize existing experiment runs into a shared group and attach a grader across every current experiment using the newcreate_experiment_group and add_grader_to_experiment_group MCP tools. Completed experiments queue only missing grader results immediately, while pending experiments use the grader when they finish. Grader assignments stay on the experiments already in the group, so experiments added later do not inherit them automatically.Dashboard
Replays for up to 5,000 recent traces
Replay runs can now select up to 5,000 recent traces at once, up from 100, so larger evaluations no longer need to be split into manual batches. Starting a replay now avoids loading every child span up front, while dataset-backed replays continue to use the dataset’s full trace list regardless of the recent-trace limit.Dashboard
Guidance for creating datasets and experiments
Datasets and experiments are created by asking your coding agent, and the dashboard now shows you how. When a datasets or experiments page has none yet, it explains what the primitive is and gives you a ready-to-copy prompt to paste into your coding agent. The “New” button on those pages opens the same guidance with an example, so you are never left on a blank screen wondering how to start.Paginated grader lists in coding agents
list_graders now returns manageable pages for functions with large grader collections, with name search and a cursor for fetching the next page. It returns active graders by default, can include archived definitions when requested, and leaves grader-training pipeline entries out of the results.Assistant runs your plan without pausing to ask
The Bitfab assistant no longer stops after each experiment to ask whether to keep going or to revert a fix. It now reports the results and automatically continues through the experiments you already approved, wrapping up on its own once the plan is done. A multi-experiment run finishes in one pass instead of prompting between rounds.Attach graders to experiments from your coding agent
You can now attach graders directly to an experiment so they run against its replay traces, using two new Bitfab MCP tools,add_graders_to_experiment and remove_graders_from_experiment. Your coding agent can manage an experiment’s grading scope without leaving its workflow, the same way it already manages dataset graders. At completion an experiment is scored against the union of these direct attachments and its dataset’s current graders.Dashboard
Experiments grouped by run
Your experiments page now groups a trace function’s runs into groups, so each iteration reads as one unit rather than a flat list of runs. Groups are ordered by their most recent run, each header links back to the dataset it was run against, and you can collapse a group to focus. Runs launched without a group are gathered under an “Ungrouped” heading.Dashboard
Live dataset grader pass rates
Dataset grader pass-rate pills now stay current as new grader results arrive or grader assignments change. The datasets list refreshes automatically, so you can monitor evaluation progress without reloading the page.Dashboard
Experiment labeling shows only the run’s graders
When you open a trace inside an experiment, the labeling panel now shows only the graders that experiment was scored against: its dataset graders plus any graders attached to the run, rather than every grader defined for the function. Experiments with no graders open straight to the pass/fail labeling panel.Grader definitions after saves
After you create or update an automated grader through Bitfab MCP tools, your coding agent now presents the full saved definition instead of a generic success message. The response includes the grader function, status, evaluation focus, and any passing or failing criteria so you can immediately verify what will be evaluated.Dashboard
Reliable Studio links
Links that open Studio now preserve all handoff parameters, including repeated values, across sign-in, datasets, experiments, trace plans, and template previews. Dataset-linked experiments also switch to the dataset organization automatically, so shared links open in the correct context.Dashboard
Manage graders from a dataset
You can now attach and remove graders directly from a dataset. Open Manage in the Graders panel on a dataset page to move graders between Available and Attached, with search and sorting to find the right one. Attached graders score every trace in the dataset, and changes take effect immediately.Dashboard
Experiment graders run across every replay
You can now attach graders directly to an experiment, and Bitfab runs them together with the dataset’s assigned graders across every trace in the completed replay. The finalized grader set stays with the experiment, so late-arriving replay traces are evaluated consistently and completed experiment results retain the exact grader coverage that ran.Dashboard
Framework labels on trace plans
Trace plan headers now show the instrumentation framework Bitfab detected for your workflow, next to the function name and language. Plans that span multiple frameworks list each one, so you can see at a glance how a workflow is instrumented.Dashboard
Consistent labels across trace lists
Workflow, dataset, and experiment trace lists now use the same current label state, so pass, fail, and skipped results stay consistent across views. List loading also avoids fetching unused trace metadata, making these pages more efficient without changing how traces are managed.Dashboard
Keep your place when switching functions
Switching trace functions from the sidebar now keeps you in the current section, such as Traces, Datasets, Graders, or Experiments, instead of sending you back to Traces. When you switch from a specific trace, dataset, or grader, Bitfab opens the matching section list for the new function so resource IDs are not carried across functions.Dashboard
Live grader results across review views
Dataset reviews, experiment cards, experiment trace rows, and trace lists now update immediately as grader results and dataset grader assignments change. Pass rates and per-grader verdicts stay current during grading and review without a manual refresh.Dashboard
More reliable dataset grader labeling
Dataset grader labels now stay consistent when automated evaluations and human reviews overlap, so one source no longer overwrites the other. Newly added completed traces are automatically graded by their assigned dataset graders, and the labeling panel preserves in-progress selections when a save or refresh fails.Dashboard
Grader pass rates on trace lists
When a dataset has graders assigned, trace rows now show how many graders passed as a pass-rate bar (for example, 3 of 4) instead of a single Pass or Fail. It appears on the dataset review page, on experiment rows, and in the experiment header, so you can see at a glance how each trace and each run scored across all of its graders. The main traces list shows the same pass rate for grader-evaluated traces.Dashboard
Run graders without tuning
Active graders now evaluate traces directly from their saved criteria, so dataset grader runs work immediately without tuning. If a tuned prompt exists, Bitfab still uses it; empty or stale prompts fall back to the grader criteria.Dashboard
Live dataset review updates
Dataset pages, trace lists, and labeling panels now stay in sync as graders are assigned or removed and labels are added. Reviewers see the current dataset grader order immediately, while removed graders disappear from active labeling workflows without deleting their historical labels.Dashboard
Label traces while reviewing experiments
You can now label traces without leaving the experiments view. Open the before/after comparison for any trace and use the new Label toggle, next to the Diff / Original / Replayed switch, to score it. When the dataset has graders, the panel shows each grader for per-grader approval or override; otherwise it is a simple pass/fail verdict with notes.Dashboard
Smoother dataset grader reruns
Dataset grader reruns now keep traces in a grading state through transient evaluation failures, so temporary issues no longer appear as permanent errors. If every retry fails, the trace still moves to an error state instead of remaining stuck in grading.Dashboard
Compare individual spans in the Diff view
When you compare a replayed trace against its original, the experiment Diff view now lets you open any span that changed, not just the whole trace. The span tree greys out spans whose input and output stayed the same and keeps the changed ones selectable, so you can jump straight to what your code change actually affected.Clearer replay diffs
The Diff view now shows the input side fully expanded, so you can read exactly what the model received, and hovering a highlighted line tells you whether the replayed run excluded or included it.Dashboard
Trace details open in a side panel
Clicking a trace on the traces page now opens a slide-in detail panel next to the list, matching how dataset and experiment review work, instead of navigating to a separate page. You can label the trace Pass, Fail, or Skip right from the panel and move through traces with Save & Next. Links to individual traces still work: sharing or reloading a trace URL opens the same list with the panel already open, and your active filters stay in the URL.Redesigned trace list rows
Trace rows across the traces, dataset, and experiment lists now share one design that separates how the run went from how it was judged. A leading icon shows the run state (running, completed, errored, or a replay), while the verdict pill shows Pass, Fail, Skip, or an agent suggestion awaiting review, with a robot glyph marking machine verdicts. Replayed traces are tinted indigo so re-runs stand out, unlabeled rows show an input and output preview, and reviewer notes appear inline on labeled rows.Re-run graders on a dataset
When a dataset has graders attached, you can now re-run them across every trace in the dataset directly from the dataset page. Open the Graders panel, choose which graders to run, and follow the run from running to finished. You can also click any grader to view its evaluation criteria and prompt.Dashboard
Label datasets grader-by-grader
When a dataset has graders assigned, reviewing a trace now shows a labeling panel with one row per grader instead of a single pass/fail. Approve an automated grader’s suggested verdict in one click or override it, add a note, and move through the dataset trace by trace. Human labels always take precedence over the automated suggestions.Dashboard
Reliable experiment history pagination
Experiment histories now load every run when several experiments start at nearly the same time. Infinite scrolling no longer skips or repeats experiments that share the same timestamp.Dashboard
Scroll through complete experiment histories
Experiment histories now keep loading as you scroll, so older runs remain available instead of stopping after the newest 50. This works across function, dataset, and experiment-group views. The loading footer stays visible while more runs remain, with a spinner while the next page arrives.Dashboard
Dataset graders now grade automatically
The graders you assign to a dataset now run automatically on that dataset’s traces, and on the replayed traces when you run an experiment against the dataset. You get grades on your dataset and experiment results without kicking off anything by hand.Create and edit graders from your coding assistant
Claude Code, Cursor, and Codex can now create and edit automated graders for a traced function with the newsave_grader and list_graders tools. Ask for a check like “the reply never invents order numbers” and your coding agent defines the grader, then renames, updates, archives, or restores it on request without leaving your workflow. Graders are saved as definitions for now; nothing runs them against new traces automatically yet.Dashboard
Diff view for replay comparisons
When you replay a trace to test a code change, the trace comparison drawer now opens on a new Diff view that shows the replayed run against the original side by side, input on the left and output on the right, with changed lines highlighted. You can tell whether the change helped without flipping between the Original and Replayed panes and holding both in your head. Use the Diff, Original, and Replayed toggle at the top of the drawer to switch views; your choice sticks as you step through the run.Assign graders to datasets from your coding assistant
Claude Code, Cursor, and Codex can now assign graders to datasets withadd_graders_to_dataset and remove_graders_from_dataset. list_datasets now includes assigned graders, so you can inspect evaluation coverage and update it without leaving your coding workflow.Dashboard
See experiment annotations in trace comparisons
Experiment trace comparisons now show the replay annotation beside the pass/fail transition, so you can see the labeler’s reasoning without returning to the trace list. Hover over a truncated annotation to read its full text in a tooltip.Keyboard navigation in dataset review and experiments
You can now move through traces and spans with the keyboard while labeling a dataset or comparing experiment results. Use the arrow keys or Vim keys (h, j, k, l) to step between traces and their spans, and press Esc to close a trace’s detail view.Start setup with a specific request
You can now pass--prompt (or -p) to bitfab init and bitfab setup to tell the setup agent what you want instrumented from the start. The prompt is forwarded into setup in Claude Code, Codex, and Cursor, so onboarding can begin with the workflow you already have in mind.Lower tracing overhead in the Python SDK
Traced functions in the Python SDK now return as soon as their trace data is captured, instead of waiting for spans to finish uploading to Bitfab. For latency-sensitive code, this takes a network round-trip out of your own request path while your traces keep uploading in the background.Dashboard
Browse past experiments from the dashboard
Every trace function now has an Experiments tab next to Traces and Datasets, listing the experiments that have run against it, newest first. Each run shows its pass rate and how many traces were fixed, regressed, still passing, or still failing versus the original, so you can tell at a glance whether a change helped. Expand a run to inspect its traces, compare the original and updated output, and view the code change that produced it.TypeScript SDKPython SDKRuby SDKDashboardPluginsCLI
TypeScript SDK v0.29.1, Python SDK v0.29.1, Ruby SDK v0.29.1, Plugins v0.8.158, CLI v0.2.203
Replay verdicts persist by the original trace
When your coding agent evaluates a replay run, its pass/fail verdicts now persist against the trace each item was replayed from: passtestRunId plus the item’s originalTraceId to update_agent_labels and Bitfab resolves them onto that run’s replay traces. Verdicts reliably reach the experiments page without the agent ever needing a server-generated replay trace id.Replay items rename source to original
Replay items, progress events, and adapt-inputs context now name the replayed-from traceoriginalTraceId and originalSpanId (snake_case in Python and Ruby); the previous sourceTraceId/sourceSpanId names keep working everywhere as deprecated aliases, so existing scripts are unaffected. An item’s traceId is now null while the run streams and is filled in with the server replay id when the run completes.Replay verdicts are saved automatically
When you replay a single trace to check whether a fix worked, the assistant now saves its pass/fail verdict onto that replay trace instead of only showing it in chat, so your conclusion sticks and appears alongside the trace. The replay path stays lightweight, with no Studio, dataset, or experiment setup. If your SDK is too old to return a replay trace ID, the verdict stays in chat with a prompt to upgrade the SDK.TypeScript SDKPython SDKRuby SDKDashboardPluginsCLI
TypeScript SDK v0.29.0, Python SDK v0.29.0, Ruby SDK v0.29.0, Plugins v0.8.156, CLI v0.2.201
Inject custom values into specific spans during replay
When you replay a trace, you can now override the output of a chosen span instead of running its real code or replaying its recorded output. Match a span by its name, function key, or type, then return a fixed value or one computed from the span’s live inputs and its original recorded output. Available in the TypeScript, Python, and Ruby SDKs.Dashboard
Long trace results stay visible
Large arrays in trace Input and Output views now open automatically, so you can see long result lists without an extra click. When those lists contain nested objects or arrays, each entry stays collapsed to keep the trace readable; compact scalar values remain visible.DashboardPluginsTypeScript SDKPython SDKRuby SDKGo SDKCLI
Plugins v0.8.155, TypeScript SDK v0.28.11, Python SDK v0.27.8, Ruby SDK v0.23.8, Go SDK v0.12.5, CLI v0.2.200
Read one span without loading the full trace
Fetch a single persisted span directly from a trace in the TypeScript, Python, Ruby, and Go SDKs. Select it by canonicalid or by name; repeated names return the last occurrence by default, with options for the first or a zero-based occurrence.Reliable nested traces from the first Node.js call
The TypeScript SDK now preserves nested span context from the first traced call in Node.js, including ESM and CommonJS applications. For async-generator streams, wrap the controller that owns the iteration in an outerwithSpan call so service-side production and consumer-side work appear under one trace.Trace plan warnings for spans that won’t replay
Trace plans now flag spans that won’t replay cleanly before you confirm. If the entry point’s input can’t be serialized, the plan shows a “Root not replayable” warning; spans mocked despite output that can’t be serialized are called out the same way. When a plan carries any of these warnings, Confirm opens a confirmation step so you accept them deliberately rather than by accident. The warnings show on the trace plan review page, and your coding agent applies the same rules when it drafts a plan.Dashboard
Structured Map and Set trace output
Trace outputs containing JavaScript Maps and Sets now render as navigable structured data instead of collapsed string representations. Maps preserve distinct keys even when their string forms collide, and nested Maps and Sets remain expandable in the trace viewer.Dashboard
More reliable dashboard startup
The Bitfab dashboard now avoids analytics initialization crashes in browsers where cookie access is unavailable or blocked. Analytics stays inactive until it is ready, so affected sessions can load the dashboard normally.Edit span templates without leaving the trace you’re viewing
When you ask your coding agent to change how spans render, Bitfab no longer pulls you onto a separate template-preview page. If you’re already looking at a trace of that function, it now offers to edit the templates in place, and your open trace re-renders live with each change you make. The dedicated preview page, with click-to-target editing on the function’s most recent trace, is still one option away when you want it. Available in the Claude, Cursor, and Codex plugins.Fix flow reverts changes that cause regressions
When you use the Bitfab assistant’s fix flow and re-run your full dataset to lock in a fix, it now checks whether your change broke traces that were passing before. If it finds real regressions, the assistant recommends reverting the fix and starting a fresh attempt, so you never ship a change that trades one fixed trace for several broken ones. The trace you were fixing stays saved in your dataset as a regression test to revisit. Available in the Claude, Cursor, and Codex plugins.Choose where fixed traces get saved
When you fix a failing trace with the Bitfab assistant (/bitfab:assistant fix), it now asks which dataset to save the fixed scenario to, instead of silently adding it to whichever dataset already existed. Pick an existing dataset, create a new one, or continue without saving. When you have several datasets it recommends the most recently used one and keeps the list short, so the choice stays quick.Dashboard
Trace plans stay available longer
Trace plans now remain available for 365 days, giving you much more time to return to an instrumentation plan before confirming it. The longer review window applies to newly created trace plans.Session length in chat summaries
Chat session summaries now show how long each coding session lasted, so you can put the product feedback in context at a glance. Slack notifications show the readable duration, while generic webhooks include the exact start time and duration in milliseconds.Dashboard
Redesigned trace input and output view
Trace spans now show Input, Output, Context, and Error as distinct, color-coded zones with sticky headers, so you always know which part of a span you’re reading as you scroll. Input and Output sit side by side and stack automatically when the panel is narrow. When a span has an error, the Output header shows a control that jumps straight to the error details.Collapsible JSON for trace payloads
Trace input and output now render as an interactive tree you can expand and collapse, instead of a flat text dump. Deeply nested objects, large arrays, and long embedding vectors stay collapsed by default, so you can drill into just the parts you care about.Dashboard
A clearer trace plan review
The trace plan review page now speaks the same visual language as its “What is a replay?” explainer. Each span shows a type-colored dot with an icon for what happens on replay (re-runs live, mocked from the recording, or skipped inside a mock), its classification written out beside the name, and a Re-run | Mock toggle on the right. Every control has a tooltip, and the tree navigates with the arrow keys.The replay entry point always re-runs
A replay starts by re-running the top traced span, so its Mock control is now disabled with an explanation instead of silently having no effect. If you untrace spans above a mocked one, the newly promoted entry point switches to re-run and its children are no longer marked as skipped. Run/mock choices also survive untracing and re-tracing a span, and closing the replay explainer with Escape no longer cancels the whole plan.DashboardTypeScript SDKPython SDKRuby SDK
TypeScript SDK v0.28.9, Python SDK v0.27.7, Ruby SDK v0.23.7
See which spans were mocked on a replay
When you replay a trace, spans set to mock on replay are served from the original trace’s recorded output instead of re-executing. The trace view now marks those spans with a badge, both in the span tree and on the span header, so you can tell at a glance which nodes were replayed from history and which re-ran live. Recording that disposition requires the latest TypeScript, Python, or Ruby SDK.A replayability check before you accept a trace plan
When you instrument an AI workflow, Bitfab now verifies the traced function’s root can be replayed before it proposes the trace plan, instead of letting you accept a plan and only then discover the root cannot be replayed. If the root’s inputs are not serializable, setup resolves it up front by moving the trace boundary inward, using a framework handler, or refactoring, so the plan you confirm is one you can actually replay against later. Available in Claude Code, Cursor, and Codex.A faster path through instrumentation setup
Bitfab setup now moves directly from finishing one instrumented workflow to choosing the next workflow, selecting another target, or finishing setup. It shows how to exercise the workflow and run its generated replay command, while replay coverage checks remain available as an explicit action. When you choose the next workflow, Bitfab refreshes its workflow scan so targeted setup runs do not miss other candidates. Available in Claude Code, Cursor, and Codex.Studio opens in your normal browser
Bitfab Studio now opens in your usual browser as a regular tab, so you keep the address bar, tab controls, and the rest of your browser workflow. When a Studio session ends on macOS, the plugin closes only its matching tab and leaves your other browser tabs untouched.Honest Studio launch reporting
When a plugin command opens Bitfab Studio, it now always surfaces a clickable link in chat, and the message no longer claims a window opened before one actually did. If a browser could not be launched at all (common on remote or SSH sessions, or when no supported browser is available), the command reports why and the surfaced link still connects the session when you click it.Automatic recovery when a Studio window never appears
Previously, if a Studio window failed to surface, commands could wait indefinitely and later opens kept pointing at the dead session until it was cleared by hand. The plugin now detects a window that never connected, ends the wait with a clear reason so your coding agent offers a retry instead of treating it as a cancel, and clears the session automatically so the next open starts a fresh window. Stale or unresponsive Studio background processes are also detected and restarted on their own, so Studio commands always run the version of the plugin you have installed.Dashboard
Animated replay explainer on the trace plan page
The trace plan’s “What is a replay?” modal now teaches by showing. A side-by-side animation plays the original run recording each call’s input and output, then a replay re-running it: injecting the recorded input, answering mocked calls straight from the recording, and skipping everything nested under a mock. Each call is annotated with its replay plan (runs live, from recording, or skipped) so you can see why the replay behaves the way it does before confirming your plan. With reduced motion enabled, the modal shows the final annotated diagram as a static picture instead.Recover a stuck Studio session with bitfab login --force
If bitfab login reported that a Studio window was recorded as open but was not responding, there was previously no way to clear it from the command line. Running bitfab login --force now clears the stale Studio session before opening a fresh window, so you can get straight back to signing in. The error message also points you to the flag whenever you hit that state.Dashboard
More reliable trace ingestion
Bitfab now handles traces and spans that contain Postgres-incompatible text in their raw payloads, so ingestion can continue instead of failing the write. This improves reliability for SDK uploads that include null bytes or malformed Unicode from upstream tools.Readable analyze-repo summaries
bitfab analyze-repo now prints a compact terminal report after uploading draft trace plans, so you can see the selected workflows, frameworks, instrumentation effort, suggested capture methods, replay mocks, and real-data value without opening plan links. The same summary output works across Claude Code, Codex, and Cursor runs, with long lines wrapped for terminal readability and skipped candidates kept in the report.More reliable trace plan setup
Asking Bitfab to create a trace plan now reliably opens it in Bitfab Studio for review, instead of occasionally rendering the plan inline in the chat. Requests like “create a trace plan” or “instrument the next function” route directly into the setup flow, and following up to instrument another function reopens the Studio confirmation UI automatically.Dashboard
More reliable trace search indexing
Bitfab now recovers from temporary upstream interruptions while preparing trace search summaries, reducing cases where newly ingested traces fail to become searchable. The retry behavior covers rate limits, provider-side failures, and network transport failures while still stopping on deterministic request errors.Steer analyze-repo with a prompt
bitfab analyze-repo now takes free-text guidance so you can point it at the parts of your codebase you care about. Pass --prompt (short form -p, or just a trailing quoted argument) with something like “focus on the billing and checkout flows” and the scan biases toward those areas when picking which AI workflows to draft trace plans for, topping up any remaining slots from the rest of the repo. Combine it with --limit to cap how many plans it uploads.Go SDK: drop() is safe to call across goroutines
Calling drop() on a trace from one goroutine while another goroutine finishes a span on the same trace no longer races on the trace’s dropped state. If your Go service drops traces from a different goroutine than the one running the traced work, that path is now safe.Dashboard
More reliable CLI sign-in handoff
Bitfab now waits for the browser sign-in handoff to confirm that the CLI received its credentials before showing success. The Studio close page stays in a finishing state while sign-in completes, and if delivery fails it leaves the page open with a retry option instead of making the terminal wait silently.TypeScript SDKPython SDKRuby SDKGo SDK
TypeScript SDK v0.28.8, Python SDK v0.27.6, Ruby SDK v0.23.6, Go SDK v0.12.3
drop() now stops later spans from being sent
Calling drop() on the current trace now prevents any span that finishes afterward from being uploaded at all, so dropping a run that carries sensitive data keeps that data local instead of sending it and clearing it server-side. Spans already sent before the drop() call are still cleared, and the trace is still marked dropped. Available in the TypeScript, Python, Ruby, and Go SDKs.TypeScript SDKPython SDKRuby SDKGo SDK
TypeScript SDK v0.28.7, Python SDK v0.27.5, Ruby SDK v0.23.5, Go SDK v0.12.2
Discard an in-flight trace with drop()
You can now discard the current trace at runtime from your own code, when you decide it shouldn’t be recorded (a health check, a cache hit, any path with no useful signal). Call drop() on the current trace and Bitfab skips it: its inputs, outputs, and spans are never stored, and anything already uploaded for that trace is cleared.getCurrentTrace().drop(), get_current_trace().drop(), and GetCurrentTrace(ctx).Drop() in Go). The call is always safe: if there’s no active trace it does nothing, and it never throws or interrupts your code.More reliable analyze-repo from the CLI
bitfab analyze-repo now runs non-interactively through Claude Code, Codex, and Cursor Agent, so you can scan a repository and upload draft trace plans without opening an editor UI. Use --editor to choose the agent and --limit to cap how many plans are drafted.The CLI also gives these headless runs clearer outcomes: uploaded plans share a Bitfab run identity, logs redact API keys safely, and terminated agent processes fail with a clear error instead of looking successful.Analyze-repo runs link plans and session logs
Thebitfab analyze-repo command now gives each scan a shared run identity, so the draft trace plans it uploads and the optional captured Claude Code session are tied together in Bitfab. This makes it easier to audit what the agent found, which plans it created, and the conversation that produced them.Codex can run analyze-repo headlessly
bitfab analyze-repo --editor codex now runs through codex exec without opening the Codex TUI, using the same non-interactive scan and draft trace-plan upload flow that was already available through Claude Code. Cursor Agent is now supported too through its --print headless mode.Analyze-repo can sign in before scanning
Thebitfab analyze-repo command now opens the Bitfab sign-in flow when an interactive run is not authenticated, then continues the scan after login succeeds. Non-interactive runs still stop with clear instructions, and if a stale environment API key blocks verification the CLI explains how to fix it.Cleaner plugin login and logout in local projects
Plugin login now keeps the Studio sign-in URL visible and reports success as its own status line, which makes local and dev sign-ins easier to follow. Logout now clears project-local credentials before falling back to global credentials, so worktree-specific logins can be reset without affecting other projects.Editor sign-in checks before setup
Thebitfab CLI now checks whether Claude Code, Codex, or Cursor is signed in before it launches setup, assistant, SDK update, or analyze-repo. If the editor agent is logged out or the editor CLI is missing, Bitfab stops early with the login command to run instead of opening an agent session that fails later.Command-specific help in the Bitfab CLI
Thebitfab CLI now shows help for individual commands, so you can check the right flags and usage without triggering the command itself. Use bitfab help <command>, <command> --help, or <command> -h to inspect commands before running onboarding, login, install, or other workflows.More accurate natural-language routing in the plugin skills
The setup and assistant skills now route free-form requests to the right mode more reliably. Phrases like “trace a new workflow,” “why aren’t my traces showing up,” or “did my fix work on this trace” land in the correct mode without you having to name it. Each skill also lists its modes and what they do up front, so the full set of things it can do is visible at a glance.Clearer plugin setup status
Bitfab plugin setup now makes local and dev authentication easier to verify. Login success messages include the non-production endpoint when the plugin is pointed away from production, and plugin update checks show the version that is already installed or was just updated.Trace plans isolate external calls by default
Setup-generated trace plans now mark mockable external reads and side effects, such as database queries, HTTP calls, and writes, to return their recorded output during replay. LLM calls and local code stay live by default, so replay still tests the model behavior you are trying to improve. If an external parent span would skip live child spans, the server now keeps that parent live and expects the smaller external boundary to be mocked instead.Replay scripts stay aligned with production roots
Bitfab setup and assistant workflows now require generated replay scripts to call the same production root wrapper for traced functions, instead of introducing a replay-only helper that can drift from runtime behavior. For handler-based integrations, replay guidance now points back to the same production framework entrypoint and includes a root-parity checklist before setup finishes.PluginsTypeScript SDKPython SDKRuby SDK
Plugins v0.8.122, TypeScript SDK v0.28.6, Python SDK v0.27.4, Ruby SDK v0.23.4
More reliable replay results
Replays run through the Bitfab plugin no longer fail to report their result when the replay script prints extra output (framework logs, env-loader noise). The TypeScript, Python, and Ruby SDKs now write the full replay result to a file the plugin reads directly, so a passing replay is never mistaken for a failed one.Scan a repo for AI workflows from the terminal
Runnpx bitfab-cli analyze-repo to headlessly scan a repository for its AI workflows and upload a draft trace plan for each of the top candidates, with no prompts and no code changes. Cap how many plans it uploads with --limit (default 5), then review and confirm the drafts in Studio. Available for Claude Code.Setup asks before rewriting your code
When/bitfab:setup instruments a function, it now pauses and asks for approval before restructuring an existing framework or SDK call to attach a trace, instead of rewriting it silently. Purely additive instrumentation (wrapping an unchanged call) proceeds as before; only a change that would modify existing code stops for your confirmation, and any rewrite you approve preserves the original behavior exactly.Dashboard
Trace plans stay valid longer
Trace plans no longer expire after 30 minutes. When you set up tracing and step away before confirming a plan, it now stays valid for 7 days, so you can pick up where you left off instead of recreating it.More reliable Studio sessions
Studio now stays put across restarts. If Studio’s background process reloads, your active session is restored and reopens on the page you were last viewing instead of being lost or opening a duplicate window. Switching between pages (a dataset, a trace plan, the experiments view) now reuses the open Studio tab instead of closing and reopening it.Verified replay labels in assistant runs
Bitfab plugins now verify replay labels immediately after persisting them, so benchmark scorecards only continue once the server reports the expected effective PASS, FAIL, or skipped state. During assistant and replay workflows,persistReplayLabels parses the update_agent_labels response and stops with verification-failed if any trace label is missing or mismatched, giving the agent a clear retry path instead of reporting partial results.Framework-aware replay mocking
The trace planner now decides which spans can be mocked on replay based on how your framework captures them. Spans your code wraps directly can be mocked; spans a framework observes from the outside (LangChain, LangGraph, and similar callback-based integrations) re-run live on replay instead, while Vercel AI SDK model calls stay mockable. This keeps model calls from being dropped or wrongly mocked when you set up tracing on a framework app with/bitfab:setup.Mock a span the planner flagged
When the trace planner marks a span as not mockable, you can now override it and mock it anyway from the trace plan page in Studio. The choice is kept as a warning rather than blocked, and the plan footer shows a warning count so you can review these before confirming.Batch-analyze a repo for what to trace
The Bitfab setup plugin can now scan a whole codebase and draft trace plans in one non-interactive pass. Run/bitfab:setup analyze-repo and it finds your AI workflows, picks the top few worth tracing, and uploads a draft trace plan for each, without prompting or changing any code. Review the drafts in Studio, then run /bitfab:setup instrument on the ones you want to wire up.Replay status is available as an MCP tool
The Claude, Cursor, and Codex plugins now exposeget_replay_status directly through the local Bitfab MCP server. During replay-based assistant runs, agents can map local replay trace IDs to server trace IDs while the run is still in progress, so they can persist per-trace verdicts incrementally without relying on a separate command wrapper.The plugin MCP tool list has also dropped the deprecated grader tools, keeping trace inspection, datasets, labeling, experiments, setup, and replay status aligned across the editor plugins and Bitfab MCP endpoint.Clearer replay label errors
Replay label persistence now fails loudly when a verdict batch includes trace IDs that do not exist in the active organization, instead of saving only part of the batch and treating the rest as skipped. The Claude, Cursor, and Codex plugins now guide agents to remap replay results to server replay trace IDs and retry, so experiment labels are less likely to disappear behind a misleading success message.Trace plans analyze context nodes before review
Bitfab setup now asks Claude, Cursor, and Codex to classify replay behavior for every trace-plan node, including surrounding context nodes that are not initially captured. When you toggle those context nodes into capture from the trace-plan review, they already have a replay decision instead of needing a second classification pass.Modify flows now backfill missing analysis on older trace plans before opening review, so expanding an existing plan uses the same per-node replay guidance.Clearer dataset wording in assistant fixes
The Bitfab assistant fix flow now describes the final step as adding the trace to a dataset with a validated failing label, instead of using capture language for dataset membership. In Claude, Cursor, and Codex, the flow still replays the target trace first, then saves it to a dataset only after the replay passes and lets you choose Studio, a full dataset rerun, another iteration, or stop.Assistant fixes ask before guessing
The Bitfab assistant fix flow now confirms why a trace is wrong before changing code when the trace or conversation does not already provide a clear failure reason. In Claude, Cursor, and Codex, targeted trace fixes reuse an existing failing label, a user-stated defect, or obvious trace evidence; otherwise the agent asks what correct behavior should be before replaying or saving the trace as a regression test.Dashboard
See when experiment results are still settling
When you open the code-change summary for an experiment that is still running, it now shows how many replays are still running or awaiting agent labels, with a note that the breakdown is provisional and will keep updating as they finish. Before, the summary presented partial results as if they were final.More accurate unpaired counts
Replays that are still running, awaiting labels, or errored are no longer counted as “unpaired” in the experiment breakdown, and each now appears as its own segment in the run progress bar. The unpaired count now reflects only replays that genuinely could not be matched to an original.Dashboard
Experiment replays update as traces finish
Experiment pages now refresh replay progress as each trace finishes, so results appear while the replay is still running instead of waiting for later labeling work. Runs outside an experiment group no longer trigger experiment updates, keeping grouped experiment views focused on the runs they are showing.Trace descriptions route to assistant fixes
The Bitfab assistant now treats requests like “find and fix the trace where…” as targeted fixes, even when you describe the bad output instead of pasting a trace ID. In Claude, Cursor, and Codex, the fix flow first verifies that the request is trace-backed, then uses local instrumentation and trace search to find the matching failing trace before making changes.Dashboard
Expired trace plans show their final state
Trace plan review now shows when an awaiting plan has expired instead of letting you try to confirm it and then showing an error. The review bar switches to a disabledPlan expired action as soon as the plan is stale, including when an already-open review page crosses its expiry time.More reliable BAML client execution
BAML client execution is more reliable across the Bitfab dashboard and TypeScript SDK. Generated OpenAI clients no longer send unsupported temperature options to GPT-5 and o-series models, while supported OpenAI, Claude, and Gemini clients keep deterministic sampling where the provider accepts it.Shared Bitfab BAML fallbacks now use current tested Gemini and Claude models, so duplicate detection, entity extraction, trace summaries, and custom BAML prompts avoid stale model endpoints that could fail before the prompt ran.Trace plans show when spans cannot be mocked
Trace plan review now distinguishes spans that can return recorded output during replay from spans that must re-run live. When Bitfab cannot mock a span because the recorded output is not serializable or the span comes from library instrumentation, the plan keeps it on re-run and shows the reason in the replay control tooltip, so reviewers know why the mock toggle is unavailable.The Claude, Cursor, and Codex plugins now pass that mockability metadata throughcreate_trace_plan, so generated plans can mark those spans before you confirm instrumentation.PluginsTypeScript SDKPython SDKRuby SDK
Plugins v0.8.105, TypeScript SDK v0.28.4, Python SDK v0.27.3, Ruby SDK v0.23.3
Replay mocks are marked by default
Replay now usesmock: "marked" by default in the TypeScript, Python, and Ruby SDKs. A span tagged with mockOnReplay: true / mock_on_replay: true now returns its recorded output during replay without also passing a mock option. Pass mock: "none" when you explicitly want every child span to run real code, or mock: "all" when you want every child span to return historical output.The docs now treat replay mocking as a first-class workflow, with a dedicated Replay Mocking guide linked from the introduction and SDK pages.Redesigned trace plan review
The trace plan you confirm when setting up tracing has a cleaner, more informative review screen. Each captured span is now labeled with what it does,code, llm call, read, or write, and whether replay re-runs it live or serves its recorded output, so you can see at a glance how your workflow will replay. The span tree is easier to scan: you can toggle spans in or out of the capture set and mark which ones to mock on replay, and a “How to review” guide walks you through it the first time.Manual Studio links in plugin flows
Bitfab plugins now print a copyable Studio link whenever they open a fresh Studio page, so you can still get to login, setup, datasets, experiments, and trace-plan flows if your editor hides the browser launch. The agent instructions for Claude, Cursor, and Codex now tell the agent to surface that link in chat asStudio opened: <url>, while the command output also includes Studio opened at: <url> for terminal visibility.Setup signs you in before instrumenting
Running/bitfab:setup instrument now signs you into Bitfab before it starts analyzing your code, so instrumenting a new AI workflow no longer stalls partway through when you aren’t logged in. Previously the flow could begin and then fail later at the trace-plan step; it now authenticates up front, matching how the other setup modes already work.Safer failed-fix regression capture
The Bitfab assistant now only marks an unresolved fix as saved after the failing trace is actually attached to the selected dataset. If a dataset attach is skipped, the Claude, Cursor, and Codex plugins keep the fix unsaved and guide the agent to choose a function-scoped dataset instead, so a failed fix cannot disappear without a real dataset entry.Codex Studio navigation also returns control to the conversation once the Studio page is open and ready to report events, instead of waiting in the foreground while you inspect the page.More reliable Codex session capture
The Codex plugin now captures active Bitfab sessions more reliably while you work, so chat-session history is less likely to miss turns during tool-heavy workflows. Final session uploads still complete when Codex stops, and late background capture work can no longer revive stale local progress after a session has closed.Dashboard
Coding-agent session summaries
Bitfab now turns ended coding-agent chat sessions into structured product-feedback summaries, so teams can see what the agent tried, where Bitfab helped, where the plugin or service got in the way, and what should be improved next. Idle sessions are closed automatically, reopened sessions are summarized again after new work, and configured webhooks can send the summary to Slack with a link back to the chat session.Trace environment in read_traces
read_traces now shows each trace’s environment alongside the trace id, function, status, and timing details, so agents can distinguish production, staging, local, and unset traces during investigations. When a trace has no stored environment tag, the tool prints unset instead of hiding the field.Test-first fixes with the assistant’s fix command
The assistant’s fix command now proves a fix before it saves anything. It diagnoses the failing trace, makes the change, and replays just that one trace; only once the replay passes does it add the trace to a dataset with a validated failing label, then offers to re-run the whole dataset (in Studio or your terminal). If the fix does not land, you can save the trace as a failing test to revisit later. When you open the fixed trace in Studio, the experiments page now lands directly on that trace’s before/after comparison instead of a list you have to click into. Available in the Claude, Cursor, and Codex plugins.Replay your first captured trace during setup
When you instrument a workflow with/bitfab:setup, Bitfab now waits for your first trace to land and then offers to replay that exact trace right away, so you can confirm your replay script works before moving on. Available in the Claude, Cursor, and Codex plugins.Codex uses project-local Bitfab credentials from cached launches
The Codex plugin now resolves its active worktree even when the MCP server starts from Codex’s plugin cache without a recorded session id. That keeps setup, login, and plugin actions pointed at the project’s local Bitfab connection instead of falling back to stale global credentials.More dependable coding-agent session capture
Bitfab now keeps coding-agent chat sessions in order across transcript compaction and retry recovery, so older turns stay attached to the same session instead of shifting sequence positions. Sessions that stop without new turns also preserve their real activity and end times, and the dashboard now recognizes Bitfab slash-command activity in the same session activity view.More reliable replay mocks
Replay mocking in the TypeScript SDK now matches older historical spans even when the recorded span tree does not include a span name, falling back to the trace function key for the mock lookup. This keepsmock: "all" and mock: "marked" replay runs working on older traces instead of rerunning child spans that were meant to use their historical outputs.The TypeScript and Python replay examples now show both mock modes with mockOnReplay and mock_on_replay, so you can verify that expensive child steps are skipped before running a full experiment.Cleaner final replay results
The Bitfab assistant now keeps the final replay event log focused on one server-backed item reference per replayed trace, even when progress events already wrote per-trace payload files. This prevents assistant evaluations from scoring the same replayed trace twice in mixed-payload runs, while live progress rows still support incremental evaluation as traces finish.PluginsTypeScript SDKPython SDKRuby SDK
Plugins v0.8.89, TypeScript SDK v0.28.1, Python SDK v0.27.1, Ruby SDK v0.23.1
Live per-item replay evaluation
The Bitfab assistant now scores and records each replayed trace as soon as it finishes, instead of waiting for the entire replay run to complete. On long replays, Studio’s experiment view fills in pass/fail verdicts trace by trace while the run is still going, so you see results as they land instead of all at once at the end.The replayonProgress callback in the TypeScript, Python, and Ruby SDKs now reports per-item detail for each trace as it settles, including the deserialized inputs, the replayed output, the original output, and token usage, so tools watching a running replay can evaluate results mid-run rather than waiting for the final ReplayResult.TypeScript SDKPython SDKRuby SDKGo SDK
TypeScript SDK v0.28.0, Python SDK v0.27.0, Ruby SDK v0.23.0, Go SDK v0.12.0
Reliable tracing regardless of API key load order
The Bitfab SDKs now resolve your API key the first time a traced function runs, instead of when the client is constructed. If your key (or.env file) loads after the client is created, tracing still activates, so a client built before your environment is ready no longer silently drops every trace.All four SDKs (TypeScript, Python, Ruby, Go) also gain a BITFAB_API_KEY environment fallback when you don’t pass a key explicitly, a callable key form for deferred resolution (e.g. new Bitfab({ apiKey: () => process.env.BITFAB_API_KEY })), and an opt-in strict mode that fails loud on a missing key instead of disabling tracing quietly.Trace-first assistant investigations
The Bitfab assistant now grounds traced-function investigations in actual trace evidence before recommending a fix or revert. When it investigates an AI workflow failure, it first derives the function key from local instrumentation, searches the matching traces, and separates what the trace proves from code-based inference, so debugging starts from the run that failed instead of only from static code.Targeted trace fixes before dataset runs
The assistant fix flow now replays only the target failing trace first, then asks whether to inspect the before/after in Studio, run the full dataset, keep iterating, or stop. This makes/bitfab:assistant fix <trace-id>, /bitfab-assistant fix <trace-id>, and $bitfab:assistant fix <trace-id> faster to validate and prevents a dataset run from hiding whether the original bug is green. If you choose the full dataset run, Bitfab opens Studio as an experiment so you can catch regressions after the target fix is proven.Clearer LangChain setup guidance
Bitfab setup guidance now recognizes LangGraph and LangChain projects earlier and recommends the callback handler instead of manually wrapping graph nodes, tools, retrievers, or model calls. The docs and setup prompts also clarify that the handler already records a replayable framework root, so you only need a same-key outer span when there is meaningful application work around the graph or chain invocation.DashboardPluginsTypeScript SDKPython SDK
Plugins v0.8.80, TypeScript SDK v0.27.2, Python SDK v0.26.2
In-progress LangGraph and LangChain traces
Bitfab now shows LangGraph and plain LangChain runs as in progress as soon as the framework callback root starts, so long-running agents appear in the dashboard before their final output is available. The TypeScript and Python callback handlers also keep configured LangChain run names on chain, model, tool, and retriever spans, making trace trees easier to line up with your graph or chain config.Cleaner framework replay inputs
Python tool spans now preserve empty structured tool inputs as{} instead of falling back to raw text, and TypeScript chain callbacks handle both LangChain callback argument orders for run names and parent IDs. This keeps handler-captured traces more accurate for replay and avoids nested framework callbacks marking the outer trace complete too early.Project-local plugin credentials from cached launches
Bitfab plugins now keep using your project-local connection settings when Claude Code, Cursor, or Codex launches the plugin from a cache or nested workspace. Localconfig.local.json and credentials.local.json files are resolved together from the same project search path, so agents are less likely to fall back to the wrong workspace or global credentials.Consistent Studio navigation across plugins
Studio navigation commands now work consistently in the Claude Code, Cursor, and Codex plugins when a Bitfab skill opens a specific Studio page. The plugins now ship matching command wrappers, so agent workflows can rely on the same Studio actions regardless of which editor plugin you use.DashboardPluginsTypeScript SDKPython SDKRuby SDK
TypeScript SDK v0.27.1, Python SDK v0.26.1, Ruby SDK v0.22.1, Plugins v0.8.77
Name replay experiments
You can now give Bitfab experiments a readable name when you create a dashboard test run or start a replay. Experiment cards show the name next to the short run ID, making it easier to compare baselines, prompt edits, and code-change runs without opening each result.Replay names in SDKs and plugins
TypeScript, Python, and Ruby replay calls now accept aname option, and replay scripts can forward it with --name. The Bitfab plugins detect whether the installed SDK and script support experiment names before using the flag, and recommend an upgrade when a project is too old.TypeScript SDKPython SDKRuby SDKPluginsDashboard
TypeScript SDK v0.27.0, Python SDK v0.26.0, Ruby SDK v0.22.0, Plugins v0.8.76
Live replay progress in setup and assistant runs
Bitfab replay runs can now stream per-trace progress in Claude, Cursor, and Codex while the replay keeps running in the background. The plugins show each trace as it settles, write a per-run event log under.bitfab/replays/<run-id>/events.jsonl, and store full per-item outputs under .bitfab/replays/<run-id>/items/ so agents can review complete outputs without mixing progress logs into the result JSON.Replay progress reporters for SDK scripts
TypeScript, Python, and Ruby SDKs now include ready-made replay progress callbacks:reportReplayProgress, report_replay_progress, and Bitfab.report_replay_progress. Pass them into replay scripts to emit plugin-readable progress events while stdout stays reserved for the final ReplayResult JSON.TypeScript SDKPython SDKRuby SDKGo SDKPluginsDashboard
TypeScript SDK v0.26.2, Python SDK v0.25.2, Ruby SDK v0.21.2, Go SDK v0.11.2, Plugins v0.8.75
Capture warnings stay out of execution errors
SDK spans that have to degrade serialization now mark those payload warnings as SDK-sourced capture notices. The dashboard shows them as a bordered “Capture incomplete” tag on the span instead of mixing them into execution errors, so real runtime failures stay distinct from lossy capture metadata.Studio opens in an app-style window from Codex
The Bitfab Codex plugin now opens Studio in the same focused app-style browser window used by the other editor plugins when Chromium supports it. If the app-window launch cannot start, the plugin falls back to your normal browser, andBITFAB_DISABLE_CHROME_APP_WINDOWS=1 still forces the tabbed fallback.Dashboard
Chat-session capture handles unsupported text characters
Bitfab now keeps coding-agent chat-session capture working when terminal output includes characters that databases cannot store in JSON fields. Unsupported characters are safely replaced so the rest of the transcript, tool calls, and usage details remain available in the dashboard.Dashboard
More reliable chat session capture
Bitfab now captures coding-agent chat sessions more reliably when turns include token usage details. This keeps session histories and usage metadata flowing into the dashboard instead of dropping affected turns during ingestion.Fix a failing trace end to end
The Bitfab assistant has a newfix mode that takes one failing trace and drives it to green. Point it at a trace (/bitfab:assistant fix <trace-id> in Claude Code, /bitfab-assistant fix <trace-id> in Cursor) and it adds the trace to a dataset with a validated failing label, edits the code, then replays until the trace passes, flagging any regressions and offering to fix the other failing traces too. It only engages when you point it at a real Bitfab trace, so an ordinary “fix this bug” on untraced code still goes through normal coding. Works across the Claude, Cursor, and Codex plugins.Plugin login reuses its sign-in window
Signing in to the Bitfab plugin no longer opens duplicate Studio windows when a login is retried before it finishes. The plugin now reattaches to the window that’s already waiting for you, so a sign-in that looks slow won’t stack up extra browser windows.Filter trace search by database snapshot
Thesearch_traces tool now accepts a hasDbSnapshot filter, so you can scope a search to traces that captured a database snapshot, or only those that didn’t. Use it to quickly find the traces that can be replayed against their historical database state. Works across the Claude, Cursor, and Codex plugins.Dashboard
Plugin sign-in with Google no longer hangs
Signing in to the Bitfab plugin with Google now completes reliably and connects your coding agent. Previously a Google sign-in could leave the plugin waiting on a login that never registered.Dashboard
Faster usage page
The usage page loads noticeably faster, especially for organizations with high trace volumes, and switching the function filter now applies instantly instead of pausing while the page catches up.Dashboard
Shareable links for usage filters
The usage page now keeps your function filter in the page URL, so a filtered view can be bookmarked or shared as a link and it stays put as you move around. Your browser’s back and forward buttons also step through filter changes.Dashboard
Accurate “Snapshot captured” badge on traces
The “Snapshot captured” badge now appears only on traces that actually captured a database snapshot, instead of on every trace once your organization connected a source database. The trace list and trace view now give an accurate at-a-glance signal of which traces can be replayed against their database state from when they ran.Dashboard
Break down usage by function
The usage page can now be scoped to a single traced function: pick a function and the totals, chart, periods table, and CSV export all narrow to just that one. A new “By function” chart mode plots one line per function so you can compare volume across functions over time. The function picker and chart mode appear once your organization has more than one traced function.Bind framework handlers to your trace key once
ThegetFunction() handle now hands you framework handlers and middleware already bound to its trace key, so an outer span and the handler share one key without repeating the string. Available for the Claude Agent SDK, LangGraph/LangChain, and (TypeScript only) the Vercel AI SDK.get_function(): pipeline.get_claude_agent_handler() and pipeline.get_langgraph_callback_handler().Live progress callbacks for replay
Replay now takes an optional progress callback, so you can show live progress while a run is in flight instead of waiting for it to finish. It fires once per trace as each one settles, with running totals you can render however you like.onProgress in the TypeScript SDK and on_progress in Python and Ruby. The totals report how many traces have finished, how many ran without error, and how many threw; pass/fail verdicts are assigned after the run, so the live totals split ran-ok versus errored. Upgrade to @bitfab/sdk v0.25.0, the bitfab Python package v0.24.0, or the Ruby gem v0.21.0.Dashboard
Clearer experiment run summaries
Experiment run rows now report accurate counts: a replay that finished with an error shows as “errored” instead of “awaiting labels”, and a trace still awaiting an agent label is no longer also counted as “unpaired”. Hover any status pill to see the full breakdown of passing, failing, awaiting, and errored traces. The row also stays readable at any panel width, collapsing to the essentials when space is tight.Replay results report the replayed run’s token usage
When you replay traces, each result item’stokens now reflects the token usage of the replayed run instead of the original trace. Compare it against the original trace’s recorded usage to see how a code change moved cost. durationMs and model stay as the original trace’s reference values, and the token counts match what the experiments view shows. Upgrade to @bitfab/sdk v0.24.1, the bitfab Python package v0.23.2, or the Ruby gem v0.20.2.Dashboard
See database snapshot status on your traces
Traces now show whether they can be replayed against the database state from when they ran, and whether a replay actually used that historical state. A “Snapshot captured” badge marks an original trace you can replay against its past database; on replays, “Snapshot replayed” means your code read the historical database branch and “Snapshot unused” means it fell back to the live database. The indicator appears on the trace list, the trace and span detail headers, and in dataset and experiment rows, and only shows for organizations with a connected source database.Watch experiment results appear live during replay
When you start a replay, its experiment now appears right away and fills in trace by trace as results come back, instead of staying on “Waiting for experiments to start…” until the whole replay finished. You can watch the pass and fail counts climb in real time as each trace completes.The TypeScript SDK never crashes or hangs your app
Tracing is now fully fail-open. If anything in the SDK’s instrumentation goes wrong (a runtime without a usablecrypto, an oversized payload, a serialization edge), your traced function still runs and returns its real value, and your Node process still exits cleanly instead of being held open by a pending timer. When tracing has to degrade, the SDK logs a one-time [bitfab] warning so a dropped span is visible rather than silent. Upgrade to @bitfab/sdk v0.24.0.Type-checking no longer needs the OpenAI Agents package
The SDK’s published type definitions no longer reference@openai/agents, so projects that don’t use the OpenAI Agents integration type-check cleanly without installing that package (previously tsc could fail with “Cannot find module ‘@openai/agents’” under skipLibCheck: false). If you use getOpenAiAgentHandler(key).wrapRun(...), its result is now typed structurally: read finalOutput and cast it to your agent’s output type.Dashboard
More reliable trace ingestion under load
Traces now ingest reliably during high-volume bursts. We fixed a case where ingesting a trace could time out and fail when its search index was being built inside the same database operation that saved the trace. Search indexing now runs after the trace is safely stored, so a slow index can never block or fail ingestion.TypeScript SDK: builds no longer fail on unused optional integrations
If you used the TypeScript SDK with one integration (like the Vercel AI SDK) but not others, your bundler could fail at build time withModule not found: Can't resolve '@openai/agents', even though you never used the OpenAI Agents integration. The SDK no longer references its optional peer dependencies (@openai/agents, @boundaryml/baml) in any way a bundler tries to resolve up front, so your app only needs to install the integrations it actually uses. Upgrade to @bitfab/sdk v0.23.3.Vercel AI SDK tracing
Bitfab now traces the Vercel AI SDK out of the box. Wrap your model withgetVercelAiMiddleware and every generateText, streamText, generateObject, and streamObject call is captured as a traced LLM span, including streaming responses and which provider answered each call (handy when you fall back between models). The Bitfab plugin also detects and instruments Vercel AI SDK projects automatically during setup.Ruby SDK guards against replay key mismatches
replay() in the Ruby SDK now raises an ArgumentError when the trace_function_key: you pass does not match the key the method is actually traced under, instead of silently fetching one function’s history and recording the run under another. This brings Ruby in line with the TypeScript and Python SDKs.Replayed traces match the original trace structure
Replaying a handler-instrumented run (OpenAI Agents, Claude Agent SDK) now produces a root span named after your trace function key, matching the original production trace instead of the replayed function’s name. For OpenAI Agents, a replayed run also nests under a single root span instead of an extra duplicated one, so a replay’s span tree lines up with the trace it replays.Closed Studio windows stay closed
When you close a Studio window, it now stays closed. Previously the plugin could reopen a Studio window on its own shortly after you dismissed it; that no longer happens. Studio windows open only when you or your coding agent explicitly ask for one.Recovering a stalled Studio session
If a Studio window crashed, was closed, or your machine went to sleep, reopening Studio could stay stuck reporting that the existing session was unreachable, even after you cleared it. Clearing a stalled session now reliably reopens a fresh Studio window.More reliable Studio sessions
Fixed an issue where leftover Studio background helpers from earlier plugin versions could pile up and cause a Studio session to hang or fail to open. The plugin now clears out stale helpers before starting a session, so opening Studio stays reliable across plugin updates.LangGraph and LangChain retriever steps are now traced
Retriever calls in your LangGraph or LangChain graphs now appear as spans, capturing the query and the documents returned, so retrieval shows up alongside the rest of your agent’s work. No code change is needed beyond the callback handler you already pass.Trace streamed OpenAI Agents runs
You can now trace streamed agent runs. In TypeScript, pass{ stream: true } to wrapRun; in Python, use the new wrap_run_streamed async generator as a drop-in for Runner.run_streamed. The run’s input and final output are recorded on the root span once the stream finishes, so streaming runs are traced without changing how you consume the events.Read the BAML Collector after a call
wrapBAML / wrap_baml now expose the BAML Collector from the most recent call through a .collector attribute, plus an onCollector / on_collector callback that runs after each invocation. Use it to inspect the prompt, model, and token usage yourself, in addition to the metadata Bitfab already captures.Dashboard
Faster trace lists for high-volume functions
The traces list now loads quickly for functions with very large trace volumes, where it could previously take several seconds or fail to load. The speedup is automatic, with nothing to configure.Replayable OpenAI Agents runs with one line
If you trace the OpenAI Agents SDK with Bitfab, you can now make agent runs replayable with a drop-in change. The new run wrapper records the run’s input as a replayable root, with the tracing processor’s spans nested underneath, so the trace replays by key with no hand-written wrapper. Keep registering the processor for the internals and swap your run call for the wrapper.get_openai_agent_handler("my-agent").wrap_run(agent, "user input").Dashboard
More reliable organization switching
Switching between organizations now reliably loads the selected organization’s data. Previously, changing orgs (including opening a Studio page for a resource that belongs to another organization you’re a member of) could leave the view stuck on your previous organization and show missing or “not found” data until you refreshed. The switch now finishes applying before the page reloads, so the correct organization’s data loads the first time.Replayable instrumentation for OpenAI Agents
When you set up Bitfab tracing, it now makes sure each traced workflow can be replayed, not just observed. For OpenAI Agents, that means wrapping the agent run in a root span that captures its input, so you can re-run the trace against your current code. The setup health check also flags any instrumented function whose root can’t be replayed and points you to the fix.Full Claude Agent SDK tracing in TypeScript
Tracing for the Claude Agent SDK now captures complete agent runs in TypeScript: every LLM turn, tool call, and subagent becomes a span, with token usage included. Get a handler withgetClaudeAgentHandler, inject the hooks with instrumentOptions, and wrap the query() stream:Replay handler-traced agent runs
Agent runs captured by the Claude Agent SDK handler are now replayable without wrapping your code in an extra span. Pass the prompt asinput to the wrap call and the handler records it as the run’s root, so replay() can re-run each historical prompt against your current code. Works in both the TypeScript and Python SDKs.Dashboard
Trace and dataset labeling panels adapt to narrow widths
The dataset review and trace detail views now reflow to fit tight spaces, so labeling traces in the Studio side panel stays comfortable even when the panel is narrow. The labeling toolbar collapses onto a compact top row, and a trace’s input, output, and context sections stack cleanly instead of overflowing or leaving uneven gaps between them.Compact span tree with hover details
When the trace panel is narrow, the span tree collapses to an icon rail that keeps the full call hierarchy while handing the freed space to the span content. Hover any span to see its name, position, duration, and token count, and the token figure follows your selected token view (all or uncached).Cost-optimize mode for the assistant
The Bitfab assistant has a dedicated cost-optimize mode for cutting token spend. Run/bitfab:assistant cost-optimize <function-key> and it first profiles where a dataset’s tokens go (prompt size, redundant context, output shape, model choice), then edits and replays against your labeled dataset to lower cost while holding the pass rate. The token-cost view turns on automatically, reporting token deltas next to pass/fail. Available across the Claude, Cursor, and Codex plugins.Before and after on every experiment trace
In the experiments view, any replay trace paired with an original now always shows the before and after comparison: the verdict change, the Original/Updated toggle, and the token trend. Values that aren’t known render as blanks instead of hiding the comparison, so still-passing and still-failing traces are as easy to inspect as fixed and regressed ones.More accurate OpenAI Agents trace capture
Traces from the OpenAI Agents SDK integration now capture tool-call and generation inputs correctly. Previously the TypeScript SDK could drop these inputs and record them as empty, and both SDKs added empty input and response fields to agent and other non-LLM spans. Tool calls, agent steps, and other spans now show exactly the data the run produced. Update to TypeScript SDK 0.21.1 or Python SDK 0.21.2 to pick up the fix.Faster labeling and evaluation on large datasets
When the assistant labels a fresh batch of traces or evaluates an experiment, it now judges the traces in parallel on larger datasets, so big batches finish noticeably faster. The speed-up kicks in automatically above roughly 20 traces; smaller batches are unchanged.Read a span field in full when a trace is truncated
When Bitfab shows you a trace, large span fields (a long input or a big tool output) are truncated so the response stays readable. The newread_span_field tool fetches the complete, untruncated value of a single field (input, output, reasoning, content, errors, or context) for one span, so your coding agent can pull the full text only when it actually needs it. read_traces now points you to it whenever a field is truncated. Available across the Claude, Cursor, and Codex plugins.Dashboard
Experiments show when a replay is awaiting labels
In the experiments view, a replay that has finished running but hasn’t been scored yet is now marked “awaiting agent labels” instead of an indefinite loading spinner. The state appears on the run’s pass-rate pill, its trace rows, and the progress bar, and stays out of the pass rate until a verdict lands, so a run that is done replaying reads as settled rather than stuck.Studio windows close more reliably at the end of a flow
Studio windows now close themselves when a flow finishes, with a backup that helps close the window if the browser does not. You will see fewer stray Studio windows left open after setup, assistant, and other flows wrap up.Reliable sign-in from the Bitfab CLI
Setting up Bitfab withnpx bitfab-cli init (or signing in with bitfab login) now completes reliably. A recent CLI version could fail to open the sign-in window and report that you were not authenticated; the CLI now opens Studio and finishes login as expected.Faster trace loading in the assistant
When the Bitfab assistant builds or labels a dataset, it now loads the dataset’s traces in parallel batches instead of one group at a time. Large datasets come into context faster and more reliably, with no change to how you run the assistant. This applies across the Claude, Cursor, and Codex plugins.Studio session daemon for persistent browser management
Bitfab plugins now include a session daemon that manages Studio browser windows as a persistent background process. The daemon keeps Studio alive across coding agent restarts: if a browser window crashes, it re-spawns automatically; if you navigate to a new page, it reuses the existing window instead of opening a second one; and if a page refresh fires a transient close event, the daemon waits briefly before treating it as a real close.Built-in help on every command
Every Bitfab plugin and CLI command now responds to-h / --help with a usage line, a short description, and per-argument details, then exits without running. You (and your coding agent) can discover how to call a command without leaving the terminal, and bitfab init -h now prints help instead of starting the full onboarding flow.Dashboard
Studio sessions reconnect after expiring
Returning to Bitfab Studio after a session has been idle now reconnects automatically instead of showing a “Could not connect to this session” error. Previously an expired session left you stuck until you signed in again; the session is now re-established transparently when you come back.No duplicate Studio window after signing in
Signing in to Bitfab from your coding agent no longer leaves a leftover Studio window or opens a duplicate. The next action reuses the window you signed in with instead of spawning a second one.More reliable tracing for Python agent frameworks
The Python SDK no longer drops a span or trace when a value that can’t be JSON-serialized (such as a Pydantic model) appears outside a span’s input or response, like a tool’s output or a trace’s metadata. Spans and traces from the OpenAI Agents, LangGraph, and Claude Agent SDK integrations now always reach Bitfab, and any value that couldn’t be captured faithfully is flagged as non-replayable instead of silently lost.Compact trace reads by default
Theread_traces tool now returns bounded, truncated span details by default (scope: "summary"), so the Bitfab assistant can scan many traces in one pass without overflowing its context. Each span keeps the head and tail of its largest fields, so big inputs and outputs stay legible at a glance; pass scope: "full" when you need complete, untruncated detail on a handful of traces.Trace streaming functions without a refactor
You can now instrument a streaming function (one that returns a live response stream) by adding a singlefinalize option, with no need to restructure your code. Your function still streams to users exactly as before, while Bitfab records a clean, replayable summary of the turn (text, token usage, tool calls) as the trace output.In the TypeScript SDK, pass finalize to withSpan and use the built-in finalizers.aiSdk helper for the Vercel AI SDK:finalizers.openai_chunks or finalizers.anthropic_events. And when you run setup on a streaming endpoint, Bitfab now guides you straight to this instead of asking you to refactor.Experiments stay linked to their dataset
When you benchmark or run an experiment against a dataset, the run is now durably attached to that dataset, so it always appears on the dataset’s experiments page, even when the underlying traces can’t be matched back by lineage. Before, the link was inferred from shared traces and could be missed.datasetId option when replaying
The TypeScript, Python, and Ruby SDKs’ replay() now takes a datasetId / dataset_id option that attributes the run to a dataset. Passed on its own, it replays exactly that dataset’s traces, so you no longer need to look up and pass the trace IDs yourself.Point Bitfab setup at a specific workflow
When you set up Bitfab tracing, the plugin now asks how you want to find what to instrument: let it scan your codebase for AI workflows, or point it straight at a specific file, function, or directory. Choose the targeted option when you already know what you want traced and want to skip the full scan. Available in the Claude, Cursor, and Codex plugins.Faster Bitfab assistant runs
The Bitfab assistant no longer pauses between steps while it reports progress to the Studio sidebar. Those activity updates are now sent in the background, so the assistant keeps moving instead of waiting on the network. The difference is most noticeable during longer flows like dataset building and experiments.Dashboard
Studio follows your work across organizations
Opening a Studio link for a dataset, trace plan, trace, test run, or experiment that lives in a different organization than your session started in now switches you into the right organization and loads the page, instead of showing a not-found error. If you belong to only one organization, nothing changes. When you are viewing a page in an organization other than the one your coding agent is connected to, a small indicator in the Studio header makes that clear.Sign-in returns you to where you were headed
Following a link to a specific trace, trace plan, or other page while signed out now takes you to that exact page after you sign in or sign up, instead of dropping you on your default traces view. This works for both email and Google sign-in.DashboardTypeScript SDKPython SDKPlugins
TypeScript SDK v0.19.1, Python SDK v0.19.1, Plugins v0.8.10
Uncached token basis in the experiments cost view
The experiments cost lens can now show “uncached” tokens (input minus cached reads, plus output), so you can see the tokens you actually pay full price for instead of the cheap cache reads. With the token lens on, switch between All and Uncached from the “Token count” control in the experiments header, and each trace’s cost trend recolors so the two views read apart at a glance. This surfaces cost regressions that prompt caching would otherwise hide behind a flat total.Claude Agent SDK reports the full prompt size in inputTokens
The Claude Agent SDK handler (TypeScript and Python) now folds cache reads and cache creation intoinputTokens, so it reports the full prompt size, consistent with the LangGraph integration. The cached portion is still reported separately as cacheReadTokens. If you read inputTokens directly for cache-heavy calls, expect a larger value than before.TypeScript SDKPython SDKRuby SDKGo SDKPlugins
TypeScript SDK v0.19.0, Python SDK v0.19.0, Ruby SDK v0.18.0, Go SDK v0.11.0, Plugins v0.8.9
Spans never silently drop on non-serializable inputs
All four SDKs now keep a span even when an input or output can’t be JSON-encoded: the offending value is replaced with a placeholder and a warning flags that the trace may not be replayable, instead of the whole span being dropped. The LangGraph, OpenAI Agents, and Claude Agent SDK handlers, along with the span decorator, also capture nested values more completely, so handler-instrumented traces stay replayable.Dashboard
Cleared the phantom “Grading” status on trace rows
Trace rows no longer show a spinning “Grading…” status when your project has no graders configured. Previously every incoming trace briefly displayed a grading indicator that never resolved. Traces that do have graders are unaffected and still show their pass and fail scores.Cached tokens in the per-span token breakdown
When the experiments token view is on, each span’s token breakdown now shows cached input tokens alongside input and output, so you can see how much of a span’s prompt was read from cache rather than processed fresh.Token usage in experiment trace results
Theget_experiment_traces tool now reports token usage (input, output, cached, and total) for each replay trace and the original it is compared against. Your coding agent can use this to reason about cost and cache-read changes between runs, not just the pass or fail verdict.Dashboard
Before/after comparison for replays with an unlabeled original
In the experiments view, a replay whose original trace was never labeled now shows a neutral “Unlabeled” badge alongside its own Pass or Fail result and opens the same before/after comparison as a scored trace, instead of hiding the comparison entirely. Open the replay to toggle between the original and updated runs and see the change in context, even while the replay is still being evaluated.DashboardTypeScript SDKPython SDKRuby SDK
TypeScript SDK v0.18.2, Python SDK v0.18.2, Ruby SDK v0.17.1
Replay tracks whether the database snapshot was actually used
When you replay traces against a historical database branch, each replayed trace now records whether a branch was provisioned for it and whether your code actually read the branch connection URL. A replay that checksenv.active but never reads env.databaseUrl (env.database_url in Python and Ruby) is recorded as not having used the branch, which catches the common silent failure where a connection pool created at module import sends the replay to your live database instead. Tracking is automatic in the TypeScript (0.18.2), Python (0.18.2), and Ruby (0.17.1) SDKs with no code changes; the Database Branching docs describe the three states a replayed trace can record.Dashboard
See your organization’s usage
A new Usage page (in the user menu) shows how many traces your organization has ingested and how much data they carry over time. Switch between daily, weekly, and monthly views, pick a date range, and read totals with trend indicators at the top; a usage-by-period table breaks it down further and exports to CSV. Dates and time buckets follow your local time zone.Ask Bitfab to cut token usage
The Bitfab assistant now responds to token and cost reduction requests. Tell your coding agent something like “use Bitfab to reduce token usage on one of my datasets” and it enters the experiment flow: it changes prompts or code, replays your labeled dataset, and checks that token usage drops without hurting the pass rate. The experiments view shows original vs. replay token totals next to each verdict, so you can see the savings as results stream in.Dashboard
Skipped replay traces are marked done, not left spinning
When you skip a replay trace during review, the experiments view now shows it with a “Skipped” badge instead of leaving the row spinning as if it were still being evaluated. The run summary reads “9/11 passing · 1 skipped” so a skipped trace is clearly counted as intentionally unscored, rather than dropped from the totals or mistaken for a result that is still loading.Dashboard
Replay environments branch your organization’s own database copy
When replaying with aReplayEnvironment, each replayed item now branches your organization’s own managed database copy (the one provisioned when you connect a database in the dashboard) at the trace’s capture time, so replays read your data as it was, never anyone else’s. Replay branches are cleaned up automatically after each run, with a background sweep catching anything an interrupted run leaves behind.Token cost in experiments and the trace viewer
When an experiment is about reducing token usage, you can now see per-trace and per-span token counts alongside pass/fail, so you can tell whether a change actually cut cost as the results stream in. It shows up in the experiments comparison and the trace viewer, with the token figures styled distinctly from the green/red verdict so cost reads as a measurement, not another pass/fail signal. The Bitfab assistant turns it on automatically when an iteration is framed around cutting tokens or cost.Guided setup for database branching in replay
A new/bitfab:setup db-branching flow walks you through replaying traces against your database as it was at trace time, instead of today’s data (TypeScript, Python, and Ruby). Connect a Postgres database once in the dashboard, and the flow polls until the branchable copy is ready, then wires a ReplayEnvironment into your replay scripts so each replayed item reads from a per-trace branch. A new connection-status check lets the flow tell you exactly when your database is connected and provisioned.Dashboard
Reliable database connection setup
Connecting a source database for trace replay now completes reliably regardless of database size. Setup runs in the background in resumable stages and the dashboard’s connection status tracks it as it progresses, instead of timing out partway through on large databases. Failures (an unreachable database, an invalid connection string) now surface promptly as a failed connection with the underlying reason, rather than leaving the status stuck on checking.Full token-usage capture for LangChain and LangGraph
LLM spans traced through the LangChain/LangGraph callback handler now record token usage from modern LangChain releases, which put counts on each message’susage_metadata rather than the legacy llm_output.token_usage location. Streaming agents are covered too: usage is read from the final aggregated chunk. Provider-native shapes from OpenAI, Anthropic, and Google are recognized as fallbacks, and the legacy location keeps working.Cached prompt tokens are now reported as cachedInputTokens on the span, and Anthropic input counts include cache reads so they reflect the true prompt size (heavily cached agents will see input counts go up; that is the previous under-count being fixed). Only provider-reported numbers are recorded, nothing is estimated. For OpenAI streaming, enable stream_usage=True (Python) or stream_options: {"include_usage": true} (TypeScript) so usage arrives on the stream.Replay for handler-instrumented workflows
Workflows traced through a framework handler (LangGraph, LangChain, Claude Agent SDK, OpenAI Agents) can now be replayed, even though they have no decorated root function in your code. Pass the handler’s trace function key plus any plain callable toreplay(), and the SDK wraps it internally so every replayed run records a trace tied to the experiment:bitfab.replay("my-agent", fn, options), where plain callables are wrapped automatically. The setup and assistant plugin skills now offer handler instrumentation as a first-class option for workflows whose entry points take live objects (database handles, billing callbacks), and write the matching replay script for you. See “Replaying handler-instrumented functions” in the Python and TypeScript SDK docs.Dashboard
Fixed database connector provisioning in production
Fixed a bug where connecting a database for trace snapshots failed in production before provisioning could start. Database connector setup from the dashboard now completes as expected.More reliable Studio sessions
Studio now keeps a single, durable connection open for the whole assistant session. Moving between pages (dataset review, experiments, trace plans) reuses the open Studio window instead of opening extra background connections, so signals like marking a dataset done or ending a session are no longer missed. The live Studio updates in the assistant flow are dependable from start to finish, even when a session is reused from an earlier run.Replay a trace against its historical database state
The Python and Ruby SDKs can now replay a recorded trace against the database as it was when the trace ran. Pass aReplayEnvironment to replay() and read its database URL inside your function; Bitfab resolves a per-trace database branch for each item and releases it when the item finishes. This brings Python and Ruby to parity with the TypeScript SDK.Trace LangChain by its own name
The Bitfab callback handler has always traced plain LangChain chains as well as LangGraph graphs (both share the same callback system), but every entry point was named after LangGraph. The SDKs now expose LangChain-named aliases that return the identical handler:getLangChainCallbackHandler() in TypeScript and get_langchain_callback_handler() in Python, with the class also exported as BitfabLangChainCallbackHandler.Keep your app’s API key in sync when you switch organizations
When you switch organizations with the Bitfab plugin, it now offers to update your project’s localBITFAB_API_KEY as well. Previously the switch only repointed the plugin, so traces your own code sent kept landing in the old organization until you updated the key by hand. After you run /bitfab:setup switch-org (or /bitfab-setup switch-org in Cursor, $bitfab:setup switch-org in Codex), the agent finds every .env file that defines the key and, with your go-ahead, updates them in place.Dashboard
Connection fields on the Integrations page no longer clear while you type
Entering a database connection string or integration secret on the Integrations page now stays put as you type. Browser password managers were treating these masked fields as login passwords and overwriting them mid-entry; they’re now opted out, so your input holds.Switch organizations from your coding agent
You can now switch which Bitfab organization a plugin reads and writes without leaving your editor. Run/bitfab:setup switch-org (or /bitfab-setup switch-org in Cursor, $bitfab:setup switch-org in Codex): the agent lists the organizations you belong to, switches to the one you pick, and swaps the plugin’s API key to the new org. The agent can also call the new list_organizations tool on its own to check which org it is currently pointed at. Your already-open browser tabs keep showing their current org until the next time the plugin opens Studio.Dashboard
Faster traces list
The traces list now loads significantly faster for functions with a large number of traces. Opening a function’s traces no longer waits on a full count of every matching trace before showing results, so the page appears as soon as the traces are ready.Sign in without a duplicate Studio window
When you run a Bitfab plugin command while signed out and a Studio window is already open, the plugin now reuses that window to sign you in instead of opening a second one. There’s only ever one Studio window now, where before a fresh window could appear and leave the original orphaned. Running login while you’re already signed in is also a no-op, and you can re-authenticate on demand by running login with--force.See a dataset’s experiment history in Studio
You can now open the experiments page scoped to a single dataset. Ask the Bitfab assistant to “show experiments for a dataset” and Studio lists every experiment that replayed one of the dataset’s traces, giving you that dataset’s full run history in one place. Previously the experiments page opened only for specific test runs or an experiment group.More reliable Studio session cleanup
Studio sessions now close themselves cleanly when an assistant or setup run finishes: the browser tab closes, background processes stop, and nothing is left running even if the window refuses to close. Recovering from an unreachable Studio window (“Open a new Studio”) also cleans up the old window and its background process instead of leaving them behind.Dashboard
Accurate error indicators in the trace viewer
Spans with no recorded errors no longer show a false “Error detected” badge or an “Execution Error” section in the trace viewer. Error detection now correctly ignores empty error data, and a related fix ensures a real execution error can no longer be hidden by an empty error list.Live updates arrive reliably on the hosted dashboard
Pages that update in real time, such as dataset review, trace lists, and experiment results, could miss updates on the hosted dashboard and only show new data after a manual refresh. Event delivery now completes reliably, so new traces, labels, template changes, and experiment results appear the moment they happen.Replay a single trace to check your fix
The assistant has a newreplay mode for the quickest version of the improvement loop: you already made a fix and just want to know whether one specific trace passes now. Run /bitfab:assistant replay <function-key> <trace-id> (or simply ask “did my fix work on <trace-id>?”) and the agent finds your replay script, re-runs that one trace through your current code, and reports a pass/fail verdict in chat. It skips everything heavier: no browser, no dataset, no labeling, and nothing is persisted, so it’s safe to run as often as you like while iterating.TypeScript SDKPython SDKRuby SDKPlugins
TypeScript SDK v0.16.1, Python SDK v0.16.1, Ruby SDK v0.16.1, Plugins v0.7.9
Replay accepts limit and trace IDs together
Passing bothlimit and traceIds (trace_ids in Python and Ruby) to replay() no longer throws. The SDK now logs a warning and ignores limit, since an explicit trace ID list already determines how many traces replay. This applies to the TypeScript, Python, and Ruby SDKs, so replay scripts that forward both flags keep working instead of crashing.TypeScript SDKPython SDKRuby SDKPlugins
TypeScript SDK v0.16.0, Python SDK v0.16.0, Ruby SDK v0.16.0, Plugins v0.7.7
Replay traces whose function signature changed
When you rename, reorder, or restructure a traced function’s arguments, replay can no longer feed it the inputs recorded against the old shape.replay() in the TypeScript, Python, and Ruby SDKs now takes an adaptInputs (adapt_inputs in Python and Ruby) hook that reshapes each trace’s recorded inputs onto the current signature, so older traces keep running.The assistant recovers replays broken by signature changes
When you iterate on a traced function in your coding agent and a replay fails because the signature drifted since the traces were captured, the Bitfab assistant now recognizes the mismatch instead of treating it as an environment error, and helps you write a small committed input adapter so those traces rejoin the run.Spot traces that can’t be replayed
The trace viewer now flags traces that won’t replay against your current code, either because they captured no top-level span or because their recorded inputs no longer fit the function’s current signature. A “Can’t replay” badge appears on dataset rows and in the trace detail view, so you can see at a glance which traces a replay will actually cover before you run it.When your coding agent opens a dataset, the Bitfab plugins pass along your function’s current input shape, and the check runs live against it. Nothing is stored, so the badge can’t go stale.TypeScript SDKPython SDKRuby SDKDashboardPlugins
TypeScript SDK v0.15.0, Python SDK v0.15.0, Ruby SDK v0.15.0, Plugins v0.7.5
Replay results now persist reliably
replay() in the TypeScript, Python, and Ruby SDKs now waits for each replayed item’s trace to be fully persisted before completing the test run. item.traceId (trace_id in Python and Ruby) is a real server trace ID you can use immediately. Previously, a race could leave every trace ID null and the experiments page empty, even though the replay appeared to succeed.Failures are no longer silent. If none of the replayed items’ traces reached the server (for example, the replayed function isn’t instrumented), replay() raises an error explaining why. If only some items fail to persist, those items return a null trace ID with a logged error and the rest of the run comes back intact, so one bad trace costs you one data point instead of the whole run.Clearer skill routing in plugin flows
The setup, assistant, and update skills now state exactly where each choice leads (“Update all → step 7”, “Skip → stop”), so coding agents follow multi-step flows more reliably instead of inferring the wiring from prose. The update skill on Claude Code also runs as chained sub-skills: each phase hands off directly to the next with the invocation mode attached, removing a class of lost-context routing mistakes in long sessions.Live dataset review in Studio
Dataset review now always happens on the dataset’s own page in Studio, which updates in real time as your coding agent adds traces and applies labels. Previously the agent could leave you on a function-level review page that only showed new activity after a manual refresh.That older function-level page now redirects to the function’s most recent dataset, so existing links and older plugin versions keep working.TypeScript SDKPython SDKRuby SDKDashboardPlugins
TypeScript SDK v0.14.0, Python SDK v0.14.0, Ruby SDK v0.14.0, Plugins v0.7.2
Replay by trace IDs no longer truncates
Replaying specific traces by ID previously capped the list at the defaultlimit, silently dropping the rest of your selection: 12 IDs in could mean only 5 replayed, skewing experiment results without warning. An explicit ID list now always replays every trace in it (up to 100).limit and traceIds are now mutually exclusive: limit means “replay my last N traces”, and an ID list speaks for itself. Passing both raises a clear error instead of guessing.trace_ids semantics in Python and Ruby. Older SDK versions get the core fix server-side: explicit ID lists are no longer truncated by a default limit.No more duplicate Studio windows
Your coding agent now keeps exact track of its Studio window. Ending a session no longer makes the next Studio command open a second window while the old one lingers: the existing window is reused, and the agent only forgets a window once the browser confirms it actually closed. Refreshing the Studio page mid-session is also safe; the connection re-establishes itself instead of being mistaken for a close.Commands that reconnect to an already-open Studio window now react only to what happens after they connect, so a previously ended session or an earlier run’s activity can no longer end a new command prematurely. If a Studio window disappears without a trace (for example the browser quit entirely), the agent detects that it is unreachable and offers to reopen instead of guessing.Watch benchmark runs live in Studio
The assistant skill’sbenchmark mode is terminal-only by default, but you can now add the studio keyword (for example, benchmark <function> studio, or just ask in natural language to “open studio”) to open Studio’s experiments page and watch each trace’s pass/fail verdict stream in as the replay runs. The default stays terminal-only, so existing benchmark runs are unchanged unless you opt in.Dashboard
Traces are flagged errored only when your code fails
A trace is now marked as errored only when your traced code throws, not when the Bitfab SDK hits a serialization or ingestion error while recording the trace. The error indicator in the trace list now reflects failures in your own functions, so SDK-side noise no longer surfaces as a failed trace.Dashboard
Errored traces open on the failing span
When you open a trace that recorded an error, the trace viewer now jumps straight to the first span that failed instead of starting on the trace root. You land on the error and its message right away, without scanning the span tree for the red marker. Traces without errors open exactly as before, and any span you deep-link to still takes precedence.Dashboard
Errored spans highlighted in the trace viewer
Spans that recorded an error are now flagged directly in the trace viewer. The span tree marks failed spans in red, and the span header shows an Error tag with the error message on hover, so you can spot failures in a trace without opening each span.Replay keeps going when an individual trace fails
The Python and Ruby SDKs now isolate per-trace errors duringreplay(): if one historical trace fails to load or its function raises, that result is marked with an error and the rest of the run still completes, instead of the whole replay aborting. This matches the TypeScript SDK’s behavior, so a single bad trace no longer costs you the entire run.Dashboard
Copy buttons in custom span templates
Custom span templates can now drop in a clipboard icon button next to any field value with{{ value | copyButton | safe }}. Pass a string for the tooltip and accessible label, e.g. {{ span.id | copyButton("Copy span id") | safe }}. The button copies the value (objects and arrays are pretty-printed as JSON), flashes “Copied”, and works inside the template’s isolated shadow DOM without any extra JavaScript or CSS in the template.More reliable Studio sign-in
When your coding agent opens Bitfab Studio and you sign in, the Studio tab now connects reliably on the first try. Previously a timing issue could leave a freshly opened session showing “Could not connect to this session” until you re-ran the command; signing in now hands off cleanly to the live session every time.TypeScript SDKPython SDKRuby SDKGo SDKPluginsDashboard
TypeScript SDK v0.13.8, Python SDK v0.13.4, Ruby SDK v0.12.4, Go SDK v0.10.2, Plugins v0.6.74
Error source classification on spans
Span errors now carry an explicit source tag so you can tell whether an error came from your code or from the SDK itself. When your traced function throws, the error is recorded withsource: "code". SDK-internal failures (like serialization errors) are tagged source: "sdk". Both types appear in the unified errors field on the span, replacing the previous split between span_data.error and the errors column.All four SDKs support this: set error_source: "code" automatically when a traced function fails.Dashboard
Connect a database for per-trace snapshots
You can now connect your Postgres database from the new Database page so Bitfab can take an isolated snapshot per trace at replay time. Paste a connection string and activate: replays run against a fresh branch of your database, so they never touch or slow down production. The page shows live status (activating, active, or failed, with a support contact if it can’t reach your database), and you can deactivate at any time from a confirmation dialog.More reliable trace reads under heavy use
Reading several traces at once through the Bitfab plugin (theread_traces tool) no longer returns intermittent errors when many requests are in flight at the same time. Trace reads and agent-label updates are now more efficient, so larger reads stay fast and reliable.Quickly add a trace to a dataset
The/bitfab:assistant command has a new lightweight add-trace mode that attaches one or more existing traces to a dataset and stops, without the full label-and-iterate flow. Run /bitfab:assistant add-trace <trace-id> (the function key is inferred from the trace) or just ask your coding agent to “add this trace to a dataset”. It picks or creates the right dataset for you, and if you point it at several traces it makes sure they all belong to the same function before attaching.Diagnose your tracing setup with /bitfab:setup inspect
A new inspect mode checks whether your Bitfab tracing is healthy: whether you’re authenticated, what’s instrumented in this repo, whether the plugin and SDK are up to date, whether your replay scripts cover every trace function, and whether traces are actually arriving. It then walks the available fixes one at a time, asking before each change. Run /bitfab:setup inspect, or just ask your coding agent something like “why aren’t my traces showing up?”.Get oriented with /bitfab:setup explain
A new read-only explain mode prints a quick overview of what Bitfab is and what each setup mode does, without authenticating or scanning your code. Run /bitfab:setup explain or ask “what is Bitfab?”.Smarter Studio window reuse
The Bitfab plugin now reuses your existing Studio window whenever it’s still open, instead of risking a duplicate window or a stale “not responding” prompt. Close the Studio tab and the next action opens a fresh window right away; only a window that’s genuinely unreachable, like after a crash or your machine sleeping, will ask whether to retry or open a new one.Clearer Studio connection status
Studio’s connection indicator now tells you exactly what’s happening: “Studio connected” when your agent is live, “Awaiting agent” while it’s away, and “Studio disconnected” if the browser loses its live connection to the session (it reconnects automatically). Reconnecting to a session you already have open now reuses that Studio tab instead of opening a second window.Benchmark a dataset against your current code
Run/bitfab:assistant benchmark <key> to replay a labeled dataset against your current code without changing anything, then read a pass/fail scorecard that shows which traces still pass, still fail, regressed, or were fixed. Use it to measure where your function stands right now, as a regression baseline or a quick check after unrelated work, instead of starting an experiment loop. You can also just say “benchmark my dataset” in plain language and the assistant routes there.More reliable Studio sessions
Studio no longer shows a stray “agent disconnected” popup on the session-complete page. When your agent disconnects mid-session, the reconnect prompt now rejoins your existing Studio session instead of starting a new one. And if you run more than one coding agent in the same project, each keeps its own Studio session instead of overwriting the other’s.Sign-in URL shown when the browser doesn’t open
When you sign in to Bitfab from your coding agent and the browser doesn’t open automatically, the login flow now prints the sign-in URL so you can open it manually. Previously it told you to visit the URL without showing one.Simplified plugin login
The setup login flow now uses a single authentication method that works everywhere, including SSH sessions, containers, and cloud IDEs. The separatelogin headless mode has been removed since the standard login already handles these environments automatically via its server-polled channel.Seamless login in the assistant flow
Running/bitfab:assistant without being authenticated now logs you in inline through Studio instead of stopping the flow. Previously, unauthenticated users were told to run a separate login command first, breaking the workflow. The assistant now opens Studio’s sign-in page directly and continues automatically once you’ve signed in.Plugin login no longer hangs for CLI users
Thebitfab init login flow now completes reliably instead of hanging after sign-in. Previously, the published CLI used a query parameter the close page didn’t recognize, so the authentication callback never fired and the CLI waited indefinitely. The close page also now displays “Login complete” instead of the generic “Session Complete” message.Experiment results show label annotations
The experiments page now displays the label annotation for each trace instead of the raw function output. This matches how the dataset page already renders traces and makes it easier to scan experiment results for what passed, what failed, and why. Annotations from both human reviewers and the agent’s automated labeling are shown.Replay script upgrades no longer interrupt the assistant flow
When the assistant detects that your replay script needs an upgrade (missing code-change or experiment-group support), it now edits the script directly instead of launching a separate setup flow. Previously, this would break the assistant’s continuity and drop you to an empty prompt. The upgrade happens inline and the experiment flow continues automatically.Dashboard
Faster disconnect detection in Studio
When an agent closes its Studio session, the browser now shows “disconnected” within milliseconds instead of up to 60 seconds. Previously, the connection status indicator could display “Agent connected” long after the agent process had exited because the server-side heartbeat lingered in cache.Studio redirects to sign-in instead of showing a session error
Opening a Studio page without being signed in now redirects you to the sign-in page instead of showing a “Could not connect to this session” error. After signing in, you’re returned to the page you originally requested with the session intact.Studio commands now open login instead of erroring
Plugin commands that open Studio pages (experiments, trace plans, datasets, template previews) no longer fail with “Not authenticated” when you haven’t logged in yet. Instead, they open Studio directly and redirect you to the sign-in page. After you sign in, you land on the page the command originally requested. For interactive commands like trace plan confirmation and dataset review, the plugin saves your credentials automatically so the bidirectional event channel works normally after login.Automatic replay script capability detection
The assistant now checks whether your replay script supports the latest experiment features before running experiments. If your script is missing support for code diffs, experiment groups, or trace ID tracking, the assistant offers to upgrade your SDK and regenerate the script in place. You can also choose to continue without the missing features. This replaces the previous behavior where outdated scripts would silently skip features or produce incomplete experiment results.Dataset mode now continues through failure diagnosis and experiments
Fixed a bug where the assistant’s dataset mode could stop after building the dataset instead of continuing to diagnose failures and run experiments. The flow’s internal instructions contradicted its routing in three places, which could cause the agent to exit early. Dataset mode now reliably progresses through the full pipeline: build dataset, diagnose failures, iterate with experiments, and wrap up.Dashboard
Clear error messages when Studio can’t connect to a session
When Studio fails to connect to an agent session or switch to the correct organization, it now shows a clear error message instead of silently loading in the wrong context. This prevents the confusing state where traces appear missing because Studio was looking in a different organization than the one your plugin authenticated against.DashboardPluginsTypeScript SDKPython SDKRuby SDK
Plugins v0.6.50, TypeScript SDK v0.13.6, Python SDK v0.13.3, Ruby SDK v0.12.3
Live experiment streaming
The experiments page now streams results in real time as replays complete. When the assistant runs experiments, it opens the experiments viewer before the first replay starts and new results appear automatically via server-sent events as each test run finishes. Previously, the experiments page only opened after all replays completed.All three SDKs now accept anexperimentGroupId parameter on replay() that groups multiple test runs into a single experiment batch:?experimentGroupId= as a query parameter, and falls back to ?testRunIds= for replay scripts that haven’t been updated yet.Inline template editing during labeling
You can now edit trace view templates in chat while labeling traces, without leaving the dataset review page. When you ask the assistant to change how a span type renders (e.g. “edit the LLM template”), it reads and updates the template inline using MCP tools, and the dataset page re-renders automatically. Previously, template editing required invoking the setup flow, which navigated Studio away from the dataset and broke the labeling session.Dashboard
Auto-update trace plan in Studio
When you modify your span capture setup and create a new trace plan, Studio now automatically navigates to the updated plan. Previously, you had to manually refresh or re-navigate to see the latest version after making changes to the capture configuration.Investigate mode continues through the full pipeline
When you run the assistant in investigate mode, the flow now continues through diagnosis and experiments after building a dataset, matching how dataset mode already works. Previously, investigate mode stopped after dataset building, requiring you to restart in experiment mode to iterate on fixes. All assistant modes now serve as entry points into the same pipeline, converging at wrap-up regardless of where they start.Graceful Studio close on flow exit
When the assistant finishes a flow (wrap-up, early stop, or sub-mode completion like dataset-only), the Studio tab now closes gracefully instead of lingering in a disconnected state. The agent navigates to a close route that lets the Studio clean up before the background process is terminated.Better error messages when agents pass wrong CLI arguments
Plugin commands now validate arguments against a declarative schema before executing, catching common agent mistakes like invented--flag syntax, missing arguments, or invalid UUIDs. When a command receives bad input, it prints a usage string showing the expected arguments and a clear error message, so the agent can self-correct on the next attempt.Dashboard
Reconnect guidance when Studio loses agent connection
When your coding agent disconnects from Studio, a popup now appears after 30 seconds with a copiable prompt you can paste into your agent to reconnect. The popup includes your session ID so the agent can rejoin the same session. If you dismiss it, the popup reappears with increasing intervals, and it auto-dismisses if the agent reconnects on its own.Dataset mode continues to diagnosis and experiments
When you run/bitfab:assistant dataset, the flow now continues past labeling into Phase 4 (diagnosis and experiments), matching the behavior of the full /bitfab:assistant flow. Previously, dataset mode stopped after labeling, requiring you to restart in the default mode to run experiments on the same dataset.Live experiment verdicts in Studio
When your replay script returns trace IDs (requires SDK v0.13.5+), the assistant now opens the experiments page in Studio before running evaluations, so you can watch pass/fail verdicts populate in real time. If your SDK predates trace ID support, the assistant prompts you to update and falls back to showing evaluation results as text in the agent.Code-change diffs in the experiment viewer
When the assistant runs experiments, it now captures before-and-after file snapshots for every edit and attaches them to the replay. The experiment viewer can then display the literal code change alongside pass/fail results, so you can see exactly what was tried in each iteration. Existing replay scripts that predate this feature continue to work; the assistant detects whether the script supports the new--code-change flag and gracefully skips the metadata if it doesn’t.TypeScript SDKPython SDKRuby SDKDashboard
TypeScript SDK v0.13.5, Python SDK v0.13.2, Ruby SDK v0.12.2, Plugins v0.6.42
Replay results include trace IDs
Each replay result item now includes atraceId (or trace_id in Python/Ruby) that links directly to the server-side trace created during replay. Previously, matching a replay result back to its trace in the dashboard required heuristics based on input similarity. Now you can navigate straight to the trace.Experiments distinguish errored traces from pending
When a replay trace fails during execution (before it can be graded), the experiments page now shows it as “errored” instead of lumping it in with pending traces. Errored traces appear with a red “Error” badge, a red row tint, and a dedicated segment in the progress bar. The pass-rate pill also now shows context-appropriate states: “N errored” when all traces errored, “N pending” with a spinner when grading is in progress, and “X/Y so far” for partially graded runs.Assistant reviews existing dataset traces before searching for new ones
When you pick a dataset that already has traces, the assistant now goes straight to the review page instead of asking how to source new candidates. Previously, datasets with unlabeled (but present) traces were treated like empty datasets, which skipped past the traces you already had. Empty datasets still get the “what kind of traces should I find?” prompt as before.Dashboard
Experiments page updates after replay completion
The experiments page now refreshes automatically when a replay finishes and as labels are applied. Previously, completing a replay didn’t trigger an update, so the page could appear empty until manually refreshed. Summary counts and individual trace labels now appear as soon as they’re available.Studio session recovery after agent interruptions
The plugin assistant now automatically reconnects to an existing Studio browser session when the background polling process is interrupted (for example, by a long conversation triggering context compaction). Previously, losing the background process meant the assistant would open a duplicate Studio window. Now it resumes the existing session seamlessly, keeping the same browser tab and session state.Experiment and dataset tools for the assistant flow
The plugin assistant can now work with experiments and datasets directly. Two new MCP tools,list_experiments and get_experiment_traces, let the assistant list recent experiments for a function and drill into individual trace verdicts (fixed, regressed, still-passing, still-failing). The search_traces tool also now accepts testRunId and datasetId parameters, so you can scope trace searches to a specific experiment run or dataset without manual filtering.Fixed investigate mode opening a broken Studio page
Running/bitfab:assistant investigate <key> no longer opens Studio to a non-existent page. The investigate mode now lands on the Studio root, which loads correctly. The investigation itself (trace reading, code exploration, findings summary) was unaffected since it runs via tool calls, not Studio navigation.Trace plan links now open correctly in Studio
Clicking a trace plan link from the plugin assistant now opens the trace plan inside the Studio session. Previously, the link pointed to a standalone route outside of Studio, which bypassed the authenticated Studio context.Dashboard
Experiment results stream in real time
Running experiments on a dataset no longer blocks until every test finishes. The experiments page opens immediately and results stream in trace by trace: the progress bar fills, pass/fail counts update, and trace rows appear as each test completes. If background execution fails, the run is marked as failed and the page updates accordingly instead of getting stuck on “pending.”Dashboard
Trace list loads reliably for high-volume functions
The traces page now loads correctly on the first visit for functions with very high trace creation rates. Previously, real-time update events could interfere with the initial page load, causing the trace list to appear empty until you navigated away and back.Reliable Studio navigation during assistant sessions
Studio navigations that include query parameters (such as opening the experiments page with specific test run IDs) no longer time out with “not responding.” The same fix also ensures that navigating to the same page with different parameters is recognized correctly, so the assistant flow proceeds without interruption.Dashboard
Dataset traces now stream in real-time in Studio
Traces added to a dataset while viewing it in Studio now appear immediately without a page refresh. A recent migration to Studio’s route tree accidentally dropped the real-time event connection, so newly added traces were invisible until you reloaded.Fixed experiment mode navigation
The plugin assistant’s experiment mode now correctly opens the experiments page. Previously, it attempted to navigate to a non-existent per-function experiments route.Dashboard
Studio reconnects automatically after sleep
Studio connections now recover automatically when your laptop wakes from sleep. Previously, closing your lid and reopening could leave the session unresponsive until you manually refreshed the page.Dashboard
Dataset page shows errors instead of misleading empty state
The dataset review page now displays a clear error message when traces fail to load, instead of incorrectly showing “No traces yet.” This helps you quickly identify loading failures, such as viewing a dataset while signed into the wrong organization.Per-trace DB branching for replay (alpha)
Replay can now run against the database state at the moment a trace was recorded, not your current production database. This raises fidelity for agents whose behavior depends on stored state, like a refund decision that read a since-cancelled order or a retrieval agent that saw last week’s index. Available in the TypeScript SDK; backed by Neon preview branches on the Bitfab service.Wire it up:Dashboard
Studio sessions survive page refreshes
Refreshing the Studio tab no longer kills your coding agent’s session. Previously, a browser refresh was indistinguishable from closing the tab, so the plugin would immediately tear down the connection. Now the plugin waits up to 10 seconds for the page to reload before ending the session, so you can refresh freely without interrupting your workflow.Plugin login respects project-local credential isolation
When your project has a.bitfab/credentials.local.json file (used to isolate credentials per project), logging in now writes the new API key to that file instead of the global credentials store. Previously, login always wrote to ~/.config/bitfab/credentials.json, which meant the project-local file stayed empty and the plugin fell back to the global key, defeating the isolation.Studio auto-switches org to match the plugin session
When your browser’s active org differs from the org bound to the plugin’s API key, Studio now detects the mismatch and automatically switches to the correct org on load. This fixes the “Awaiting agent” stuck state that could occur when you belong to multiple organizations and your browser happened to be on a different one than your plugin.If you’re not a member of the session’s org, the plugin now cleanly aborts and tells you why, instead of hanging indefinitely.Dashboard
Studio agent connection recovers after laptop sleep
Studio now correctly restores the agent connection indicator after your laptop sleeps and wakes. Previously, the “agent disconnected” banner could get stuck even though the agent had successfully reconnected. The fix ensures heartbeat tracking refreshes on every poll and that the browser verifies the actual connection state when resuming after a gap.TypeScript SDK is now @bitfab/sdk
The TypeScript SDK package has moved from bitfab to @bitfab/sdk. New installs should use the scoped name:bitfab package continues to work but now prints a deprecation warning on import. Running the plugin’s update command (/bitfab:update) detects the legacy package and warns you to switch, even if the version number is current.Studio is now the single browser surface for all plugin flows
Every plugin CLI flow (login, trace plan confirmation, dataset review, template preview) now opens inside Studio instead of launching a separate browser window. If Studio is already open, the plugin navigates it in place rather than opening a new window. This means fewer browser tabs, a consistent UI, and the ability to stay in one window while working with the assistant.Headless login is now available for environments where a browser can’t reach your terminal (SSH, cloud IDEs, CI). Visit/studio/auth/claude in any browser, sign in, copy the token, and paste it back into your coding agent.Closing Studio during a trace plan confirmation now cleanly cancels the operation instead of leaving the CLI in an error state.Automatic package rename in update flow
Running/bitfab:update now detects the legacy bitfab npm package and offers to switch it to @bitfab/sdk. The update flow removes the old package, installs the new one, and rewrites imports in your source files. If you’re already on @bitfab/sdk, nothing changes; the flow works as before.Studio URL guard and auth verification
Studio now always opens at the correct/studio path. Previously, certain launch conditions could cause the Studio window to open at the site root instead of the Studio interface. A path guard now normalizes the URL before the browser window opens.The plugin’s auth status check now verifies your API key against the current server. If you switch between servers (e.g., local development to production), the status command correctly reports that re-authentication is needed instead of showing a stale “authenticated” state.Live-streaming dataset pages in Studio
When the Studio assistant creates or picks a dataset, the dataset review page now opens immediately instead of waiting for all traces to be labeled and attached first. Traces appear on the page in real time as the agent finds, labels, and attaches them. Label changes on traces already in a dataset also update live, so you can watch rows move between the “Agent labeled,” “Labeled,” and “Unlabeled” sections without refreshing.If the page is empty while the agent is still working, a “Building your dataset” indicator shows that traces are on the way.Dashboard
Trace plan review page scrolls
The trace plan review page now scrolls when a plan has more captured nodes than fit on screen. Before, long plans clipped the Advanced selection toggle and any inline error messages below the fold; opening a plan with around 40 captured nodes now scrolls cleanly through the full call tree.TypeScript SDK available as @bitfab/sdk
The TypeScript SDK is now published under the scoped package name@bitfab/sdk in addition to the existing bitfab package. Both names resolve to the same code and will stay in sync on every release. If you prefer scoped package names for clarity in your package.json, you can switch your import at any time:import { Bitfab } from "bitfab" and import { Bitfab } from "@bitfab/sdk" work identically.TypeScript SDKPython SDKRuby SDKPlugins
TypeScript SDK v0.13.1, Python SDK v0.13.1, Ruby SDK v0.12.1, Plugins v0.6.14
SDK serialization hardening
Trace spans now ship reliably even when function inputs or outputs are difficult to serialize. Objects with circular references, oversized payloads (over 512 KB), or classes that throw during serialization no longer cause lost spans. Instead, the SDK replaces the problematic value with a descriptive<unserializable: ClassName (reason)> stub so the span still appears in your traces with full timing and metadata.This fix applies to the TypeScript, Python, and Ruby SDKs. No code changes are needed on your side; update to the latest SDK version to get the improvement automatically.Plugin can query Studio browser state
Plugins can now check whether the Studio browser tab is connected and which page is currently active via the newgetStudioState function. This lets the plugin make smarter decisions before navigating, for example skipping a navigation command when Studio is already on the target page, or surfacing a connection warning when the browser tab has been closed.Studio crash recovery and agent navigation guardrails
Studio now shows a recoverable error screen when a runtime error or unexpected crash occurs during a session. Instead of a blank page or a full 404, you see a “Try again” button that retries without losing your session context.Agent-initiated navigation is now validated against a known route whitelist. When an agent tries to navigate to an invalid or out-of-scope path, it receives an immediatenavigation-blocked event with a reason string instead of waiting for a 12-second timeout. This helps agents self-correct faster when a requested page doesn’t exist.Studio stays connected through long conversations
The Bitfab plugin now persists the link between your coding agent conversation and your Studio session. Previously, when a long conversation triggered context compaction, the agent lost track of which Studio window it had opened, requiring you to reopen Studio manually. Now the mapping is written to disk and recovered automatically after compaction, so Studio commands continue working seamlessly in extended sessions.Dashboard
Code change diffs in experiments
When an experiment replays traces against a code change, you can now view the exact diff that was tested. Click the file stats on any experiment card or the code-change pill in the trace detail header to open a side-by-side diff modal. The modal also shows how the dataset reacted overall (fixed, regressed, still passing, still failing) or, when opened from a single trace, whether that specific trace flipped.Dashboard
Fixed plugin login falling through to manual paste flow
Signing in to a Bitfab plugin (Claude Code, Cursor, or Codex) via the browser now reliably completes the automatic handoff back to your terminal. Previously, the login page could lose the callback parameters during a redirect, causing every login to fall through to the manual “copy and paste this token” flow even when the browser and terminal were on the same machine.Investigate a trace function with /bitfab:assistant
Run/bitfab:assistant investigate [<key>] to characterize an issue in a trace function without going through the full assistant flow. The agent reads recent traces and your code based on what you describe, then offers three follow-ups: stop with an in-chat summary, save a written report under .bitfab/analysis/, or hand off to dataset building when the findings include reproducible failures worth labeling. The function key is optional; when omitted, the agent picks it from your description or asks.Agent-initiated Studio session close
Agents can now programmatically end a Studio session when their work is complete. ThecloseStudio() helper sends a completion event with an optional message, and the browser automatically closes or shows a “Session Complete” screen with the agent’s message. This replaces the need for users to manually click “End session” when the agent is done.Trace plan confirmation lands inside Studio
The Bitfab plugin’s trace-plan confirmation page (where you review which spans your function will capture) now renders inside your existing Studio tab during/bitfab:setup instead of spawning a second browser window. Studio’s header and agent indicator stay visible while you decide; Confirm or Cancel keeps the tab open for the rest of the flow, no more orphan windows.If no Studio is running (you invoked /bitfab:setup outside an /bitfab:assistant session), the confirmation falls back to the standalone chromeless window as before.Annotate a closed trace from any process
The TypeScript and Python SDKs now expose a detachedclient.getTrace(id) handle that lets you add context, merge metadata, or set the session id on a trace after its root span has closed. The handle works from any process, thread, or agent that knows the trace id, with no shared in-memory state. Useful when a downstream worker or a forked AI agent needs to attach information to the original conversation’s trace.bitfab v0.13.0 for TypeScript and Python.Smarter Studio session management
The assistant flow now reuses an existing Studio session instead of opening a new browser window each time. If the Studio becomes unresponsive (tab closed, page crashed), the agent detects this within 12 seconds and offers options to refresh the tab or open a fresh session.Live activity progress in Studio
Studio now shows which phase the assistant is working on in real time. As the skill progresses through steps like identifying the trace function, building a dataset, or running experiments, the header displays the active phase name with a live elapsed timer. When one phase completes and the next begins, you see the previous phase’s duration before it transitions.If the agent disconnects or crashes mid-phase, the activity indicator automatically resets within 30 seconds instead of showing stale state indefinitely.Open a trace plan from inside /bitfab:assistant
Ask the assistant to “open the trace plan for X” (or “show me what’s captured”) and it now routes your open Studio tab to that function’s most recent trace plan in place. The Studio shell stays mounted around the plan, so your agent session, header, and connection indicator persist across the navigation, and no new browser tab pops up. The canonical /trace-plan/[id] URL still works as a standalone shareable link outside Studio.Dashboard
Click any trace while reviewing a dataset
Fixed a bug that blocked clicks while a trace detail was open. You can now switch traces or press Done without closing the open one first.Redesigned trace planner
The trace planner now leads with what you actually need to know: a validation summary at the top that calls out anything blocking replay (live writes inside captured spans, missing samples, disconnected roots), then a flow diagram of the captured spans and a sample-trace preview of how the recorded trace will look in the viewer. The legacy two-pane tree picker is still there, tucked behind an Advanced selection toggle for power-users. Confirm and Cancel still flow through the same Cmd+Enter / Esc handoff, so muscle memory carries over.Live agent connection indicator in Studio
Studio now shows a real-time connection status in the header. A green dot with “Agent connected” appears when the coding agent is actively polling, and transitions to a gray dot with “Awaiting agent” if the agent disconnects. The indicator updates instantly when the agent reconnects, with no page refresh needed.Mutual presence detection for plugins
The agent plugin now receivesbrowserConnected in its poll response, indicating whether a user has Studio open in the browser. This enables plugins to adapt their behavior based on whether someone is actively watching the session.Studio is now the default assistant mode
The/assistant skill now opens Studio automatically on every invocation. You no longer need to pass a studio argument to get the companion browser surface. Studio is always there, from start to finish.Studio opens directly at the relevant page
When you start in dataset or experiment mode (/assistant dataset <key> or /assistant experiment <key>), Studio now opens directly at that function’s datasets or experiments page instead of opening at the root and navigating after. This shaves a few seconds off each focused session and puts you in context immediately.Dashboard
Logout redirects to sign-in page
Signing out no longer lands on a blank page. You’re now redirected to the sign-in page, where you can immediately log back in or close the tab.Session log capture fix and standalone opt-in
Session log capture now works correctly after opting in during setup. A configuration mismatch previously caused the plugin to silently skip session capture even when you’d consented, so no session data was being collected. You can also now toggle session log capture on or off by running/bitfab:setup session-logs, a standalone mode that doesn’t require authentication.Dashboard
Trace viewer skips empty spans on open
Opening any trace now lands on the first span that has data instead of a blank trace root or an empty span. This applies across the dashboard: trace detail pages, the labeling panel, the experiments comparison view, the dataset detail panel, and the template preview studio. The hard template filter still hides non-matching spans; you just no longer have to scroll past empty ones to see meaningful content.Dashboard
Studio navigation events for coding agents
When you navigate between pages in Studio, the coding agent now receives real-time navigation events with the current path. This gives the agent immediate awareness of where you are in Studio, so it can tailor its responses and actions to the page you’re viewing without needing to ask.Studio sign-in stays within the Studio shell
When your coding agent opens Studio and you’re not signed in, you now see a branded sign-in page inside the Studio window instead of being redirected to the main Bitfab login. The session context persists across the sign-in flow, so the agent picks up exactly where it left off once you authenticate. The CLI receives real-timeauth-required and authenticated events, letting it wait for sign-in without polling.Accurate offline SDK update checks
The plugin’s session-start update check now always reports the correct latest SDK versions. Previously the baked version snapshot could lag behind by one release, causing the plugin to miss update notifications or report you were up to date when a newer SDK was available.Hill-climb from existing labels in /bitfab:assistant
When you start a new dataset for a function that already has validated labels, the assistant now offers a Reuse option that seeds the dataset with those labels instead of starting from scratch. Pick Reuse when you’re spinning up a different cut for experimentation but want to keep the labeling work you already trust. Define and Open are still there for the from-scratch and broad-sample cases.Replay verdicts persist with a coverage gate
After a replay in Phase 5, the assistant writes its pass/fail verdicts on the replay traces through a bundled script that verifies every replay trace got a verdict before moving on. Previously a verdict could die mid-session if the agent forgot to persist it; now the script enforces full coverage before continuing. If a trace is genuinely ambiguous, you can record it as an explicit skip rather than leaving it silently unverdicted.Plugins surface which Bitfab org they’re writing to
The plugin MCP now flags which Bitfab org it reads and writes from, so you’ll catch mismatches between your project’sBITFAB_API_KEY and the org open in your Studio tab before traces land somewhere unexpected. Coding agents now call get_api_key_context at the start of a plugin MCP session, and again whenever you mention data you just wrote isn’t visible in Studio. The same tightening applies to the remote MCP server in the Dashboard for direct (non-plugin) callers.Ruby SDK: skip child spans during replay
When you replay historical traces throughclient.replay(...), you can now have child spans return their recorded outputs instead of running real code. Three strategies control which children get short-circuited:mock: "none"(default) reruns every child span as before.mock: "all"returns historical output for every child.mock: "marked"returns historical output only for spans declared withmock_on_replay: true, and runs everything else real.
mock: "marked" to iterate on agent logic without paying for the marked child calls on each replay. Use mock: "all" for the cheapest possible replay (only the root function runs real code). Brings the Ruby SDK to parity with the existing mock option in the Python and TypeScript SDKs.Ruby SDK: fluent wrapper for shared trace function keys
client.get_function(key) returns a wrapper bound to that trace function key, so you can wrap multiple methods or classes without repeating the key on every call.client.get_function in the Python SDK and client.getFunction in TypeScript.DashboardPlugins
Accurate experiment counts with multi-label traces
Experiment pass/fail counts now correctly deduplicate traces that have labels from multiple sources (human review, approved agent, unapproved agent). Previously, a trace with both a human and an agent label could be double-counted in experiment totals. The viewer now picks the highest-priority label per trace: human labels take precedence over approved agent labels, which take precedence over unapproved ones.Experiments auto-label replayed traces
When you run an experiment through the assistant, replayed traces now receive agent labels automatically. The experiment viewer shows pass/fail results immediately after a replay completes, without requiring a manual labeling step first.Replay with mocks: shared-key spans return the correct output
Fixed an off-by-one in the replay mockTree when the function under test and one of its children share onetraceFunctionKey (the canonical getFunction(key).withSpan(...) pattern). The marked child was returning the root’s historical output instead of its own. The mockTree is now keyed by (traceFunctionKey, spanName, callIndex), which also unblocks recursive same-key replays.Mocked non-async Promise-returning functions stay Promises
If you wrap afunction fetchX() { return fetch(...) } (no async, but returns a Promise) and mock it during replay, the mocked return is now a Promise, not a raw value. Downstream .then(...) callers no longer crash. Detected at wrap time./bitfab:assistant experiments auto-pick parallel or serial
Phase 5 of the assistant skill now checks whether subagent worktrees inherit bypass permissions before forking parallel experiments. If permissions.defaultMode: "bypassPermissions" is set in committed .claude/settings.json or ~/.claude/settings.json, experiments fork to worktree-isolated subagents; otherwise they run serially in the main agent. Cursor and Codex always run serial since they don’t support worktree-isolated subagent calls.Dashboard
Organization switcher fix
Fixed the organization switcher dropdown not appearing in the header. After upgrading to Clerk v7, the switcher silently returned no memberships, making it impossible to switch between teams. The switcher now reliably shows all your organizations, with your personal workspace listed first and the rest sorted alphabetically.Live agent activity in Studio
The Studio home page now shows what your coding agent is doing in real time. While the assistant is working, the agent card highlights green and displays the current tool action (e.g., “Reading traces…”, “Creating grader…”). When the agent finishes or goes idle, the card fades back to its neutral state. Activity persists across page navigation within Studio, so you won’t lose track of the agent’s progress.Studio detects which coding agent opened it
When Studio is launched from Cursor or Codex, the UI now shows that agent’s logo and name instead of defaulting to Claude Code. The welcome page, header, and “Return to” button all reflect the agent that started the session.Replay failure handling in the assistant skill
/bitfab:assistant now separates infrastructure failures (missing DB rows, rejected writes) from real regressions during replay, and keeps unreplayable traces out of the pass-rate. When a child span fails environmentally, it suggests either flipping the span to mockOnReplay or pointing replay at the trace’s source environment.Replay mocks return the correct child span’s output
Fixed ordering bugs inmock: "marked" that caused a marked child span to return a sibling’s historical output instead of its own. Upgrade to TypeScript SDK 0.12.1 if you’re using mock: "marked" on 0.12.0.Mock child spans during replay
When you replay a recorded trace against new code, child spans sometimes fail locally for reasons unrelated to what you’re iterating on, like a paid API key you don’t have set, a flaky external service, or a production database row that isn’t seeded in your local environment. Replay now supports skipping those children and returning their recorded outputs instead, so the root function can still run.Pass amock strategy to replay() to control it. "none" (default) runs every child for real. "all" returns the historical output for every descendant. "marked" only short-circuits descendants you’ve tagged at definition time, leaving everything else to run real, which is the iteration-friendly mode.Tag a span with mockOnReplay: true in TypeScript or mock_on_replay=True in Python:mock: "marked" (TS) or mock="marked" (Python). The flagged child returns its recorded output and downstream spans run real code, so you can iterate on the analysis or formatting steps without standing up the upstream dependency.When the assistant skill is replaying a function and a child span fails environmentally, it’ll now suggest this fix directly. Full docs: TypeScript SDK and Python SDK reference under “Mocking child spans during replay”.Reliable focus restoration for macOS terminals
When clicking “Return to coding agent” in Studio, focus now reliably returns to the correct terminal app. The previous approach could target the wrong window if the terminal’s environment was modified (common inside Claude Code). The plugin now identifies your terminal by walking the process tree to find the parent application. For iTerm2 users with multiple windows, focus targets the exact session pane.Persistent Studio session for the assistant flow
Addstudio to any /bitfab:assistant invocation (e.g., /bitfab:assistant studio) to keep a single Studio window open for the entire flow. Dataset review and experiment results open inside the same window instead of launching separate ones, so you stay in one place while iterating. Without the studio argument, the flow works exactly as before.See which template renders each span at a glance
Iterating on the right template is faster when you can tell which one runs for the span you’re looking at. In the template preview, click or arrow-key through any span in the trace viewer and the matching card in the left rail lights up in that span’s color. That’s the template to edit.Know when your coding agent is mid-edit
Stay out of the agent’s way and watch its work land in context. When your agent saves a template, the studio names who is editing (Claude Code, Cursor, or Codex), pulses the affected card in the rail, and outlines the exact region inside the rendered span, even on instant saves, so you don’t miss it.Chat session capture
Bitfab plugins can now capture your coding-agent chat sessions and send them to the dashboard. Session capture is opt-in: enable it by settingBITFAB_CAPTURE_SESSIONS=true or adding "captureSessions": true to ~/.config/bitfab/config.json. Nothing is captured until you explicitly turn it on.Once enabled, sessions are only recorded after you invoke a Bitfab tool or slash command in the same conversation, so ordinary non-Bitfab conversations are never captured. Works across Claude Code, Cursor, and Codex.Cross-platform focus restoration
When a plugin opens a browser window (OAuth login, Studio preview), focus now returns to your terminal or editor automatically on Linux and Windows. Previously this only worked on macOS. If platform tools aren’t available (e.g., Wayland on Linux), the handoff completes normally without focus restoration.Studio connection errors surface immediately
When your coding agent opens the Studio preview, connection problems (expired API key, network timeout) are now caught before the browser window opens. Previously, errors could surface mid-session after you’d already started editing.Click-to-target template editing
In the template preview studio, you can now click directly on a rendered span to tell your coding agent exactly which region you want changed. No more “make the user message smaller” guesswork: point at the element and describe the change.Live preview auto-refresh
Templates saved in the studio now re-render in the preview automatically. Previously, you had to reload the page to see your changes.Template reference for coding agents
The newget_template_reference MCP tool returns a catalog of every editable region in the standard template, so coding agents can discover what’s available without you having to describe it.Dashboard
Template preview is faster on large functions
The template preview page loads significantly faster for functions with many spans. Pages that previously made dozens of parallel requests now resolve in a single batched call.Dashboard
Template rendering page: template-first layout
The template rendering page now starts from the templates instead of starting from a trace. You pick a template and see exactly which spans it affects.The new three-column layout shows all templates for a function on the left, the current trace in the center (with non-matching spans dimmed), and affected spans across recent traces on the right. If the current trace has no spans for the selected template, the viewer auto-navigates to one that does.API key context
Coding agents can now callget_api_key_context to find out which organization and environment their API key belongs to before sending traces. No more guesswork.API key descriptions
You can now add a description when creating API keys in the dashboard. Descriptions show up in the key list and are returned byget_api_key_context, so your coding agent can tell you which key it’s using without you having to check.