Available in the TypeScript, Python, Ruby, and Go SDKs. Go mocking applies to closure-style
Client.Span calls, which own execution; manual Start/End spans cannot prevent caller-owned code from running.When to use it
Use replay mocking when a child span is not the thing you are trying to improve:- An unsafe side effect must not execute again, such as sending an email, charging a card, publishing to a queue, or writing to a database.
- An LLM call is expensive, slow, or rate-limited.
- An external API is flaky or needs credentials you do not have locally.
- A database read depends on production-only rows.
- A retrieval or preprocessing step should stay fixed while you iterate on later logic.
How it works
Every replay has a mock strategy. If you omit the option, Bitfab uses"marked".
The root function always runs real code. Mocking only applies to descendant spans.
Python async-generator spans are not mockable yet. When
mock="all", mock="marked" selects one, or an override matches, the replay item fails without iterating the real generator. Move unsafe work inside the generator to a mockable sync or coroutine descendant.threading.Thread need Bitfab(trace_across_threads=True); pre-created consumers and other processes are outside that propagation. Ruby replay context is thread-local, so move work dispatched to another thread/process behind a boundary that runs in the replay thread. Traced Ruby methods that return an Enumerator bridge their parent span and replay context into same-thread Enumerator.new / enum_for source fibers. In TypeScript, a synchronous span cannot consume the lazy output used by mock: "marked"; use an async/Promise-returning boundary, or eager mock: "all" only when freezing every matched recorded child is the intended experiment.
Experimental selective replay (all SDKs)
The SDKs keep explicitly namedmustRun nodes live and reuse
recorded outputs around them. It always invokes the root. The server protects each
selected node’s entire subtree and its ancestors, then adds assertion protection
for each trace. At each other call, the
SDK compares its current inputs with the recording before returning a recorded output.
Different inputs cause eligible downstream code to execute.
experimental_selective_replay: {must_run: [...]} with
{trace_function_key: "checkout", span_name: "calculatePrice"} identities and
snake-case report keys. Go uses:
mustRun is the only execution manifest (1–100 node identities). Include changed
nodes, callers affected by changes in untraced helpers, constants, configuration,
or dependencies since capture, and unchanged nodes whose effects are needed later
(for example, populating a cache). The agent supplies this set; assertion requirements
are added by the server, not supplied as another list. Every matching occurrence is
protected, including new runtime descendants. Ancestors stay live to reach the
selected subtree, without automatically protecting unrelated siblings.
codeChangeFiles remains experiment context and does not automatically populate
this manifest. If a required call is not reached, the report does not claim it was tested.
Reuse eligibility is configured once on the traced function, not supplied as a
second list on every replay:
replayReusable: true declares that the entire subtree is output-only and depends
on the values of its JSON arguments. It must not read hidden mutable state, mutate
inputs or shared state, depend on object identity, or perform effects needed later.
Its implementation and dependencies must be unchanged except for nodes included in
mustRun. Do not declare arbitrary database reads or LLM calls reusable unless
those dependencies are explicitly fixed by your replay model. Use existing
mockOnReplay safety mocks for effects that must never run.
The SDK records this declaration as span_data.replay_reusable. The server reads
that metadata and combines it with each trace’s required subtrees, assertions, and
available recordings. Both the recorded declaration and the current function’s
declaration must allow reuse. Old traces without this metadata and unmarked calls
remain live; removing the current declaration also prevents reuse at that boundary.
The option works on TypeScript withSpan/span and trace-owned withNode/node configuration.
Python supports span, get_function(key).span, and trace-owned node configuration.
Ruby supports bitfab_span, fluent wrap, and bitfab_node with replay_reusable: true.
Go supports closure-style Span with WithReplayReusable(true) and trace-owned
NodeOptions{ReplayReusable: true}. Automatic Ruby descendants stay live when
reached; configure a node to give it a reusable interception boundary.
It has no effect on ordinary replay. There is no reusableNodes replay parameter.
The prototype requires the default mock: "marked" strategy. It rejects mock
overrides. TypeScript and Python also reject combining it with resume.
If mustRun, ancestry, or assertions require a safety-mocked call to
execute, the item fails without executing it. A safety mock whose recording no
longer matches also fails closed. Catching that error inside customer code does
not make the replay successful.
The server batches recordings into the start response. Matching is scoped to the
recorded parent and uses inputs to disambiguate reordered calls. Ambiguous repeated
calls execute, subject to safety restrictions. A required identity absent from the
recording disables selective reuse for that trace. Missing content, recorded errors,
and non-JSON values cannot be reused. Each reused output is copied for that invocation.
This opt-in supports SDK span/trace instrumentation in TypeScript, Python, Ruby, and Go.
Framework integrations, generators, custom replay outputs, and finalized streaming
outputs are outside this prototype. It supports at most 20 traces per run, 500 spans
per trace, and a 2 MB plan per trace. A compatible server-owned-plan (version 4) server
is required. The earlier changedNodes / changed_nodes options and older plans
are rejected rather than silently reinterpreted. Ordinary replay is unchanged.
Ruby and Go require new recordings with a validated replay_recording and
replay_json_safe: true; older recordings remain live. Ruby accepts lossless JSON
values, preserves positional/keyword separation, and rejects symbol-keyed outputs,
custom objects, aliases, and integral floats. Calls with Ruby blocks stay live
because the block’s behavior is not recorded; safety-mocked calls with blocks fail closed.
Go excludes custom serialization,
pointers, structs, aliases, and unsafe numbers. Use WithMockOutputType[T]() for
closure outputs needing a concrete Go type (for example int); without it, only
outputs retaining their exact runtime type through JSON are eligible. Generated
Go nodes infer the output type for one value, optionally followed by an error;
multi-value output boundaries remain live. Finalized outputs are not reusable.
Ruby and Go selective replay currently require in-process execution; process
registry launchers are rejected. Safety mocks without usable recordings fail closed.
Python keeps positional arguments separate from keyword arguments during matching.
The capture also records whether the original runtime values can be reused without
loss through JSON. Legacy Python recordings without that check remain live.
Safety mocks without a usable recording fail closed. Tuples, custom objects,
aliased containers, large integers, integral floats, and integer-index dictionary
keys are excluded from reuse. Their Python types or ordering may not survive the
server’s JSON representation. Finalizers, generators, and variant-bearing nodes
are not reusable boundaries in this prototype.
Python selective replay supports synchronous and asynchronous functions, concurrent
items, and repeated attempts through replay(). It does not yet support the
process-isolated registry launcher. Automatically observed Python descendants
always execute when reached. They contribute to execution coverage, but need
@bitfab.node(replay_reusable=True) to become a reusable boundary.
Before execution, the server combines every non-archived assertion on the selected
trace and all its replay ancestors. Pending, approved, and rejected assertions all
protect execution. Assertions on a newer replay never replace an ancestor’s
assertions for mocking decisions. Archived assertions are excluded. This protection
does not approve assertions or change which assertions the grader evaluates.
No additional replay option or per-call agent decision is needed. The assertion’s
structured target determines the conservative protection:
{ kind: "output" }: the root runs; eligible branches may still reuse values.{ kind: "span", name: "..." }: every matching call, its subtree, and its ancestors stay live, even when markedreplayReusable. All occurrences are protected so changes in call order cannot silently remove first/last/index targets.- No target (whole trace), or a span name/occurrence absent from the recording: selective reuse is disabled for the trace.
Opt a trace into descendant mocking
Subtree tracing can opt into descendant-default mocking without changing the global replay strategy. Set the trace option totrue, then annotate only the
code that should keep running with a node-level false.
applyNewPolicy runs because its @node opts out. Without
mockOnReplayDefault: true,
subtree mocking remains opt-in exactly as before. Python exposes the analogous
@trace(..., mock_on_replay_default=True) option, but only configured @node()
descendants can inherit it because interpreter monitoring cannot skip an
unwrapped call. Replay mock: "all" retains its original all-descendants meaning.
Mark child spans
Client.Span returns any, and recorded JSON objects otherwise decode as map[string]any. Add WithMockOutputType[T]() when a mocked child returns a struct, slice, or other concrete type; the SDK decodes the substituted output into T before returning it.
Replay with marked mocks
fetch-article-from-db returns the output recorded in each historical trace. summarize-article runs your current code, so the experiment tests the code you are changing.
What Bitfab matches
Bitfab matches mocked child calls against the recorded span tree for each source trace. A call is matched by span name and variant first, then by call order, so the first call gets the first historical output, the second call gets the second, and so on. Current SDKs emit strictly increasing microsecond timestamps so rapid sibling calls retain that order; the service falls back to ingestion order and span ID for legacy traces whose timestamps tie exactly. A variant is the value set withnode(variant=...) in Python to tell apart calls sharing one span name, such as one function running several times at once. A mocked call beneath a node with a variant only matches recordings beneath the same name and variant, so concurrent calls each get their own recorded outputs whatever order they run in. Recordings with no variants keep matching by name and call order exactly as before.
If a strategy selects a child for mocking but its recorded occurrence is missing or exhausted, Bitfab fails that replay item without executing the real child. Unselected calls still run real code.
During replay,
mockOverride / mock_override / MockOverrides checks each non-root call that passes through a Bitfab span wrapper (withSpan, @span, bitfab_span, or Go’s closure-style Client.Span). It does not iterate over framework spans that Bitfab only observes.Inject custom values (overrides)
Mock overrides are available in the TypeScript, Python, Ruby, and Go SDKs.
match + value pair. match selects which calls it applies to (by span name, type, trace function key, or any structural field); value is the substitution — a flat value used as-is, or a function that produces one. Full replacement: value becomes the call’s output.
The trace function key passed to replay() selects the workflow’s historical root traces; it does not by itself identify the descendant to override. In this example every call is bound to process-article, so the override matches the descendant by its span name. If your calls use separate trace function keys, a matcher can select by that field instead.
value is a function it receives the span’s live replay inputs, plus get_original_output to read the recorded output when you want to tweak it rather than replace it wholesale. In the Python and Ruby SDKs the context also carries the live keyword args (ctx.kwargs / ctx[:kwargs]) alongside the positional inputs:
value (one that never calls get_original_output) fetches no recorded output at all. Pass a single override or an array (first matcher wins). To apply overrides to every replay on a client, register them:
mockOverride / mock_override option. Precedence for each span: a per-call override wins, then a registered override, then the base mock strategy. The sentinel declines only the current resolver, so lower-priority overrides still get a chance before the base strategy.
In the TypeScript SDK, a synchronous wrapped function cannot wait for any Promise-returning resolver, including one that eventually resolves to
NO_MOCK_OVERRIDE. For a global resolver used across mixed sync/async spans, use a non-async routing function: return NO_MOCK_OVERRIDE synchronously for sync keys, and return a Promise only for async keys. getOriginalOutput() also fetches asynchronously, so a synchronous span cannot use it; make the span async, or use mock: "all". A flat value or synchronous function works on synchronous spans. Python, Ruby, and Go read the recorded output synchronously, so this restriction does not apply there; Go’s fetch blocks only that replay worker.Parameterize registry mocks
Keep reusable mock construction in the replay registry instead of adding application-specific flags to the SDK command. Supply JSON values with--params scenario.json and override individual values with repeatable --param name=value. Values that parse as JSON retain their type; other values remain strings.
Common patterns
Keep expensive model calls fixed
Mark a paid classification or extraction span withmockOnReplay / mock_on_replay, then iterate on the decision logic that consumes its output.
Stabilize infrastructure while changing prompts
Mock data loading, retrieval, or external tools, then keep the prompt or formatter real. This isolates prompt changes from local setup problems.Pair with database branching
Replay mocking is about replacing child function outputs. Database branching is about replaying against the database state that existed when the trace was captured. Use both when a replay needs production-like state and selected child calls should still be short-circuited.Mocking a seeded trace
A seeded trace is one written from a case you already held rather than captured from production. What mocking can do with it depends on how it was seeded.
For a case-seeded trace, use overrides to supply values for calls that must not run for real. An override injects a value you write, so it does not need a recorded output to work from.
Neither kind of seeded trace carries a database pin, so
dbBranch / db_branch refuses it rather than silently pinning the wrong moment.