Skip to main content
Replay runs historical inputs through your current code. Unmocked child calls run real code too: LLM calls, API calls, database fetches, tools, and any other wrapped work. Replay mocking lets selected child spans return their recorded outputs instead. The root function still runs real code, so you can change the orchestration, prompts, validation, formatting, or downstream logic and test it against production scenarios without paying for every upstream dependency again.
Mocking is the replay safety boundary; the deployment environment is not. Put every unsafe side-effecting call behind a replay-mockable descendant span and mark it. Unmarked calls, root-inline work, and import-time side effects run real code. mock: "none" deliberately disables recorded-output mocking.
Available in the TypeScript, Python, Ruby, and Go SDKs. Go mocking applies to closure-style Client.Span calls, which own execution; manual Start/End spans cannot prevent caller-owned code from running.

When to use it

Use replay mocking when a child span is not the thing you are trying to improve:
  • An unsafe side effect must not execute again, such as sending an email, charging a card, publishing to a queue, or writing to a database.
  • An LLM call is expensive, slow, or rate-limited.
  • An external API is flaky or needs credentials you do not have locally.
  • A database read depends on production-only rows.
  • A retrieval or preprocessing step should stay fixed while you iterate on later logic.
Do not mock the part of the system you are actively changing. If you are improving the summarizer, keep the summarizer real and mock only the setup work around it.

How it works

Every replay has a mock strategy. If you omit the option, Bitfab uses "marked". The root function always runs real code. Mocking only applies to descendant spans.
Python async-generator spans are not mockable yet. When mock="all", mock="marked" selects one, or an override matches, the replay item fails without iterating the real generator. Move unsafe work inside the generator to a mockable sync or coroutine descendant.
Mocking also requires the child to execute as a descendant in the replay context. Python thread pools and threading.Thread need Bitfab(trace_across_threads=True); pre-created consumers and other processes are outside that propagation. Ruby replay context is thread-local, so move work dispatched to another thread/process behind a boundary that runs in the replay thread. Traced Ruby methods that return an Enumerator bridge their parent span and replay context into same-thread Enumerator.new / enum_for source fibers. In TypeScript, a synchronous span cannot consume the lazy output used by mock: "marked"; use an async/Promise-returning boundary, or eager mock: "all" only when freezing every matched recorded child is the intended experiment.
Python’s automatically discovered @bitfab.trace() descendants cannot be mocked because interpreter monitoring observes rather than wraps them. Add @bitfab.node() to any descendant that needs a replay boundary, or use @bitfab.span() for an explicit replay boundary.

Experimental selective replay (all SDKs)

The SDKs keep explicitly named mustRun nodes live and reuse recorded outputs around them. It always invokes the root. The server protects each selected node’s entire subtree and its ancestors, then adds assertion protection for each trace. At each other call, the SDK compares its current inputs with the recording before returning a recorded output. Different inputs cause eligible downstream code to execute.
Python uses the same server plan with snake-case option and report names:
Ruby uses experimental_selective_replay: {must_run: [...]} with {trace_function_key: "checkout", span_name: "calculatePrice"} identities and snake-case report keys. Go uses:
mustRun is the only execution manifest (1–100 node identities). Include changed nodes, callers affected by changes in untraced helpers, constants, configuration, or dependencies since capture, and unchanged nodes whose effects are needed later (for example, populating a cache). The agent supplies this set; assertion requirements are added by the server, not supplied as another list. Every matching occurrence is protected, including new runtime descendants. Ancestors stay live to reach the selected subtree, without automatically protecting unrelated siblings. codeChangeFiles remains experiment context and does not automatically populate this manifest. If a required call is not reached, the report does not claim it was tested. Reuse eligibility is configured once on the traced function, not supplied as a second list on every replay:
replayReusable: true declares that the entire subtree is output-only and depends on the values of its JSON arguments. It must not read hidden mutable state, mutate inputs or shared state, depend on object identity, or perform effects needed later. Its implementation and dependencies must be unchanged except for nodes included in mustRun. Do not declare arbitrary database reads or LLM calls reusable unless those dependencies are explicitly fixed by your replay model. Use existing mockOnReplay safety mocks for effects that must never run. The SDK records this declaration as span_data.replay_reusable. The server reads that metadata and combines it with each trace’s required subtrees, assertions, and available recordings. Both the recorded declaration and the current function’s declaration must allow reuse. Old traces without this metadata and unmarked calls remain live; removing the current declaration also prevents reuse at that boundary. The option works on TypeScript withSpan/span and trace-owned withNode/node configuration. Python supports span, get_function(key).span, and trace-owned node configuration. Ruby supports bitfab_span, fluent wrap, and bitfab_node with replay_reusable: true. Go supports closure-style Span with WithReplayReusable(true) and trace-owned NodeOptions{ReplayReusable: true}. Automatic Ruby descendants stay live when reached; configure a node to give it a reusable interception boundary. It has no effect on ordinary replay. There is no reusableNodes replay parameter. The prototype requires the default mock: "marked" strategy. It rejects mock overrides. TypeScript and Python also reject combining it with resume. If mustRun, ancestry, or assertions require a safety-mocked call to execute, the item fails without executing it. A safety mock whose recording no longer matches also fails closed. Catching that error inside customer code does not make the replay successful. The server batches recordings into the start response. Matching is scoped to the recorded parent and uses inputs to disambiguate reordered calls. Ambiguous repeated calls execute, subject to safety restrictions. A required identity absent from the recording disables selective reuse for that trace. Missing content, recorded errors, and non-JSON values cannot be reused. Each reused output is copied for that invocation. This opt-in supports SDK span/trace instrumentation in TypeScript, Python, Ruby, and Go. Framework integrations, generators, custom replay outputs, and finalized streaming outputs are outside this prototype. It supports at most 20 traces per run, 500 spans per trace, and a 2 MB plan per trace. A compatible server-owned-plan (version 4) server is required. The earlier changedNodes / changed_nodes options and older plans are rejected rather than silently reinterpreted. Ordinary replay is unchanged. Ruby and Go require new recordings with a validated replay_recording and replay_json_safe: true; older recordings remain live. Ruby accepts lossless JSON values, preserves positional/keyword separation, and rejects symbol-keyed outputs, custom objects, aliases, and integral floats. Calls with Ruby blocks stay live because the block’s behavior is not recorded; safety-mocked calls with blocks fail closed. Go excludes custom serialization, pointers, structs, aliases, and unsafe numbers. Use WithMockOutputType[T]() for closure outputs needing a concrete Go type (for example int); without it, only outputs retaining their exact runtime type through JSON are eligible. Generated Go nodes infer the output type for one value, optionally followed by an error; multi-value output boundaries remain live. Finalized outputs are not reusable. Ruby and Go selective replay currently require in-process execution; process registry launchers are rejected. Safety mocks without usable recordings fail closed. Python keeps positional arguments separate from keyword arguments during matching. The capture also records whether the original runtime values can be reused without loss through JSON. Legacy Python recordings without that check remain live. Safety mocks without a usable recording fail closed. Tuples, custom objects, aliased containers, large integers, integral floats, and integer-index dictionary keys are excluded from reuse. Their Python types or ordering may not survive the server’s JSON representation. Finalizers, generators, and variant-bearing nodes are not reusable boundaries in this prototype. Python selective replay supports synchronous and asynchronous functions, concurrent items, and repeated attempts through replay(). It does not yet support the process-isolated registry launcher. Automatically observed Python descendants always execute when reached. They contribute to execution coverage, but need @bitfab.node(replay_reusable=True) to become a reusable boundary. Before execution, the server combines every non-archived assertion on the selected trace and all its replay ancestors. Pending, approved, and rejected assertions all protect execution. Assertions on a newer replay never replace an ancestor’s assertions for mocking decisions. Archived assertions are excluded. This protection does not approve assertions or change which assertions the grader evaluates. No additional replay option or per-call agent decision is needed. The assertion’s structured target determines the conservative protection:
  • { kind: "output" }: the root runs; eligible branches may still reuse values.
  • { kind: "span", name: "..." }: every matching call, its subtree, and its ancestors stay live, even when marked replayReusable. All occurrences are protected so changes in call order cannot silently remove first/last/index targets.
  • No target (whole trace), or a span name/occurrence absent from the recording: selective reuse is disabled for the trace.
Free-text assertions are not interpreted by an LLM to guess a narrower scope. Safety mocks never execute to satisfy these requirements; a conflict fails the item. New calls inside an assertion-protected subtree also remain live. The report lists assertion IDs, unresolved recorded targets, and requested span occurrences not reached during replay. These are execution-coverage diagnostics, not assertion verdicts or proof of complete evidence. Existing grading/agent scoring still judges the replay separately. Missing targets must not be counted as passing. A mocked output is recorded evidence, not fresh execution.

Opt a trace into descendant mocking

Subtree tracing can opt into descendant-default mocking without changing the global replay strategy. Set the trace option to true, then annotate only the code that should keep running with a node-level false.
Every transformed TypeScript descendant is marked by this trace option; applyNewPolicy runs because its @node opts out. Without mockOnReplayDefault: true, subtree mocking remains opt-in exactly as before. Python exposes the analogous @trace(..., mock_on_replay_default=True) option, but only configured @node() descendants can inherit it because interpreter monitoring cannot skip an unwrapped call. Replay mock: "all" retains its original all-descendants meaning.

Mark child spans

Go’s Client.Span returns any, and recorded JSON objects otherwise decode as map[string]any. Add WithMockOutputType[T]() when a mocked child returns a struct, slice, or other concrete type; the SDK decodes the substituted output into T before returning it.

Replay with marked mocks

During this replay, fetch-article-from-db returns the output recorded in each historical trace. summarize-article runs your current code, so the experiment tests the code you are changing.

What Bitfab matches

Bitfab matches mocked child calls against the recorded span tree for each source trace. A call is matched by span name and variant first, then by call order, so the first call gets the first historical output, the second call gets the second, and so on. Current SDKs emit strictly increasing microsecond timestamps so rapid sibling calls retain that order; the service falls back to ingestion order and span ID for legacy traces whose timestamps tie exactly. A variant is the value set with node(variant=...) in Python to tell apart calls sharing one span name, such as one function running several times at once. A mocked call beneath a node with a variant only matches recordings beneath the same name and variant, so concurrent calls each get their own recorded outputs whatever order they run in. Recordings with no variants keep matching by name and call order exactly as before. If a strategy selects a child for mocking but its recorded occurrence is missing or exhausted, Bitfab fails that replay item without executing the real child. Unselected calls still run real code.
During replay, mockOverride / mock_override / MockOverrides checks each non-root call that passes through a Bitfab span wrapper (withSpan, @span, bitfab_span, or Go’s closure-style Client.Span). It does not iterate over framework spans that Bitfab only observes.

Inject custom values (overrides)

Mock overrides are available in the TypeScript, Python, Ruby, and Go SDKs.
Marking a span replays its recorded output. An override goes further: it substitutes a value you supply for a matched span, then lets downstream real code run against that substitution. Use it for “what if this step returned X” experiments, without editing the traced code. An override is a match + value pair. match selects which calls it applies to (by span name, type, trace function key, or any structural field); value is the substitution — a flat value used as-is, or a function that produces one. Full replacement: value becomes the call’s output. The trace function key passed to replay() selects the workflow’s historical root traces; it does not by itself identify the descendant to override. In this example every call is bound to process-article, so the override matches the descendant by its span name. If your calls use separate trace function keys, a matcher can select by that field instead.
When value is a function it receives the span’s live replay inputs, plus get_original_output to read the recorded output when you want to tweak it rather than replace it wholesale. In the Python and Ruby SDKs the context also carries the live keyword args (ctx.kwargs / ctx[:kwargs]) alongside the positional inputs:
A flat or synthetic value (one that never calls get_original_output) fetches no recorded output at all. Pass a single override or an array (first matcher wins). To apply overrides to every replay on a client, register them:
Scope a client registration directly to one trace function key by passing the key first and either a resolver or another match-and-value override second. A keyed resolver only runs for spans with that key. Return the opt-out sentinel to continue to the next override and then the base strategy; ordinary nullish values remain valid mocked outputs. Omit the key to register one global resolver for every child span.
The resolver form also works directly in one replay’s mockOverride / mock_override option. Precedence for each span: a per-call override wins, then a registered override, then the base mock strategy. The sentinel declines only the current resolver, so lower-priority overrides still get a chance before the base strategy.
In the TypeScript SDK, a synchronous wrapped function cannot wait for any Promise-returning resolver, including one that eventually resolves to NO_MOCK_OVERRIDE. For a global resolver used across mixed sync/async spans, use a non-async routing function: return NO_MOCK_OVERRIDE synchronously for sync keys, and return a Promise only for async keys. getOriginalOutput() also fetches asynchronously, so a synchronous span cannot use it; make the span async, or use mock: "all". A flat value or synchronous function works on synchronous spans. Python, Ruby, and Go read the recorded output synchronously, so this restriction does not apply there; Go’s fetch blocks only that replay worker.

Parameterize registry mocks

Keep reusable mock construction in the replay registry instead of adding application-specific flags to the SDK command. Supply JSON values with --params scenario.json and override individual values with repeatable --param name=value. Values that parse as JSON retain their type; other values remain strings.
The parameters are configuration for the registry’s option factory only. The command never appends them to the historical inputs passed into the production function.

Common patterns

Keep expensive model calls fixed

Mark a paid classification or extraction span with mockOnReplay / mock_on_replay, then iterate on the decision logic that consumes its output.

Stabilize infrastructure while changing prompts

Mock data loading, retrieval, or external tools, then keep the prompt or formatter real. This isolates prompt changes from local setup problems.

Pair with database branching

Replay mocking is about replacing child function outputs. Database branching is about replaying against the database state that existed when the trace was captured. Use both when a replay needs production-like state and selected child calls should still be short-circuited.

Mocking a seeded trace

A seeded trace is one written from a case you already held rather than captured from production. What mocking can do with it depends on how it was seeded. For a case-seeded trace, use overrides to supply values for calls that must not run for real. An override injects a value you write, so it does not need a recorded output to work from. Neither kind of seeded trace carries a database pin, so dbBranch / db_branch refuses it rather than silently pinning the wrong moment.

SDK references