Replay Mocking

Replay runs historical inputs through your current code. Unmocked child calls run real code too: LLM calls, API calls, database fetches, tools, and any other wrapped work. Replay mocking lets selected child spans return their recorded outputs instead. The root function still runs real code, so you can change the orchestration, prompts, validation, formatting, or downstream logic and test it against production scenarios without paying for every upstream dependency again.

Mocking is the replay safety boundary; the deployment environment is not. Put every unsafe side-effecting call behind a replay-mockable descendant span and mark it. Unmarked calls, root-inline work, and import-time side effects run real code. mock: "none" deliberately disables recorded-output mocking.

Available in the TypeScript, Python, Ruby, and Go SDKs. Go mocking applies to closure-style Client.Span calls, which own execution; manual Start/End spans cannot prevent caller-owned code from running.

When to use it

Use replay mocking when a child span is not the thing you are trying to improve:

An unsafe side effect must not execute again, such as sending an email, charging a card, publishing to a queue, or writing to a database.
An LLM call is expensive, slow, or rate-limited.
An external API is flaky or needs credentials you do not have locally.
A database read depends on production-only rows.
A retrieval or preprocessing step should stay fixed while you iterate on later logic.

Do not mock the part of the system you are actively changing. If you are improving the summarizer, keep the summarizer real and mock only the setup work around it.

How it works

Every replay has a mock strategy. If you omit the option, Bitfab uses "marked".

Strategy	Behavior	Best for
`"marked"`	Only spans tagged at definition time are mocked.	The normal iteration loop, and the default.
`"none"`	Every child span runs real code.	Runs where you intentionally want every dependency real and have verified that is safe.
`"all"`	Every matched recorded descendant span returns its historical output; a missing occurrence fails the item closed.	Fast sanity checks where only the root logic needs to run.

The root function always runs real code. Mocking only applies to descendant spans.

Python async-generator spans are not mockable yet. When mock="all", mock="marked" selects one, or an override matches, the replay item fails without iterating the real generator. Move unsafe work inside the generator to a mockable sync or coroutine descendant.

Mocking also requires the child to execute as a descendant in the replay context. Python thread pools and threading.Thread need Bitfab(trace_across_threads=True); pre-created consumers and other processes are outside that propagation. Ruby replay context is thread-local, so move work dispatched to another thread/process behind a boundary that runs in the replay thread. Traced Ruby methods that return an Enumerator bridge their parent span and replay context into same-thread Enumerator.new / enum_for source fibers. In TypeScript, a synchronous span cannot consume the lazy output used by mock: "marked"; use an async/Promise-returning boundary, or eager mock: "all" only when freezing every matched recorded child is the intended experiment.

Python’s automatically discovered @bitfab.trace() descendants cannot be mocked because interpreter monitoring observes rather than wraps them. Add @bitfab.node() to any descendant that needs a replay boundary, or use @bitfab.span() for an explicit replay boundary.

Opt a trace into descendant mocking

Subtree tracing can opt into descendant-default mocking without changing the global replay strategy. Set the trace option to true, then annotate only the code that should keep running with a node-level false.

class SupportAgent {
  @bitfab.node({ mockOnReplay: false })
  async applyNewPolicy(input: Input): Promise<Decision> {
    return applyNewPolicy(input)
  }

  @bitfab.trace("support-agent", { mockOnReplayDefault: true })
  async run(input: Input): Promise<Result> {
    return this.applyNewPolicy(input)
  }
}

await bitfab.replay("support-agent", agent.run.bind(agent))

Every transformed TypeScript descendant is marked by this trace option; applyNewPolicy runs because its @node opts out. Without mockOnReplayDefault: true, subtree mocking remains opt-in exactly as before. Python exposes the analogous @trace(..., mock_on_replay_default=True) option, but only configured @node() descendants can inherit it because interpreter monitoring cannot skip an unwrapped call. Replay mock: "all" retains its original all-descendants meaning.

Mark child spans

const articlePipeline = bitfab.getFunction("process-article")

const fetchArticle = articlePipeline.withSpan(
  { name: "fetch-article-from-db", mockOnReplay: true },
  async (id: string) => db.articles.findById(id),
)

const summarize = articlePipeline.withSpan(
  { name: "summarize-article" },
  async (article: Article) => summarizeWithNewPrompt(article),
)

const processArticle = articlePipeline.withSpan(
  { name: "process-article" },
  async (id: string) => summarize(await fetchArticle(id)),
)

article_pipeline = bitfab.get_function("process-article")


@article_pipeline.span(name="fetch-article-from-db", mock_on_replay=True)
def fetch_article_from_db(article_id: str) -> Article:
    return db.articles.find_by_id(article_id)


@article_pipeline.span(name="summarize-article")
def summarize_article(article: Article) -> Summary:
    return summarize_with_new_prompt(article)


@article_pipeline.span(name="process-article")
def process_article(article_id: str) -> Summary:
    return summarize_article(fetch_article_from_db(article_id))

class ArticleProcessor
  include Bitfab::Traceable
  bitfab_function "process-article"

  bitfab_span :fetch_article_from_db, name: "fetch-article-from-db", mock_on_replay: true
  def fetch_article_from_db(article_id)
    db.articles.find_by_id(article_id)
  end

  bitfab_span :summarize_article, name: "summarize-article"
  def summarize_article(article)
    summarize_with_new_prompt(article)
  end

  bitfab_span :process_article, name: "process-article"
  def process_article(article_id)
    summarize_article(fetch_article_from_db(article_id))
  end
end

func fetchArticleFromDB(ctx context.Context, articleID string) (Article, error) {
    value, err := client.Span(
        ctx,
        "process-article",
        func(ctx context.Context) (any, error) {
            return db.Articles.FindByID(ctx, articleID)
        },
        bitfab.WithName("fetch-article-from-db"),
        bitfab.WithInput(articleID),
        bitfab.WithMockOnReplay(true),
        bitfab.WithMockOutputType[Article](),
    )
    if err != nil {
        return Article{}, err
    }
    return value.(Article), nil
}

Go’s Client.Span returns any, and recorded JSON objects otherwise decode as map[string]any. Add WithMockOutputType[T]() when a mocked child returns a struct, slice, or other concrete type; the SDK decodes the substituted output into T before returning it.

Replay with marked mocks

const result = await bitfab.replay("process-article", processArticle, {
  limit: 10,
})

console.log(result.testRunUrl)

result = bitfab.replay(process_article, limit=10)
print(result["test_run_url"])

processor = ArticleProcessor.new
result = Bitfab.client.replay(
  processor, :process_article,
  trace_function_key: "process-article",
  limit: 10,
)

puts result[:test_run_url]

result, err := client.Replay(
    context.Background(),
    "process-article",
    processArticle,
    &bitfab.ReplayOptions{Limit: 10},
)

During this replay, fetch-article-from-db returns the output recorded in each historical trace. summarize-article runs your current code, so the experiment tests the code you are changing.

What Bitfab matches

Bitfab matches mocked child calls against the recorded span tree for each source trace. Repeated calls to the same span are matched by call order, so the first call gets the first historical output, the second call gets the second, and so on. Current SDKs emit strictly increasing microsecond timestamps so rapid sibling calls retain that order; the service falls back to ingestion order and span ID for legacy traces whose timestamps tie exactly. If a strategy selects a child for mocking but its recorded occurrence is missing or exhausted, Bitfab fails that replay item without executing the real child. Unselected calls still run real code.

During replay, mockOverride / mock_override / MockOverrides checks each non-root call that passes through a Bitfab span wrapper (withSpan, @span, bitfab_span, or Go’s closure-style Client.Span). It does not iterate over trace-plan nodes or framework spans that Bitfab only observes.

Inject custom values (overrides)

Mock overrides are available in the TypeScript, Python, Ruby, and Go SDKs.

Marking a span replays its recorded output. An override goes further: it substitutes a value you supply for a matched span, then lets downstream real code run against that substitution. Use it for “what if this step returned X” experiments, without editing the traced code. An override is a match + value pair. match selects which calls it applies to (by span name, type, trace function key, or any structural field); value is the substitution — a flat value used as-is, or a function that produces one. Full replacement: value becomes the call’s output. The trace function key passed to replay() selects the workflow’s historical root traces; it does not by itself identify the descendant to override. In this example every call is bound to process-article, so the override matches the descendant by its span name. If your calls use separate trace function keys, a matcher can select by that field instead.

const result = await bitfab.replay("process-article", processArticle, {
  mock: "none", // run everything real...
  mockOverride: {
    // ...except this one call, which gets a value you supply
    match: (node) => node.spanName === "fetch-article-from-db",
    value: { id: "fixed", title: "Fixed title" }, // a flat value
  },
})

from bitfab import MockOverride

result = bitfab.replay(
    process_article,
    mock="none",  # run everything real...
    mock_override=MockOverride(
        # ...except this one call, which gets a value you supply
        match=lambda node: node.span_name == "fetch-article-from-db",
        value={"id": "fixed", "title": "Fixed title"},  # a flat value
    ),
)

result = Bitfab.client.replay(
  processor, :process_article,
  trace_function_key: "process-article",
  mock: "none", # run everything real...
  mock_override: {
    # ...except this one call, which gets a value you supply
    match: ->(node) { node[:span_name] == "fetch-article-from-db" },
    value: { id: "fixed", title: "Fixed title" } # a flat value
  }
)

result, err := client.Replay(ctx, "process-article", processArticle, &bitfab.ReplayOptions{
    Mock: bitfab.MockNone,
    MockOverrides: []bitfab.MockOverride{{
        Match: func(node bitfab.SpanNodeMeta) bool {
            return node.SpanName == "fetch-article-from-db"
        },
        Value: Article{ID: "fixed", Title: "Fixed title"},
    }},
})

When value is a function it receives the span’s live replay inputs, plus get_original_output to read the recorded output when you want to tweak it rather than replace it wholesale. In the Python and Ruby SDKs the context also carries the live keyword args (ctx.kwargs / ctx[:kwargs]) alongside the positional inputs:

// getOriginalOutput() is async in the TypeScript SDK
value: async ({ getOriginalOutput }) => {
  const original = await getOriginalOutput()
  return { ...original, score: 1 } // override one field, keep the rest
}

# get_original_output() is synchronous in the Python SDK
value=lambda ctx: {**ctx.get_original_output(), "score": 1}

# get_original_output is synchronous in the Ruby SDK
value: ->(ctx) { ctx[:get_original_output].call.merge(score: 1) }

Resolve: func(ctx bitfab.MockOverrideContext) (any, error) {
    original, err := ctx.GetOriginalOutput()
    if err != nil {
        return nil, err
    }
    value := original.(Result)
    value.Score = 1
    return value, nil
}

A flat or synthetic value (one that never calls get_original_output) fetches no recorded output at all. Pass a single override or an array (first matcher wins). To apply overrides to every replay on a client, register them:

bitfab.registerMockOverride({
  match: (node) => node.type === "llm",
  value: { label: "refund" },
})
// Ordered form (equivalent): registerMockOverride(match, value)
bitfab.clearMockOverrides() // reset

bitfab.register_mock_override(
    MockOverride(match=lambda node: node.type == "llm", value={"label": "refund"})
)
# Ordered form (equivalent): register_mock_override(match, value)
bitfab.clear_mock_overrides()  # reset

Bitfab.client.register_mock_override(
  match: ->(node) { node[:type] == "llm" },
  value: { label: "refund" }
)
# Positional form (equivalent): register_mock_override(match, value)
Bitfab.client.clear_mock_overrides # reset

err := client.RegisterMockOverride(bitfab.MockOverride{
    Match: func(node bitfab.SpanNodeMeta) bool { return node.Type == "llm" },
    Value: map[string]any{"label": "refund"},
})
client.ClearMockOverrides()

Scope a client registration directly to one trace function key by passing the key first and either a resolver or another match-and-value override second. A keyed resolver only runs for spans with that key. Return the opt-out sentinel to continue to the next override and then the base strategy; ordinary nullish values remain valid mocked outputs. Omit the key to register one global resolver for every child span.

import { NO_MOCK_OVERRIDE } from "@bitfab/sdk"

bitfab.registerMockOverride("checkout-agent", ({ node }) =>
  node.spanName === "Classifier"
    ? { label: "refund" }
    : NO_MOCK_OVERRIDE,
)

from bitfab import NO_MOCK_OVERRIDE

client.register_mock_override(
    "checkout-agent",
    lambda ctx: (
        {"label": "refund"}
        if ctx.node.span_name == "Classifier"
        else NO_MOCK_OVERRIDE
    )
)

client.register_mock_override("checkout-agent", lambda do |ctx|
  if ctx[:node][:span_name] == "Classifier"
    {label: "refund"}
  else
    Bitfab::NO_MOCK_OVERRIDE
  end
end)

The resolver form also works directly in one replay’s mockOverride / mock_override option. Precedence for each span: a per-call override wins, then a registered override, then the base mock strategy. The sentinel declines only the current resolver, so lower-priority overrides still get a chance before the base strategy.

In the TypeScript SDK, a synchronous wrapped function cannot wait for any Promise-returning resolver, including one that eventually resolves to NO_MOCK_OVERRIDE. For a global resolver used across mixed sync/async spans, use a non-async routing function: return NO_MOCK_OVERRIDE synchronously for sync keys, and return a Promise only for async keys. getOriginalOutput() also fetches asynchronously, so a synchronous span cannot use it; make the span async, or use mock: "all". A flat value or synchronous function works on synchronous spans. Python, Ruby, and Go read the recorded output synchronously, so this restriction does not apply there; Go’s fetch blocks only that replay worker.

Parameterize registry mocks

Keep reusable mock construction in the replay registry instead of adding application-specific flags to the SDK command. Supply JSON values with --params scenario.json and override individual values with repeatable --param name=value. Values that parse as JSON retain their type; other values remain strings.

optionsFactory: ({ params }) => ({
  mockOverride: createIntentMock(params.forcedLabel),
})

options_factory=lambda ctx: {
    "mock_override": create_intent_mock(ctx.params.get("forcedLabel"))
}

options_factory: ->(ctx) {
  {mock_override: IntentMocks.build(ctx.params["forcedLabel"])}
}

bitfab-replay --registry scripts/replayRegistry.ts agent \
  --params scenarios/refund.json \
  --param 'forcedLabel="refund"' \
  --param confidence=0.95

The parameters are configuration for the registry’s option factory only. The command never appends them to the historical inputs passed into the production function.

Common patterns

Keep expensive model calls fixed

Mark a paid classification or extraction span with mockOnReplay / mock_on_replay, then iterate on the decision logic that consumes its output.

Stabilize infrastructure while changing prompts

Mock data loading, retrieval, or external tools, then keep the prompt or formatter real. This isolates prompt changes from local setup problems.

Pair with database branching

Replay mocking is about replacing child function outputs. Database branching is about replaying against the database state that existed when the trace was captured. Use both when a replay needs production-like state and selected child calls should still be short-circuited.

Mocking a seeded trace

A seeded trace is one written from a case you already held rather than captured from production. What mocking can do with it depends on how it was seeded.

Seeded how	What replay mocking can substitute
By running the function once (`seedTrace(key, fn, ...)`, `seed_trace(key, fn, ...)`)	Everything a captured trace offers. The run recorded a full first-party subtree, so marked children, `mock: "all"`, and overrides all behave normally
From a case without running (TypeScript `seedTrace(key, case)`)	Nothing recorded. The trace has only a root span, so `mock: "marked"` and `mock: "all"` find no recorded child output to return

For a case-seeded trace, use overrides to supply values for calls that must not run for real. An override injects a value you write, so it does not need a recorded output to work from. Neither kind of seeded trace carries a database pin, so dbBranch / db_branch refuses it rather than silently pinning the wrong moment.

Getting Started

Primitives

Account

SDKs

Framework Integrations

Integrations

Web Portal

Reference

Changelog

Replay Mocking

When to use it

How it works

Opt a trace into descendant mocking

Mark child spans

Replay with marked mocks

What Bitfab matches

Inject custom values (overrides)

Parameterize registry mocks

Common patterns

Keep expensive model calls fixed

Stabilize infrastructure while changing prompts

Pair with database branching

Mocking a seeded trace

SDK references

​When to use it

​How it works

​Opt a trace into descendant mocking

​Mark child spans

​Replay with marked mocks

​What Bitfab matches

​Inject custom values (overrides)

​Parameterize registry mocks

​Common patterns

​Keep expensive model calls fixed

​Stabilize infrastructure while changing prompts

​Pair with database branching

​Mocking a seeded trace

​SDK references

When to use it

How it works

Opt a trace into descendant mocking

Mark child spans

Replay with marked mocks

What Bitfab matches

Inject custom values (overrides)

Parameterize registry mocks

Common patterns

Keep expensive model calls fixed

Stabilize infrastructure while changing prompts

Pair with database branching

Mocking a seeded trace

SDK references