Replay Mocking

Replay runs historical inputs through your current code. By default, every child span runs real code too: LLM calls, API calls, database fetches, tools, and any other wrapped work. Replay mocking lets selected child spans return their recorded outputs instead. The root function still runs real code, so you can change the orchestration, prompts, validation, formatting, or downstream logic and test it against production scenarios without paying for every upstream dependency again.

Available in the TypeScript, Python, and Ruby SDKs. The Go SDK does not support replay yet, so replay mocking does not apply there.

When to use it

Use replay mocking when a child span is not the thing you are trying to improve:

An LLM call is expensive, slow, or rate-limited.
An external API is flaky or needs credentials you do not have locally.
A database read depends on production-only rows.
A retrieval or preprocessing step should stay fixed while you iterate on later logic.

Do not mock the part of the system you are actively changing. If you are improving the summarizer, keep the summarizer real and mock only the setup work around it.

How it works

Every replay has a mock strategy. If you omit the option, Bitfab uses "marked".

Strategy	Behavior	Best for
`"marked"`	Only spans tagged at definition time are mocked.	The normal iteration loop, and the default.
`"none"`	Every child span runs real code.	Full-fidelity replay when your local environment is complete.
`"all"`	Every descendant span returns its historical output.	Fast sanity checks where only the root logic needs to run.

The root function always runs real code. Mocking only applies to descendant spans.

Mark child spans

const fetchArticle = bitfab.withSpan(
  "fetch-article-from-db",
  { mockOnReplay: true },
  async (id: string) => db.articles.findById(id),
)

const summarize = bitfab.withSpan(
  "summarize-article",
  async (article: Article) => summarizeWithNewPrompt(article),
)

const processArticle = bitfab.withSpan(
  "process-article",
  async (id: string) => summarize(await fetchArticle(id)),
)

@bitfab.span("fetch-article-from-db", mock_on_replay=True)
def fetch_article_from_db(article_id: str) -> Article:
    return db.articles.find_by_id(article_id)


@bitfab.span("summarize-article")
def summarize_article(article: Article) -> Summary:
    return summarize_with_new_prompt(article)


@bitfab.span("process-article")
def process_article(article_id: str) -> Summary:
    return summarize_article(fetch_article_from_db(article_id))

class ArticleProcessor
  include Bitfab::Traceable
  bitfab_function "process-article"

  bitfab_span :fetch_article_from_db, mock_on_replay: true
  def fetch_article_from_db(article_id)
    db.articles.find_by_id(article_id)
  end

  bitfab_span :summarize_article
  def summarize_article(article)
    summarize_with_new_prompt(article)
  end

  bitfab_span :process_article
  def process_article(article_id)
    summarize_article(fetch_article_from_db(article_id))
  end
end

Replay with marked mocks

const result = await bitfab.replay("process-article", processArticle, {
  limit: 10,
})

console.log(result.testRunUrl)

result = bitfab.replay(process_article, limit=10)
print(result["testRunUrl"])

processor = ArticleProcessor.new
result = Bitfab.client.replay(
  processor, :process_article,
  trace_function_key: "process-article",
  limit: 10,
)

puts result[:testRunUrl]

During this replay, fetch-article-from-db returns the output recorded in each historical trace. summarize-article runs your current code, so the experiment tests the code you are changing.

What Bitfab matches

Bitfab matches mocked child calls against the recorded span tree for each source trace. Repeated calls to the same span are matched by call order, so the first call gets the first historical output, the second call gets the second, and so on. If no historical span matches a child call, Bitfab falls through to the real function. It never silently omits the call.

Inject custom values (overrides)

Mock overrides are available in the TypeScript, Python, and Ruby SDKs.

Marking a span replays its recorded output. An override goes further: it substitutes a value you supply for a matched span, then lets downstream real code run against that substitution. Use it for “what if this step returned X” experiments, without editing the traced code. An override is a match + value pair. match selects which spans it applies to (by trace_function_key, type, or any structural field); value is the substitution — a flat value used as-is, or a function that produces one. Full replacement: value becomes the span’s output.

const result = await bitfab.replay("process-article", processArticle, {
  mock: "none", // run everything real...
  mockOverride: {
    // ...except this one span, which gets a value you supply
    match: (node) => node.traceFunctionKey === "fetch-article-from-db",
    value: { id: "fixed", title: "Fixed title" }, // a flat value
  },
})

result = client.replay(
    process_article,
    mock="none",  # run everything real...
    mock_override=MockOverride(
        # ...except this one span, which gets a value you supply
        match=lambda node: node.trace_function_key == "fetch-article-from-db",
        value={"id": "fixed", "title": "Fixed title"},  # a flat value
    ),
)

result = client.replay(
  processor, :process_article,
  trace_function_key: "process-article",
  mock: "none", # run everything real...
  mock_override: {
    # ...except this one span, which gets a value you supply
    match: ->(node) { node[:trace_function_key] == "fetch-article-from-db" },
    value: { id: "fixed", title: "Fixed title" } # a flat value
  }
)

When value is a function it receives the span’s live replay inputs, plus get_original_output to read the recorded output when you want to tweak it rather than replace it wholesale. In the Python and Ruby SDKs the context also carries the live keyword args (ctx.kwargs / ctx[:kwargs]) alongside the positional inputs:

// getOriginalOutput() is async in the TypeScript SDK
value: async ({ getOriginalOutput }) => {
  const original = await getOriginalOutput()
  return { ...original, score: 1 } // override one field, keep the rest
}

# get_original_output() is synchronous in the Python SDK
value=lambda ctx: {**ctx.get_original_output(), "score": 1}

# get_original_output is synchronous in the Ruby SDK
value: ->(ctx) { ctx[:get_original_output].call.merge(score: 1) }

A flat or synthetic value (one that never calls get_original_output) fetches no recorded output at all. Pass a single override or an array (first matcher wins). To apply overrides to every replay on a client, register them:

bitfab.registerMockOverride({
  match: (node) => node.type === "llm",
  value: { label: "refund" },
})
// Ordered form (equivalent): registerMockOverride(match, value)
bitfab.clearMockOverrides() // reset

client.register_mock_override(
    MockOverride(match=lambda node: node.type == "llm", value={"label": "refund"})
)
# Ordered form (equivalent): register_mock_override(match, value)
client.clear_mock_overrides()  # reset

client.register_mock_override(
  match: ->(node) { node[:type] == "llm" },
  value: { label: "refund" }
)
# Positional form (equivalent): register_mock_override(match, value)
client.clear_mock_overrides # reset

Precedence for each span: a per-call mockOverride wins, then a registered override, then the base mock strategy. A span no override matches falls back to that strategy, so you can combine overrides with "marked" or "all".

In the TypeScript SDK, getOriginalOutput() fetches asynchronously, so a synchronous wrapped function cannot be mocked with a value that must be fetched (a recorded-output reuse, or a value function that awaits getOriginalOutput). Make the span async, or use mock: "all". A flat value (or a synchronous function) works on synchronous spans. The Python and Ruby SDKs read the recorded output synchronously, so this restriction does not apply there.

Common patterns

Keep expensive model calls fixed

Mark a paid classification or extraction span with mockOnReplay / mock_on_replay, then iterate on the decision logic that consumes its output.

Stabilize infrastructure while changing prompts

Mock data loading, retrieval, or external tools, then keep the prompt or formatter real. This isolates prompt changes from local setup problems.

Pair with database branching

Replay mocking is about replacing child function outputs. Database branching is about replaying against the database state that existed when the trace was captured. Use both when a replay needs production-like state and selected child calls should still be short-circuited.

Getting Started

Account

SDKs

Framework Integrations

Integrations

Web Portal

Reference

Changelog

Replay Mocking

When to use it

How it works

Mark child spans

Replay with marked mocks

What Bitfab matches

Inject custom values (overrides)

Common patterns

Keep expensive model calls fixed

Stabilize infrastructure while changing prompts

Pair with database branching

SDK references

​When to use it

​How it works

​Mark child spans

​Replay with marked mocks

​What Bitfab matches

​Inject custom values (overrides)

​Common patterns

​Keep expensive model calls fixed

​Stabilize infrastructure while changing prompts

​Pair with database branching

​SDK references

When to use it

How it works

Mark child spans

Replay with marked mocks

What Bitfab matches

Inject custom values (overrides)

Common patterns

Keep expensive model calls fixed

Stabilize infrastructure while changing prompts

Pair with database branching

SDK references