Skip to main content
Replay runs historical inputs through your current code. By default, every child span runs real code too: LLM calls, API calls, database fetches, tools, and any other wrapped work. Replay mocking lets selected child spans return their recorded outputs instead. The root function still runs real code, so you can change the orchestration, prompts, validation, formatting, or downstream logic and test it against production scenarios without paying for every upstream dependency again.
Available in the TypeScript, Python, and Ruby SDKs. The Go SDK does not support replay yet, so replay mocking does not apply there.

​
When to use it

Use replay mocking when a child span is not the thing you are trying to improve:
  • An LLM call is expensive, slow, or rate-limited.
  • An external API is flaky or needs credentials you do not have locally.
  • A database read depends on production-only rows.
  • A retrieval or preprocessing step should stay fixed while you iterate on later logic.
Do not mock the part of the system you are actively changing. If you are improving the summarizer, keep the summarizer real and mock only the setup work around it.

​
How it works

Every replay has a mock strategy. If you omit the option, Bitfab uses "marked".
StrategyBehaviorBest for
"marked"Only spans tagged at definition time are mocked.The normal iteration loop, and the default.
"none"Every child span runs real code.Full-fidelity replay when your local environment is complete.
"all"Every descendant span returns its historical output.Fast sanity checks where only the root logic needs to run.
The root function always runs real code. Mocking only applies to descendant spans.

​
Mark child spans

const fetchArticle = bitfab.withSpan(
  "fetch-article-from-db",
  { mockOnReplay: true },
  async (id: string) => db.articles.findById(id),
)

const summarize = bitfab.withSpan(
  "summarize-article",
  async (article: Article) => summarizeWithNewPrompt(article),
)

const processArticle = bitfab.withSpan(
  "process-article",
  async (id: string) => summarize(await fetchArticle(id)),
)

@bitfab.span("fetch-article-from-db", mock_on_replay=True)
def fetch_article_from_db(article_id: str) -> Article:
    return db.articles.find_by_id(article_id)


@bitfab.span("summarize-article")
def summarize_article(article: Article) -> Summary:
    return summarize_with_new_prompt(article)


@bitfab.span("process-article")
def process_article(article_id: str) -> Summary:
    return summarize_article(fetch_article_from_db(article_id))

class ArticleProcessor
  include Bitfab::Traceable
  bitfab_function "process-article"

  bitfab_span :fetch_article_from_db, mock_on_replay: true
  def fetch_article_from_db(article_id)
    db.articles.find_by_id(article_id)
  end

  bitfab_span :summarize_article
  def summarize_article(article)
    summarize_with_new_prompt(article)
  end

  bitfab_span :process_article
  def process_article(article_id)
    summarize_article(fetch_article_from_db(article_id))
  end
end

​
Replay with marked mocks

const result = await bitfab.replay("process-article", processArticle, {
  limit: 10,
})

console.log(result.testRunUrl)

result = bitfab.replay(process_article, limit=10)
print(result["testRunUrl"])

processor = ArticleProcessor.new
result = Bitfab.client.replay(
  processor, :process_article,
  trace_function_key: "process-article",
  limit: 10,
)

puts result[:testRunUrl]
During this replay, fetch-article-from-db returns the output recorded in each historical trace. summarize-article runs your current code, so the experiment tests the code you are changing.

​
What Bitfab matches

Bitfab matches mocked child calls against the recorded span tree for each source trace. Repeated calls to the same span are matched by call order, so the first call gets the first historical output, the second call gets the second, and so on. If no historical span matches a child call, Bitfab falls through to the real function. It never silently omits the call.

​
Common patterns

​
Keep expensive model calls fixed

Mark a paid classification or extraction span with mockOnReplay / mock_on_replay, then iterate on the decision logic that consumes its output.

​
Stabilize infrastructure while changing prompts

Mock data loading, retrieval, or external tools, then keep the prompt or formatter real. This isolates prompt changes from local setup problems.

​
Pair with database branching

Replay mocking is about replacing child function outputs. Database branching is about replaying against the database state that existed when the trace was captured. Use both when a replay needs production-like state and selected child calls should still be short-circuited.

​
SDK references