TypeScript SDK

The Bitfab TypeScript SDK captures your AI function calls to automatically generate evaluations. Re-run your prompts with different models, parameters, and inputs to iterate faster.

Installation

# npm
npm install @bitfab/sdk

# pnpm
pnpm add @bitfab/sdk

# yarn
yarn add @bitfab/sdk

Quick Start

import { Bitfab } from "@bitfab/sdk"

const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })

Need an API key? Get one from the Bitfab dashboard or see the API Keys guide for detailed setup instructions.

Coding Agent Prompt (Cursor, Claude Code)

Copy this prompt into your coding agent (tested with Cursor and Claude Code using Sonnet 4.5):

Modify existing TypeScript code to add Bitfab tracing.
Do NOT browse or web search. Use ONLY the API described below.

Bitfab TypeScript SDK (authoritative excerpt):
- Install: `npm install @bitfab/sdk` or `pnpm add @bitfab/sdk`
- Init:
  import { Bitfab } from "@bitfab/sdk"
  const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
- Framework integrations:
  If the codebase uses LangGraph or LangChain (`@langchain/langgraph`, `@langchain/core`, or other `@langchain/*` packages), use the callback handler instead of manually wrapping graph nodes, tools, retrievers, or model calls:
    const handler = bitfab.getLangGraphCallbackHandler("<trace_function_key>")
    await graph.invoke(input, { callbacks: [handler] })
  For plain LangChain chains, `getLangChainCallbackHandler("<trace_function_key>")` is an identical alias. The handler records a replayable root from the framework input, so no outer `withSpan` root is needed when the workflow is just the graph/chain invocation. Add a same-key outer root only for meaningful surrounding application work.
- Manual instrumentation (when no framework handler applies, or for meaningful work around a framework call):
  // Declare trace function key once
  const myService = bitfab.getFunction("<trace_function_key>")

  // Wrap functions with withSpan
  const tracedFn = myService.withSpan(originalFunction)

  // Or with options:
  const tracedFn = myService.withSpan({ name: "DisplayName", type: "function" }, originalFunction)

  // Span types: "llm", "agent", "function", "guardrail", "handoff", "custom"
- DO NOT modify the original function.
- DO NOT extract helper methods.

Task:
1) Ensure bitfab is installed and initialization exists.
2) Read the codebase and identify ALL AI workflows (LLM calls, agent runs, AI-driven decisions). Check for LangGraph/LangChain before planning manual instrumentation.
3) Present me with a numbered list of workflows you found. For each, describe:
   - What it does
   - Why it's worth instrumenting -- what visibility tracing gives you into each step
4) After I choose which workflow(s) to instrument:
   - If it uses LangGraph/LangChain, add the Bitfab callback handler to the framework invoke config instead of wrapping framework-managed internals. Use `getFunction("<trace_function_key>").getLangGraphCallbackHandler()` only when a same-key outer `withSpan` root is needed for surrounding application work.
   - For non-framework workflows, create a function wrapper with `bitfab.getFunction("<trace_function_key>")`
   - Wrap non-framework functions with `myService.withSpan(originalFunction)`
   - Instrument intermediate steps (not just the final output) so each trace has enough context to diagnose issues
   - Replace usages of manually wrapped non-framework functions with the traced versions
5) Do not change function signature, behavior, or return value. Minimal diff.

Output:
- First: your numbered list of workflows with why each is worth instrumenting
- After my selection: minimal diffs for dependencies, initialization, and the function wrapping

Basic Configuration

new Bitfab({ apiKey: string })

// Omit apiKey entirely: the SDK reads BITFAB_API_KEY from the environment
new Bitfab({})

// Disable tracing (functions still execute, but no spans are sent)
new Bitfab({ apiKey: string, enabled: false })

Missing API key doesn’t crash. If the API key is missing, empty, or whitespace-only, the SDK automatically disables tracing and logs a one-time warning at first use. All wrapped functions still execute normally — no spans are sent, no errors are thrown. You don’t need any conditional logic around the API key.

API key resolution

The key is resolved lazily, the first time a span runs, not when the client is constructed. This matters in scripts: with ES modules, an import that constructs the client is evaluated before the importing file’s body runs dotenv.config(), so a key read at construction would be empty even though it is set moments later. Resolving at first use reads the key after env loading has happened.

// Pass a function to defer resolution explicitly (resolved at first use):
new Bitfab({ apiKey: () => process.env.BITFAB_API_KEY })

When no key is passed (or it resolves empty), the SDK falls back to reading BITFAB_API_KEY from the environment, again at first use. For standalone scripts where a run that emits no traces should be treated as a failure rather than silently skipped, set strict:

// Throws on the first traced call if no key resolves, instead of disabling quietly
new Bitfab({ apiKey: process.env.BITFAB_API_KEY, strict: true })

If you load env with dotenv in a script, prefer loading it before the module graph is evaluated, for example node --env-file=.env script.ts, so every module-level read sees the key.

Tracing

Custom (Recommended)

Using `getFunction()` to Link Spans

Declare the trace function key once and wrap multiple functions:

const orderService = bitfab.getFunction("order-processing")

async function processOrder(orderId: string) {
  return { orderId }
}

async function validateOrder(orderId: string) {
  return { valid: true }
}

// Wrap functions - all share the same trace function key
const tracedProcessOrder = orderService.withSpan(processOrder)
const tracedValidateOrder = orderService.withSpan(validateOrder)

Multi-File Projects

For projects with instrumented functions spread across multiple files, create a dedicated file that initializes Bitfab and exports the function. Import it wherever you need to instrument.

// lib/bitfab.ts -- single source of truth
import { Bitfab } from "@bitfab/sdk"
const bitfab = new Bitfab({ apiKey: process.env.BITFAB_API_KEY })
export const orderService = bitfab.getFunction("order-processing")

// services/processOrder.ts
import { orderService } from "../lib/bitfab"

async function processOrder(orderId: string) {
  return { orderId }
}

export const tracedProcessOrder = orderService.withSpan(processOrder)

// services/validateOrder.ts
import { orderService } from "../lib/bitfab"

async function validateOrder(orderId: string) {
  return { valid: true }
}

export const tracedValidateOrder = orderService.withSpan(validateOrder)

Spans from different files are automatically linked as parent-child when one wrapped function calls another.

Wrapping Existing Functions Inline

When wrapping a function you didn’t define (e.g. an SDK or library call), pass it directly to withSpan and call the result immediately. This ensures the arguments are captured as span input.

// ✅ GOOD -- pass function directly, arguments are captured as span input
const result = await orderService.withSpan(
  { name: "ProcessOrder", type: "function" },
  processOrder,
)(orderId)

// ❌ BAD -- anonymous wrapper loses all input capture (span has no input)
const result = await orderService.withSpan(
  { name: "ProcessOrder", type: "function" },
  async () => processOrder(orderId),
)()

Never wrap functions in an anonymous function like async () => fn(args). The SDK captures the wrapper function’s arguments as span input — an anonymous wrapper has no arguments, so the span records nothing.

Using `withSpan()` Directly

For a single span without linking to a function group:

const standaloneTask = bitfab.withSpan("one-off-operation", () => {
  return "done"
})

Automatic Nesting

Spans nest automatically based on call stack:

const outer = bitfab.withSpan("outer", { type: "agent" }, async () => {
  await inner()  // Becomes a child of "outer"
})

const inner = bitfab.withSpan("inner", { type: "function" }, async () => {
  // ...
})

Span Options

Parameters:

traceFunctionKey (required): String identifier for grouping spans
name (optional): Display name. Defaults to function name, then trace function key
type (optional): Span type. Defaults to "custom"
finalize (optional): (result) => serializableView. Record a serializable view of a non-serializable result (a live stream). See Tracing streaming functions

Span Types:

type SpanType =
  | "llm"        // LLM calls
  | "agent"      // Agent workflows
  | "function"   // Function calls
  | "guardrail"  // Safety checks
  | "handoff"    // Human handoffs
  | "custom"     // Default

Examples:

// Function name is automatically captured as span name
async function processOrder(orderId: string) {
  return { orderId }
}
const traced = bitfab.withSpan("order-processing", processOrder)
// Span name: "processOrder"

// Override with name option
const traced = bitfab.withSpan(
  "order-processing",
  { name: "OrderProcessor" },
  processOrder
)
// Span name: "OrderProcessor"

// Set span type
const checkSafety = bitfab.withSpan(
  "safety-check",
  { type: "guardrail" },
  async (content: string) => ({ safe: true })
)

// With getFunction()
const service = bitfab.getFunction("order-processing")
const traced = service.withSpan({ name: "CustomName", type: "function" }, processOrder)

Tracing Streaming Functions

A streaming function returns a live stream object that the caller consumes directly (an SSE response, a UI message stream). That object isn’t serializable as a trace output, and awaiting it to completion before returning would break streaming and first-byte latency. The finalize option solves this: withSpan hands the live stream back to the caller unchanged, but records await finalize(result) as the span output, a drained, serializable, replayable value such as { text, usage, toolCalls }. For the Vercel AI SDK, use the prebuilt finalizers.aiSdk helper. Reading the result’s text / totalUsage promises does not consume the live stream (the AI SDK tees internally), so your own streaming is unaffected. (For automatic per-call llm spans with no withSpan at all, see the Vercel AI SDK framework integration; finalizers.aiSdk is for an explicit root span around the call.)

import { streamText } from "ai"
import { finalizers } from "@bitfab/sdk"

const runChatTurn = bitfab.withSpan(
  "chat-turn",
  { type: "agent", finalize: finalizers.aiSdk },
  () => streamText({ model, messages }),
)

// In your route handler:
const result = runChatTurn()                  // live StreamTextResult
return result.toUIMessageStreamResponse()     // stream to the user as usual
// The span records { text, usage, finishReason, toolCalls } in the background.

Provide your own finalize to record a specific shape:

const runChatTurn = bitfab.withSpan(
  "chat-turn",
  {
    type: "agent",
    finalize: async (r) => ({ answer: await r.text, tokens: await r.totalUsage }),
  },
  () => streamText({ model, messages }),
)

For a raw ReadableStream, use finalizers.readableStream, which tee()s the stream and collects its chunks; the caller must use the live branch it hands back:

import { finalizers } from "@bitfab/sdk"

let live: ReadableStream
const traced = bitfab.withSpan(
  "render",
  { finalize: (r) => finalizers.readableStream(r, (s) => { live = s }) },
  () => makeReadableStream(),
)
traced()
return new Response(live!)

finalize runs in the background and never affects the caller’s value; a finalize that throws records an error on the span instead of crashing the host. It is ignored for async-generator results, which are captured automatically. Inputs to the wrapped function must still be serializable for the trace to replay.

Span Context

Use getCurrentSpan() to get a handle to the active span, then call .addContext() to attach contextual key-value pairs from inside a traced function — useful for runtime values like request IDs, computed scores, or dynamic context:

import { getCurrentSpan } from "@bitfab/sdk"

async function processOrder(orderId: string) {
  const userId = await getCurrentUser()
  getCurrentSpan()?.addContext({ user_id: userId, order_id: orderId })
  return { orderId, status: "completed" }
}

const traced = bitfab.withSpan("order-processing", { type: "function" }, processOrder)

Each addContext call pushes the entire object as one entry. Multiple calls accumulate entries:

getCurrentSpan()?.addContext({ user_id: "u-123" })
getCurrentSpan()?.addContext({ request_id: "req-789" })
// Result: contexts: [{ user_id: "u-123" }, { request_id: "req-789" }]

Span Trace ID

Access the current trace ID from within a span using getCurrentSpan().traceId. This is useful for capturing trace IDs to use with replay or for logging:

import { getCurrentSpan } from "@bitfab/sdk"

const traced = bitfab.withSpan("my-function", async () => {
  const traceId = getCurrentSpan().traceId  // UUID string
  console.log("Current trace:", traceId)
  return { traceId }
})

Outside a span context, getCurrentSpan().traceId returns an empty string.

Span Prompt

Use getCurrentSpan() to set the prompt string on the current span. This is stored in span_data.prompt and is useful for capturing the exact prompt text sent to an LLM:

import { getCurrentSpan } from "@bitfab/sdk"

async function classifyText(text: string) {
  const prompt = `Classify the following text: ${text}`
  getCurrentSpan()?.setPrompt(prompt)
  const result = await llm.complete(prompt)
  return result
}

const traced = bitfab.withSpan("classification", { type: "llm" }, classifyText)

The last setPrompt call wins — it overwrites any previously set prompt on the span. Calling setPrompt outside a span context is a no-op (it never crashes).

Framework Integrations

Bitfab provides automatic tracing for popular AI frameworks. See the dedicated guides for full API references:

LangGraph / LangChain

Callback handler that records a replayable framework root plus graph nodes, LLM calls, tools, and retrievers

OpenAI Agents SDK

Trace processor for agent runs

BAML

Auto-capture prompts and LLM metadata

Claude Agent SDK

Capture LLM turns, tool calls, and subagents

Vercel AI SDK

Language model middleware for every generateText / streamText call

Trace Context

Use getCurrentTrace() to set context that applies to the entire trace (all spans within a single execution). This is useful for grouping traces by session or attaching trace-level metadata:

import { getCurrentTrace } from "@bitfab/sdk"

const traced = bitfab.withSpan("order-processing", { type: "function" }, async () => {
  const trace = getCurrentTrace()

  // Set session ID (stored as database column, filterable in dashboard)
  trace?.setSessionId("session-123")

  // Set trace metadata (stored in raw trace data)
  trace?.setMetadata({ region: "us-west-2", environment: "production" })

  // Add context entries (stored as key-value pairs, accumulates across calls)
  trace?.addContext({ workflow: "checkout-flow", batch_id: "batch-2024-01" })

  return { status: "completed" }
})

setSessionId(id) — Groups traces by user session. Stored as a database column for efficient filtering.
setMetadata(obj) — Arbitrary key-value metadata on the trace. Merges with existing metadata.
addContext(obj) — Key-value context entries. Accumulates across multiple calls.

Dropping a Trace

Call .drop() on the current-trace handle to discard the in-flight trace. Once flagged, spans that complete afterward are not uploaded at all, and the flag rides out on the completion payload, so when the trace completes the server scrubs any payloads that already raced out (the trace, its external trace, and sibling spans), deletes the archived S3 objects, and marks it dropped instead of completed, keeping only a skeleton audit row. Use it to discard runs you never want stored (health checks, test traffic) or a run you know carries sensitive data.

import { getCurrentTrace } from "@bitfab/sdk"

const traced = bitfab.withSpan("order-processing", { type: "function" }, async () => {
  if (isHealthCheck) {
    getCurrentTrace()?.drop()
  }

  return { status: "completed" }
})

Safe to call outside a trace (a no-op), and never throws into your application.

Detached Trace

Use client.getTrace(traceId) to get a handle to a trace that has already closed. This lets you add context, merge metadata, or set the session ID from any process, thread, or agent that knows the trace ID, with no shared in-memory state.

const trace = client.getTrace("order_abc_123")
await trace.addContext({ refund_status: "approved" })
await trace.setMetadata({ region: "us-west" })
await trace.setSessionId("session_xyz")

The traceId must be the caller-supplied ID used when the trace was originally created. All methods are fire-and-forget (return Promises you may await or ignore). Pending requests are tracked so flushTraces() waits for them.

addContext(context) — Appends a context entry. Existing entries are preserved.
setMetadata(metadata) — Shallow-merges new keys into existing metadata.
setSessionId(sessionId) — Replaces any existing session ID.

Error Handling

Errors are captured in the span and re-raised:

const risky = bitfab.withSpan("risky-service", () => {
  throw new Error("error")
})

try {
  risky()
} catch (e) {
  // Span records error and timing
}

Each error is classified by source. Errors thrown by your code are recorded with error_source: "code". SDK-internal errors (e.g. serialization failures) are recorded with source: "sdk". Both appear in the span’s errors array in the Bitfab dashboard.

Advanced Configuration

new Bitfab({
  apiKey: string,                    // Required
  serviceUrl?: string,               // Default: https://bitfab.ai
  timeout?: number,                  // Request timeout in ms (default: 120000)
  envVars?: { OPENAI_API_KEY: string },  // For native function execution
  enabled?: boolean,                 // Default: true
  bamlClient?: unknown               // Generated BAML client (for wrapBAML)
})

timeout: Request timeout in milliseconds for API calls. Defaults to 120000 (2 minutes).
envVars: Pass LLM provider API keys for native function execution via call().
enabled: When false, all tracing is disabled. Wrapped functions still execute normally but no spans are sent.
bamlClient: The generated BAML client instance (e.g., b from @baml). See BAML framework guide for full usage.

Replay

A trace is replayable when its root span has serializable inputs, or when the workflow is instrumented through a framework handler (whose recorded root input is itself serializable). One of these must hold for replay to work. Replay historical traces through an updated function version to compare outputs:

const pipeline = bitfab.getFunction("my-function")

const updatedFn = pipeline.withSpan(
  { name: "UpdatedPipeline", type: "function" },
  async (input: string) => {
    // New implementation to test
    return { result: input.toUpperCase() }
  },
)

// Replay all traces (up to limit)
const result = await bitfab.replay("my-function", updatedFn, {
  name: "Uppercase candidate",
  limit: 10,
  maxConcurrency: 5,  // Default: 10
})

// Replay specific traces by ID
const result2 = await bitfab.replay("my-function", updatedFn, {
  traceIds: ["trace-id-1", "trace-id-2"],
})

// Result structure
console.log(result.testRunId)   // Test run identifier
console.log(result.testRunUrl)  // Dashboard URL
for (const item of result.items) {
  console.log(item.input)           // Original input
  console.log(item.result)          // New output
  console.log(item.originalOutput)  // Original output
  console.log(item.error)           // Error if any
  console.log(item.durationMs)      // Original trace duration in ms (or null)
  console.log(item.tokens)          // { input, output, cached, total } or null
  console.log(item.model)           // Original model name, or null
  console.log(item.traceId)         // Server trace ID for the replayed execution
}

Pass replay() either an already-withSpan-wrapped function (it carries its trace function key, so replay() runs it as-is) or a plain callable (which replay() wraps under the key for you). Do not wrap an already-instrumented function in a fresh closure: a plain arrow like (input) => myWrappedFn(input) carries no trace function key, so replay() adds its own root span around it while myWrappedFn records its own span underneath, nesting a duplicate. If your root is already wrapped, pass it directly: bitfab.replay("my-function", myWrappedFn, ...).

Replay waits for each item’s trace (spans + completion) to be persisted server-side before completing the test run, so item.traceId is a real server trace ID for completed items. Plain callables are wrapped in withSpan internally, so every replayed invocation records a trace. If NO completed item’s trace persisted (uploads wholesale failed), replay() throws a BitfabError instead of silently returning null trace IDs. If only SOME items’ traces are missing (a transient per-item upload failure), those items get null trace IDs with a loud console.error and the rest of the run is returned intact. item.traceId is also null for errored (unreplayable) items, and for all items when the server predates the trace-ID mapping (a console warning explains which). Per-item durationMs and model come from the historical trace that fed this replay item. tokens is the replayed run’s token usage (the same numbers Studio’s experiments view shows), so comparing each item’s tokens.total against the original trace’s recorded usage tells you how your change moved cost. Each field is null when it wasn’t captured. Options:

limit — Maximum number of traces to replay (default: 5). Ignored when traceIds is passed (with a warning): an explicit ID list already determines how many traces replay.
traceIds — Specific trace IDs to replay (max 100). The ID count determines how many traces replay; limit is ignored when both are passed.
name — Optional display name for the resulting experiment/test run.
maxConcurrency — Number of traces to replay in parallel (default: 10)
codeChangeDescription — Optional rationale for the code change being tested in this replay (stored on the experiment)
codeChangeFiles — Optional list of edited files, each as { path, before, after } (use "" for newly created or deleted files)
mock — Mock strategy for child spans during replay: "marked" (default, only return historical output for child spans declared with mockOnReplay: true), "none" (run real code for every child), or "all" (return historical output for every child). See Mocking child spans during replay below.
experimentGroupId — Optional UUID string that groups multiple replay runs into a single experiment batch. Pass the same ID across successive replay() calls to link them together in the dashboard.
adaptInputs — Optional hook to reshape recorded inputs onto the function’s current signature when its shape changed after the traces were captured. See Adapting inputs after a signature change below.
onProgress — Optional callback fired once per item as it settles, with running totals plus the settled item payload (source trace id, local replay trace id, input, result, original output, error, duration, tokens/model metadata). Use it to render live progress or start evaluating completed items while replay runs. A throwing callback never crashes the run. Bitfab plugin replay scripts can pass the SDK’s ready-made reportReplayProgress callback straight in (onProgress: reportReplayProgress); it writes the event to stderr, which the Bitfab plugin polls to report live progress and write per-item result files while replay runs (stdout remains available for direct-run ReplayResult JSON).
environment — Optional ReplayEnvironment. When passed, the Bitfab server resolves a per-trace database branch from each source trace’s captured snapshot reference, and the SDK exposes that branch’s URL via environment.databaseUrl inside the replayed function (releasing the branch after each item). Read environment.active to fall back to your live database when no branch was resolved (e.g. the trace predates snapshot capture, or DB branching isn’t configured). Construct one with new ReplayEnvironment() and read it only inside the replayed function.

Replaying handler-instrumented functions

Workflows instrumented through a framework handler (getLangGraphCallbackHandler, getLangChainCallbackHandler, getClaudeAgentHandler, getOpenAiAgentHandler) have no withSpan-wrapped root in the application code: the handler (or run wrapper) records the framework invocation itself as the root span, with the framework’s own input (a LangGraph initial state, an agent prompt, the run input) as the recorded root input. LangGraph/LangChain roots are registered as pending traces when the root callback starts and completed when it ends, so long-running runs can appear before final output is available. These traces are fully replayable. Pass the handler’s trace function key plus any plain callable that re-invokes the framework entrypoint:

The OpenAI Agents SDK uses getOpenAiAgentHandler(key).wrapRun(agent, input) (a drop-in for run) for the replayable root; the bare getOpenAiTracingProcessor captures internals only and records an empty-input root. The Claude Agent SDK handler needs a hint: the prompt is not present in the message stream, so pass it explicitly, wrapQuery(stream, { input: prompt }) (or wrapResponse(stream, { input })), for the handler to record a replayable root.

// scripts/replay.ts
import { graph } from "../src/agent"          // the compiled LangGraph graph
import { bitfab } from "../src/bitfabClient"  // same client as instrumentation

const handler = bitfab.getLangGraphCallbackHandler("my-agent")  // same key

const replayMyAgent = async (state: AgentState) =>
  graph.invoke(state, {
    callbacks: [handler],
    configurable: buildReplayConfig(),
  })

const result = await bitfab.replay("my-agent", replayMyAgent, { limit: 10 })

How it fits together:

replay() fetches the handler-recorded production traces by the key string, and wraps a plain callable in withSpan under that key internally so each replayed invocation records a trace tied to the test run. The key is the only link; it does not matter that production traces were written by the handler and the callable was written today.
Passing an already-withSpan-wrapped function under the same key also works (older SDKs require this form); a wrapped function whose key contradicts the replay key throws.
The recorded root input is whatever the handler captured at the framework boundary (a LangGraph state object arrives as a single argument).
Attaching the handler inside the callable makes the replayed graph’s node/LLM/tool spans nest under the replay span, so replay traces have the same tree as production ones.
The callable rebuilds the runtime environment the trace never captured: framework configurable, dependency objects, API keys. Use safe no-op substitutes for side-effectful wiring (billing or credit callbacks, notification senders); replay should never charge or notify anyone.

Mocking child spans during replay

For the workflow-level guide, see Replay Mocking. When iterating on a root function, child spans sometimes fail in your local environment for reasons unrelated to the code under test: a paid API key is missing, an external service is flaky, or a production-only DB row isn’t seeded locally. The mock option lets the child return its recorded output so the root function can still run. Three strategies on replay():

"marked" (default): only descendants declared with mockOnReplay: true are short-circuited; everything else runs real. This is the iteration-friendly mode.
"none": every child span runs real code. Use when your local environment can faithfully reproduce the trace.
"all": every descendant withSpan returns its historical output. The root function still runs real, but every child is short-circuited. Useful for a quick sanity-check against recorded data; not the recommended iteration strategy because changes to descendants won’t actually execute.

Per-span opt-in via SpanOptions.mockOnReplay:

const fetchArticle = bitfab.withSpan(
  "fetch-article-from-db",
  { mockOnReplay: true },
  async (id: string) => db.articles.findById(id),
)

const summarize = bitfab.withSpan(
  "summarize-article",
  async (article: Article) => /* real summarization, no flag */
)

const processArticle = bitfab.withSpan(
  "process-article",
  async (id: string) => summarize(await fetchArticle(id)),
)

// During replay, fetch-article-from-db returns its recorded output;
// summarize-article runs real so you can iterate on it.
const result = await bitfab.replay("process-article", processArticle, {
  limit: 10,
})

mockOnReplay is a per-span tag at definition time — it has no effect outside replay, and it’s read by the default mock: "marked" strategy. The root function always runs real code; only descendants can be mocked. When no historical span matches a child call (e.g. the recorded trace didn’t reach that branch), execution falls through to the real function — never silent omission.

Adapting inputs after a signature change

Replay deserializes each trace’s inputs exactly as they were captured against the function’s signature at trace time, then spreads them into the current function. If the signature drifted since capture (a param renamed, reordered, collapsed into an options object, or a new required arg added), that spread no longer lines up and the call throws. The adaptInputs hook reshapes the recorded inputs onto the current signature so replay can still run:

const result = await bitfab.replay("my-function", updatedFn, {
  // Recorded as (userId, limit); current signature is ({ userId, limit }).
  adaptInputs: (inputs, ctx) => {
    const [userId, limit] = inputs as [string, number]
    return [{ userId, limit }]
  },
})

The hook runs once per item, inside the same error boundary as the function: if it throws, that item’s error is set and the run continues, so a single unmappable trace never crashes the batch. The array it returns is what gets spread into the function and what item.input reports. ctx carries { traceId, sourceSpanId } so a table-driven adapter can look up a per-trace transform (ctx.traceId is the original Bitfab trace ID). This is the escape hatch for reshapes that need judgement rather than mechanical rearrangement: compute the adapted inputs per trace up front, then have the hook look them up by traceId — keeping replay deterministic instead of calling a model mid-replay. When the new signature has a genuinely new required input with no analog in the recorded trace, don’t fabricate one — there’s nothing faithful to map it to. Leave those traces unmapped (let them error) rather than inventing test inputs. For anything beyond a one-liner, keep the adapter in its own file next to the replay script and import it — the AdaptInputsFn type is exported for this:

// scripts/replay-adapters/extraction.ts
import type { AdaptInputsFn } from "@bitfab/sdk"

export const adaptInputs: AdaptInputsFn = (inputs, ctx) => {
  const [userId, limit] = inputs as [string, number]
  return [{ userId, limit }]
}

// scripts/replay.ts
import { adaptInputs } from "./replay-adapters/extraction"

await bitfab.replay("my-function", updatedFn, { limit, adaptInputs })

That keeps the transform versioned and reviewable alongside the function it adapts, and you add the import only when a drift actually needs it.

Attaching a Code Change

Each replay creates an experiment (test run). When you’re iterating on a function and replaying after every edit, attach the change so the dashboard can show exactly what was edited alongside the results. The agent reads each file before editing, edits, then reads it again — the two strings go straight into codeChangeFiles. There’s no diff format to construct.

import { readFileSync } from "node:fs"

const before = readFileSync("src/foo.ts", "utf8")
// ...edit src/foo.ts...
const after = readFileSync("src/foo.ts", "utf8")

const result = await bitfab.replay("my-function", updatedFn, {
  name: "Retry fix candidate",
  codeChangeDescription: "fix off-by-one in retry logic",
  codeChangeFiles: [{ path: "src/foo.ts", before, after }],
})

Both options are optional and independent — you can pass just codeChangeDescription for a quick rationale-only annotation, or just codeChangeFiles to record the literal edits. Notes:

Use a single Bitfab client across instrumentation and replay. If your instrumented module constructs new Bitfab() at import and your replay script constructs another, they do not share registered trace functions — import the client from the instrumented module (or a shared singleton) rather than constructing a new one in the replay script.

Replay Output Contract

Replay results are typically consumed by automation (CI logs, code reviewers, and coding agents). When BITFAB_REPLAY_RESULT_PATH is set, bitfab.replay() automatically writes the full ReplayResult JSON to that file. For direct/manual runs, emit the full ReplayResult as a single stdout JSON block so a consumer can JSON.parse it and reason about every field, including the new per-item durationMs, tokens, and model. Never print only lengths, counts, hashes, or truncated previews, and never replace the JSON block with ad-hoc per-field log lines. Recommended script tail (TypeScript):

const result = await bitfab.replay("my-function", updatedFn, { limit })

// Human-readable summary goes to stderr, so stdout stays pure JSON.
console.error(`Test run: ${result.testRunUrl}`)
console.error(`Items:    ${result.items.length}`)

// Full structured dump to stdout, ready for JSON.parse.
console.log(JSON.stringify(result, null, 2))

The dumped object includes every item’s input, result, originalOutput, error, durationMs, tokens, model, and traceId, plus testRunId and testRunUrl. When the Bitfab plugin runs this script, it sets BITFAB_REPLAY_RESULT_PATH; the SDK writes the final result there, and the plugin reads that file into the replay run’s .bitfab/replays/<run-id>/events.jsonl while writing large per-item payloads under .bitfab/replays/<run-id>/items/. Per-item errors are part of the contract. If the wrapped function throws on a given trace, bitfab.replay catches it, sets item.error, leaves item.result undefined, and continues. Treat items with item.error set as unreplayable, not as failing outputs — compute pass/fail only over items where it’s unset. This matters most for DB reads/writes: a stale FK, missing record, or rejected write is infra failure, not a regression. Don’t swallow per-item errors in the script. A custom try/catch that returns a placeholder turns infra failures into fake successes. Let the SDK record them. The only allowed top-level catch is a fatal handler around main() that exits non-zero, so callers can tell a whole-replay crash from a clean run with some unreplayable items. Environment. Replay executes in the app’s own process — the instrumented function is imported as a library, and its DB clients, env vars, config loaders, and model IDs resolve from whatever environment the replay script is run under. The script must bootstrap the same environment the app uses (e.g. import "dotenv/config" at the top, or run via pnpm with-env tsx scripts/replay.ts). Do not mock these — they’re the same dependencies the app resolves in production. For replay to see the same DB rows the trace was captured against, point the script at the trace’s source environment (the environment field on the trace — production / staging / development). Input serialization caveat. Replay deserializes historical span inputs and passes them back to your function. This works for strings, numbers, and plain objects. If your span wraps a function that takes hydrated domain objects (ORM models, class instances, DB records), they won’t round-trip through serialization — move the span to where inputs are IDs or plain data and let the function fetch objects internally, or reshape arguments in the wrapper.

Replay Script

Create a standalone script to regression-test your trace functions against production data with one command. The script maps pipeline names to their replay functions, accepts CLI flags, and prints a side-by-side comparison with delta summaries.

/**
 * Replay production traces through instrumented functions.
 *
 * Uses bitfab.replay() to fetch real traces and re-run them through
 * the current code, creating a test run for side-by-side comparison.
 *
 * Usage:
 *   npx tsx scripts/replay.ts <pipeline>
 *   npx tsx scripts/replay.ts <pipeline> --limit 20
 *   npx tsx scripts/replay.ts <pipeline> --trace-ids id1,id2
 */
import "dotenv/config"
import { reportReplayProgress } from "@bitfab/sdk"
import { bitfab } from "../lib/bitfab"
import { extractMemories } from "../services/extraction"
import { searchDocuments } from "../services/search"

const FUNCTIONS = {
  extraction: "my-extraction-pipeline",
  search: "my-search-pipeline",
} as const

type Pipeline = keyof typeof FUNCTIONS

const pipeline = process.argv[2] as Pipeline | undefined
const args = process.argv.slice(3)

if (!pipeline || !FUNCTIONS[pipeline]) {
  console.error(
    `Usage: npx tsx scripts/replay.ts <${Object.keys(FUNCTIONS).join("|")}> [--limit N] [--trace-ids id1,id2]`,
  )
  process.exit(1)
}

let limit = 10
let traceIds: string[] | undefined

for (let i = 0; i < args.length; i++) {
  if (args[i] === "--limit" && args[i + 1]) {
    limit = Number.parseInt(args[i + 1], 10)
    i++
  } else if (args[i] === "--trace-ids" && args[i + 1]) {
    traceIds = args[i + 1].split(",").map((id) => id.trim())
    i++
  }
}

// Each pipeline gets its own replay function -- replay deserializes
// historical inputs and spreads them into the function. If a signature drifts
// after traces were captured, add an input adapter (see the "Adapting inputs"
// section): write scripts/replay-adapters/<name>.ts, import its `adaptInputs`,
// and pass it here, e.g.
//   import { adaptInputs } from "./replay-adapters/extraction"
//   return bitfab.replay(FUNCTIONS.extraction, fn, { limit, adaptInputs })

async function replayExtraction() {
  const fn = async (conversation: string, existingItems: unknown[]) => {
    return extractMemories(conversation, existingItems)
  }
  // limit is ignored when traceIds is passed -- an explicit ID list
  // already determines how many traces replay.
  return bitfab.replay(FUNCTIONS.extraction, fn, {
    ...(traceIds ? { traceIds } : { limit }),
    // Live progress the Bitfab plugin polls while replay runs in the background.
    onProgress: reportReplayProgress,
  })
}

async function replaySearch() {
  const fn = async (query: string, opts: Record<string, unknown>) => {
    return searchDocuments(query, {
      userId: opts.userId as string,
      limit: (opts.limit as number) ?? 10,
    })
  }
  return bitfab.replay(FUNCTIONS.search, fn, {
    ...(traceIds ? { traceIds } : { limit }),
    // Live progress the Bitfab plugin polls while replay runs in the background.
    onProgress: reportReplayProgress,
  })
}

const REPLAY_FNS: Record<Pipeline, () => ReturnType<typeof bitfab.replay>> = {
  extraction: replayExtraction,
  search: replaySearch,
}

async function main() {
  const functionKey = FUNCTIONS[pipeline!]
  // Human-readable output goes to stderr; stdout carries only the ReplayResult JSON.
  console.error(`[replay] Replaying ${traceIds?.length ?? limit} traces from "${functionKey}"...\n`)

  const result = await REPLAY_FNS[pipeline!]()
  console.error(`Test run: ${result.testRunUrl}\n`)

  let changed = 0, same = 0, errors = 0

  for (const item of result.items) {
    const input = item.input as unknown[] | undefined
    const label = typeof input?.[0] === "string"
      ? input[0].slice(0, 80)
      : JSON.stringify(input?.[0] ?? "unknown").slice(0, 80)

    if (item.error) {
      console.error(`  ✗ "${label}"`)
      console.error(`    Error: ${item.error}`)
      errors++
    } else {
      const origStr =
        typeof item.originalOutput === "string"
          ? item.originalOutput
          : JSON.stringify(item.originalOutput)
      const newStr =
        typeof item.result === "string"
          ? item.result
          : JSON.stringify(item.result)
      const isSame = origStr === newStr
      const marker = isSame ? "=" : "Δ"

      console.error(`  ${marker} "${label}"`)
      console.error(`    Original: ${origStr}`)
      console.error(`    New:      ${newStr}`)

      isSame ? same++ : changed++
    }
  }

  console.error(`\n─── Summary ───`)
  console.error(`  Pipeline: ${pipeline}`)
  console.error(`  Replayed: ${result.items.length}`)
  console.error(`  Same:     ${same}`)
  console.error(`  Changed:  ${changed}`)
  if (errors > 0) console.error(`  Errors:   ${errors}`)
  console.error(`\n  ${result.testRunUrl}`)

  // Full ReplayResult as JSON to stdout (the Replay Output Contract).
  console.log(JSON.stringify(result, null, 2))
}

main().catch((err) => {
  console.error("[replay] Fatal error:", err)
  process.exit(1)
})

Adapt the imports, pipeline names, and per-pipeline replay functions to match your project’s instrumented workflows.

​Installation

​Quick Start

​Basic Configuration

​API key resolution

​Tracing

​Custom (Recommended)

​Using getFunction() to Link Spans

​Multi-File Projects

​Wrapping Existing Functions Inline

​Using withSpan() Directly

​Automatic Nesting

​Span Options

​Tracing Streaming Functions

​Span Context

​Span Trace ID

​Span Prompt

​Framework Integrations

LangGraph / LangChain

OpenAI Agents SDK

BAML

Claude Agent SDK

Vercel AI SDK

​Trace Context

​Dropping a Trace

​Detached Trace

​Error Handling

​Advanced Configuration

​Replay

​Replaying handler-instrumented functions

​Mocking child spans during replay

​Adapting inputs after a signature change

​Attaching a Code Change

​Replay Output Contract

​Replay Script

Installation

Quick Start

Basic Configuration

API key resolution

Tracing

Custom (Recommended)

Using `getFunction()` to Link Spans

Multi-File Projects

Wrapping Existing Functions Inline

Using `withSpan()` Directly

Automatic Nesting

Span Options

Tracing Streaming Functions

Span Context

Span Trace ID

Span Prompt

Framework Integrations

Trace Context

Dropping a Trace

Detached Trace

Error Handling

Advanced Configuration

Replay

Replaying handler-instrumented functions

Mocking child spans during replay

Adapting inputs after a signature change

Attaching a Code Change

Replay Output Contract

Replay Script