Ruby SDK - Bitfab

The Bitfab Ruby SDK captures your AI function calls to automatically generate evaluations. Re-run your prompts with different models, parameters, and inputs to iterate faster.

Framework-native adapters (LangGraph, OpenAI Agents, BAML, Claude Agent SDK) are not yet available for Ruby. See Frameworks overview for current coverage. Instrument Ruby code manually via Bitfab::Traceable or Bitfab.span.

Installation

# Bundler
bundle add bitfab

# Gem
gem install bitfab

Quick Start

require "bitfab"

Bitfab.configure(api_key: ENV.fetch("BITFAB_API_KEY"))

Need an API key? Get one from the Bitfab dashboard or see the API Keys guide for detailed setup instructions.

Coding Agent Prompt (Cursor, Claude Code)

Copy this prompt into your coding agent (tested with Cursor and Claude Code using Sonnet 4.5):

Modify existing Ruby code to add Bitfab tracing.
Do NOT browse or web search. Use ONLY the API described below.

Bitfab Ruby SDK (authoritative excerpt):
- Install: `gem install bitfab` or `bundle add bitfab`
- Init:
  require "bitfab"
  Bitfab.configure(api_key: ENV.fetch("BITFAB_API_KEY"))
- Instrumentation (ONLY allowed form):
  class MyService
    include Bitfab::Traceable
    bitfab_function "<trace_function_key>"

    bitfab_span :method_name, type: "function"
    def method_name
      # ...
    end
  end
  (bitfab_span must be placed immediately ABOVE the `def` it instruments.)
- Span types: "llm", "agent", "function", "guardrail", "handoff", "custom"
- DO NOT use a block form of bitfab_span.
- DO NOT extract helper methods.

Task:
1) Ensure the bitfab gem is added and initialization exists (Gemfile + initializer).
2) Read the codebase and identify ALL AI workflows (LLM calls, agent runs, AI-driven decisions).
3) Present me with a numbered list of workflows you found. For each, describe:
   - What it does
   - Why it's worth instrumenting: what visibility tracing gives you into each step
4) After I choose which workflow(s) to instrument:
   - Add `bitfab_span` directly ABOVE each method's `def`
   - Produce a SPAN TREE, not one span. The workflow root gets a span, and so does every step inside it:
     each model call, each read of external state (DB query, HTTP GET, storage, vector search, cache),
     each transform of the model output (parsing, validation, ranking, formatting), each retry or loop
     iteration, and each external write. A single span around the outer function records one input and
     one output for the whole workflow, which leaves replay mocking and per-step diagnosis with nothing
     to work on. Do not wrap trivial in-memory helpers or per-item work inside a large loop.
   - Mark reads and writes to mock on replay so a replayed trace does not repeat side effects
   - Ensure each class includes `Bitfab::Traceable` and has `bitfab_function` set
5) Do not change method signature, behavior, or return value. Minimal diff.

Output:
- First: your numbered list of workflows with why each is worth instrumenting
- After my selection: minimal diffs for Gemfile, initializer, and the method changes

Basic Configuration

Bitfab.configure(api_key: "...")

# Omit api_key entirely: the SDK reads ENV["BITFAB_API_KEY"]
Bitfab.configure

# Disable tracing (methods still execute, but no spans are sent)
Bitfab.configure(api_key: "...", enabled: false)

# Leave it out and BITFAB_CAPTURE_ENABLED=0 or 1 decides
Bitfab.configure(api_key: "...")

BITFAB_CAPTURE_ENABLED decides when your code does not. Set it to 0 to turn capture off or 1 to turn it on, and the switch applies to every client that was built without one. A value passed in code always wins over the variable.

Missing API key doesn’t crash. If the API key is missing, empty, or whitespace-only, the SDK automatically disables tracing and logs a one-time warning at first use. All instrumented methods still execute normally: no spans are sent, no errors are thrown. You don’t need any conditional logic around the API key.

API key resolution

The key is resolved lazily, the first time a span runs, not when configure is called. Resolution order is the configured value (a Proc is called while still unresolved), then a fallback read of ENV["BITFAB_API_KEY"].

# Pass a Proc to defer resolution explicitly (resolved at first use):
Bitfab.configure(api_key: -> { ENV["BITFAB_API_KEY"] })

For standalone scripts where a run that emits no traces should be treated as a failure rather than silently skipped, set strict:

# Raises on the first traced call if no key resolves, instead of disabling quietly
Bitfab.configure(api_key: ENV["BITFAB_API_KEY"], strict: true)

Tracing

Trace the whole workflow, not just its entrypoint. A single bitfab_span around the outer function records one input and one output for everything inside it, which leaves replay mocking, per-step diagnosis, and prompt iteration with nothing to work on. Spans exist only where you create them: nesting is automatic, but only between spans that exist.

Give a step its own span when any of these is true:

It calls a model. Always. This is the span you iterate on, compare across experiments, and attach graders to.
It reads external mutable state (DB query, HTTP GET, object storage, vector search, cache). These are the spans you will want to mock on replay.
It writes external state (DB write, queue publish, email, charge, file write). Mark these to mock on replay so a replayed trace does not repeat the side effect.
It transforms the model output (parsing, validation, ranking, formatting), so a quality regression points at the model or at your post-processing.
It retries or loops, one span per attempt or iteration, so a trace shows how many attempts it really took.

Skip trivial in-memory helpers, per-item work inside a large loop (wrap the loop or the batch), and internals a framework integration already captures. Worked examples, replay-mocking decisions, and common pitfalls: Instrumentation.

Custom (Recommended)

Using `Bitfab::Traceable` to Link Spans

Include Bitfab::Traceable in a class, declare the trace function key once with bitfab_function, then use bitfab_span to wrap methods. Three declaration styles are supported:

class OrderService
  include Bitfab::Traceable
  bitfab_function "order-processing"

  # The root calls the traced methods, so they record as its children
  bitfab_span :process_order, type: "agent"
  def process_order(order_id)
    order = load_order(order_id)
    classification = classify_order(order)
    validate_order(classification)
  end

  # Style 1: Before-def (recommended): declare bitfab_span above the method
  bitfab_span :load_order, mock_on_replay: true
  def load_order(order_id)
    Order.find(order_id)
  end

  # Style 2: Inline: wrap the def directly
  bitfab_span def classify_order(order)
    client.responses.parse(model: MODEL, input: build_prompt(order))
  end, type: "llm"

  # Style 3: After-def: declare bitfab_span after the method definition
  def validate_order(classification)
    { valid: classification.confidence > 0.8 }
  end
  bitfab_span :validate_order, type: "guardrail"
end

Calling process_order(id) records one trace with four spans:

process_order
├── load_order
├── classify_order
└── validate_order

Tracing only process_order would record the same work as a single node, with the model call, the database read, and the validation collapsed into the root’s input and output. All three styles are equivalent. The before-def style is recommended for readability.

Experimental: Trace a Whole Call Tree

bitfab_trace records one root method plus every first-party Ruby method it calls, without adding a declaration to each descendant:

class OrderService
  include Bitfab::Traceable
  bitfab_function "order-processing"

  bitfab_trace :process_order, type: "agent"
  def process_order(order_id)
    order = load_order(order_id)
    summarize(order)
  end

  def load_order(order_id)
    Order.find(order_id)
  end

  def summarize(order)
    { id: order.id, total: order.total }
  end
end

Ruby’s TracePoint hook is enabled only during active capture. It records project methods across files, recursion, modules, inherited methods, define_method, and same-thread child Fibers; aliases use the name that was invoked. It skips Ruby/runtime code, dependency gems, Bitfab SDK internals, blocks, lambdas, forwarding decorator methods, and child-thread calls, and continues through lazy Enumerator consumption. If the traced root belongs to an installed application gem, that gem is the first-party boundary; neighboring gems remain excluded. Use max_depth: (default 30), max_captured_subtree_spans: (default 500), exclude:, and include_wrappers: to control capture. There is no cap on the total number of spans one trace root sends. max_spans: was removed, and passing it raises ArgumentError. max_captured_subtree_spans: caps discovered calls sent with their inputs and outputs. It does not count bitfab_node calls, nested trace roots, or spans the loaded sim plan turned capture off for, so those keep recording past it. Spans sent without inputs and outputs only because the sim plan could not be read still count toward it. Past it, further discovered calls with capture on record nothing. Once the plan has loaded, the SDK does not send a span whose capture it turned off, and that span’s children carry the nearest captured ancestor the SDK resolved while the trace ran, so they nest under the closest call above them whose capture is on. The trace’s root span, a framework integration span, and a span that recorded an error are still sent, and the SDK never builds their inputs or outputs. A bitfab_node call and a nested trace root ignore the plan and always keep their inputs and outputs. A discovered call that shares a name with a bitfab_node call is still dropped. The client starts reading the sim plan when it is built. A bitfab_trace root that starts while that first read is still running waits for it, for at most five seconds. Once the first read has finished, whether it succeeded or failed, no later root waits. Nested roots and the calls beneath a root never wait, and nothing waits when the plan is disabled. If that first read fails, spans other than each trace’s root span, bitfab_node calls, and nested trace roots are sent without inputs and outputs until a read succeeds, and they still count toward max_captured_subtree_spans:. The client reads the plan only when an API key resolves and the client is recording, meaning capture is on or the call runs inside a replay or seed. Otherwise it sends no read and prints no warning. Child Fibers created during capture inherit node policies, trace identity, parenting, and depth limits while keeping separate call stacks. Pre-existing Fibers and Fiber.new(storage: nil) do not inherit subtree context. When an inner capture ends, child Fibers retain any still-active outer capture; inherited context expires once all enclosing captures end. While a root remains active, hitting a span limit stops new capture but allows already-open spans in every Fiber to finish. Independent roots in interleaved Fibers remain isolated; untraced scheduler calls between them are not captured. When capture ends, already-open automatic calls in suspended Fibers are emitted once with a nil output and Subtree capture ended before the call returned as the error, unless a real error was already observed. Their inputs and parenting are retained. The SDK does not resume Fibers, and later resumption cannot add spans to the ended capture. If hot reload moves the root to an unresolved source, the last valid first-party boundary and its exclusions remain in use. Returning an Enumerator keeps capture pending until iteration finishes or raises, including calls already paused in child Fibers. Capture is inactive between the initial return and consumption. Automatic spans record lazy return values as placeholders without iterating them; the traced root records the yielded values. Nesting traced roots. Each nested bitfab_trace starts an independent trace during ordinary capture, including recursion and calls using the same key. Its complete subtree also remains in every enclosing trace. The recordings have distinct span IDs and carry links between the enclosing span and nested root. Every copy of a span also carries the contexts and prompt the call added, so each trace’s model and token totals include the calls that ran inside it. Each trace applies its own limits and exclusions. During replay and seed execution, nested annotations stay in the managed trace instead of creating independent traces. Use bitfab_node to configure methods beneath bitfab_trace. Mixing bitfab_span or Traceable.wrap with subtree tracing raises Bitfab::MixedTracingError. Default names include the class name when available, such as OrderService.process_order. Configure important calls. bitfab_node changes how a method is captured only when it runs beneath bitfab_trace. Standalone calls remain completely untraced, and the enclosing trace owns the node’s function key, parent, limits, and lifecycle.

bitfab_node :summarize,
  name: "Model call",
  type: "llm",
  mock_on_replay: true,
  finalize: ->(response) { {text: response.text} }
def summarize(order)
  model.generate(order)
end

Use name: and type: to identify the call. Set capture: false to omit it while transparently reparenting captured descendants. Use experiment_id: for attribution. Set mock_on_replay: true to reuse recorded output under marked replay. A node inherits mock_on_replay_default: from its trace when the node omits the option. An explicit false overrides the trace default under marked replay. Explicit all-mocking and custom overrides retain precedence. Use finalize: to record a serializable view without changing the caller’s return value. Finalization runs once per call, and every enclosing trace records the same configured name, type, attribution, and finalized output. For an Enumerator, finalization receives collected values after iteration finishes. Finalizer failures are recorded on the node and do not crash the caller. The before-def, inline, and after-def forms preserve visibility. Configure an external method for one client with client.node(MyClass, :method). Performance. The SDK prepares source boundaries at declaration, refreshes a boundary only if hot reload moves the root definition, caches canonical paths, and shares one TracePoint dispatcher across active roots on a thread. After a span limit is reached, it stops reading inputs and outputs immediately, then stops processing events for that session once its captured calls finish. It must still inspect every Ruby method event while capture is active and serialize one span per captured call. Keep bitfab_trace for discovery, set conservative limits, and prefer explicit bitfab_span declarations on production hot paths.

bitfab_trace :build_order_detail, trace_function_key: "order-detail"
def build_order_detail(order)
  summarize(order)
end

bitfab_trace :process_order, trace_function_key: "order-processing"
def process_order(order_id)
  build_order_detail(load_order(order_id))
end

This API is experimental. Automatically captured calls cannot be replay-mocked. Declare a call with bitfab_node when replay must substitute it. Each enclosing trace records the configured node once and parents automatically captured descendants beneath it.

Multi-File Projects

For projects with instrumented methods spread across multiple files, create an initializer that configures Bitfab, then include Bitfab::Traceable in any class that needs tracing.

# config/initializers/bitfab.rb (single source of truth)
require "bitfab"
Bitfab.configure(api_key: ENV.fetch("BITFAB_API_KEY"))

# app/services/process_order_service.rb
class ProcessOrderService
  include Bitfab::Traceable
  bitfab_function "order-processing"

  bitfab_span :process_order, type: "function"
  def process_order(order_id)
    ValidateOrderService.new.validate_order(order_id)
    { order_id: }
  end
end

# app/services/validate_order_service.rb
class ValidateOrderService
  include Bitfab::Traceable
  bitfab_function "order-processing"

  bitfab_span :validate_order, type: "guardrail"
  def validate_order(order_id)
    { valid: true }
  end
end

Classes sharing the same bitfab_function key are grouped together. Spans from different classes are automatically linked as parent-child when one instrumented method calls another.

Using `bitfab_span` for a Single Span

Declare the trace function on the class, then the span macro picks it up:

class StandaloneService
  include Bitfab::Traceable
  bitfab_function "one-off-operation"

  bitfab_span :standalone_task
  def standalone_task
    "done"
  end
end

bitfab_span :standalone_task, trace_function_key: "one-off-operation" is equally supported and does the same thing. Pass the key per method when one class spans more than one trace function.

Automatic Nesting

Spans nest automatically based on call stack:

class Pipeline
  include Bitfab::Traceable
  bitfab_function "pipeline"

  bitfab_span :outer, type: "agent"
  def outer
    inner  # Becomes a child of "outer"
  end

  bitfab_span :inner, type: "function"
  def inner
    # ...
  end
end

For reusable helpers that should appear only inside an existing trace, set capture_when: "nested":

class Pipeline
  include Bitfab::Traceable
  bitfab_function "pipeline"

  bitfab_span :helper, type: "function", capture_when: "nested"
  def helper
    # ...
  end
end

Pipeline.new.helper # Runs normally without creating a trace when no parent is active

Span Options

Parameters:

method_name (required): Symbol of the method to wrap
trace_function_key (optional): Override class-level bitfab_function
name (optional): Display name. Defaults to the qualified class and method name when the class has a name
type (optional): Span type. Defaults to "custom". A label only, used to organize and filter spans in the dashboard; it does not change how the span is traced, replayed, or evaluated
capture_when (optional): "always" / :always (default) or "nested" / :nested. Nested-only spans are captured under an active parent and run untraced when called standalone. Unknown values warn once and default to "always"

bitfab_span, Bitfab::Traceable.wrap, and client.get_function(key).wrap also accept finalize: and experiment_id:. test_run_id: remains a deprecated alias for experiment_id:. Finalizers change only the recorded output. An Enumerator remains lazy and finalizes its collected values once iteration finishes. Span Types:

SPAN_TYPES = %w[
  llm        # LLM calls
  agent      # Agent workflows
  function   # Function calls
  guardrail  # Safety checks
  handoff    # Human handoffs
  custom     # Default
]

Examples:

class SafetyService
  include Bitfab::Traceable
  bitfab_function "safety-service"

  # Method name is automatically captured as span name
  bitfab_span :check_safety, type: "guardrail"
  def check_safety(content)
    { safe: !content.include?("unsafe") }
  end

  # Override with name option
  bitfab_span :validate_input, name: "InputValidator", type: "guardrail"
  def validate_input(input)
    { valid: !input.empty? }
  end
end

Span Context

Use Bitfab.current_span to get a handle to the active span, then call .add_context() to attach contextual key-value pairs from inside a traced method, useful for runtime values like request IDs, computed scores, or dynamic context:

class OrderService
  include Bitfab::Traceable
  bitfab_function "order-processing"

  bitfab_span :process_order, type: "function"
  def process_order(order_id)
    user_id = current_user_id
    Bitfab.current_span.add_context("user_id" => user_id, "order_id" => order_id)
    { order_id: order_id, status: "completed" }
  end
end

Each add_context call pushes the entire hash as one entry. Multiple calls accumulate entries:

Bitfab.current_span.add_context("user_id" => "u-123")
Bitfab.current_span.add_context("request_id" => "req-789")
# Result: contexts: [{ "user_id" => "u-123" }, { "request_id" => "req-789" }]

You can also access the canonical Bitfab span and trace IDs via Bitfab.current_span.id and Bitfab.current_span.trace_id (both return an empty string outside a span):

span_id = Bitfab.current_span.id
trace_id = Bitfab.current_span.trace_id

Span Prompt

Use Bitfab.current_span to set the prompt string on the current span. This is stored in span_data.prompt and is useful for capturing the exact prompt text sent to an LLM:

class ClassificationService
  include Bitfab::Traceable
  bitfab_function "classification"

  bitfab_span :classify_text, type: "llm"
  def classify_text(text)
    prompt = "Classify the following text: #{text}"
    Bitfab.current_span.set_prompt(prompt)
    llm.complete(prompt)
  end
end

The prompt is metadata only. It records the prompt text for display and reference in the dashboard; it does not send the prompt to any model or change what the span executes. The last set_prompt call wins — it overwrites any previously set prompt on the span. Calling set_prompt outside a span context is a no-op (it never crashes).

Trace Context

Use Bitfab.current_trace to set context that applies to the entire trace (all spans within a single execution). This is useful for grouping traces by session or attaching trace-level metadata:

class OrderService
  include Bitfab::Traceable
  bitfab_function "order-processing"

  bitfab_span :process_order, type: "function"
  def process_order(order_id)
    trace = Bitfab.current_trace

    # Set session ID (stored as database column, filterable in dashboard)
    trace.set_session_id("session-123")

    # Name the trace (its title in Bitfab, searchable and filterable)
    trace.set_name("Order #{order_id}")

    # Set trace metadata (indexed for exact-match search)
    trace.set_metadata("region" => "us-west-2", "environment" => "production")

    # Add context entries (stored as key-value pairs, accumulates across calls)
    trace.add_context("workflow" => "checkout-flow", "batch_id" => "batch-2024-01")

    { order_id: order_id, status: "completed" }
  end
end

set_session_id(id): Groups traces by user session. Stored as a database column for efficient filtering.
set_name(name): The trace’s title in Bitfab, and a field you can search and filter on. Use it for the case, ticket, or record the run is about. Unset, the trace is titled by its trace function key.
set_metadata(hash): Searchable exact-match metadata on the trace. Merges with existing metadata.
add_context(hash): Key-value context entries. Accumulates across multiple calls.

Caller metadata accepts up to 50 keys. Keys are 1-40 characters using letters, numbers, _, ., :, or -; values are strings, numbers, or booleans up to 500 characters after conversion to text. Metadata filters match exact key/value pairs case-insensitively. Search it from the SDK with client.traces.search. Up to 10 supplied pairs must all match; pass result["nextCursor"] back as cursor: for another page.

page = Bitfab.client.traces.search(
  caller_metadata: {"ticket_id" => "ABC-123", "tenant" => "tanagram"},
  trace_function_key: "order-processing",
  limit: 50
)

The same method searches by the name set_name recorded. name: matches the name’s words in order, so name: set to "Order ABC" also finds "Order ABC 123". name_contains: matches any part of the name, including inside a word, and needs at least 3 characters. Both are case-insensitive, and every filter you supply has to match.

exact = Bitfab.client.traces.search(name: "Order ABC-123")
partial = Bitfab.client.traces.search(name_contains: "order", limit: 50)

Pass experiment_id: to list only the replay traces one experiment produced. Every result also carries "experimentId" (nil for a captured trace), "durationMs" (the trace’s wall-clock time, nil when it has no recorded end time), and "tokens" (a Hash of "input", "output", "cached", and "total" summed across the trace’s spans, nil when none were recorded). That is enough to compare two experiments on time and token spend without opening each trace:

[baseline_experiment_id, candidate_experiment_id].each do |experiment_id|
  traces = Bitfab.client.traces.search(experiment_id: experiment_id, limit: 500)["traces"]
  total_ms = traces.sum { |t| t["durationMs"] || 0 }
  total_tokens = traces.sum { |t| t.dig("tokens", "total") || 0 }
  puts "#{experiment_id}: #{traces.length} traces, #{total_ms} ms, #{total_tokens} tokens"
end

At least one of caller_metadata:, name:, name_contains:, or experiment_id: is required, so a search can never return the organization’s traces unfiltered. Reach for caller metadata when you are looking up a run by an identifier your own code owns, for the name when you are hunting for a run by the title a person reads, and for the experiment when you want what one replay run produced.

Read One Persisted Span

Fetch one span without loading the full trace. Look it up by id: alone, or by name: and/or type: ("llm", "agent", "function", "guardrail", "handoff", or "custom"). Repeated matches default to the last span.

span = Bitfab.client.get_trace_span(trace_id, name: "GenerateAnswer")
first = Bitfab.client.get_trace_span(trace_id, name: "GenerateAnswer", occurrence: "first")
first_model_call = Bitfab.client.get_trace_span(trace_id, type: "llm", occurrence: "first")
exact = Bitfab.client.get_trace_span(trace_id, id: span_id)

Both IDs are canonical Bitfab IDs; ingestion source IDs are not accepted. occurrence also accepts a zero-based integer. Passing id: together with name: or type:, or passing none of them, raises ArgumentError, and so does an unknown type. A missing trace or span returns nil. To walk every span in a trace, use spans.list. It needs no filter, returns spans in start order with "id", "traceId", "parentSpanId", "name", "type", "startedAt", and "endedAt", and includes only the payload fields you ask for. Narrow it with span_type: or span_name:, and pass "nextCursor" back as cursor: for the next page.

page = Bitfab.client.spans.list(trace_id, fields: ["input"])
page["spans"].each { |span| puts "#{span["name"]}: #{span["input"]}" }

When you only have a span ID, use get_span. It takes Bitfab’s span ID or the span ID the SDK sent, and the span it returns carries its "traceId". A span outside your API key’s organization returns nil.

span = Bitfab.client.get_span(span_id)

Dropping a Trace

Call .drop on the current-trace handle to discard the in-flight trace. Once flagged, spans that complete afterward are not uploaded at all, and the flag rides out on the completion payload, so when the trace completes the server scrubs any payloads that already raced out (the trace, its external trace, and sibling spans), deletes the archived S3 objects, and marks it dropped instead of completed, keeping only a skeleton audit row. Use it to discard runs you never want stored (health checks, test traffic) or a run you know carries sensitive data.

class OrderService
  include Bitfab::Traceable
  bitfab_function "order-processing"

  bitfab_span :process_order, type: "function"
  def process_order(order_id)
    Bitfab.current_trace.drop if health_check?(order_id)

    { order_id: order_id, status: "completed" }
  end
end

Safe to call outside a trace (a no-op), and never raises into your application.

Error Handling

Errors are captured in the span and re-raised:

class RiskyService
  include Bitfab::Traceable
  bitfab_function "risky-service"

  bitfab_span :risky
  def risky
    raise "error"
  end
end

begin
  RiskyService.new.risky
rescue => e
  # Span records error and timing
end

Each error is classified by source. Errors raised by your code are recorded with error_source: "code". SDK-internal errors are recorded with source: "sdk". Both appear in the span’s errors array in the Bitfab dashboard.

Flushing Traces

raise "Bitfab traces were not delivered before the deadline" unless Bitfab.flush_traces(timeout: 30)

The return value is false when an export fails or the deadline expires. Traces flush automatically on process exit via at_exit hook.

OpenTelemetry Transport

Variable	Default	Purpose
`BITFAB_OTEL_EXPORT_CONCURRENCY`	`32`	Concurrent direct requests, `1` through `64`.
`BITFAB_OTEL_MAX_REQUEST_BYTES`	`3000000`	Request-size target. Can only be lowered.

The Ruby SDK lazily creates one private OpenTelemetry provider and bounded BatchSpanProcessor per client; it does not replace your application’s global OTel provider, and an unused client starts no OTel worker. Traced methods submit the same replay-safe Bitfab payloads through a transport interface. Batches are sent to Bitfab as OTLP/JSON. Each carrier is encoded once and the request body is assembled from those encodings, so a batch is never re-encoded to measure its size. Live and replay traces share the same OTel pipeline. Before completing a replay experiment, the SDK flushes OTel and uses delivery acknowledgments to confirm every carrier that reached Bitfab. If any delivery is uncertain, it polls Bitfab until every submitted replay trace completion and expected span count is persisted. The SDK partitions count-based OTel exports into requests of at most about 3 MB. Each request contains at most eight carriers and up to 32 run concurrently. Set BITFAB_OTEL_MAX_REQUEST_BYTES (a positive integer no greater than 3000000) for a stricter proxy, and BITFAB_OTEL_EXPORT_CONCURRENCY (1 through 64) to tune request concurrency. Invalid values warn once and fall back to the defaults. A single span may use up to 7,800,000 carrier bytes when its dedicated request gzips below the 3,000,000-byte wire target and remains below the 8,000,000-byte decompressed ingress limit. The carrier is the payload re-escaped into the OTLP attribute. If it does not compress enough, compression is unavailable, or it exceeds the raw ceiling, the SDK replaces its largest fields with <unserializable: too_large_N_bytes> placeholders until it fits the 2,800,000-byte fallback budget. The trim is recorded on the span’s errors so the trace is flagged as incomplete. The queue holds 8,192 carriers (a span and its trace completion are two), and a burst that queues faster than the exporter drains loses the excess rather than growing your process’s memory. Bitfab.flush_traces reports on what reached the exporter, so it returns true even when the queue had to drop carriers; keep bursts inside that capacity when every span matters. A long-running process that builds transient clients should release each client’s batch worker when it is done with it; a shared client is closed at process exit.

client = Bitfab::Client.new(api_key: ENV.fetch("BITFAB_API_KEY"))
begin
  # ... traced work ...
ensure
  client.close(timeout: 30)
end

For the full ownership model, carrier format, live and replay flows, batching limits, lifecycle, and failure semantics, see OpenTelemetry Transport Architecture.

Wrapping Third-Party Methods

Use Bitfab::Traceable.wrap to trace methods on external classes:

require "openai"

fn = Bitfab.client.get_function("openai")
fn.wrap(OpenAI::Client, :chat, name: "Chat", type: "llm")

# Now all calls to client.chat are traced
client = OpenAI::Client.new(access_token: ENV["OPENAI_API_KEY"])
client.chat(parameters: { model: "gpt-4", messages: [...] })

Bitfab::Traceable.wrap(OpenAI::Client, :chat, trace_function_key: "openai", name: "Chat", type: "llm") is equally supported and is what the bound handle calls underneath. Making the function first pays off as soon as a second method needs the same key. For experimental whole-subtree capture on an external class, use fn.trace(klass, method_name, ...), or Bitfab::Traceable.trace with an explicit trace_function_key:, plus the subtree limit options.

Replay

A trace is replayable when its root span has serializable inputs (the recorded inputs must round-trip through to_json). Framework handlers are not yet available for Ruby, so serializable root inputs are the only path to a replayable trace; instrument the outer workflow method so its inputs serialize. Replay historical traces through a method to create experiments. This re-runs past inputs through your updated code and compares the results.

client = Bitfab.client

# Pass an instance for instance methods
service = OrderService.new(api_key: "...", db: db)
result = client.replay(
  service, :process_order,
  trace_function_key: "order-processing",
  limit: 5
)

# Or pass a Class for class methods
result = client.replay(
  OrderService, :process_order,
  trace_function_key: "order-processing",
  limit: 5
)

puts "Experiment URL: #{result[:experiment_url]}"
puts "Experiment ID:  #{result[:experiment_id]}"

result[:items].each do |item|
  puts "Input: #{item[:input]}, Result: #{item[:result]}, Error: #{item[:error]}"
  puts "Duration (ms): #{item[:original_duration_ms]}"
  puts "Tokens: #{item[:tokens]}"    # { input:, output:, cached:, total: } or nil
  puts "Model: #{item[:model]}"
  puts "Trace ID: #{item[:trace_id]}" # Server trace ID for the replayed execution
end

Replay waits for each item’s trace (spans + completion) to be persisted server-side before completing the experiment, so :trace_id is a real server trace ID for completed items. If NO completed item’s trace persisted (uploads wholesale failed, or the replayed method isn’t traced), replay raises a RuntimeError instead of silently returning nil trace IDs. If only SOME items’ traces are missing (a transient per-item upload failure), those items get nil trace IDs with a loud warning and the rest of the run is returned intact. :trace_id is also nil for errored (unreplayable) items, and for all items when the server predates the trace-ID mapping (a warning explains which). Anything describing the trace being replayed carries the original_ prefix: :original_duration_ms, :original_model, and :original_tokens. Unprefixed fields are the replay’s own: :duration_ms is how long this run’s call took, and :tokens is the replayed run’s usage (the same numbers Bitfab’s experiments view shows). Comparing :tokens[:total] against :original_tokens[:total] tells you how your change moved cost. Each field is nil when it wasn’t captured. :model remains as a deprecated alias for :original_model. Note that :duration_ms changed meaning: it used to report the original trace’s duration and now reports the replay’s. Parameters:

receiver (required): An instance for instance methods, or a Class for class methods
method_name (required): Symbol of the method to replay
trace_function_key (required): The trace function key
limit (optional): Max recent traces to replay (default: 5; maximum: 5,000). Ignored when trace_ids, dataset_id or dataset_ids is passed, since an explicit ID list or a dataset already determines how many traces replay. To replay part of a dataset selection, name the members to run in trace_ids.
trace_ids (optional): Array of specific trace IDs to replay (max 100). The ID count determines how many traces replay, and limit is ignored when both are passed. Passed alongside a dataset selector it pins which members of that selection replay, and the server rejects any ID none of those datasets contains.
name (optional): Display name for the resulting experiment.
notes (optional): Run conditions Bitfab cannot see on its own, such as an environment override or a forced feature flag. Kept on the experiment next to its name (max 10,000 characters).
max_concurrency (optional): Positive integer bounding threads for parallel replay (default: 10). Pass nil for unlimited concurrency. Other values raise before an experiment starts.
code_change_description (optional): Rationale for the code change being tested in this replay (stored on the experiment); when supplied alone, it is preserved while files are captured automatically
code_change_files (optional): Array of edited files, each as { path:, before:, after: } (use "" for newly created or deleted files); omit to capture automatically or pass nil to suppress capture
mock (optional): Mock strategy for child spans during replay. One of "marked" (default, only spans tagged with mock_on_replay: true return historical output), "none" (every child runs real code), or "all" (every matched recorded child returns its historical output; a missing occurrence fails the item closed)
mock_override (optional): One { match:, value: } hash, or an array of them, that substitutes a supplied output for matched spans. Per-call overrides take precedence over registered overrides and the base mock strategy.
experiment_group_id (optional): UUID string that groups multiple replay runs into a single experiment batch. Pass the same ID across successive replay() calls to link them together in the dashboard. An ID no group has yet creates the group.
experiment_group_name (optional): Display name for the experiment group, applied when this replay creates the group or the group has no name yet. A group that already has a name keeps it. Requires experiment_group_id; passing it alone raises ArgumentError before any request.
dataset_id (optional): UUID of the dataset this replay runs against, durably attributing the experiment to that dataset. Pass trace_ids alongside it to replay only those members.
dataset_ids (optional): UUIDs of the datasets this replay runs against, for benchmarking one method against several corpora in a single run. Replays the union of their traces, graded by the union of their graders, and attributes the experiment to every one of them.
grader_ids (optional): Array of grader UUIDs (max 100) attached directly to this replay run, independent of the dataset’s own graders. The resulting experiment is graded by the union of these and the dataset’s runnable graders. Use it to grade a single run with a check you don’t want to add to the dataset permanently. Each must be an active grader in the same organization and trace function, or the replay is rejected with a 400. A replay with no dataset can still carry graders this way.
adapt_inputs (optional): A callable ->(args, kwargs, ctx) that reshapes recorded inputs onto the method’s current signature when its shape changed after the traces were captured. See Adapting inputs after a signature change below.
on_item_start (optional): A callable fired when a worker begins processing an item, before replay setup and customer code run. Pair it with on_item_finish to distinguish queued items from in-flight items whose callback has not returned. A raising callback never crashes the run.
on_item_finish (optional): A callable ->(progress) fired exactly once per item as it finishes, always with that item plus running totals (original/source trace id, the server replay :trace_id read back per trace off the OTLP ingest response and surfaced as each item finishes (its trace is flushed on finish, so the id is in hand at the callback, not only after the whole run), input, result, original output, error, duration, tokens/model metadata). It never emits a whole-run completion event. Use it to render live progress or start evaluating completed items while replay runs. A raising callback never crashes the run. The deprecated on_progress callback receives the same per-item events plus its legacy item-less terminal complete event, and is ignored when both are supplied. The installed bitfab-replay command passes Bitfab.method(:report_replay_progress) as both on_item_start and on_item_finish; it writes lifecycle events to stderr, which the plugin uses to identify in-flight traces, report finished items, and write per-item result files while stdout remains available for direct-run ReplayResult JSON.
db_branch (optional): true or a Hash. db_branch: true requests a DB branch per replay item with the mirror’s own sizing; pass a hash to tune it, and false or nil leaves branching off. Each replay worker resolves its branch from the source trace’s captured snapshot reference, so max_concurrency also bounds live branches. Bitfab.current_replay_branch hands the branch to you inside the replayed method, and the SDK releases it after the item. The reader returns nil when no branch was resolved (e.g. the trace predates snapshot capture, or DB branching isn’t configured), so branch ? branch.database_url : ENV["DATABASE_URL"] falls back to your live database. The keys tune the branch, symbol or string: min_cu and max_cu are the compute’s autoscaling floor and ceiling (0.25 to 56). Equal values pin a fixed size, allowed up to 56; an autoscaling range may not span more than 8 CU or exceed 16 CU; setting them equal pins the size, so one item can’t post a better number purely because it ran against an already-scaled endpoint. warmup_sql is appended to the branch’s readiness check, so the cache is warm before your method sees the branch and the warm-up is never charged to the replayed call. Omit them and the branch keeps the mirror’s own defaults.

Notes:

receiver + method_name must resolve to the method that carries the traceable decoration. Passing a plain wrapper around it will not resolve the trace function key.
trace_function_key must match the method’s declared key. It is read from the bitfab_span / Bitfab::Traceable.wrap declaration on the method you point at. If the key you pass contradicts that declared key, replay raises an ArgumentError: it would otherwise fetch one function’s historical traces but record the replay under the method’s own key, producing an incoherent experiment. (Ruby has no plain-callable replay form, so there is no reason to pass a non-matching key.)
trace_function_key is passed explicitly, so instance methods and class methods are disambiguated by the receiver.
Use a single Bitfab::Client across instrumentation and replay. If your instrumented module constructs a client at load and your replay registry constructs another, they do not share registered trace functions; import the client from the instrumented module (or a shared singleton) rather than constructing a new one in the registry.

Replay specific traces:

service = OrderService.new(api_key: "...")
result = client.replay(
  service, :process_order,
  trace_function_key: "order-processing",
  trace_ids: ["trace-id-1", "trace-id-2"]
)

Attaching a code change: Each replay creates an experiment. When you’re iterating on a method and replaying after every edit, attach the change so the dashboard can show exactly what was edited alongside the results. Read each file before editing, edit, then read it again: the two strings go straight into code_change_files. There’s no diff format to construct.

before = File.read("lib/order_service.rb")
# ...edit lib/order_service.rb...
after = File.read("lib/order_service.rb")

result = client.replay(
  service, :process_order,
  trace_function_key: "order-processing",
  code_change_description: "fix off-by-one in retry logic",
  code_change_files: [
    {path: "lib/order_service.rb", before:, after:}
  ]
)

Both options are optional and independent: pass just code_change_description for a quick rationale-only annotation, or just code_change_files to record the literal edits. If you omit code_change_files, replay falls back to capturing your uncommitted working-tree changes against HEAD (best-effort, only inside a git repo), so an experiment shows the edit you are trying rather than your whole branch. When the working tree is clean, it diffs against the branch yours was cut from instead (the parent branch in a stack, otherwise the point where you split from the default branch: origin/HEAD, else origin/main, origin/master, main, or master), so changes you already committed still show. Set BITFAB_CODE_CHANGE_BASE to a ref such as origin/main to diff against its merge-base instead. This fallback uses Git rename detection: a renamed-and-edited file is compared once under its destination path, while an unchanged rename adds no content diff. A supplied code_change_description is preserved while the files are captured. Passing code_change_files explicitly always wins and is the way to record a precise per-edit before/after. To opt out for one replay run, pass code_change_files: nil (and optionally code_change_description: nil if you also want no description). Set BITFAB_DISABLE_CODE_CHANGE_CAPTURE to turn the fallback off for every replay in the process.

Mock child spans during replay

For the workflow-level guide, see Replay Mocking. By default replay uses "marked": child spans tagged with mock_on_replay: true return their historical outputs, while every other child runs real code. Three mock strategies control this behavior: Traced methods that return an Enumerator preserve their parent span and replay context inside same-thread Enumerator.new / enum_for source fibers, so marked descendants remain mockable while the stream is consumed. Enable trace_across_threads: true for explicit spans and replay mocks in worker threads. Process replay initializes each child from its assigned item.

# "marked" (default): only spans declared with mock_on_replay: true return historical output
client.replay(service, :process_order, trace_function_key: "order-processing")

# "none": every child span runs real code
client.replay(service, :process_order, trace_function_key: "order-processing", mock: "none")

# "all": every matched recorded child returns its historical output; only the root runs real code
client.replay(service, :process_order, trace_function_key: "order-processing", mock: "all")

Tag the child spans you want mocked at definition time:

class OrderService
  include Bitfab::Traceable
  bitfab_function "order-processing"

  # Paid LLM call: skip during replay by default
  bitfab_span :classify_intent, type: "llm", mock_on_replay: true
  def classify_intent(prompt); end

  # Cheap, deterministic: keep running real
  bitfab_span :persist, type: "function"
  def persist(order); end

  bitfab_span :process_order, type: "agent"
  def process_order(order)
    classify_intent(order.description)
    persist(order)
  end
end

Use the default mock: "marked" behavior when you want to iterate on process_order’s logic without paying for the LLM call on each replay. Use mock: "all" when the goal is the cheapest possible replay (every matched recorded child span returns its recorded output; only the root function executes real code). Calls are matched by trace function key and span name. Repeated calls with the same key and name are distinguished by call order, so their first, second, and third invocations receive the corresponding historical outputs. Calls that share a trace function key but have different span names are tracked independently. If a strategy selects a call for mocking but its recorded occurrence is missing or exhausted, the item errors and the real method does not execute.

Injecting custom values with overrides

A mock override substitutes a value you supply for a matched span, so downstream real code runs against it — for “what if this step returned X” experiments without editing the traced code. An override is a { match:, value: } hash: match is a callable selecting spans by structural metadata (node[:trace_function_key], node[:span_name], node[:type], node[:original_span_id]); value is a flat value injected as-is, or a callable that returns one. The trace_function_key: passed to replay selects the workflow’s historical root traces; it does not by itself identify the descendant to override. Because bitfab_function "order-processing" binds every method in this example to the same key, match classify_intent by its span name.

result = client.replay(
  service, :process_order,
  trace_function_key: "order-processing",
  mock: "none", # run everything real...
  mock_override: {
    # ...except this span, which gets the value you supply
    match: ->(node) { node[:span_name] == "classify_intent" },
    value: { label: "refund" } # flat value
  }
)

A callable value receives a context hash with the live positional :inputs, the live keyword :kwargs (empty when the call used none), and :get_original_output (synchronous in Ruby; its first call may fetch the recorded output lazily) to tweak the recorded output instead of replacing it:

value: ->(ctx) { ctx[:get_original_output].call.merge(score: 1) }

Register overrides on the client to apply them to every replay (keyword or positional form), and reset with clear_mock_overrides:

client.register_mock_override(
  match: ->(node) { node[:type] == "llm" },
  value: { label: "refund" }
)
# Positional form (equivalent): client.register_mock_override(match, value)
client.clear_mock_overrides

Use the keyed form when one registration belongs to a known trace function. The second argument can be a resolver or { match:, value: }; for an override hash, both the registered trace function key and its match predicate must match:

client.register_mock_override(
  "classify-intent",
  ->(ctx) { {label: ctx[:inputs].first.to_s} }
)

client.register_mock_override(
  "shared-workflow",
  {match: ->(node) { node[:span_name] == "Classifier" }, value: {label: "refund"}}
)

Pass one callable directly for a client-wide resolver that routes by trace function key. Return Bitfab::NO_MOCK_OVERRIDE to decline a span; nil remains a valid mocked output:

client.register_mock_override(lambda do |ctx|
  if ctx[:node][:trace_function_key] == "classify-intent"
    {label: ctx[:inputs].first.to_s}
  else
    Bitfab::NO_MOCK_OVERRIDE
  end
end)

Precedence per span: per-call mock_override:, then registered overrides, then the base mock: strategy. Bitfab::NO_MOCK_OVERRIDE continues at the next override, then falls back to that base strategy. Pass a single override hash, resolver, or array.

Fluent API: `client.get_function`

Bind a trace_function_key once and wrap multiple classes or methods against it. Mirrors client.get_function in the Python SDK and client.getFunction in TypeScript.

fn = Bitfab.client.get_function("openai")

fn.wrap(OpenAI::Client, :chat, name: "Chat", type: "llm")
fn.wrap(OpenAI::Client, :embeddings, name: "Embed", type: "llm")

#wrap accepts the same options as Bitfab::Traceable.wrap (name, type, mock_on_replay), but the trace_function_key is fixed to the one bound on the returned Bitfab::BitfabFunction. The experimental #trace method binds the same key while capturing an external method’s first-party call subtree:

fn.trace(OrderService, :process_order, type: "agent", max_captured_subtree_spans: 200)

Adapting inputs after a signature change

Replay deserializes each trace’s inputs exactly as they were captured against the method’s signature at trace time, then calls the current method with them. If the signature drifted since capture (an argument renamed, reordered, folded into a hash, or a new required argument added), the call no longer lines up and raises. The adapt_inputs hook reshapes the recorded inputs onto the current signature so replay can still run:

# Recorded as positional (user_id, limit); current signature takes a hash.
adapt = ->(args, _kwargs, _ctx) { [[{user_id: args[0], limit: args[1]}], {}] }

result = client.replay(
  service, :process_order,
  trace_function_key: "order-processing",
  adapt_inputs: adapt
)

The hook receives the deserialized (args, kwargs) plus a per-trace ctx ({ original_trace_id:, original_span_id: }, with deprecated source_* aliases) and returns [new_args, new_kwargs]. item[:input] reports the returned positional arguments followed by a keyword-argument hash when keywords are present. It runs once per item, inside the same rescue as the method: if it raises, that item’s :error is set and the run continues, so one unmappable trace never crashes the batch. ctx[:original_trace_id] (the original Bitfab trace ID) lets a table-driven adapter look up a per-trace transform. That’s the escape hatch for reshapes that need judgement rather than mechanical rearrangement: compute the adapted inputs per trace up front, then have the hook look them up by original_trace_id, keeping replay deterministic instead of calling a model mid-replay. When the new signature has a genuinely new required argument with no analog in the recorded trace, don’t fabricate one. There’s nothing faithful to map it to, so leave those traces unmapped (let them raise) rather than inventing test inputs. For anything beyond a one-liner, keep the adapter in its own file next to the replay registry and require it:

# scripts/replay_adapters/extraction.rb
EXTRACTION_ADAPTER = ->(args, _kwargs, _ctx) do
  user_id, limit = args
  [[{user_id:, limit:}], {}]
end

# scripts/replay_registry.rb
require_relative "../config/initializers/bitfab"
require_relative "../services/extraction_service"
require_relative "replay_adapters/extraction"

REGISTRY = Bitfab::ReplayRegistry.new
REGISTRY.register(
  "workflow",
  ExtractionService.new,
  :extract_memories,
  trace_function_key: "my-function",
  adapt_inputs: EXTRACTION_ADAPTER
)

That keeps the transform versioned and reviewable alongside the method it adapts, and you add the require only when a drift actually needs it.

Programmatic Replay Output

The installed bitfab-replay command owns result serialization and output for registry-based replay. If you call Bitfab.replay directly from custom programmatic tooling, BITFAB_REPLAY_RESULT_PATH writes the full replay result JSON to that file. For a direct custom command, emit the full replay result hash as a single stdout JSON block so a consumer can JSON.parse it and reason about every field, including the new per-item :duration_ms, :tokens, and :model. Never print only lengths, counts, hashes, or truncated previews, and never replace the JSON block with ad-hoc per-field log lines. Direct API example:

result = Bitfab.client.replay(service, :process_order, trace_function_key: "order-processing", limit:)

# Human-readable summary goes to stderr, so stdout stays pure JSON.
warn "Experiment: #{result[:experiment_url]}"
warn "Items:      #{result[:items].length}"

# Full structured dump to stdout, ready for JSON.parse. The SDK serializer
# retains useful fields from trace_error and replay_error exception objects.
puts Bitfab.serialize_replay_result(result)

The dumped object includes every item’s :input, :result, :original_output, :error, structured :trace_error and :replay_error, :duration_ms, :tokens, :model, and :trace_id, plus :experiment_id and :experiment_url (and their deprecated aliases :test_run_id and :test_run_url). Use Bitfab.serialize_replay_result; JSON.pretty_generate(result) raises when a result contains an exception object. The installed command sets BITFAB_REPLAY_RESULT_PATH and writes the same structured JSON there. Per-item errors are part of the contract. If the wrapped method raises while executing the replayed trace, Bitfab.replay retains the actual exception in item[:trace_error], copies its message to item[:error], leaves item[:result] as nil, and continues. If replay setup fails before the method starts (for example database warmup or input loading), the actual exception is instead in item[:replay_error]. A database branch resolution failure is a Bitfab::DbBranchReplayError; inspect its code, message, and original_trace_id to distinguish failures such as branch_create_failed, snapshot_from_replaced_origin, and invalid_snapshot_ref without parsing item[:error]. A lease-endpoint HTTP, timeout, or network failure uses lease_request_failed and retains the original client exception as cause; unexpected resolver failures use internal_error. Treat either error kind as unreplayable, not as a failing output. If the whole run later raises, Bitfab::ReplayError#items still contains every collected item and cause retains the whole-run exception. Don’t swallow per-item errors in the registered method or a direct API caller. A custom begin/rescue that returns a placeholder turns infrastructure failures into fake successes. Let the SDK record them. Input serialization caveat. Replay deserializes historical span inputs and passes them back to your method. This works for strings, numbers, and plain hashes. If your span wraps a method that takes hydrated domain objects (ActiveRecord models, class instances, DB records), they won’t round-trip through serialization; move the span to where inputs are IDs or plain data and let the method fetch objects internally, or reshape arguments in the wrapper.

Replay Registry

For experimental execution on your own GitHub Actions, see Cloud replay. Create a small registry module. Your project owns only the requires and registrations; installing the gem also installs the standard bitfab-replay executable, which owns command-line flags, lifecycle progress, code-change loading, result serialization, and summary output.

require "dotenv/load"
require_relative "../config/initializers/bitfab"
require_relative "replay_mocks"
require_relative "../services/extraction_pipeline"
require_relative "../services/search_pipeline"

REGISTRY = Bitfab::ReplayRegistry.new
REGISTRY.register("extraction", ExtractionPipeline.new, :extract_memories)
REGISTRY.register(
  "search",
  SearchPipeline.new,
  :search_documents,
  # Plain handler methods need the key explicitly. A bitfab_span method carries
  # its key, so trace_function_key: can be omitted.
  trace_function_key: "my-search-pipeline",
  mock: "marked",
  options_factory: lambda { |ctx|
    {mock_override: ReplayMocks.search_mock(ctx.params["scenario"])}
  }
)

Keep non-trivial executable configuration in a sibling module and require it from the registry. For example, replay_mocks.rb can define a mock factory whose { match:, value: } result uses live replay inputs and caller-supplied parameters. options_factory: receives JSON values loaded from --params <file> and repeated --param name=value; direct parameters override file values. Run the executable with bundle exec bitfab-replay --registry scripts/replay_registry.rb <pipeline>. The registry module must define REGISTRY.

Flag	Value	Effect
`--limit`	`N`	Most traces to select. It bounds a `--trace-ids` or `--dataset-ids` selection instead of replacing it
`--trace-ids`	`id1,id2`	Replay exactly these traces. Mutually exclusive with `--dataset-ids`
`--dataset-ids`	`uuid1,uuid2`	Replay the membership of one or more datasets, as their deduped union. Mutually exclusive with `--trace-ids`. `--dataset-id` is the same flag
`--name`	`NAME`	What this run is testing, such as `baseline`. Bitfab already records the commit, branch, and datasets
`--notes`	`TEXT`	Run conditions Bitfab cannot see, such as a forced feature flag
`--concurrency`, `--max-concurrency`	`N`	Items in flight at once
`--attempts`	`N`	Attempts per source, from 1 to 100
`--primitive`	`async` or `process`	Execution mode
`--memory-throttle`, `--no-memory-throttle`		Configure process memory admission
`--only-with-assertions`		Select sources with approved own assertions
`--skip-assertion-judging`		Don’t judge approved assertions on each replay. Judging is on by default and costs model calls
`--dry-run`		Resolve inputs without executing customer code
`--fail-on-error`		Exit 1 after printing the result when any replayed item errored, printing `[replay] N of M replayed items errored` to stderr. Without it, errored items are reported in the summary and the command still exits 0. Under `--dry-run`, items whose inputs failed to resolve count
`--experiment-group-id`	`UUID`	Add this run to an experiment group. A new UUID creates the group
`--experiment-group-name`	`NAME`	Name the experiment group when this run creates it or it has no name yet. A group that already has a name keeps it. Requires `--experiment-group-id`
`--grader-ids`	`id1,id2`	Attach graders to this run only
`--code-change`	`PATH`	Load a code-change description from a file
`--no-code-change`		Record no code change, overriding a registry default
`--mock`	`none\|all\|marked`	Which recorded child spans return their historical output
`--db-branch`, `--no-db-branch`		Turn per-item database branching on or off
`--params`	`PATH`	JSON file of values passed to `options_factory:`
`--param`	`name=value`	One value passed to `options_factory:`. Repeatable, and overrides `--params`
`-h`, `--help`		Print usage

As soon as the server creates the experiment, the command prints [replay] Experiment <id>: <url> to stderr. If the run later fails with Bitfab::ReplayError, it prints Experiment <id>: <url> again before the error message, so the experiment ID is never lost. Pass static function-specific behavior such as adapt_inputs:, mock_override:, or db_branch: to register; build parameterized behavior with options_factory:. Command-line values override overlapping scalar defaults without removing unrelated executable options. Unknown registry option names fail when the module loads instead of reaching replay as misspelled keywords.

When the traced method needs the app’s dependencies

Most agents need tools, clients, and config that the app builds at startup. The registry gets them the way production does, by calling the app’s own code.

Put the registry where production builds the call. That is the app that builds the tools, clients, and config and then calls the traced method. Its Gemfile already includes all of them. Run bundle exec bitfab-replay from that app’s directory so its bundle resolves. When the traced method lives in a lower-level gem, the registry still goes in the app that calls it. Add bitfab to the app’s Gemfile if it is not there yet. Never copy the app’s dependencies into the gem.
Build the receiver the way production builds it. Call the factory or initializer that production calls at startup, and register the receiver it returns. Replay runs that factory, so give it the settings it requires, such as URLs and API keys, even for tools whose calls are mocked. Never write a smaller toolset or a simplified agent for replay. The replay would test that copy instead of your code.
Rebuild the arguments the recording cannot keep. A live object passed as an argument, such as a tool with its callable, a client, or a callback, is recorded as data only. Rebuild these arguments in the entry’s adapt_inputs: with the same code production uses. Never replace them with an empty list, a stub, or nil. When production builds them inline, move that code into a method that production and the registry both call.
Know which tool calls run for real. Under mock: "marked", a call inside a span marked mock_on_replay: true returns its recorded output and never runs. mock: "all" does the same for every recorded call. Every other tool call runs for real during replay. Put every call with an unsafe side-effect, such as sending an email or charging a card, behind a marked span before you replay. See Mark child spans.
Keep experiment settings out of the registry. Set a model swap or any other experiment setting through an environment variable the app already reads. Never add a second registry entry with the setting written into it.
Check the wiring before a real run. --dry-run resolves every item’s inputs without calling the method. Then replay one trace with --limit 1. With your code unchanged, the replay should call the same tools the original called.

The app builds the agent in a factory. The registry builds the agent the same way and registers its traced method.

# apps/agent/lib/support_agent.rb
class SupportAgent
  include Bitfab::Traceable

  bitfab_function "support-agent"

  def initialize(ticket_gateway)
    @ticket_gateway = ticket_gateway
  end

  bitfab_span :lookup_ticket, name: "lookup-ticket", type: "tool", mock_on_replay: true
  def lookup_ticket(ticket_id)
    @ticket_gateway.lookup(ticket_id)
  end

  bitfab_span :answer_question, name: "answer-question", type: "agent"
  def answer_question(question)
    ticket = lookup_ticket(parse_ticket_id(question))
    {"reply" => "Ticket #{ticket["id"]} is #{ticket["status"]}."}
  end

  def self.build(config)
    new(TicketGatewayClient.new(config.ticket_gateway_url))
  end
end

# apps/agent/scripts/replay_registry.rb
require "dotenv/load"
require_relative "../lib/support_agent"

REGISTRY = Bitfab::ReplayRegistry.new.register(
  "support-agent",
  SupportAgent.build(AgentConfig.load),
  :answer_question,
  mock: "marked"
)

cd apps/agent
bundle exec bitfab-replay --registry scripts/replay_registry.rb support-agent --dry-run
bundle exec bitfab-replay --registry scripts/replay_registry.rb support-agent --limit 1

A gem runs the agent and the app passes in the tools. The traced method is AgentRunner#run in the agent_core gem. The app builds the runner with SupportAssistant.build_runner and the tools with SupportAssistant.build_tools. The registry lives in the app. It registers the runner production builds and rebuilds the tools for each replayed call.

# apps/support/scripts/replay_registry.rb
require_relative "../config/environment"

REBUILD_TOOLS = lambda do |args, kwargs, _ctx|
  prompt, recorded_options = args
  [[prompt, recorded_options.merge("tools" => SupportAssistant.build_tools)], kwargs]
end

REGISTRY = Bitfab::ReplayRegistry.new.register(
  "agent-run",
  SupportAssistant.build_runner,
  :run,
  mock: "marked",
  adapt_inputs: REBUILD_TOOLS
)

Datasets

client.datasets creates, reads, and modifies datasets programmatically, with the same operations your coding agent reaches through the Bitfab MCP tools. A dataset is a named bucket of traces under one trace function. Experiments replay against it and its graders score its members.

client = Bitfab.client
saved = client.datasets.save(
  trace_function_key: "checkout-agent",
  name: "Refund failures",
  description: "Checkout runs where the refund was declined"
)
dataset_id = saved["dataset"]["id"]
client.datasets.update(dataset_id, name: "Declined refunds", description: "")

added = client.datasets.add_traces(dataset_id, [trace_id])
warn "not in this trace function: #{added["skippedTraceIds"]}" if added["skippedTraceIds"].any?

trace_ids = client.datasets.list_traces(dataset_id)["traceIds"]
client.datasets.add_graders(dataset_id, [grader_id])
run = client.datasets.rerun_graders(dataset_id)["run"]
puts run["status"], run["result"]

save is an upsert on the dataset name within its trace function, so re-running a script does not accumulate duplicates. update renames a dataset or changes its description while keeping its id, traces, and graders. An empty description clears it. Membership and grader calls accept up to 100 ids and report ids they skipped rather than failing the whole call. remove_traces only drops membership. Traces are never deleted. rerun_graders blocks until the run finishes and returns the completed run. It raises Bitfab::GraderRerunError when the run errors, and Bitfab::GraderRerunTimeoutError when the run is still going after timeout: seconds (90 by default). The timeout does not stop the run on Bitfab. Read it later with get_grader_rerun. See the reference for every method and result shape.

Assertions, categories, and verdicts

Read a trace’s assertions before judging its replay. Categories group related assertions and scenarios across the organization, and verdicts record the pass or fail on an individual assertion.

category = client.categories.save(
  "Timing",
  description: "Checks involving itinerary timing"
)

assertion = client.traces.save_assertions(original_trace_id, [{
  "assertion" => "The itinerary lands before 9am local time",
  "category_id" => category.fetch("id"),
  "justification" => [{"spanId" => evidence_span_id, "text" => "Shows the arrival time"}],
  "categoryJustification" => [{"spanId" => evidence_span_id, "text" => "This is a timing check"}],
  "targetOnEvaluatedTrace" => {"kind" => "output"}
}]).first

client.verdicts.save(
  trace_id: replay_trace_id,
  assertion_id: assertion.fetch("id"),
  verdict: "pass",
  annotation: "The arrival is 06:40 local time"
)

categories.get, .list, and .delete complete the category lifecycle. Deleting a category detaches its assignments while preserving assertions and verdicts. Assertion hashes omit "category_id" to preserve an assignment and pass nil to clear it. A scenario can be filed under any number of categories: client.traces.save_scenario(trace_id, category_ids: [category.fetch("id")]) replaces its whole set, leaving category_ids: out keeps the set, and [] clears it. The returned scenario lists them under "categories". Two justifications cite the spans supporting the assertion and its category assignment. Assertion reads return one approval decision covering both; only a person in Bitfab can set it. Categories themselves are not approved. Every agent verdict scores one approved assertion, so verdicts.save and verdicts.skip take a required assertion_id: keyword, and verdicts.save_all raises ArgumentError before any request for an entry without "assertionId". verdicts.archive may omit it to clear an old whole-trace agent verdict. verdicts.save_human takes a required assertion_id: keyword too, and verdicts.save_human_all checks every entry the same way. Verdicts have no approval step, so agent and human verdicts both count as soon as they are written. See the reference for assertions and categories, verdicts, and grader verdicts.

Experiments

client.experiments reads runs back after the fact, for a dashboard or a report outside the replay process. list pages newest first with every filter optional and combinable, get reads one with its provenance, and get_rollup / get_rollup_all return the pass/fail rollup the dashboard shows, with one row per category. get_rollup_all takes up to 100 ids in one request, so read a month of runs in one call rather than one per run.

page = client.experiments.list(dataset_id: dataset_id, git_branch: "dev", metadata: {"schedule" => "eod"}, status: "completed", limit: 30)
rollups = client.experiments.get_rollup_all(page["experiments"].map { |experiment| experiment["id"] })
rollups.each do |result|
  result["rollup"]["categories"].each do |row|
    verdicts = row["verdicts"]
    puts "#{row["category"]["title"]}: #{verdicts["passing"]}/#{verdicts["passing"] + verdicts["failing"]}"
  end
end

Pass metadata: on replay, or --metadata KEY=VALUE on the replay command, to tag a run with your own strings. They come back on every read and list matches them by containment. See the reference for every method.

BAML execution and collectors

client.call("function-name", text: "hello") fetches the function’s current BAML prompt and executes it locally. String inputs are coerced according to the BAML signature. Parsed structured results remain Ruby objects. The result and native collector are submitted through the client’s tracing transport. Install BAML with gem install baml sorbet-runtime. The adapter loads BAML only when called. RubyGems currently publishes BAML 0.90.2; the adapter is verified with that native runtime. This does not change the repository’s TypeScript and Python BAML version. Provider credentials come from env_vars: or the process environment. The canonical runtime credential filter currently forwards only OPENAI_API_KEY. For a generated client, enrich the active span with the rendered prompt, model, provider, token counts, and duration:

client = Bitfab::Client.new(api_key: ENV["BITFAB_API_KEY"], baml_client: Baml.Client)
wrapped = client.wrap_baml(:ExtractResume, on_collector: ->(collector) { puts collector.usage })
result = wrapped.call(resume: text)
collector = wrapped.collector

Call the wrapper inside a bitfab_span or configured bitfab_node method. You can also pass client.wrap_baml(Baml.Client, :ExtractResume) or a bound method. The wrapper preserves arguments and the return value. Its collector reader exposes the most recently completed invocation. The fluent client.get_function(key).wrap_baml helper enriches the active span without creating a root for its bound key. If BAML is absent, wrappers call the generated method directly. Local prompt execution requires the native gem.

​Installation

​Quick Start

​Basic Configuration

​API key resolution

​Tracing

​Custom (Recommended)

​Using Bitfab::Traceable to Link Spans

​Experimental: Trace a Whole Call Tree

​Multi-File Projects

​Using bitfab_span for a Single Span

​Automatic Nesting

​Span Options

​Span Context

​Span Prompt

​Trace Context

​Read One Persisted Span

​Dropping a Trace

​Error Handling

​Flushing Traces

​OpenTelemetry Transport

​Wrapping Third-Party Methods

​Replay

​Mock child spans during replay

​Injecting custom values with overrides

​Fluent API: client.get_function

​Adapting inputs after a signature change

​Programmatic Replay Output

​Replay Registry

​When the traced method needs the app’s dependencies

​Datasets

​Assertions, categories, and verdicts

​Experiments

​BAML execution and collectors

Installation

Quick Start

Basic Configuration

API key resolution

Tracing

Custom (Recommended)

Using `Bitfab::Traceable` to Link Spans

Experimental: Trace a Whole Call Tree

Multi-File Projects

Using `bitfab_span` for a Single Span

Automatic Nesting

Span Options

Span Context

Span Prompt

Trace Context

Read One Persisted Span

Dropping a Trace

Error Handling

Flushing Traces

OpenTelemetry Transport

Wrapping Third-Party Methods

Replay

Mock child spans during replay

Injecting custom values with overrides

Fluent API: `client.get_function`

Adapting inputs after a signature change

Programmatic Replay Output

Replay Registry

When the traced method needs the app’s dependencies

Datasets

Assertions, categories, and verdicts

Experiments

BAML execution and collectors