> ## Documentation Index > Fetch the complete documentation index at: https://docs.bitfab.ai/llms.txt > Use this file to discover all available pages before exploring further. # Ruby SDK > Generate evaluations and iterate on your AI applications The Bitfab Ruby SDK captures your AI function calls to automatically generate evaluations. Re-run your prompts with different models, parameters, and inputs to iterate faster. Framework-native adapters (LangGraph, OpenAI Agents, BAML, Claude Agent SDK) are not yet available for Ruby. See [Frameworks overview](/frameworks/overview) for current coverage. Instrument Ruby code manually via `Bitfab::Traceable` or `Bitfab.span`. ## Installation ```bash theme={null} # Bundler bundle add bitfab # Gem gem install bitfab ``` ## Quick Start ```ruby theme={null} require "bitfab" Bitfab.configure(api_key: ENV.fetch("BITFAB_API_KEY")) ``` Need an API key? Get one from the [Bitfab dashboard](https://bitfab.ai/setup) or see the [API Keys guide](/api-keys) for detailed setup instructions. Copy this prompt into your coding agent (tested with Cursor and Claude Code using Sonnet 4.5): ```text theme={null} Modify existing Ruby code to add Bitfab tracing. Do NOT browse or web search. Use ONLY the API described below. Bitfab Ruby SDK (authoritative excerpt): - Install: `gem install bitfab` or `bundle add bitfab` - Init: require "bitfab" Bitfab.configure(api_key: ENV.fetch("BITFAB_API_KEY")) - Instrumentation (ONLY allowed form): class MyService include Bitfab::Traceable bitfab_function "" bitfab_span :method_name, type: "function" def method_name # ... end end (bitfab_span must be placed immediately ABOVE the `def` it instruments.) - Span types: "llm", "agent", "function", "guardrail", "handoff", "custom" - DO NOT use a block form of bitfab_span. - DO NOT extract helper methods. Task: 1) Ensure the bitfab gem is added and initialization exists (Gemfile + initializer). 2) Read the codebase and identify ALL AI workflows (LLM calls, agent runs, AI-driven decisions). 3) Present me with a numbered list of workflows you found. For each, describe: - What it does - Why it's worth instrumenting: what visibility tracing gives you into each step 4) After I choose which workflow(s) to instrument: - Add `bitfab_span` directly ABOVE each method's `def` - Instrument intermediate steps (not just the final output) so each trace has enough context to diagnose issues - Ensure each class includes `Bitfab::Traceable` and has `bitfab_function` set 5) Do not change method signature, behavior, or return value. Minimal diff. Output: - First: your numbered list of workflows with why each is worth instrumenting - After my selection: minimal diffs for Gemfile, initializer, and the method changes ``` ## Basic Configuration ```ruby theme={null} Bitfab.configure(api_key: "...") # Omit api_key entirely: the SDK reads ENV["BITFAB_API_KEY"] Bitfab.configure # Disable tracing (methods still execute, but no spans are sent) Bitfab.configure(api_key: "...", enabled: false) ``` **Missing API key doesn't crash.** If the API key is missing, empty, or whitespace-only, the SDK automatically disables tracing and logs a one-time warning at first use. All instrumented methods still execute normally: no spans are sent, no errors are thrown. You don't need any conditional logic around the API key. ### API key resolution The key is resolved **lazily, the first time a span runs**, not when `configure` is called. Resolution order is the configured value (a `Proc` is called while still unresolved), then a fallback read of `ENV["BITFAB_API_KEY"]`. ```ruby theme={null} # Pass a Proc to defer resolution explicitly (resolved at first use): Bitfab.configure(api_key: -> { ENV["BITFAB_API_KEY"] }) ``` For standalone scripts where a run that emits no traces should be treated as a failure rather than silently skipped, set `strict`: ```ruby theme={null} # Raises on the first traced call if no key resolves, instead of disabling quietly Bitfab.configure(api_key: ENV["BITFAB_API_KEY"], strict: true) ``` ## Tracing ### Custom (Recommended) #### Using `Bitfab::Traceable` to Link Spans Include `Bitfab::Traceable` in a class, declare the trace function key once with `bitfab_function`, then use `bitfab_span` to wrap methods. Three declaration styles are supported: ```ruby theme={null} class OrderService include Bitfab::Traceable bitfab_function "order-processing" # Style 1: Before-def (recommended): declare bitfab_span above the method bitfab_span :process_order, type: "function" def process_order(order_id) { order_id: order_id } end # Style 2: Inline: wrap the def directly bitfab_span def validate_order(order_id) { valid: true } end, type: "guardrail" # Style 3: After-def: declare bitfab_span after the method definition def enrich_order(order_id) { enriched: true } end bitfab_span :enrich_order, type: "function" end ``` All three styles are equivalent. The before-def style is recommended for readability. #### Multi-File Projects For projects with instrumented methods spread across multiple files, create an initializer that configures Bitfab, then include `Bitfab::Traceable` in any class that needs tracing. ```ruby theme={null} # config/initializers/bitfab.rb (single source of truth) require "bitfab" Bitfab.configure(api_key: ENV.fetch("BITFAB_API_KEY")) ``` ```ruby theme={null} # app/services/process_order_service.rb class ProcessOrderService include Bitfab::Traceable bitfab_function "order-processing" bitfab_span :process_order, type: "function" def process_order(order_id) { order_id: order_id } end end ``` ```ruby theme={null} # app/services/validate_order_service.rb class ValidateOrderService include Bitfab::Traceable bitfab_function "order-processing" bitfab_span :validate_order, type: "guardrail" def validate_order(order_id) { valid: true } end end ``` Classes sharing the same `bitfab_function` key are grouped together. Spans from different classes are automatically linked as parent-child when one instrumented method calls another. #### Using `bitfab_span` with Explicit Key For a single span with an explicit trace function key: ```ruby theme={null} class StandaloneService include Bitfab::Traceable bitfab_span :standalone_task, trace_function_key: "one-off-operation" def standalone_task "done" end end ``` #### Automatic Nesting Spans nest automatically based on call stack: ```ruby theme={null} class Pipeline include Bitfab::Traceable bitfab_function "pipeline" bitfab_span :outer, type: "agent" def outer inner # Becomes a child of "outer" end bitfab_span :inner, type: "function" def inner # ... end end ``` #### Span Options **Parameters:** * `method_name` (required): Symbol of the method to wrap * `trace_function_key` (optional): Override class-level `bitfab_function` * `name` (optional): Display name. Defaults to method name * `type` (optional): Span type. Defaults to `"custom"`. A label only, used to organize and filter spans in the dashboard; it does not change how the span is traced, replayed, or evaluated **Span Types:** ```ruby theme={null} SPAN_TYPES = %w[ llm # LLM calls agent # Agent workflows function # Function calls guardrail # Safety checks handoff # Human handoffs custom # Default ] ``` **Examples:** ```ruby theme={null} class SafetyService include Bitfab::Traceable bitfab_function "safety-service" # Method name is automatically captured as span name bitfab_span :check_safety, type: "guardrail" def check_safety(content) { safe: !content.include?("unsafe") } end # Override with name option bitfab_span :validate_input, name: "InputValidator", type: "guardrail" def validate_input(input) { valid: !input.empty? } end end ``` #### Span Context Use `Bitfab.current_span` to get a handle to the active span, then call `.add_context()` to attach contextual key-value pairs from inside a traced method, useful for runtime values like request IDs, computed scores, or dynamic context: ```ruby theme={null} class OrderService include Bitfab::Traceable bitfab_function "order-processing" bitfab_span :process_order, type: "function" def process_order(order_id) user_id = current_user_id Bitfab.current_span.add_context("user_id" => user_id, "order_id" => order_id) { order_id: order_id, status: "completed" } end end ``` Each `add_context` call pushes the entire hash as one entry. Multiple calls accumulate entries: ```ruby theme={null} Bitfab.current_span.add_context("user_id" => "u-123") Bitfab.current_span.add_context("request_id" => "req-789") # Result: contexts: [{ "user_id" => "u-123" }, { "request_id" => "req-789" }] ``` You can also access the canonical Bitfab span and trace IDs via `Bitfab.current_span.id` and `Bitfab.current_span.trace_id` (both return an empty string outside a span): ```ruby theme={null} span_id = Bitfab.current_span.id trace_id = Bitfab.current_span.trace_id ``` #### Span Prompt Use `Bitfab.current_span` to set the prompt string on the current span. This is stored in `span_data.prompt` and is useful for capturing the exact prompt text sent to an LLM: ```ruby theme={null} class ClassificationService include Bitfab::Traceable bitfab_function "classification" bitfab_span :classify_text, type: "llm" def classify_text(text) prompt = "Classify the following text: #{text}" Bitfab.current_span.set_prompt(prompt) llm.complete(prompt) end end ``` The prompt is metadata only. It records the prompt text for display and reference in the dashboard; it does not send the prompt to any model or change what the span executes. The last `set_prompt` call wins -- it overwrites any previously set prompt on the span. Calling `set_prompt` outside a span context is a no-op (it never crashes). #### Trace Context Use `Bitfab.current_trace` to set context that applies to the entire trace (all spans within a single execution). This is useful for grouping traces by session or attaching trace-level metadata: ```ruby theme={null} class OrderService include Bitfab::Traceable bitfab_function "order-processing" bitfab_span :process_order, type: "function" def process_order(order_id) trace = Bitfab.current_trace # Set session ID (stored as database column, filterable in dashboard) trace.set_session_id("session-123") # Set trace metadata (stored in raw trace data) trace.set_metadata("region" => "us-west-2", "environment" => "production") # Add context entries (stored as key-value pairs, accumulates across calls) trace.add_context("workflow" => "checkout-flow", "batch_id" => "batch-2024-01") { order_id: order_id, status: "completed" } end end ``` * `set_session_id(id)`: Groups traces by user session. Stored as a database column for efficient filtering. * `set_metadata(hash)`: Arbitrary key-value metadata on the trace. Merges with existing metadata. * `add_context(hash)`: Key-value context entries. Accumulates across multiple calls. #### Read One Persisted Span Fetch one span without loading the full trace. Repeated name matches default to the last span. ```ruby theme={null} span = Bitfab.client.get_trace_span(trace_id, name: "GenerateAnswer") first = Bitfab.client.get_trace_span(trace_id, name: "GenerateAnswer", occurrence: "first") exact = Bitfab.client.get_trace_span(trace_id, id: span_id) ``` Both IDs are canonical Bitfab IDs; ingestion source IDs are not accepted. `occurrence` also accepts a zero-based integer. A missing trace or span returns `nil`. #### Dropping a Trace Call `.drop` on the current-trace handle to discard the in-flight trace. Once flagged, spans that complete afterward are not uploaded at all, and the flag rides out on the completion payload, so when the trace completes the server scrubs any payloads that already raced out (the trace, its external trace, and sibling spans), deletes the archived S3 objects, and marks it `dropped` instead of `completed`, keeping only a skeleton audit row. Use it to discard runs you never want stored (health checks, test traffic) or a run you know carries sensitive data. ```ruby theme={null} class OrderService include Bitfab::Traceable bitfab_function "order-processing" bitfab_span :process_order, type: "function" def process_order(order_id) Bitfab.current_trace.drop if health_check?(order_id) { order_id: order_id, status: "completed" } end end ``` * Safe to call outside a trace (a no-op), and never raises into your application. #### Error Handling Errors are captured in the span and re-raised: ```ruby theme={null} class RiskyService include Bitfab::Traceable bitfab_function "risky-service" bitfab_span :risky def risky raise "error" end end begin RiskyService.new.risky rescue => e # Span records error and timing end ``` Each error is classified by source. Errors raised by your code are recorded with `error_source: "code"`. SDK-internal errors are recorded with `source: "sdk"`. Both appear in the span's `errors` array in the Bitfab dashboard. #### Flushing Traces ```ruby theme={null} Bitfab.flush_traces(timeout: 30) # Default: 30s ``` Traces flush automatically on process exit via `at_exit` hook. #### Wrapping Third-Party Methods Use `Bitfab::Traceable.wrap` to trace methods on external classes: ```ruby theme={null} require "openai" Bitfab::Traceable.wrap( OpenAI::Client, :chat, trace_function_key: "openai", name: "Chat", type: "llm" ) # Now all calls to client.chat are traced client = OpenAI::Client.new(access_token: ENV["OPENAI_API_KEY"]) client.chat(parameters: { model: "gpt-4", messages: [...] }) ``` ### Replay A trace is replayable when its root span has serializable inputs (the recorded inputs must round-trip through `to_json`). Framework handlers are not yet available for Ruby, so serializable root inputs are the only path to a replayable trace; instrument the outer workflow method so its inputs serialize. Replay historical traces through a method to create test runs. This re-runs past inputs through your updated code and compares the results. ```ruby theme={null} client = Bitfab.client # Pass an instance for instance methods service = OrderService.new(api_key: "...", db: db) result = client.replay( service, :process_order, trace_function_key: "order-processing", limit: 5 ) # Or pass a Class for class methods result = client.replay( OrderService, :process_order, trace_function_key: "order-processing", limit: 5 ) puts "Test Run URL: #{result[:test_run_url]}" puts "Test Run ID: #{result[:test_run_id]}" result[:items].each do |item| puts "Input: #{item[:input]}, Result: #{item[:result]}, Error: #{item[:error]}" puts "Duration (ms): #{item[:duration_ms]}" puts "Tokens: #{item[:tokens]}" # { input:, output:, cached:, total: } or nil puts "Model: #{item[:model]}" puts "Trace ID: #{item[:trace_id]}" # Server trace ID for the replayed execution end ``` Replay waits for each item's trace (spans + completion) to be persisted server-side before completing the test run, so `:trace_id` is a real server trace ID for completed items. If NO completed item's trace persisted (uploads wholesale failed, or the replayed method isn't traced), `replay` raises a `RuntimeError` instead of silently returning `nil` trace IDs. If only SOME items' traces are missing (a transient per-item upload failure), those items get `nil` trace IDs with a loud warning and the rest of the run is returned intact. `:trace_id` is also `nil` for errored (unreplayable) items, and for all items when the server predates the trace-ID mapping (a warning explains which). Per-item `:duration_ms` and `:model` come from the historical trace that fed the item. `:tokens` is the **replayed run's** token usage (the same numbers Studio's experiments view shows), so comparing each item's `:tokens[:total]` against the original trace's recorded usage tells you how your change moved cost. Each field is `nil` when it wasn't captured. **Parameters:** * `receiver` (required): An instance for instance methods, or a Class for class methods * `method_name` (required): Symbol of the method to replay * `trace_function_key` (required): The trace function key * `limit` (optional): Max recent traces to replay (default: 5; maximum: 5,000). Ignored when `trace_ids` or `dataset_id` is passed: an explicit ID list or dataset already determines how many traces replay. * `trace_ids` (optional): Array of specific trace IDs to replay (max 100). The ID count determines how many traces replay; `limit` is ignored when both are passed. * `name` (optional): Display name for the resulting experiment/test run. * `max_concurrency` (optional): Max threads for parallel replay (default: 10) * `code_change_description` (optional): Rationale for the code change being tested in this replay (stored on the experiment) * `code_change_files` (optional): Array of edited files, each as `{ path:, before:, after: }` (use `""` for newly created or deleted files) * `mock` (optional): Mock strategy for child spans during replay. One of `"marked"` (default, only spans tagged with `mock_on_replay: true` return historical output), `"none"` (every child runs real code), or `"all"` (every child returns its historical output) * `experiment_group_id` (optional): UUID string that groups multiple replay runs into a single experiment batch. Pass the same ID across successive `replay()` calls to link them together in the dashboard. * `grader_ids` (optional): Array of grader UUIDs (max 100) attached directly to this replay run, independent of the dataset's own graders. The resulting experiment is graded by the union of these and the dataset's runnable graders. Use it to grade a single run with a check you don't want to add to the dataset permanently. Each must be an active grader in the same organization and trace function, or the replay is rejected with a 400. A replay with no dataset can still carry graders this way. * `adapt_inputs` (optional): A callable `->(args, kwargs, ctx)` that reshapes recorded inputs onto the method's current signature when its shape changed after the traces were captured. See **Adapting inputs after a signature change** below. * `on_progress` (optional): A callable `->(progress)` fired once per item as it settles, with running totals plus the settled item payload (source trace id, local replay trace id, input, result, original output, error, duration, tokens/model metadata). Use it to render live progress or start evaluating completed items while replay runs. A raising callback never crashes the run. Bitfab plugin replay scripts can pass the SDK's ready-made reporter straight in (`on_progress: Bitfab.method(:report_replay_progress)`); it writes the event to stderr, which the Bitfab plugin polls to report live progress and write per-item result files while replay runs (stdout remains available for direct-run ReplayResult JSON). * `environment` (optional): A `Bitfab::ReplayEnvironment`. When passed, the Bitfab server resolves a per-trace database branch from each source trace's captured snapshot reference, and the SDK exposes that branch's URL via `environment.database_url` inside the replayed method (releasing the branch after each item). Read `environment.active?` to fall back to your live database when no branch was resolved (e.g. the trace predates snapshot capture, or DB branching isn't configured). Construct one with `Bitfab::ReplayEnvironment.new` and read it only inside the replayed method. **Notes:** * `receiver` + `method_name` must resolve to the method that carries the `traceable` decoration. Passing a plain wrapper around it will not resolve the trace function key. * **`trace_function_key` must match the method's declared key.** It is read from the `bitfab_span` / `Bitfab::Traceable.wrap` declaration on the method you point at. If the key you pass contradicts that declared key, `replay` raises an `ArgumentError`: it would otherwise fetch one function's historical traces but record the replay under the method's own key, producing an incoherent test run. (Ruby has no plain-callable replay form, so there is no reason to pass a non-matching key.) * `trace_function_key` is passed explicitly, so instance methods and class methods are disambiguated by the `receiver`. * **Use a single `Bitfab::Client` across instrumentation and replay.** If your instrumented module constructs a client at load and your replay script constructs another, they do not share registered trace functions; import the client from the instrumented module (or a shared singleton) rather than constructing a new one in the replay script. **Replay specific traces:** ```ruby theme={null} service = OrderService.new(api_key: "...") result = client.replay( service, :process_order, trace_function_key: "order-processing", trace_ids: ["trace-id-1", "trace-id-2"] ) ``` **Attaching a code change:** Each replay creates an experiment (test run). When you're iterating on a method and replaying after every edit, attach the change so the dashboard can show *exactly what was edited* alongside the results. Read each file before editing, edit, then read it again: the two strings go straight into `code_change_files`. There's no diff format to construct. ```ruby theme={null} before = File.read("lib/order_service.rb") # ...edit lib/order_service.rb... after = File.read("lib/order_service.rb") result = client.replay( service, :process_order, trace_function_key: "order-processing", code_change_description: "fix off-by-one in retry logic", code_change_files: [ {path: "lib/order_service.rb", before:, after:} ] ) ``` Both options are optional and independent: pass just `code_change_description` for a quick rationale-only annotation, or just `code_change_files` to record the literal edits. If you pass neither, `replay` falls back to capturing your working-tree diff against the trunk merge-base (best-effort, only inside a git repo), so an experiment still shows a diff. Passing `code_change_files` explicitly always wins and is the way to record a precise per-edit before/after. Set `BITFAB_DISABLE_CODE_CHANGE_CAPTURE` to turn the fallback off. #### Mock child spans during replay For the workflow-level guide, see [Replay Mocking](/replay-mocking). By default replay uses `"marked"`: child spans tagged with `mock_on_replay: true` return their historical outputs, while every other child runs real code. Three mock strategies control this behavior: ```ruby theme={null} # "marked" (default): only spans declared with mock_on_replay: true return historical output client.replay(service, :process_order, trace_function_key: "order-processing") # "none": every child span runs real code client.replay(service, :process_order, trace_function_key: "order-processing", mock: "none") # "all": every child span returns its historical output, real code never runs client.replay(service, :process_order, trace_function_key: "order-processing", mock: "all") ``` Tag the child spans you want mocked at definition time: ```ruby theme={null} class OrderService include Bitfab::Traceable bitfab_function "order-processing" # Paid LLM call: skip during replay by default bitfab_span :classify_intent, type: "llm", mock_on_replay: true def classify_intent(prompt); end # Cheap, deterministic: keep running real bitfab_span :persist, type: "function" def persist(order); end bitfab_span :process_order, type: "agent" def process_order(order) classify_intent(order.description) persist(order) end end ``` Use the default `mock: "marked"` behavior when you want to iterate on `process_order`'s logic without paying for the LLM call on each replay. Use `mock: "all"` when the goal is the cheapest possible replay (every child span returns its recorded output; only the root function executes real code). Repeated calls to the same `trace_function_key` are distinguished by call order, so `step:0`, `step:1`, `step:2` correspond to the first, second, and third invocations. Unmarked spans still advance the counter, so a marked sibling that runs *after* an unmarked one lines up with the right historical entry. #### Injecting custom values with overrides A **mock override** substitutes a value you supply for a matched span, so downstream real code runs against it -- for "what if this step returned X" experiments without editing the traced code. An override is a `{ match:, value: }` hash: `match` is a callable selecting spans by structural metadata (`node[:trace_function_key]`, `node[:span_name]`, `node[:type]`, `node[:original_span_id]`); `value` is a flat value injected as-is, or a callable that returns one. ```ruby theme={null} result = client.replay( processor, :process_order, trace_function_key: "order-processing", mock: "none", # run everything real... mock_override: { # ...except this span, which gets the value you supply match: ->(node) { node[:trace_function_key] == "classify-intent" }, value: { label: "refund" } # flat value } ) ``` A callable `value` receives a context hash with the live positional `:inputs`, the live keyword `:kwargs` (empty when the call used none), and `:get_original_output` (synchronous in Ruby -- the recorded output is already on the replay tree) to tweak the recorded output instead of replacing it: ```ruby theme={null} value: ->(ctx) { ctx[:get_original_output].call.merge(score: 1) } ``` Register overrides on the client to apply them to every replay (keyword or positional form), and reset with `clear_mock_overrides`: ```ruby theme={null} client.register_mock_override( match: ->(node) { node[:type] == "llm" }, value: { label: "refund" } ) # Positional form (equivalent): client.register_mock_override(match, value) client.clear_mock_overrides ``` Precedence per span: per-call `mock_override:`, then registered overrides, then the base `mock:` strategy (a span no override matches falls back to it). Pass a single override hash or an array (first matcher wins). ### Fluent API: `client.get_function` Bind a `trace_function_key` once and wrap multiple classes or methods against it. Mirrors `client.get_function` in the Python SDK and `client.getFunction` in TypeScript. ```ruby theme={null} fn = Bitfab.client.get_function("openai") fn.wrap(OpenAI::Client, :chat, name: "Chat", type: "llm") fn.wrap(OpenAI::Client, :embeddings, name: "Embed", type: "llm") ``` `#wrap` accepts the same options as `Bitfab::Traceable.wrap` (`name`, `type`, `mock_on_replay`), but the `trace_function_key` is fixed to the one bound on the returned `Bitfab::BitfabFunction`. #### Adapting inputs after a signature change Replay deserializes each trace's inputs exactly as they were captured against the method's signature **at trace time**, then calls the current method with them. If the signature drifted since capture (an argument renamed, reordered, folded into a hash, or a new required argument added), the call no longer lines up and raises. The `adapt_inputs` hook reshapes the recorded inputs onto the current signature so replay can still run: ```ruby theme={null} # Recorded as positional (user_id, limit); current signature takes a hash. adapt = ->(args, _kwargs, _ctx) { [[{user_id: args[0], limit: args[1]}], {}] } result = client.replay( service, :process_order, trace_function_key: "order-processing", adapt_inputs: adapt ) ``` The hook receives the deserialized `(args, kwargs)` plus a per-trace `ctx` (`{ original_trace_id:, original_span_id: }`, with deprecated `source_*` aliases) and returns `[new_args, new_kwargs]`. The returned `args` is what `item[:input]` reports. It runs once per item, **inside the same rescue as the method**: if it raises, that item's `:error` is set and the run continues, so one unmappable trace never crashes the batch. `ctx[:original_trace_id]` (the original Bitfab trace ID) lets a table-driven adapter look up a per-trace transform. That's the escape hatch for reshapes that need judgement rather than mechanical rearrangement: compute the adapted inputs per trace up front, then have the hook look them up by `original_trace_id`, keeping replay deterministic instead of calling a model mid-replay. When the new signature has a genuinely new **required** argument with no analog in the recorded trace, don't fabricate one. There's nothing faithful to map it to, so leave those traces unmapped (let them raise) rather than inventing test inputs. For anything beyond a one-liner, keep the adapter in its own file next to the replay script and require it: ```ruby theme={null} # scripts/replay_adapters/extraction.rb EXTRACTION_ADAPTER = ->(args, _kwargs, _ctx) do user_id, limit = args [[{user_id:, limit:}], {}] end ``` ```ruby theme={null} # scripts/replay.rb require_relative "replay_adapters/extraction" client.replay(service, :extract_memories, trace_function_key: "my-function", adapt_inputs: EXTRACTION_ADAPTER) ``` That keeps the transform versioned and reviewable alongside the method it adapts, and you add the require only when a drift actually needs it. #### Replay Output Contract Replay results are typically consumed by automation (CI logs, code reviewers, and coding agents). When `BITFAB_REPLAY_RESULT_PATH` is set, `Bitfab.replay` automatically writes the full replay result JSON to that file. For direct/manual runs, **emit the full replay result hash as a single stdout JSON block** so a consumer can `JSON.parse` it and reason about every field, including the new per-item `:duration_ms`, `:tokens`, and `:model`. Never print only lengths, counts, hashes, or truncated previews, and never replace the JSON block with ad-hoc per-field log lines. Recommended script tail: ```ruby theme={null} result = Bitfab.client.replay(service, :process_order, trace_function_key: "order-processing", limit:) # Human-readable summary goes to stderr, so stdout stays pure JSON. warn "Test run: #{result[:test_run_url]}" warn "Items: #{result[:items].length}" # Full structured dump to stdout, ready for JSON.parse. puts JSON.pretty_generate(result) ``` The dumped object includes every item's `:input`, `:result`, `:original_output`, `:error`, `:duration_ms`, `:tokens`, `:model`, and `:trace_id`, plus `:test_run_id` and `:test_run_url`. When the Bitfab plugin runs this script, it sets `BITFAB_REPLAY_RESULT_PATH`; the SDK writes the final result there, and the plugin reads that file into the replay run's `.bitfab/replays//events.jsonl` while writing large per-item payloads under `.bitfab/replays//items/`. **Per-item errors are part of the contract.** If the wrapped method raises on a given trace, `Bitfab.replay` rescues it, sets `item[:error]`, leaves `item[:result]` as `nil`, and continues. Treat items with `item[:error]` set as **unreplayable**, not as failing outputs; compute pass/fail only over items where it's `nil`. This matters most for DB reads/writes: a stale FK, missing record, or rejected write is infra failure, not a regression. **Don't swallow per-item errors in the script.** A custom `begin/rescue` that returns a placeholder turns infra failures into fake successes. Let the SDK record them. The only allowed top-level `rescue` is a fatal handler around `main` that exits non-zero, so callers can tell a whole-replay crash from a clean run with some unreplayable items. **Environment.** Replay executes in the app's own process: the instrumented method is loaded as a library, and its DB clients, env vars, config loaders, and model IDs resolve from whatever environment the replay script is run under. The script must bootstrap the same environment the app uses (e.g. `require "dotenv/load"` at the top, or run via `bundle exec dotenv ruby scripts/replay.rb`). Do not mock these; they're the same dependencies the app resolves in production. For replay to see the same DB rows the trace was captured against, point the script at the trace's source environment (the `:environment` field on the trace: production / staging / development). **Input serialization caveat.** Replay deserializes historical span inputs and passes them back to your method. This works for strings, numbers, and plain hashes. If your span wraps a method that takes hydrated domain objects (ActiveRecord models, class instances, DB records), they won't round-trip through serialization; move the span to where inputs are IDs or plain data and let the method fetch objects internally, or reshape arguments in the wrapper. #### Replay Script Create a standalone script to regression-test your trace functions against production data with one command. The script maps pipeline names to their replay functions, accepts CLI flags, and prints a side-by-side comparison with delta summaries. ```ruby theme={null} #!/usr/bin/env ruby # Replay production traces through instrumented functions. # # Uses Bitfab.client.replay to fetch real traces and re-run them # through the current code, creating a test run for side-by-side comparison. # # Usage: # ruby scripts/replay.rb # ruby scripts/replay.rb --limit 20 # ruby scripts/replay.rb --trace-ids id1,id2 require "dotenv/load" require "json" require_relative "../config/initializers/bitfab" require_relative "../services/extraction_pipeline" require_relative "../services/search_pipeline" FUNCTIONS = { "extraction" => "my-extraction-pipeline", "search" => "my-search-pipeline", }.freeze pipeline = ARGV[0] unless pipeline && FUNCTIONS.key?(pipeline) warn "Usage: ruby scripts/replay.rb <#{FUNCTIONS.keys.join('|')}> [--limit N] [--trace-ids id1,id2]" exit 1 end limit = 10 trace_ids = nil ARGV[1..].each_with_index do |arg, i| case arg when "--limit" limit = ARGV[i + 2].to_i when "--trace-ids" trace_ids = ARGV[i + 2].split(",").map(&:strip) end end # Each pipeline gets its own replay method. If a signature drifts after traces # were captured, add an input adapter (see the "Adapting inputs" section): # write scripts/replay_adapters/.rb defining an adapter lambda, require it, # and pass it here, e.g. # require_relative "replay_adapters/extraction" # defines EXTRACTION_ADAPTER # adapt_inputs: EXTRACTION_ADAPTER # limit is ignored when trace_ids is passed -- an explicit ID list already # determines how many traces replay. def replay_extraction(limit:, trace_ids:) service = ExtractionPipeline.new opts = trace_ids ? {trace_ids:} : {limit:} Bitfab.client.replay( service, :extract_memories, trace_function_key: FUNCTIONS["extraction"], # Live progress the Bitfab plugin polls while replay runs in the background. on_progress: Bitfab.method(:report_replay_progress), **opts ) end def replay_search(limit:, trace_ids:) service = SearchPipeline.new opts = trace_ids ? {trace_ids:} : {limit:} Bitfab.client.replay( service, :search_documents, trace_function_key: FUNCTIONS["search"], on_progress: Bitfab.method(:report_replay_progress), **opts ) end REPLAY_FNS = { "extraction" => method(:replay_extraction), "search" => method(:replay_search), }.freeze function_key = FUNCTIONS[pipeline] # Human-readable output goes to stderr; stdout carries only the ReplayResult JSON. $stderr.puts "[replay] Replaying #{trace_ids&.length || limit} traces from \"#{function_key}\"...\n" result = REPLAY_FNS[pipeline].call(limit:, trace_ids:) $stderr.puts "Test run: #{result[:test_run_url]}\n" changed = same = errors = 0 result[:items].each do |item| raw_input = item[:input] || [] label = (raw_input.first || "unknown").to_s[0, 80] if item[:error] $stderr.puts " ✗ \"#{label}\"" $stderr.puts " Error: #{item[:error]}" errors += 1 else orig = item[:original_output] new_val = item[:result] orig_str = orig.is_a?(String) ? orig : orig.to_json new_str = new_val.is_a?(String) ? new_val : new_val.to_json is_same = orig_str == new_str marker = is_same ? "=" : "Δ" $stderr.puts " #{marker} \"#{label}\"" $stderr.puts " Original: #{orig_str}" $stderr.puts " New: #{new_str}" is_same ? same += 1 : changed += 1 end end $stderr.puts "\n─── Summary ───" $stderr.puts " Pipeline: #{pipeline}" $stderr.puts " Replayed: #{result[:items].length}" $stderr.puts " Same: #{same}" $stderr.puts " Changed: #{changed}" $stderr.puts " Errors: #{errors}" if errors > 0 $stderr.puts "\n #{result[:test_run_url]}" # Full ReplayResult as JSON to stdout (the Replay Output Contract). puts JSON.pretty_generate(result) ``` Adapt the imports, pipeline names, and per-pipeline replay methods to match your project's instrumented workflows.