Skip to main content

​
What is Bitfab?

Bitfab helps you test code changes and run experiments on your AI features by replaying traces and handling unsafe side-effects: a replay never re-sends an email, re-charges a card, or writes to your production database. You capture traces with our SDK, then simulate and judge changes directly from your coding agent - with the web portal available to dive deeper when you need it.
1

Capture

Add the SDK to capture inputs, outputs, errors, and context for every AI call. 1-shot setup with the Claude plugin or MCP - your coding agent reads your codebase, finds AI workflows, instruments them, and creates replay scripts.
2

Simulate

Replay production traces through your changed code. Bitfab’s simulation engine reproduces cases that process messy real-world data through a mix of regular code and LLM calls, and handles unsafe side-effects: replay mocking returns recorded outputs for the calls you don’t want to re-run, and database snapshots restore the database state at capture time.
3

Judge

Score every replay with graders and run experiments across whole datasets. Each replayed trace comes back fixed, regressed, still passing, or still failing, so you know a change works before you ship.

​
Quick Start

Install the CLI and run init:
This detects your editor (Claude Code, Codex, or Cursor), installs the Bitfab plugin, authenticates you, and launches the setup workflow. See the Claude Code plugin docs for full details. For agents without a dedicated plugin, connect via MCP or visit the setup page.

​
How It Works

​
Capture

Your coding agent handles the entire setup:
  1. Instrument + create replay scripts - Reads your codebase, finds AI workflows, and in parallel adds tracing with minimal diffs and generates scripts so you can regression-test against production data with one command
  2. Get traces - Run your app and traces flow into Bitfab automatically
Instrumenting by hand instead? Read Instrumentation first. It covers the two decisions you make by choosing spans: capturing enough to see and evaluate each step, and deciding what gets mocked on replay.

​
Simulate

Once traces are flowing, test changes directly from your coding agent:
  1. Build a dataset - Tell your agent which traces to test against, in plain language:
  2. Fix and replay - Your agent suggests fixes and replays the dataset through your updated code. The simulation engine reproduces each case as it happened, even when the workflow mixes regular code and LLM calls over messy real-world data.
  3. Contain side-effects - Mock the calls you’re not testing so LLM calls, API calls, and database lookups return their recorded outputs instead of re-running for real, and snapshot your database so every replay sees the database state at capture time.

​
Judge

Every replay gets scored, so verification is automatic:
  1. Grade replays - Graders mark each replay pass or fail on the properties that matter
  2. Run experiments - Replay a whole dataset against your changed code and see each trace come back fixed, regressed, still passing, or still failing
  3. Dive deeper - Use the web portal to inspect individual traces, compare outputs side-by-side, and explore patterns across your data

​
Core Workflows

Primitives

Traces, datasets, graders, and experiments: the four objects Bitfab is built from and how they chain into an iteration loop.

Instrumentation

Choose your spans: capture enough to see and evaluate each step, and decide what gets mocked on replay.

Replay Mocking

Replace selected child calls with recorded outputs so you can iterate on the root behavior without paying for every dependency again.

Database Snapshots

Replay each trace against the database state at capture time.