Skip to main content
The Bitfab Claude Code plugin brings the full evaluation workflow into Claude Code. It provides MCP tools for trace inspection, datasets, labeling, experiments, and setup, slash commands for authentication, and automatic notifications — so you never have to leave your editor.

​
Installation

Run the CLI from your project directory: 
npx bitfab-cli init --editor claude
This installs the Bitfab plugin, opens your browser to log in, and launches /bitfab:setup. The CLI checks that Claude Code is signed in before it launches the Bitfab agent. If needed, run claude auth login first.
Add the Bitfab plugin marketplace and install:
> /plugin marketplace add Project-White-Rabbit/bitfab-claude-plugin
> /plugin install bitfab
> /exit
Then restart Claude Code and run the setup command to authenticate and instrument your codebase:
claude --continue

> /bitfab:setup

​
What the Plugin Does

​
Automatic Setup

The /bitfab:setup command runs a multi-phase workflow:
  1. Login — Opens your browser for OAuth authentication, saves credentials securely
  2. Instrument + Replay (in parallel, per workflow) — Reads your codebase, finds all AI workflows (LLM calls, agents, AI-driven decisions), and presents them as a numbered list. You choose which to instrument — it adds tracing with minimal diffs, and at the same time generates a replay script so you can regression-test your trace functions against production data with one command
You can run individual phases: 
> /bitfab:setup explain      # Explain what Bitfab is (read-only, no login)
> /bitfab:setup login        # Auth only
> /bitfab:setup instrument   # Trace instrumentation only
> /bitfab:setup inspect      # Diagnose (and offer to fix) your tracing setup
> /bitfab:setup replay       # Replay script creation only
> /bitfab:setup analyze-repo # Scan the repo and upload draft trace plans without prompts
The setup is interactive — it presents 2-5 concrete options per decision point with a recommended choice, so you stay in control throughout.

​
Assistant

The /bitfab:assistant command turns production traces into code improvements, whether the goal is correctness (improving pass rates) or efficiency (cutting token usage and cost). Your agent will do the mechanical work and collaborate with you on three steps:
  1. Build a dataset from production traces — search for failures, label them with expected outcomes
  2. Experiment against that dataset — make isolated code changes, replay, compare results
  3. Hill climb — repeat until the best change is found, then present results
Run it with an optional trace function key: 
> /bitfab:assistant
> /bitfab:assistant order-processing

​
Building the Dataset

Your coding agent does the data wrangling — it searches production traces for failures, reads full inputs and outputs, and identifies edge cases. It then presents edge cases for your judgment: is this a failure (and what should the output be), correct, or irrelevant? This labeled dataset becomes the benchmark for all experiments. The plugin opens a rich UI for navigating and labeling the dataset, then brings you back to your coding agent so you stay in flow. You can label every trace yourself, or label a few and let the agent classify the rest based on the patterns you’ve established.

​
Running Experiments

The command reads your code, diagnoses failure patterns, and categorizes proposed changes:
  • Code fixes — deterministic bugs, bundled into one experiment as a foundation
  • Judgment-based fixes — prompt changes, search tuning, output formatting — each gets its own experiment
  • Infrastructure proposals — larger changes noted for future work, not experimented on
Independent experiments run in parallel — each in its own isolated subagent on a separate git worktree. Each subagent edits the code, runs the replay script against your labeled dataset, and compares new outputs to expected outcomes.

​
Results

After each round, you see which traces now match expected outcomes, which still diverge, and whether any regressions occurred. You decide whether to keep iterating or stop. The final summary shows pass rate improvement and all files changed — uncommitted in your working tree for review.

​
MCP Tools

The plugin registers MCP tools that Claude Code can call during conversations. These let you inspect traces, manage datasets and labels, run experiments, and improve your code without leaving the editor.

​
Core

get_bitfab_api_key
Retrieve your API key for SDK initialization and environment variable configuration.
get_api_key_context
Returns which Bitfab org the plugin reads/writes to (it can differ from the project’s BITFAB_API_KEY and from the org open in Studio). Call before the first plugin write of a session, or when data you wrote isn’t visible in Studio. 
"Which org is this plugin connected to?"
list_organizations
List the Bitfab organizations available to the signed-in user, marking the current plugin org.
get_database_connection_status
Reports whether the org has a database connected for per-trace replay branching (none, checking, connected, or failed). Used by /bitfab:setup db-snapshot to tell when the branchable copy is ready. 
"Is my database connected for replay branching?"

​
Trace Inspection

list_trace_functions
List all traced functions in your organization. 
"Show me all my traced functions"
search_traces
Search and filter traces with full-text search, date ranges, status filters, regex matching, environment, and label filters. Supports drill-down to narrow results progressively. Filter by datasetId to scope to a specific dataset, or by testRunId to scope to a specific experiment’s replay traces. Filter by hasDbSnapshot to scope to traces that captured (or lack) a database snapshot reference. 
"Find all failed traces for order-processing from the last week"
read_traces
Read one or more traces by ID with the trace environment plus summary (truncated) or full span details (input, output, reasoning, context, errors, per-span duration, tokens, and model). 
"Show me the full details of trace abc-123"
read_trace_labels
Read just the labels for one or more traces by ID: each trace’s pass/fail verdict, its annotation, and whether the label is human-validated. No span content is loaded, so one call accepts up to 100 IDs, ideal for loading a whole dataset’s verdicts at once. 
"Load the labels for every trace in this dataset"
read_span_field
Fetch the complete, untruncated value of a single span field (input, output, reasoning, content, errors, or contexts) when read_traces truncated it. Pass the trace ID, the span’s [ID: ...], and the field name. 
"Get the full input for span def-456 in trace abc-123"

​
Labeling and Datasets

update_agent_labels
Set, skip, or archive the agent’s pass/fail verdict on one or more traces. Supports confidence levels (VeryLow through VeryHigh) and annotations shown to human reviewers. New verdicts start unapproved; once a human approves one in the UI, the label joins the validated dataset. 
"Label these traces as passing with high confidence"
set_human_labels
Write a validated human pass/fail verdict on one or more traces, with an optional annotation. Unlike update_agent_labels, these labels are validated the instant they’re written (no UI approval step). Used by assistant fix before adding the failing trace to a dataset once the fix is verified. 
"Save this failing trace as a validated test"
create_dataset
Create a named dataset for a traced function. Datasets are buckets of traces that humans review and that experiments replay against.
list_datasets
List all datasets for a traced function with IDs, names, descriptions, and trace counts.
add_traces_to_dataset / remove_traces_from_dataset
Add or remove traces from a dataset (idempotent, up to 100 per call). Removing a trace from a dataset does not delete the trace itself.

​
Experiments

list_experiments
List experiments (replay test runs) for a traced function with status, pass/fail totals, and delta (fixed, regressed, still passing, still failing) for each. 
"Show me past experiments for order-processing"
get_experiment_traces
Get individual trace results for an experiment, including each replay trace’s verdict by comparing against the original trace’s label, plus token usage (input, output, cached, total) for the replay and the paired original. 
"Show me the trace-level results for experiment abc-123"
get_replay_status
Read the current replay test run status, including the mapping from local replay trace IDs to server trace IDs while a replay is still running.

​
Templates

get_template_reference
Read the agent-facing reference for Bitfab span templates: the Nunjucks engine, render-context schema, registered filters, and common patterns. Call once per session before editing templates.
get_template
Read the rendering template for a span type (llm, agent, function, guardrail, handoff, custom), scoped to a trace function key or org-global.
update_template
Upsert a rendering template for a span type. Controls how span input/output renders in the Bitfab UI.

​
Instrumentation

create_trace_plan
Post a tracing instrumentation plan (call tree with recommended captured nodes) and get a URL the user opens to confirm or adjust selections.
confirm_trace_plan
Confirm a trace plan without the browser, for the inline “Continue” path where the user reviewed the ASCII plan in chat and skipped Studio. Persists it as the confirmed plan for its trace function key so setup view and setup modify can find it later, and returns the final captured set and per-node replay decisions.
get_trace_plan
Read a trace plan by ID (after user confirmation) or by trace function key (to bootstrap a Modify cycle from a prior plan).

​
Slash Commands

CommandDescription
/bitfab:setupFull setup workflow — authenticate, instrument, create replay scripts
/bitfab:setup loginAuth only
/bitfab:assistantBuild a dataset from traces, experiment with code changes, improve pass rates or cut token costs
/bitfab:assistant <key>Iterate on a specific trace function
/bitfab:assistant fix <trace-id>Fix one failing trace: diagnose it, make the focused code change, and replay just that trace; once it passes, add it to a dataset, then choose to inspect the before/after in Studio or re-run the full dataset
/bitfab:logoutRemove saved credentials
/bitfab:statusCheck auth status, plugin version, and available updates
/bitfab:updateUpdate the plugin to the latest version

​
Session Notifications

The plugin runs a hook on every session start and resume that checks:
  • Authentication: If you’re not logged in, it reminds you to run /bitfab:setup
  • Updates: If a new plugin version is available, it tells you how to update (or auto-updates if you’ve enabled it)

​
Example Workflows

​
Instrument a new project

> /bitfab:setup
The agent detects your project language, finds AI workflows, presents options, and instruments your chosen workflows — all interactively.

​
Diagnose and fix a failing function

Ask Claude Code naturally: 
"My order-processing traces are failing. What's going wrong and can you fix it?"
The plugin calls search_traces and read_traces to inspect failing traces, then suggests code fixes directly. For a specific failing trace, run /bitfab:assistant fix <trace-id>. The agent diagnoses the failure, confirms why the original trace is wrong before editing when the trace or conversation does not already make that clear, makes the focused code fix, and replays only that trace first. Once the fix passes, it adds that trace to a dataset with a validated failing label, then branches: inspect the before/after in Studio, re-run the full dataset (in Studio or terminal-only), keep iterating, or stop. If the replay still fails, it offers to keep iterating or save the trace as a failing test instead.

​
Iterate on a trace function

> /bitfab:assistant memory-search
The agent finds failing traces, walks you through labeling them with expected outcomes, diagnoses the failure patterns in your code, then runs experiments — editing prompts or code, replaying against your labeled dataset, and reporting what improved. You stay in control at every decision point.

​
Replay after a code change

After updating a function, run your replay script to test against production data: 
npx tsx scripts/replay.ts extraction --limit 20
Or ask Claude Code to do it for you: it can run the script and interpret the results. While the replay runs, Claude runs it in the background and reports progress to you as it goes: one line per trace as it finishes (a pass/fail mark, the running count, and how long that trace took), with any error reason inline, plus a periodic “still running” heartbeat when a slow trace takes a while so the run never looks stuck, then a summary with the total and average time. Full per-item outputs are written under that replay run’s .bitfab/replays/<run-id>/items/ folder and referenced from .bitfab/replays/<run-id>/events.jsonl.

​
Configuration

​
Credentials

Credentials are stored in .bitfab/credentials.local.json when that project-local file exists, otherwise in ~/.config/bitfab/credentials.json (created by /bitfab:setup login with owner-readable permissions).

​
Environment Variables

VariableDescription
BITFAB_API_KEYOverride the stored API key

​
Troubleshooting

​
Not authenticated

If you see “Not authenticated” on session start:
  1. Run /bitfab:setup login to authenticate via browser
  2. Check that ~/.config/bitfab/credentials.json exists and contains your API key
  3. If using an environment variable, verify BITFAB_API_KEY is set

​
MCP tools not available

If Claude Code can’t access the Bitfab tools:
  1. Run /bitfab:status to check connection status
  2. Try restarting Claude Code — the MCP server initializes on startup
  3. Verify the plugin is installed: check /plugin list

​
Stale session

The plugin automatically detects and recovers from stale MCP sessions. If tools stop working mid-conversation, they’ll reconnect on the next call.

​
Plugin updates

Run /bitfab:status to check for updates, then /bitfab:update to install the latest version. Restart Claude Code after updating.