Claude Plugin - Bitfab

The Bitfab Claude Code plugin brings the full evaluation workflow into Claude Code. It provides MCP tools for trace inspection, datasets, labeling, experiments, and setup, slash commands for authentication, and automatic notifications — so you never have to leave your editor.

Installation

Run the CLI from your project directory:

npx bitfab-cli init --editor claude

This installs the Bitfab plugin, opens your browser to log in, and launches /bitfab:setup. Pass an initial setup request with --prompt (or -p) to send it straight to the agent:

npx bitfab-cli init --editor claude --prompt "instrument the chat workflow"

The CLI checks that Claude Code is signed in before it launches the Bitfab agent. If needed, run claude auth login first.

Manual installation (without the CLI)

Add the Bitfab plugin marketplace and install:

> /plugin marketplace add Project-White-Rabbit/bitfab-claude-plugin
> /plugin install bitfab
> /exit

Then restart Claude Code and run the setup command to authenticate and instrument your codebase:

claude --continue

> /bitfab:setup

What the Plugin Does

Automatic Setup

The /bitfab:setup command runs a multi-phase workflow:

Login — Opens your browser for OAuth authentication, saves credentials securely
Instrument + Replay (in parallel, per workflow) — Reads your codebase, finds all AI workflows (LLM calls, agents, AI-driven decisions), and presents them as a numbered list. You choose which to instrument — it adds tracing with minimal diffs, and at the same time generates a replay script so you can regression-test your trace functions against production data with one command

You can run individual phases:

> /bitfab:setup explain      # Explain what Bitfab is (read-only, no login)
> /bitfab:setup login        # Auth only
> /bitfab:setup instrument   # Trace instrumentation only
> /bitfab:setup inspect      # Diagnose (and offer to fix) your tracing setup
> /bitfab:setup replay       # Replay script creation only
> /bitfab:setup analyze-repo # Scan the repo and upload draft trace plans without prompts

The setup is interactive — it presents 2-5 concrete options per decision point with a recommended choice, so you stay in control throughout.

Assistant

The /bitfab:assistant command turns production traces into code improvements, whether the goal is correctness (improving pass rates) or efficiency (cutting token usage and cost). Your agent will do the mechanical work and collaborate with you on three steps:

Build a dataset from production traces — search for failures, label them with expected outcomes
Experiment against that dataset — make isolated code changes, replay, compare results
Hill climb — repeat until the best change is found, then present results

Run it with an optional trace function key:

> /bitfab:assistant
> /bitfab:assistant order-processing

Building the Dataset

Your coding agent does the data wrangling — it searches production traces for failures, reads full inputs and outputs, and identifies edge cases. It then presents edge cases for your judgment: is this a failure (and what should the output be), correct, or irrelevant? This labeled dataset becomes the benchmark for all experiments. The plugin opens a rich UI for navigating and labeling the dataset, then brings you back to your coding agent so you stay in flow. You can label every trace yourself, or label a few and let the agent classify the rest based on the patterns you’ve established.

Running Experiments

The command reads your code, diagnoses failure patterns, and categorizes proposed changes:

Code fixes — deterministic bugs, bundled into one experiment as a foundation
Judgment-based fixes — prompt changes, search tuning, output formatting — each gets its own experiment
Infrastructure proposals — larger changes noted for future work, not experimented on

Independent experiments run in parallel — each in its own isolated subagent on a separate git worktree. Each subagent edits the code, runs the replay script against your labeled dataset, and compares new outputs to expected outcomes.

Results

After each round, you see which traces now match expected outcomes, which still diverge, and whether any regressions occurred. The assistant works through the planned experiments in turn without pausing to ask whether to keep going, then wraps up once the plan is complete. The final summary shows pass rate improvement and all files changed, uncommitted in your working tree for review.

MCP Tools

The plugin registers MCP tools that Claude Code can call during conversations. These let you inspect traces, manage datasets and labels, run experiments, and improve your code without leaving the editor.

Core

`get_bitfab_api_key`

Retrieve your API key for SDK initialization and environment variable configuration.

`get_api_key_context`

Returns which Bitfab org the plugin reads/writes to (it can differ from the project’s BITFAB_API_KEY and from the org open in Studio). Call before the first plugin write of a session, or when data you wrote isn’t visible in Studio.

"Which org is this plugin connected to?"

`list_organizations`

List the Bitfab organizations available to the signed-in user, marking the current plugin org.

`get_database_connection_status`

Reports whether the org has a database connected for per-trace replay branching (none, checking, connected, or failed). Used by /bitfab:setup db-snapshot to tell when the branchable copy is ready.

"Is my database connected for replay branching?"

Trace Inspection

`list_trace_functions`

List all traced functions in your organization.

"Show me all my traced functions"

`search_traces`

Search and filter traces with full-text search, date ranges, status filters, regex matching, environment, and label filters. Supports drill-down to narrow results progressively. Filter by datasetId to scope to a specific dataset, or by testRunId to scope to a specific experiment’s replay traces. Filter by hasDbSnapshot to scope to traces that captured (or lack) a database snapshot reference.

"Find all failed traces for order-processing from the last week"

`read_traces`

Read one or more traces by ID with the trace environment plus summary (truncated) or full span details (input, output, reasoning, context, errors, per-span duration, tokens, and model).

"Show me the full details of trace abc-123"

`read_trace_labels`

Read just the labels for one or more traces by ID: each trace’s pass/fail verdict, its annotation, and whether the label is human-validated. No span content is loaded, so one call accepts up to 100 IDs, ideal for loading a whole dataset’s verdicts at once.

"Load the labels for every trace in this dataset"

`read_span_field`

Fetch the complete, untruncated value of a single span field (input, output, reasoning, content, errors, or contexts) when read_traces truncated it. Pass the trace ID, the span’s [ID: ...], and the field name.

"Get the full input for span def-456 in trace abc-123"

Labeling and Datasets

`update_agent_labels`

Set, skip, or archive the agent’s pass/fail verdict on one or more traces. Supports confidence levels (VeryLow through VeryHigh) and annotations shown to human reviewers. New verdicts start unapproved; once a human approves one in the UI, the label joins the validated dataset.

"Label these traces as passing with high confidence"

`set_human_labels`

Write a validated human pass/fail verdict on one or more traces, with an optional annotation. Unlike update_agent_labels, these labels are validated the instant they’re written (no UI approval step). Used by assistant fix before adding the failing trace to a dataset once the fix is verified.

"Save this failing trace as a validated test"

`create_dataset`

Create a named dataset for a traced function. Datasets are buckets of traces that humans review and that experiments replay against.

`list_datasets`

List all datasets for a traced function with IDs, names, descriptions, trace counts, and assigned graders.

`add_traces_to_dataset` / `remove_traces_from_dataset`

Add or remove traces from a dataset (idempotent, up to 100 per call). Removing a trace from a dataset does not delete the trace itself.

`add_graders_to_dataset` / `remove_graders_from_dataset`

Assign or remove graders from a dataset (idempotent, up to 100 per call). Assignments are restricted to graders in the same organization and trace function as the dataset. Removing an assignment does not delete the grader.

`save_grader` / `list_graders`

Create or edit an automated grader (an LLM-as-judge pass/fail check) for a traced function, or list the graders defined for one. save_grader upserts by grader id or by name, supports renaming, clearing pass/fail criteria, and archive/restore. list_graders returns each grader’s name, status, evaluation focus, and criteria; supports case-insensitive name search; and paginates newest-first with 20 results by default, up to 50 per page. Pass the returned cursor to retrieve the next page. Archived graders are hidden by default.

Experiments

`create_experiment_group`

Assign or reuse one shared group UUID across selected experiments (test runs).

`add_grader_to_experiment_group`

Assign one grader directly to every experiment currently in a group. Completed experiments queue any missing evaluations immediately; pending experiments use the grader when they complete. Because assignments live on experiments rather than the group, experiments added later do not inherit the grader automatically.

`list_experiments`

List experiments (replay test runs) for a traced function with status, pass/fail totals, and delta (fixed, regressed, still passing, still failing) for each.

"Show me past experiments for order-processing"

`get_experiment_traces`

Get individual trace results for an experiment, including each replay trace’s verdict by comparing against the original trace’s label, plus token usage (input, output, cached, total) for the replay and the paired original.

"Show me the trace-level results for experiment abc-123"

`add_graders_to_experiment` / `remove_graders_from_experiment`

Attach or detach graders directly on an experiment (test run) so they run against its replay traces (idempotent, up to 100 per call). Attachments are restricted to active graders in the same organization and trace function as the experiment. The effective grader set at completion is the union of these direct attachments and the dataset’s current runnable graders. Detaching does not delete the grader. Its effect depends on run state: for an in-progress experiment a detached grader that is also a dataset grader is re-added from the dataset at completion (so it still runs), while for an already-completed experiment the grader set is a finalized snapshot with no dataset re-union, so detaching permanently drops that grader from the run’s results. Attaching to an already-completed experiment persists the assignment but does not immediately grade its existing replay traces; those traces are graded only the next time the experiment runs completion again (e.g. an SDK replay retry) or a fresh replay runs.

"Attach the no-fabricated-order-ids grader to experiment abc-123"

`get_replay_status`

Read the current replay test run status, including the mapping from local replay trace IDs to server trace IDs while a replay is still running.

Templates

`get_template_reference`

Read the agent-facing reference for Bitfab span templates: the Nunjucks engine, render-context schema, registered filters, and common patterns. Call once per session before editing templates.

`get_template`

Read the rendering template for a span type (llm, agent, function, guardrail, handoff, custom), scoped to a trace function key or org-global.

`update_template`

Upsert a rendering template for a span type. Controls how span input/output renders in the Bitfab UI.

Instrumentation

`create_trace_plan`

Post a tracing instrumentation plan (call tree with recommended captured nodes) and get a URL the user opens to confirm or adjust selections.

`confirm_trace_plan`

Confirm a trace plan without the browser, for the inline “Continue” path where the user reviewed the ASCII plan in chat and skipped Studio. Persists it as the confirmed plan for its trace function key so setup view and setup modify can find it later, and returns the final captured set and per-node replay decisions.

`get_trace_plan`

Read a trace plan by ID (after user confirmation) or by trace function key (to bootstrap a Modify cycle from a prior plan).

Slash Commands

Command	Description
`/bitfab:setup`	Full setup workflow — authenticate, instrument, create replay scripts
`/bitfab:setup login`	Auth only
`/bitfab:assistant`	Build a dataset from traces, experiment with code changes, improve pass rates or cut token costs
`/bitfab:assistant <key>`	Iterate on a specific trace function
`/bitfab:assistant fix <trace-id>`	Fix one failing trace: diagnose it, make the focused code change, and replay just that trace; once it passes, add it to a dataset, then choose to inspect the before/after in Studio or re-run the full dataset
`/bitfab:logout`	Remove saved credentials
`/bitfab:status`	Check auth status, plugin version, and available updates
`/bitfab:update`	Update the plugin to the latest version

Session Notifications

The plugin runs a hook on every session start and resume that checks:

Authentication: If you’re not logged in, it reminds you to run /bitfab:setup
Updates: If a new plugin version is available, it tells you how to update (or auto-updates if you’ve enabled it)

Example Workflows

Instrument a new project

> /bitfab:setup

The agent detects your project language, finds AI workflows, presents options, and instruments your chosen workflows — all interactively.

Diagnose and fix a failing function

Ask Claude Code naturally:

"My order-processing traces are failing. What's going wrong and can you fix it?"

The plugin calls search_traces and read_traces to inspect failing traces, then suggests code fixes directly. For a specific failing trace, run /bitfab:assistant fix <trace-id>. The agent diagnoses the failure, confirms why the original trace is wrong before editing when the trace or conversation does not already make that clear, makes the focused code fix, and replays only that trace first. Once the fix passes, it adds that trace to a dataset with a validated failing label, then branches: inspect the before/after in Studio, re-run the full dataset (in Studio or terminal-only), keep iterating, or stop. If that full-dataset re-run reveals real regressions (previously-passing traces the fix broke), it reports them and keeps the target trace saved as a red test to revisit. If the replay still fails, it offers to keep iterating or save the trace as a failing test instead.

Iterate on a trace function

> /bitfab:assistant memory-search

The agent finds failing traces, walks you through labeling them with expected outcomes, diagnoses the failure patterns in your code, then runs experiments — editing prompts or code, replaying against your labeled dataset, and reporting what improved. You stay in control at every decision point.

Replay after a code change

After updating a function, run your replay script to test against production data:

npx tsx scripts/replay.ts extraction --limit 20

Or ask Claude Code to do it for you: it can run the script and interpret the results. While the replay runs, Claude runs it in the background and reports progress to you as it goes: one line per trace as it finishes (a pass/fail mark, the running count, and how long that trace took), with any error reason inline, plus a periodic “still running” heartbeat when a slow trace takes a while so the run never looks stuck, then a summary with the total and average time. Full per-item outputs are written under that replay run’s .bitfab/replays/<run-id>/items/ folder and referenced from .bitfab/replays/<run-id>/events.jsonl.

Configuration

Credentials

Credentials are stored in .bitfab/credentials.local.json when that project-local file exists, otherwise in ~/.config/bitfab/credentials.json (created by /bitfab:setup login with owner-readable permissions).

Environment Variables

Variable	Description
`BITFAB_API_KEY`	Override the stored API key

Troubleshooting

Not authenticated

If you see “Not authenticated” on session start:

Run /bitfab:setup login to authenticate via browser
Check that ~/.config/bitfab/credentials.json exists and contains your API key
If using an environment variable, verify BITFAB_API_KEY is set

MCP tools not available

If Claude Code can’t access the Bitfab tools:

Run /bitfab:status to check connection status
Try restarting Claude Code — the MCP server initializes on startup
Verify the plugin is installed: check /plugin list

Stale session

The plugin automatically detects and recovers from stale MCP sessions. If tools stop working mid-conversation, they’ll reconnect on the next call.

Plugin updates

Run /bitfab:status to check for updates, then /bitfab:update to install the latest version. Restart Claude Code after updating.

​Installation

​What the Plugin Does

​Automatic Setup

​Assistant

​Building the Dataset

​Running Experiments

​Results

​MCP Tools

​Core

get_bitfab_api_key

get_api_key_context

list_organizations

get_database_connection_status

​Trace Inspection

list_trace_functions

search_traces

read_traces

read_trace_labels

read_span_field

​Labeling and Datasets

update_agent_labels

set_human_labels

create_dataset

list_datasets

add_traces_to_dataset / remove_traces_from_dataset

add_graders_to_dataset / remove_graders_from_dataset

save_grader / list_graders

​Experiments

create_experiment_group

add_grader_to_experiment_group

list_experiments

get_experiment_traces

add_graders_to_experiment / remove_graders_from_experiment

get_replay_status

​Templates

get_template_reference

get_template

update_template

​Instrumentation

create_trace_plan

confirm_trace_plan

get_trace_plan

​Slash Commands

​Session Notifications

​Example Workflows

​Instrument a new project

​Diagnose and fix a failing function

​Iterate on a trace function

​Replay after a code change

​Configuration

​Credentials

​Environment Variables

​Troubleshooting

​Not authenticated

​MCP tools not available

​Stale session

​Plugin updates