> ## Documentation Index > Fetch the complete documentation index at: https://docs.bitfab.ai/llms.txt > Use this file to discover all available pages before exploring further. # Claude Plugin > Install the Bitfab plugin for Claude Code to get tracing, diagnostics, replay, and improvement tools directly in your editor The Bitfab Claude Code plugin brings the full evaluation workflow into Claude Code. It provides MCP tools for trace inspection, datasets, labeling, experiments, and setup, slash commands for authentication, and automatic notifications -- so you never have to leave your editor. ## Installation Run the CLI from your project directory: ```bash theme={null} npx bitfab-cli init --editor claude ``` This installs the Bitfab plugin, opens your browser to log in, and launches `/bitfab:setup`. Pass an initial setup request with `--prompt` (or `-p`) to send it straight to the agent: ```bash theme={null} npx bitfab-cli init --editor claude --prompt "instrument the chat workflow" ``` The CLI checks that Claude Code is signed in before it launches the Bitfab agent. If needed, run `claude auth login` first. Add the Bitfab plugin marketplace and install: ``` > /plugin marketplace add Project-White-Rabbit/bitfab-claude-plugin > /plugin install bitfab > /exit ``` Then restart Claude Code and run the setup command to authenticate and instrument your codebase: ```bash theme={null} claude --continue ``` ``` > /bitfab:setup ``` ## What the Plugin Does ### Automatic Setup The `/bitfab:setup` command runs a multi-phase workflow: 1. **Login** -- Opens your browser for OAuth authentication, saves credentials securely 2. **Instrument + Replay** (in parallel, per workflow) -- Reads your codebase, finds all AI workflows (LLM calls, agents, AI-driven decisions), and presents them as a numbered list. You choose which to instrument -- it adds tracing with minimal diffs, and at the same time generates a replay script so you can regression-test your trace functions against production data with one command You can run individual phases: ``` > /bitfab:setup explain # Explain what Bitfab is (read-only, no login) > /bitfab:setup login # Auth only > /bitfab:setup instrument # Trace instrumentation only > /bitfab:setup inspect # Diagnose (and offer to fix) your tracing setup > /bitfab:setup replay # Replay script creation only > /bitfab:setup analyze-repo # Scan the repo and upload draft trace plans without prompts ``` The setup is interactive -- it presents 2-5 concrete options per decision point with a recommended choice, so you stay in control throughout. ### Assistant The `/bitfab:assistant` command turns production traces into code improvements, whether the goal is correctness (improving pass rates) or efficiency (cutting token usage and cost). Your agent will do the mechanical work and collaborate with you on three steps: 1. **Build a dataset** from production traces -- search for failures, label them with expected outcomes 2. **Experiment** against that dataset -- make isolated code changes, replay, compare results 3. **Hill climb** -- repeat until the best change is found, then present results Run it with an optional trace function key: ``` > /bitfab:assistant > /bitfab:assistant order-processing ``` #### Building the Dataset Your coding agent does the data wrangling -- it searches production traces for failures, reads full inputs and outputs, and identifies edge cases. It then presents edge cases for your judgment: is this a failure (and what should the output be), correct, or irrelevant? This labeled dataset becomes the benchmark for all experiments. The plugin opens a rich UI for navigating and labeling the dataset, then brings you back to your coding agent so you stay in flow. You can label every trace yourself, or label a few and let the agent classify the rest based on the patterns you've established. #### Running Experiments The command reads your code, diagnoses failure patterns, and categorizes proposed changes: * **Code fixes** -- deterministic bugs, bundled into one experiment as a foundation * **Judgment-based fixes** -- prompt changes, search tuning, output formatting -- each gets its own experiment * **Infrastructure proposals** -- larger changes noted for future work, not experimented on Independent experiments run in parallel -- each in its own isolated subagent on a separate git worktree. Each subagent edits the code, runs the replay script against your labeled dataset, and compares new outputs to expected outcomes. #### Results After each round, you see which traces now match expected outcomes, which still diverge, and whether any regressions occurred. The assistant works through the planned experiments in turn without pausing to ask whether to keep going, then wraps up once the plan is complete. The final summary shows pass rate improvement and all files changed, uncommitted in your working tree for review. ### MCP Tools The plugin registers MCP tools that Claude Code can call during conversations. These let you inspect traces, manage datasets and labels, run experiments, and improve your code without leaving the editor. #### Core ##### `get_bitfab_api_key` Retrieve your API key for SDK initialization and environment variable configuration. ##### `get_api_key_context` Returns which Bitfab org the plugin reads/writes to (it can differ from the project's `BITFAB_API_KEY` and from the org open in Studio). Call before the first plugin write of a session, or when data you wrote isn't visible in Studio. ``` "Which org is this plugin connected to?" ``` ##### `list_organizations` List the Bitfab organizations available to the signed-in user, marking the current plugin org. ##### `get_database_connection_status` Reports whether the org has a database connected for per-trace replay branching (`none`, `checking`, `connected`, or `failed`). Used by `/bitfab:setup db-snapshot` to tell when the branchable copy is ready. ``` "Is my database connected for replay branching?" ``` #### Trace Inspection ##### `list_trace_functions` List all traced functions in your organization. ``` "Show me all my traced functions" ``` ##### `search_traces` Search and filter traces with full-text search, date ranges, status filters, regex matching, environment, and label filters. Supports drill-down to narrow results progressively. Filter by `datasetId` to scope to a specific dataset, or by `testRunId` to scope to a specific experiment's replay traces. Filter by `hasDbSnapshot` to scope to traces that captured (or lack) a database snapshot reference. ``` "Find all failed traces for order-processing from the last week" ``` ##### `read_traces` Read one or more traces by ID with the trace environment plus summary (truncated) or full span details (input, output, reasoning, context, errors, per-span duration, tokens, and model). ``` "Show me the full details of trace abc-123" ``` ##### `read_trace_labels` Read just the labels for one or more traces by ID: each trace's pass/fail verdict, its annotation, and whether the label is human-validated. No span content is loaded, so one call accepts up to 100 IDs, ideal for loading a whole dataset's verdicts at once. ``` "Load the labels for every trace in this dataset" ``` ##### `read_span_field` Fetch the complete, untruncated value of a single span field (input, output, reasoning, content, errors, or contexts) when `read_traces` truncated it. Pass the trace ID, the span's `[ID: ...]`, and the field name. ``` "Get the full input for span def-456 in trace abc-123" ``` #### Labeling and Datasets ##### `update_agent_labels` Set, skip, or archive the agent's pass/fail verdict on one or more traces. Supports confidence levels (`VeryLow` through `VeryHigh`) and annotations shown to human reviewers. New verdicts start unapproved; once a human approves one in the UI, the label joins the validated dataset. ``` "Label these traces as passing with high confidence" ``` ##### `set_human_labels` Write a validated human pass/fail verdict on one or more traces, with an optional annotation. Unlike `update_agent_labels`, these labels are validated the instant they're written (no UI approval step). Used by `assistant fix` before adding the failing trace to a dataset once the fix is verified. ``` "Save this failing trace as a validated test" ``` ##### `create_dataset` Create a named dataset for a traced function. Datasets are buckets of traces that humans review and that experiments replay against. ##### `list_datasets` List all datasets for a traced function with IDs, names, descriptions, trace counts, and assigned graders. ##### `add_traces_to_dataset` / `remove_traces_from_dataset` Add or remove traces from a dataset (idempotent, up to 100 per call). Removing a trace from a dataset does not delete the trace itself. ##### `add_graders_to_dataset` / `remove_graders_from_dataset` Assign or remove graders from a dataset (idempotent, up to 100 per call). Assignments are restricted to graders in the same organization and trace function as the dataset. Removing an assignment does not delete the grader. ##### `save_grader` / `list_graders` Create or edit an automated grader (an LLM-as-judge pass/fail check) for a traced function, or list the graders defined for one. `save_grader` upserts by grader id or by name, supports renaming, clearing pass/fail criteria, and archive/restore. `list_graders` returns each grader's name, status, evaluation focus, and criteria; supports case-insensitive name search; and paginates newest-first with 20 results by default, up to 50 per page. Pass the returned cursor to retrieve the next page. Archived graders are hidden by default. #### Experiments ##### `create_experiment_group` Assign or reuse one shared group UUID across selected experiments (test runs). ##### `add_grader_to_experiment_group` Assign one grader directly to every experiment currently in a group. Completed experiments queue any missing evaluations immediately; pending experiments use the grader when they complete. Because assignments live on experiments rather than the group, experiments added later do not inherit the grader automatically. ##### `list_experiments` List experiments (replay test runs) for a traced function with status, pass/fail totals, and delta (fixed, regressed, still passing, still failing) for each. ``` "Show me past experiments for order-processing" ``` ##### `get_experiment_traces` Get individual trace results for an experiment, including each replay trace's verdict by comparing against the original trace's label, plus token usage (input, output, cached, total) for the replay and the paired original. ``` "Show me the trace-level results for experiment abc-123" ``` ##### `add_graders_to_experiment` / `remove_graders_from_experiment` Attach or detach graders directly on an experiment (test run) so they run against its replay traces (idempotent, up to 100 per call). Attachments are restricted to active graders in the same organization and trace function as the experiment. The effective grader set at completion is the union of these direct attachments and the dataset's current runnable graders. Detaching does not delete the grader. Its effect depends on run state: for an in-progress experiment a detached grader that is also a dataset grader is re-added from the dataset at completion (so it still runs), while for an already-completed experiment the grader set is a finalized snapshot with no dataset re-union, so detaching permanently drops that grader from the run's results. Attaching to an already-completed experiment persists the assignment but does not immediately grade its existing replay traces; those traces are graded only the next time the experiment runs completion again (e.g. an SDK replay retry) or a fresh replay runs. ``` "Attach the no-fabricated-order-ids grader to experiment abc-123" ``` ##### `get_replay_status` Read the current replay test run status, including the mapping from local replay trace IDs to server trace IDs while a replay is still running. #### Templates ##### `get_template_reference` Read the agent-facing reference for Bitfab span templates: the Nunjucks engine, render-context schema, registered filters, and common patterns. Call once per session before editing templates. ##### `get_template` Read the rendering template for a span type (llm, agent, function, guardrail, handoff, custom), scoped to a trace function key or org-global. ##### `update_template` Upsert a rendering template for a span type. Controls how span input/output renders in the Bitfab UI. #### Instrumentation ##### `create_trace_plan` Post a tracing instrumentation plan (call tree with recommended captured nodes) and get a URL the user opens to confirm or adjust selections. ##### `confirm_trace_plan` Confirm a trace plan without the browser, for the inline "Continue" path where the user reviewed the ASCII plan in chat and skipped Studio. Persists it as the confirmed plan for its trace function key so `setup view` and `setup modify` can find it later, and returns the final captured set and per-node replay decisions. ##### `get_trace_plan` Read a trace plan by ID (after user confirmation) or by trace function key (to bootstrap a Modify cycle from a prior plan). ### Slash Commands | Command | Description | | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `/bitfab:setup` | Full setup workflow -- authenticate, instrument, create replay scripts | | `/bitfab:setup login` | Auth only | | `/bitfab:assistant` | Build a dataset from traces, experiment with code changes, improve pass rates or cut token costs | | `/bitfab:assistant ` | Iterate on a specific trace function | | `/bitfab:assistant fix ` | Fix one failing trace: diagnose it, make the focused code change, and replay just that trace; once it passes, add it to a dataset, then choose to inspect the before/after in Studio or re-run the full dataset | | `/bitfab:logout` | Remove saved credentials | | `/bitfab:status` | Check auth status, plugin version, and available updates | | `/bitfab:update` | Update the plugin to the latest version | ### Session Notifications The plugin runs a hook on every session start and resume that checks: * **Authentication**: If you're not logged in, it reminds you to run `/bitfab:setup` * **Updates**: If a new plugin version is available, it tells you how to update (or auto-updates if you've enabled it) ## Example Workflows ### Instrument a new project ``` > /bitfab:setup ``` The agent detects your project language, finds AI workflows, presents options, and instruments your chosen workflows -- all interactively. ### Diagnose and fix a failing function Ask Claude Code naturally: ``` "My order-processing traces are failing. What's going wrong and can you fix it?" ``` The plugin calls `search_traces` and `read_traces` to inspect failing traces, then suggests code fixes directly. For a specific failing trace, run `/bitfab:assistant fix `. The agent diagnoses the failure, confirms why the original trace is wrong before editing when the trace or conversation does not already make that clear, makes the focused code fix, and replays only that trace first. Once the fix passes, it adds that trace to a dataset with a validated failing label, then branches: inspect the before/after in Studio, re-run the full dataset (in Studio or terminal-only), keep iterating, or stop. If that full-dataset re-run reveals real regressions (previously-passing traces the fix broke), it reports them and keeps the target trace saved as a red test to revisit. If the replay still fails, it offers to keep iterating or save the trace as a failing test instead. ### Iterate on a trace function ``` > /bitfab:assistant memory-search ``` The agent finds failing traces, walks you through labeling them with expected outcomes, diagnoses the failure patterns in your code, then runs experiments -- editing prompts or code, replaying against your labeled dataset, and reporting what improved. You stay in control at every decision point. ### Replay after a code change After updating a function, run your replay script to test against production data: ```bash theme={null} npx tsx scripts/replay.ts extraction --limit 20 ``` Or ask Claude Code to do it for you: it can run the script and interpret the results. While the replay runs, Claude runs it in the background and reports progress to you as it goes: one line per trace as it finishes (a pass/fail mark, the running count, and how long that trace took), with any error reason inline, plus a periodic "still running" heartbeat when a slow trace takes a while so the run never looks stuck, then a summary with the total and average time. Full per-item outputs are written under that replay run's `.bitfab/replays//items/` folder and referenced from `.bitfab/replays//events.jsonl`. ## Configuration ### Credentials Credentials are stored in `.bitfab/credentials.local.json` when that project-local file exists, otherwise in `~/.config/bitfab/credentials.json` (created by `/bitfab:setup login` with owner-readable permissions). ### Environment Variables | Variable | Description | | ---------------- | --------------------------- | | `BITFAB_API_KEY` | Override the stored API key | ## Troubleshooting ### Not authenticated If you see "Not authenticated" on session start: 1. Run `/bitfab:setup login` to authenticate via browser 2. Check that `~/.config/bitfab/credentials.json` exists and contains your API key 3. If using an environment variable, verify `BITFAB_API_KEY` is set ### MCP tools not available If Claude Code can't access the Bitfab tools: 1. Run `/bitfab:status` to check connection status 2. Try restarting Claude Code -- the MCP server initializes on startup 3. Verify the plugin is installed: check `/plugin list` ### Stale session The plugin automatically detects and recovers from stale MCP sessions. If tools stop working mid-conversation, they'll reconnect on the next call. ### Plugin updates Run `/bitfab:status` to check for updates, then `/bitfab:update` to install the latest version. Restart Claude Code after updating.