Amp Plugin - Bitfab

The Bitfab Amp plugin brings the full evaluation workflow into Amp. It registers the Bitfab tools directly with Amp, ships the setup, assistant, and update skills, and bundles the local commands those skills call, so instrumenting, replaying, and improving your AI workflows all happen in Amp.

Installation

Run the CLI from your project directory:

npx bitfab-cli init --editor amp

This clones the plugin into Amp’s plugin directory (~/.config/amp/plugins/bitfab), confirms Amp loaded it, opens your browser to log in, and starts Amp with the bitfab:setup invocation to type. Pass an initial setup request with --prompt (or -p) and the CLI includes it in that invocation:

npx bitfab-cli init --editor amp --prompt "instrument the chat workflow"

For the experimental terminal-native flow, run npx bitfab-cli init --v2 instead. It does not launch Amp: setup choices and tool permissions stay in the CLI while the Claude Agent SDK performs repository analysis and approved edits. Set ANTHROPIC_API_KEY or configure a supported Agent SDK cloud provider first. Use npx bitfab-cli setup --v2 instrument to run one setup mode, or add --diagram to print its declarative state graph without starting the agent.

Manual installation (without the CLI)

Clone the plugin into Amp’s system plugin directory:

git clone --depth 1 https://github.com/Project-White-Rabbit/bitfab-amp-plugin.git ~/.config/amp/plugins/bitfab

Then run plugins: reload from the command palette (Ctrl-O in the Amp CLI, Cmd/Alt-Shift-A in the editor extensions), or restart Amp. Confirm it loaded:

amp plugins list

Ask Amp for bitfab:setup login to sign in.

Use .amp/plugins/bitfab instead of the system directory to install for one project only.

Requirements

Node 18 or newer. The plugin’s local commands (login, Bitfab, replay progress) run under node.
A Bitfab account. The setup skill opens a browser to sign in when it needs to.

What the Plugin Does

Automatic Setup

The bitfab:setup skill runs a multi-phase workflow:

Login: opens your browser for OAuth authentication and saves credentials securely
Explain: walks through opt-out withTrace/trace, withNode/node, supported withSpan fallbacks, and replay, including the five ways replay can change a method’s execution. Everything after this asks you to make per-method decisions, so it comes first
Approach: asks whether the agent should walk you through instrumenting or hand you the docs so you can do it yourself
Instrument + Replay (in parallel, per workflow): reads your codebase, finds all AI workflows (LLM calls, agents, AI-driven decisions), and presents them as a numbered list. You choose which to instrument, or name a file, function, or directory yourself and it reads only that instead of scanning. Either way it adds tracing with minimal diffs and creates a registry module for the replay command shipped with the SDK

You can run individual phases by asking Amp for the skill with a mode:

bitfab:setup explain      # Explain opt-out tracing + replay and list the modes (read-only, no login)
bitfab:setup login        # Auth only
bitfab:setup instrument   # Trace instrumentation only
bitfab:setup inspect      # Diagnose (and offer to fix) your tracing setup
bitfab:setup replay       # Replay registry creation only
bitfab:setup analyze-repo # Scan the repo and report workflows to instrument without prompts

The setup is interactive: it presents 2-5 concrete options per decision point with a recommended choice, so you stay in control throughout. Setup reads the selected workflow and writes instrumentation and its replay registry directly. It asks only for missing choices or a required refactor.

Assistant

The bitfab:assistant skill turns production traces into code improvements, whether the goal is correctness (improving pass rates) or efficiency (cutting token usage and cost). Amp does the mechanical work and collaborates with you on three steps:

Build a dataset from production traces: search for failures, label them with expected outcomes
Experiment against that dataset: make isolated code changes, replay, compare results
Hill climb: repeat until the best change is found, then present results

Run it with an optional trace function key:

bitfab:assistant
bitfab:assistant order-processing

Building the Dataset

Amp does the data wrangling: it searches production traces for failures, reads full inputs and outputs, and identifies edge cases. It then presents edge cases for your judgment: is this a failure (and what should the output be), correct, or irrelevant? This labeled dataset becomes the benchmark for all experiments. The plugin opens a rich UI for navigating and labeling the dataset, then brings you back to Amp so you stay in flow. You can label every trace yourself, or label a few and let the agent classify the rest based on the patterns you’ve established.

Running Experiments

The skill reads your code, diagnoses failure patterns, and categorizes proposed changes:

Code fixes: deterministic bugs, bundled into one experiment as a foundation
Judgment-based fixes: prompt changes, search tuning, output formatting, each gets its own experiment
Infrastructure proposals: larger changes noted for future work, not experimented on

Experiments run one at a time in your working tree. Each one edits the code, runs the SDK’s replay command with your registry module and labeled dataset, and compares new outputs to expected outcomes.

Results

After each round, you see which traces now match expected outcomes, which still diverge, and whether any regressions occurred. The assistant works through the planned experiments in turn without pausing to ask whether to keep going, then wraps up once the plan is complete. The final summary shows pass rate improvement and all files changed, uncommitted in your working tree for review.

Tools

The plugin registers these tools directly with Amp under the names below, so Amp can call them in any conversation. There is no MCP server in the path: the plugin’s tool bridge talks to Bitfab for you. Ask Amp to search your traces and it can, without invoking a skill.

Core

Tool	Description
`get_bitfab_api_key`	Retrieve your API key for SDK initialization and environment variable configuration
`get_api_key_context`	Returns which Bitfab org the plugin reads/writes to. Call before the first plugin write, or when data you wrote isn’t visible in Bitfab
`list_organizations`	List the Bitfab organizations available to the signed-in user, marking the current plugin org
`get_database_connection_status`	Report whether the org has connected a database for per-trace replay branching (`none`, `checking`, `connected`, or `failed`), identify it as direct Neon or a managed Postgres mirror, and include the pinned project name and ID for direct Neon connections

Trace Inspection

Tool	Description
`list_trace_functions`	List all traced functions in your organization
`search_traces`	Search and filter traces with keyword search, date ranges, status filters, regex, environment, label filters, dataset, test run, and db-snapshot scoping; replay traces are excluded by default unless `includeReplays` is set or a `testRunId` is provided
`get_traces`	Read one or more traces by ID with the trace environment plus summary (truncated) or full span details (input, output, reasoning, context, errors, per-span duration and tokens)
`get_trace_labels`	Read just the labels for up to 100 traces by ID in one call, no span content: each trace’s verdict, annotation, and approved flag, plus one line per scored assertion carrying that assertion’s own verdict, annotation, confidence, and author
`get_grader_labels`	Read the individual verdicts each automated grader recorded (reason, failure diagnostic, confidence, human or grader run), by trace IDs, by grader ID, or both
`get_span_field`	Fetch the complete, untruncated value of a single span field (input, output, reasoning, content, errors, or contexts) when `get_traces` truncated it

Labeling and Datasets

Tool	Description
`save_agent_labels`	Set, skip, or archive agent pass/fail verdicts on traces. Supports confidence levels and annotations for human review. Pass an `assertionId` to score one assertion, omit it for the whole-trace verdict
`save_human_labels`	Write validated human pass/fail verdicts (with annotations) on traces. Validated immediately with no UI approval step; used by `assistant fix` before adding a trace to a dataset
`save_trace_assertions`	Record what a trace SHOULD do when replayed: one assertion per statement, with optional pass/fail criteria, target, and people-only human note. Agents can edit the note but must never use it as assessment evidence
`get_trace_assertions`	Read the assertions and people-only human notes on up to 100 traces by ID, no span content. Notes are never assessment evidence; a replay with no assertions of its own reads its original’s
`archive_trace_assertions`	Retire assertions on a trace so later replays stop checking them. Non-destructive and all-or-nothing: one unknown, already-archived, or wrong-trace id fails the call and archives nothing
`save_dataset`	Create a labeled dataset for a traced function (named buckets of traces for review and replay)
`list_datasets`	List all datasets for a traced function with trace counts and assigned graders
`add_traces_to_dataset`	Add traces to a dataset (idempotent, 1-100 per call)
`remove_traces_from_dataset`	Remove traces from a dataset without deleting the traces themselves
`add_graders_to_dataset`	Assign graders to a dataset (idempotent, organization- and function-scoped, 1-100 per call)
`remove_graders_from_dataset`	Remove grader assignments from a dataset without deleting the graders
`save_grader`	Create or edit an automated grader (LLM-as-judge pass/fail check) for a traced function: upsert by id or name, rename, clear pass/fail criteria, archive/restore, select the judge model
`list_graders`	List automated graders for a traced function with optional name search and cursor pagination (20 by default, 50 maximum; archived hidden by default)

Experiments

Tool	Description
`save_experiment_group`	Create a group from selected experiments with optional name and notes, or update an existing group’s metadata without changing membership
`list_experiment_groups`	List recent experiment groups with their names, notes, and member experiment ids
`get_experiment_group`	Get one experiment group with its metadata and detailed member experiment summaries
`save_experiment`	Update an existing experiment’s name or notes, move it into an existing experiment group, or remove it from its group
`add_graders_to_experiment_group`	Assign one or more graders directly to every current experiment in a group and queue missing evaluations for completed runs; later experiments do not inherit them automatically
`remove_graders_from_experiment_group`	Detach one or more graders from every current experiment in a group; unassigned ids are ignored and the graders themselves are not deleted
`list_experiments`	List experiments (replay test runs) for a traced function with name, notes, status, totals, and delta (fixed/regressed/still passing/still failing)
`get_experiment`	Get a single experiment by id with name, notes, status, totals, delta, experiment group, and grader results, including a per-grader passing/failing breakdown
`list_experiment_traces`	Get individual trace results for an experiment with each replay trace’s verdict compared to the original, plus token usage (input, output, cached, total) for the replay and paired original
`add_graders_to_experiment` / `remove_graders_from_experiment`	Attach or detach active graders on an experiment so they run against its replay traces (idempotent, organization- and function-scoped, 1-100 per call); the effective set at completion is the union with the dataset’s runnable graders; detaching a dataset-overlapping grader from an in-progress run is re-added at completion (still runs) but from a completed run permanently drops it from the finalized snapshot, and attaching to a completed experiment grades existing traces only on the next completion/replay
`rerun_graders_on_dataset` / `rerun_graders_on_experiment`	Re-score a dataset’s traces or an experiment’s completed replays with their attached graders, overwriting the previous verdicts (defaults to every attached grader; a subset may be named, and ids that are not attached are rejected); waits up to 90s and reports traces graded, and a repeat call reports the running job instead of starting a second one
`get_replay_status`	Read a replay test run’s current status and local replay trace ID to server trace ID mapping while replay is still running

Templates

Tool	Description
`get_template_reference`	Read the Nunjucks template engine reference, render-context schema, and available filters. Call once per session before editing templates
`get_template`	Read the rendering template for a span type, scoped to a trace function key or org-global
`save_template`	Upsert a rendering template for a span type. Controls how span data renders in the Bitfab UI

Instrumentation

Tool	Description
`get_sim_plan`	Read a trace function’s sim plan (alpha): its span nodes from recent traces with type, call count, average payload, estimated monthly cost, share of traces, and whether content is captured. A node is its span name inside the trace function key
`save_sim_plan`	Turn content capture off or on per span node (alpha). The span itself, its name, type, timing, and errors are always recorded; content off strips only inputs and outputs. A node recorded by a framework integration, a node that is the root of its traces, a node imported from another platform, or a node named by Bitfab rather than the SDK keeps its content, and turning it off is refused with the reason. Turning a node off also turns content off for every node beneath it in the sim plan; a node beneath it that keeps its content stays on, and a node named in the same call keeps the value given to it. Turning a node back on leaves the nodes beneath it as they are. Returns the updated sim plan

Skills

Amp lists every skill for the model by name and description and loads one when it becomes relevant. Ask for one by name, or invoke it from the command palette with skill: invoke.

Skill	Description
`bitfab:setup`	Full setup workflow: authenticate, instrument, create replay registries
`bitfab:setup login`	Auth only
`bitfab:setup analyze-repo`	Non-interactively scan the repo, pick traceable workflows, and report recommended workflows
`bitfab:assistant`	Build a dataset from traces, experiment with code changes, improve pass rates or cut token costs
`bitfab:assistant <key>`	Iterate on a specific trace function
`bitfab:assistant fix <trace-id>`	Fix one failing trace: diagnose it, make the focused code change, and replay just that trace; once it passes, add it to a dataset, then choose to inspect the before/after in Bitfab or re-run the full dataset
`bitfab:update`	Update the plugin and the Bitfab SDK to the latest versions

Example Workflows

Instrument a new project

bitfab:setup

The agent detects your project language, finds AI workflows, presents options, and instruments your chosen workflows, all interactively.

Analyze the repo headlessly

npx bitfab-cli analyze-repo --editor amp --limit 3

This runs Amp headless (amp -x) with the setup skill in analyze-repo mode. It ranks the AI workflows in the repository, reports source recommendations for each, and prints a report. It edits no code. bitfab:setup instrument picks those drafts up later and reconciles each against current code.

Diagnose and fix a failing function

Ask Amp naturally:

"My order-processing traces are failing. What's going wrong and can you fix it?"

The plugin calls search_traces and get_traces to inspect failing traces and suggests code fixes directly. For a specific failing trace, run bitfab:assistant fix <trace-id>. The agent diagnoses the failure, confirms why the original trace is wrong before editing when the trace or conversation does not already make that clear, makes the focused code fix, and replays only that trace first. Once the fix passes, it adds that trace to a dataset with a validated failing label, then branches: inspect the before/after in Bitfab, re-run the full dataset (in Bitfab or terminal-only), keep iterating, or stop. If that full-dataset re-run reveals real regressions (previously-passing traces the fix broke), it reports them and keeps the target trace saved as a red test to revisit. If the replay still fails, it offers to keep iterating or save the trace as a failing test instead.

Iterate on a trace function

bitfab:assistant memory-search

The agent finds failing traces, walks you through labeling them with expected outcomes, diagnoses the failure patterns in your code, then runs experiments: editing prompts or code, replaying against your labeled dataset, and reporting what improved. You stay in control at every decision point.

Replay after a code change

After updating a function, pass your registry module to the replay command installed by the SDK:

pnpm exec bitfab-replay --registry scripts/replayRegistry.ts extraction --limit 20

Or ask Amp to do it for you. While the replay runs, Amp runs it as a background shell command and reports progress to you as it goes: one line per trace as it finishes (a pass/fail mark, the running count, and how long that trace took), with any error reason inline, plus a periodic “still running” heartbeat when a slow trace takes a while so the run never looks stuck, then a summary with the total and average time. Full per-item outputs are written under that replay run’s .bitfab/replays/<run-id>/items/ folder and referenced from .bitfab/replays/<run-id>/events.jsonl. If the replay command succeeds but its local result cannot be captured, the plugin reports the outcome as unverified instead of failed, then checks the server test run to recover the final result.

Configuration

Credentials

Credentials are stored in .bitfab/credentials.local.json when that project-local file exists, otherwise in ~/.config/bitfab/credentials.json (created by bitfab:setup login with owner-readable permissions).

Environment Variables

Variable	Description
`BITFAB_API_KEY`	Override the stored API key

Updating

npx bitfab-cli update --editor amp plugin

This pulls the latest plugin into ~/.config/amp/plugins/bitfab. Run plugins: reload in Amp afterward. npx bitfab-cli update --editor amp sdk starts Amp with bitfab:update sdk to type, which walks each workspace’s SDK upgrade in Amp. Asking Amp for bitfab:update does both.

What differs from the other hosts

Capability	Amp	Claude / Cursor / Codex
Setup, assistant, update skills	Yes	Yes
Bitfab tools available to the host agent	Yes (registered by the plugin, bare names)	Yes (MCP)
Template preview links	Yes	Yes
Session log capture	Not yet	Yes
Auth and update banner at session start	Not yet	Yes (Claude, Cursor)
`--skip-permissions` from the CLI	No flag. Set `amp.dangerouslyAllowAll` in Amp settings	Yes

Session log capture and the session-start banner ship in a follow-up release through Amp’s agent.start and agent.end events.

Troubleshooting

Not authenticated

If a skill reports “Not authenticated”:

Ask Amp for bitfab:setup login to authenticate via browser
Check that ~/.config/bitfab/credentials.json exists and contains your API key
If using an environment variable, verify BITFAB_API_KEY is set

Bitfab tools not available

If Amp cannot see tools like search_traces, the plugin did not load:

Run amp plugins list. The Bitfab entry shows whether it loaded and, if not, why
Run plugins: reload from the command palette, or restart Amp
Check that the plugin directory is ~/.config/amp/plugins/bitfab (or .amp/plugins/bitfab in the project) and contains index.js
Make sure node is on your PATH. The skills’ local commands run under node even though the plugin itself runs under Amp’s runtime

Plugin updates

Ask Amp for bitfab:update, or run npx bitfab-cli update --editor amp plugin and then plugins: reload.

​Installation

​Requirements

​What the Plugin Does

​Automatic Setup

​Assistant

​Building the Dataset

​Running Experiments

​Results

​Tools

​Core

​Trace Inspection

​Labeling and Datasets

​Experiments

​Templates

​Instrumentation

​Skills

​Example Workflows

​Instrument a new project

​Analyze the repo headlessly

​Diagnose and fix a failing function

​Iterate on a trace function

​Replay after a code change

​Configuration

​Credentials

​Environment Variables

​Updating

​What differs from the other hosts

​Troubleshooting

​Not authenticated

​Bitfab tools not available

​Plugin updates

Installation

Requirements

What the Plugin Does

Automatic Setup

Assistant

Building the Dataset

Running Experiments

Results

Tools

Core

Trace Inspection

Labeling and Datasets

Experiments

Templates

Instrumentation

Skills

Example Workflows

Instrument a new project

Analyze the repo headlessly

Diagnose and fix a failing function

Iterate on a trace function

Replay after a code change

Configuration

Credentials

Environment Variables

Updating

What differs from the other hosts

Troubleshooting

Not authenticated

Bitfab tools not available

Plugin updates