Codex Plugin

The Bitfab Codex plugin brings the full evaluation workflow into Codex. It provides MCP tools for trace inspection and diagnostics, skills for authentication and setup, and automatic notifications so you never have to leave your terminal.

Installation

Run the CLI from your project directory:

npx bitfab-cli init --editor codex

This installs the Bitfab plugin, opens your browser to log in, and launches $bitfab:setup. Pass an initial setup request with --prompt (or -p) to send it straight to the agent:

npx bitfab-cli init --editor codex --prompt "instrument the chat workflow"

The CLI checks that Codex is signed in before it launches the Bitfab agent. If needed, run codex login first.

For the experimental terminal-native flow, run npx bitfab-cli init --v2 instead. It does not launch Codex: plan review, setup decisions, and edit approvals stay in the CLI while the Claude Agent SDK performs repository analysis and approved edits. Set ANTHROPIC_API_KEY or configure a supported Agent SDK cloud provider first. Use npx bitfab-cli setup --v2 instrument to run one setup mode, or add --diagram to print its declarative state graph without starting the agent.

Manual installation (without the CLI)

Set your API key as an environment variable:

export BITFAB_API_KEY="YOUR_API_KEY"

Then add this to ~/.codex/config.toml (global) or .codex/config.toml (project-level):

[mcp_servers.bitfab]
url = "https://bitfab.ai/mcp"
bearer_token_env_var = "BITFAB_API_KEY"

Visit the setup page to get your API key and see the full configuration steps.

What the Plugin Does

Automatic Setup

The $bitfab:setup skill runs a multi-phase workflow:

Login — Opens your browser for OAuth authentication, saves credentials securely
Explain — Walks through the two primitives you instrument with, withSpan and replay, including the five ways replay can change a method’s execution. Everything after this asks you to make per-method decisions, so it comes first
Approach — Asks whether the agent should walk you through instrumenting or hand you the docs so you can do it yourself
Instrument + Replay (in parallel, per workflow) — Reads your codebase, finds all AI workflows (LLM calls, agents, AI-driven decisions), and presents them as a numbered list. You choose which to instrument, or name a file, function, or directory yourself and it reads only that instead of scanning. Either way it adds tracing with minimal diffs and creates a registry module for the replay command shipped with the SDK

You can run individual phases:

$bitfab:setup explain      # Explain withSpan + replay and list the modes (read-only, no login)
$bitfab:setup login        # Auth only
$bitfab:setup instrument   # Trace instrumentation only
$bitfab:setup inspect      # Diagnose (and offer to fix) your tracing setup
$bitfab:setup replay       # Replay registry creation only
$bitfab:setup analyze-repo # Scan the repo and upload draft trace plans without prompts

The setup is interactive: it presents 2-5 concrete options per decision point with a recommended choice, so you stay in control throughout.

Assistant

The $bitfab:assistant skill turns production traces into code improvements, whether the goal is correctness (improving pass rates) or efficiency (cutting token usage and cost). Your agent will do the mechanical work and collaborate with you on three steps:

Build a dataset from production traces: search for failures, label them with expected outcomes
Experiment against that dataset: make isolated code changes, replay, compare results
Hill climb: repeat until the best change is found, then present results

Run it with an optional trace function key:

$bitfab:assistant
$bitfab:assistant order-processing

Building the Dataset

Your coding agent does the data wrangling: it searches production traces for failures, reads full inputs and outputs, and identifies edge cases. It then presents edge cases for your judgment: is this a failure (and what should the output be), correct, or irrelevant? This labeled dataset becomes the benchmark for all experiments. The plugin opens a rich UI for navigating and labeling the dataset, then brings you back to your coding agent so you stay in flow. You can label every trace yourself, or label a few and let the agent classify the rest based on the patterns you’ve established.

Running Experiments

The skill reads your code, diagnoses failure patterns, and categorizes proposed changes:

Code fixes: deterministic bugs, bundled into one experiment as a foundation
Judgment-based fixes: prompt changes, search tuning, output formatting, each gets its own experiment
Infrastructure proposals: larger changes noted for future work, not experimented on

Independent experiments run in parallel, each in its own isolated subagent on a separate git worktree. Each subagent edits the code, runs the SDK’s replay command with your registry module and labeled dataset, and compares new outputs to expected outcomes.

Results

After each round, you see which traces now match expected outcomes, which still diverge, and whether any regressions occurred. The assistant works through the planned experiments in turn without pausing to ask whether to keep going, then wraps up once the plan is complete. The final summary shows pass rate improvement and all files changed, uncommitted in your working tree for review.

MCP Tools

The plugin registers MCP tools that Codex can call during conversations. These let you inspect traces, diagnose failures, and improve your code without leaving the terminal.

Core

Tool	Description
`get_bitfab_api_key`	Retrieve your API key for SDK initialization and environment variable configuration
`get_api_key_context`	Returns which Bitfab org the plugin reads/writes to. Call before the first plugin write, or when data you wrote isn’t visible in Studio
`list_organizations`	List the Bitfab organizations available to the signed-in user, marking the current plugin org
`get_database_connection_status`	Report whether the org has connected a database for per-trace replay branching (`none`, `checking`, `connected`, or `failed`), identify it as direct Neon or a managed Postgres mirror, and include the pinned project name and ID for direct Neon connections

Trace Inspection

Tool	Description
`list_trace_functions`	List all traced functions in your organization
`search_traces`	Search and filter traces with keyword search, date ranges, status filters, regex, environment, label filters, dataset, test run, and db-snapshot scoping; replay traces are excluded by default unless `includeReplays` is set or a `testRunId` is provided
`get_traces`	Read one or more traces by ID with the trace environment plus summary (truncated) or full span details (input, output, reasoning, context, errors, per-span duration and tokens)
`get_trace_labels`	Read just the labels for up to 100 traces by ID in one call, no span content: each trace’s verdict, annotation, and approved flag, plus one line per scored assertion carrying that assertion’s own verdict, annotation, confidence, and author
`get_grader_labels`	Read the individual verdicts each automated grader recorded (reason, failure diagnostic, confidence, human or grader run), by trace IDs, by grader ID, or both
`get_span_field`	Fetch the complete, untruncated value of a single span field (input, output, reasoning, content, errors, or contexts) when `get_traces` truncated it

Labeling and Datasets

Tool	Description
`save_agent_labels`	Set, skip, or archive agent pass/fail verdicts on traces. Supports confidence levels and annotations for human review. Pass an `assertionId` to score one assertion, omit it for the whole-trace verdict
`save_human_labels`	Write validated human pass/fail verdicts (with annotations) on traces. Validated immediately with no UI approval step; used by `assistant fix` before adding a trace to a dataset
`save_trace_assertions`	Record what a trace SHOULD do when replayed: one assertion per statement, with optional pass/fail criteria and a target scoping the check to the output or one span. Distinct from a verdict, which judges a run that already happened
`get_trace_assertions`	Read the assertions on up to 100 traces by ID, no span content. Call it before judging a replay so the verdict is measured against what the user asked for; a replay with none of its own reads its original’s
`archive_trace_assertions`	Retire assertions on a trace so later replays stop checking them. Non-destructive and all-or-nothing: one unknown, already-archived, or wrong-trace id fails the call and archives nothing
`save_dataset`	Create a labeled dataset for a traced function (named buckets of traces for review and replay)
`list_datasets`	List all datasets for a traced function with trace counts and assigned graders
`add_traces_to_dataset`	Add traces to a dataset (idempotent, 1-100 per call)
`remove_traces_from_dataset`	Remove traces from a dataset without deleting the traces themselves
`add_graders_to_dataset`	Assign graders to a dataset (idempotent, organization- and function-scoped, 1-100 per call)
`remove_graders_from_dataset`	Remove grader assignments from a dataset without deleting the graders
`save_grader`	Create or edit an automated grader (LLM-as-judge pass/fail check) for a traced function: upsert by id or name, rename, clear pass/fail criteria, archive/restore, select the judge model
`list_graders`	List automated graders for a traced function with optional name search and cursor pagination (20 by default, 50 maximum; archived hidden by default)

Experiments

Tool	Description
`save_experiment_group`	Create a group from selected experiments with optional name and notes, or update an existing group’s metadata without changing membership
`list_experiment_groups`	List recent experiment groups with their names, notes, and member experiment ids
`get_experiment_group`	Get one experiment group with its metadata and detailed member experiment summaries
`save_experiment`	Update an existing experiment’s name or notes, move it into an existing experiment group, or remove it from its group
`add_graders_to_experiment_group`	Assign one or more graders directly to every current experiment in a group and queue missing evaluations for completed runs; later experiments do not inherit them automatically
`remove_graders_from_experiment_group`	Detach one or more graders from every current experiment in a group; unassigned ids are ignored and the graders themselves are not deleted
`list_experiments`	List experiments (replay test runs) for a traced function with name, notes, status, totals, and delta (fixed/regressed/still passing/still failing)
`get_experiment`	Get a single experiment by id with name, notes, status, totals, delta, experiment group, and grader results, including a per-grader passing/failing breakdown
`list_experiment_traces`	Get individual trace results for an experiment with each replay trace’s verdict compared to the original, plus token usage (input, output, cached, total) for the replay and paired original
`add_graders_to_experiment` / `remove_graders_from_experiment`	Attach or detach active graders on an experiment so they run against its replay traces (idempotent, organization- and function-scoped, 1-100 per call); the effective set at completion is the union with the dataset’s runnable graders; detaching a dataset-overlapping grader from an in-progress run is re-added at completion (still runs) but from a completed run permanently drops it from the finalized snapshot, and attaching to a completed experiment grades existing traces only on the next completion/replay
`rerun_graders_on_dataset` / `rerun_graders_on_experiment`	Re-score a dataset’s traces or an experiment’s completed replays with their attached graders, overwriting the previous verdicts (defaults to every attached grader; a subset may be named, and ids that are not attached are rejected); waits up to 90s and reports traces graded, and a repeat call reports the running job instead of starting a second one
`get_replay_status`	Read a replay test run’s current status and local replay trace ID to server trace ID mapping while replay is still running

Templates

Tool	Description
`get_template_reference`	Read the Nunjucks template engine reference, render-context schema, and available filters. Call once per session before editing templates
`get_template`	Read the rendering template for a span type, scoped to a trace function key or org-global
`save_template`	Upsert a rendering template for a span type. Controls how span data renders in the Bitfab UI

Instrumentation

Tool	Description
`save_trace_plan`	Create a tracing plan, or structurally update an existing plan (its root included) in place by passing its plan ID
`confirm_trace_plan`	Confirm a plan without the browser (the continue path, or a plan page left without saving) so `setup view`/`setup modify` can find it by key later
`get_trace_plan`	Read a trace plan by ID (after confirmation) or by trace function key; Modify preserves prior decisions while reconciling the plan and its root with current instrumentation
`list_trace_plans`	List the org’s trace plans, newest first, filterable by source, status, and trace function key; used to reuse unconfirmed `analyze-repo` drafts instead of re-scanning, and to find the plan a key already has so it is updated rather than duplicated
`cancel_trace_plan`	Retire an unconfirmed plan nobody will act on (already instrumented, or the workflow is gone) so it stops coming back as a reusable draft
`get_sim_plan`	Read a trace function’s sim plan (alpha): its span nodes from recent traces with type, call count, average payload, estimated monthly cost, share of traces, and whether content is captured. A node is its span name inside the trace function key
`save_sim_plan`	Turn content capture off or on per span node (alpha). The span itself, its name, type, timing, and errors are always recorded; content off strips only inputs and outputs. A node recorded by a framework integration, a node that is the root of its traces, a node imported from another platform, or a node named by Bitfab rather than the SDK keeps its content, and turning it off is refused with the reason. Turning a node off also turns content off for every node beneath it in the sim plan, the way untracing a span in a trace plan drops the spans under it; a node beneath it that keeps its content stays on, and a node named in the same call keeps the value given to it. Turning a node back on leaves the nodes beneath it as they are. Returns the updated sim plan

Skills

Skill	Description
`$bitfab:setup`	Full setup workflow: authenticate, instrument, create replay registries
`$bitfab:setup login`	Auth only
`$bitfab:setup analyze-repo`	Non-interactively scan the repo, pick traceable workflows, and upload draft trace plans
`$bitfab:assistant`	Build a dataset from traces, experiment with code changes, improve pass rates or cut token costs
`$bitfab:assistant <key>`	Iterate on a specific trace function
`$bitfab:assistant fix <trace-id>`	Fix one failing trace: diagnose it, make the focused code change, and replay just that trace; once it passes, add it to a dataset, then choose to inspect the before/after in Studio or re-run the full dataset
`$bitfab:update`	Update the plugin to the latest version

Example Workflows

Instrument a new project

$bitfab:setup

The agent detects your project language, finds AI workflows, presents options, and instruments your chosen workflows, all interactively.

Diagnose and fix a failing function

Ask Codex naturally:

"My order-processing traces are failing. What's going wrong and can you fix it?"

The plugin calls search_traces and get_traces to inspect failing traces and suggests code fixes directly. For a specific failing trace, run $bitfab:assistant fix <trace-id>. The agent diagnoses the failure, confirms why the original trace is wrong before editing when the trace or conversation does not already make that clear, makes the focused code fix, and replays only that trace first. Once the fix passes, it adds that trace to a dataset with a validated failing label, then branches: inspect the before/after in Studio, re-run the full dataset (in Studio or terminal-only), keep iterating, or stop. If that full-dataset re-run reveals real regressions (previously-passing traces the fix broke), it reports them and keeps the target trace saved as a red test to revisit. If the replay still fails, it offers to keep iterating or save the trace as a failing test instead.

Iterate on a trace function

$bitfab:assistant memory-search

The agent finds failing traces, walks you through labeling them with expected outcomes, diagnoses the failure patterns in your code, then runs experiments: editing prompts or code, replaying against your labeled dataset, and reporting what improved. You stay in control at every decision point.

Replay after a code change

After updating a function, pass your registry module to the replay command installed by the SDK:

pnpm exec bitfab-replay --registry scripts/replayRegistry.ts extraction --limit 20

Or ask Codex to do it for you. While the replay runs, Codex runs it in the background and reports progress to you as it goes: one line per trace as it finishes (a pass/fail mark, the running count, and how long that trace took), with any error reason inline, plus a periodic “still running” heartbeat when a slow trace takes a while so the run never looks stuck, then a summary with the total and average time. Full per-item outputs are written under that replay run’s .bitfab/replays/<run-id>/items/ folder and referenced from .bitfab/replays/<run-id>/events.jsonl. If the replay command succeeds but its local result cannot be captured, the plugin reports the outcome as unverified instead of failed, then checks the server test run to recover the final result.

Configuration

Credentials

Credentials are stored in .bitfab/credentials.local.json when that project-local file exists, otherwise in ~/.config/bitfab/credentials.json (created by $bitfab:setup login with owner-readable permissions).

Environment Variables

Variable	Description
`BITFAB_API_KEY`	Override the stored API key

Troubleshooting

Not authenticated

If you see “Not authenticated” on session start:

Run $bitfab:setup login to authenticate via browser
Check that ~/.config/bitfab/credentials.json exists and contains your API key
If using an environment variable, verify BITFAB_API_KEY is set

MCP tools not available

If Codex can’t access the Bitfab tools:

Verify the MCP server configuration in ~/.codex/config.toml or .codex/config.toml
Check that BITFAB_API_KEY is set in your environment
Try restarting Codex

Plugin updates

Run $bitfab:update to install the latest version.

Getting Started

Primitives

Account

SDKs

Framework Integrations

Integrations

Web Portal

Reference

Changelog

Installation

What the Plugin Does

Automatic Setup

Assistant

Building the Dataset

Running Experiments

Results

MCP Tools

Core

Trace Inspection

Labeling and Datasets

Experiments

Templates

Instrumentation

Skills

Example Workflows

Instrument a new project

Diagnose and fix a failing function

Iterate on a trace function

Replay after a code change

Configuration

Credentials

Environment Variables

Troubleshooting

Not authenticated

MCP tools not available

Plugin updates

​Installation

​What the Plugin Does

​Automatic Setup

​Assistant

​Building the Dataset

​Running Experiments

​Results

​MCP Tools

​Core

​Trace Inspection

​Labeling and Datasets

​Experiments

​Templates

​Instrumentation

​Skills

​Example Workflows

​Instrument a new project

​Diagnose and fix a failing function

​Iterate on a trace function

​Replay after a code change

​Configuration

​Credentials

​Environment Variables

​Troubleshooting

​Not authenticated

​MCP tools not available

​Plugin updates

Installation

What the Plugin Does

Automatic Setup

Assistant

Building the Dataset

Running Experiments

Results

MCP Tools

Core

Trace Inspection

Labeling and Datasets

Experiments

Templates

Instrumentation

Skills

Example Workflows

Instrument a new project

Diagnose and fix a failing function

Iterate on a trace function

Replay after a code change

Configuration

Credentials

Environment Variables

Troubleshooting

Not authenticated

MCP tools not available

Plugin updates