Codex Plugin

The Bitfab Codex plugin brings the full evaluation workflow into Codex. It provides MCP tools for trace inspection and diagnostics, skills for authentication and setup, and automatic notifications so you never have to leave your terminal.

Installation

Run the CLI from your project directory:

npx bitfab-cli init --editor codex

This installs the Bitfab plugin, opens your browser to log in, and launches $bitfab:setup. Pass an initial setup request with --prompt (or -p) to send it straight to the agent:

npx bitfab-cli init --editor codex --prompt "instrument the chat workflow"

The CLI checks that Codex is signed in before it launches the Bitfab agent. If needed, run codex login first.

Manual installation (without the CLI)

Set your API key as an environment variable:

export BITFAB_API_KEY="YOUR_API_KEY"

Then add this to ~/.codex/config.toml (global) or .codex/config.toml (project-level):

[mcp_servers.bitfab]
url = "https://bitfab.ai/mcp"
bearer_token_env_var = "BITFAB_API_KEY"

Visit the setup page to get your API key and see the full configuration steps.

What the Plugin Does

Automatic Setup

The $bitfab:setup skill runs a multi-phase workflow:

Login — Opens your browser for OAuth authentication, saves credentials securely
Instrument + Replay (in parallel, per workflow) — Reads your codebase, finds all AI workflows (LLM calls, agents, AI-driven decisions), and presents them as a numbered list. You choose which to instrument: it adds tracing with minimal diffs and generates a replay script so you can regression-test your trace functions against production data with one command

You can run individual phases:

$bitfab:setup explain      # Explain what Bitfab is (read-only, no login)
$bitfab:setup login        # Auth only
$bitfab:setup instrument   # Trace instrumentation only
$bitfab:setup inspect      # Diagnose (and offer to fix) your tracing setup
$bitfab:setup replay       # Replay script creation only
$bitfab:setup analyze-repo # Scan the repo and upload draft trace plans without prompts

The setup is interactive: it presents 2-5 concrete options per decision point with a recommended choice, so you stay in control throughout.

Assistant

The $bitfab:assistant skill turns production traces into code improvements, whether the goal is correctness (improving pass rates) or efficiency (cutting token usage and cost). Your agent will do the mechanical work and collaborate with you on three steps:

Build a dataset from production traces: search for failures, label them with expected outcomes
Experiment against that dataset: make isolated code changes, replay, compare results
Hill climb: repeat until the best change is found, then present results

Run it with an optional trace function key:

$bitfab:assistant
$bitfab:assistant order-processing

Building the Dataset

Your coding agent does the data wrangling: it searches production traces for failures, reads full inputs and outputs, and identifies edge cases. It then presents edge cases for your judgment: is this a failure (and what should the output be), correct, or irrelevant? This labeled dataset becomes the benchmark for all experiments. The plugin opens a rich UI for navigating and labeling the dataset, then brings you back to your coding agent so you stay in flow. You can label every trace yourself, or label a few and let the agent classify the rest based on the patterns you’ve established.

Running Experiments

The skill reads your code, diagnoses failure patterns, and categorizes proposed changes:

Code fixes: deterministic bugs, bundled into one experiment as a foundation
Judgment-based fixes: prompt changes, search tuning, output formatting, each gets its own experiment
Infrastructure proposals: larger changes noted for future work, not experimented on

Independent experiments run in parallel, each in its own isolated subagent on a separate git worktree. Each subagent edits the code, runs the replay script against your labeled dataset, and compares new outputs to expected outcomes.

Results

After each round, you see which traces now match expected outcomes, which still diverge, and whether any regressions occurred. The assistant works through the planned experiments in turn without pausing to ask whether to keep going, then wraps up once the plan is complete. The final summary shows pass rate improvement and all files changed, uncommitted in your working tree for review.

MCP Tools

The plugin registers MCP tools that Codex can call during conversations. These let you inspect traces, diagnose failures, and improve your code without leaving the terminal.

Core

Tool	Description
`get_bitfab_api_key`	Retrieve your API key for SDK initialization and environment variable configuration
`get_api_key_context`	Returns which Bitfab org the plugin reads/writes to. Call before the first plugin write, or when data you wrote isn’t visible in Studio
`list_organizations`	List the Bitfab organizations available to the signed-in user, marking the current plugin org
`get_database_connection_status`	Report whether the org has connected a database for per-trace replay branching (`none`, `checking`, `connected`, or `failed`)

Trace Inspection

Tool	Description
`list_trace_functions`	List all traced functions in your organization
`search_traces`	Search and filter traces with keyword search, date ranges, status filters, regex, environment, label filters, dataset, test run, and db-snapshot scoping
`read_traces`	Read one or more traces by ID with the trace environment plus summary (truncated) or full span details (input, output, reasoning, context, errors, per-span duration and tokens)
`read_trace_labels`	Read just the labels (verdict, annotation, approved flag) for up to 100 traces by ID in one call, no span content, for loading a dataset’s verdicts at once
`read_span_field`	Fetch the complete, untruncated value of a single span field (input, output, reasoning, content, errors, or contexts) when `read_traces` truncated it

Labeling and Datasets

Tool	Description
`update_agent_labels`	Set, skip, or archive agent pass/fail verdicts on traces. Supports confidence levels and annotations for human review
`set_human_labels`	Write validated human pass/fail verdicts (with annotations) on traces. Validated immediately with no UI approval step; used by `assistant fix` before adding a trace to a dataset
`create_dataset`	Create a labeled dataset for a traced function (named buckets of traces for review and replay)
`list_datasets`	List all datasets for a traced function with trace counts and assigned graders
`add_traces_to_dataset`	Add traces to a dataset (idempotent, 1-100 per call)
`remove_traces_from_dataset`	Remove traces from a dataset without deleting the traces themselves
`add_graders_to_dataset`	Assign graders to a dataset (idempotent, organization- and function-scoped, 1-100 per call)
`remove_graders_from_dataset`	Remove grader assignments from a dataset without deleting the graders
`save_grader`	Create or edit an automated grader (LLM-as-judge pass/fail check) for a traced function: upsert by id or name, rename, clear pass/fail criteria, archive/restore
`list_graders`	List automated graders for a traced function with optional name search and cursor pagination (20 by default, 50 maximum; archived hidden by default)

Experiments

Tool	Description
`create_experiment_group`	Assign or reuse one shared group UUID across selected experiments (test runs)
`add_grader_to_experiment_group`	Assign one grader directly to every current experiment in a group and queue missing evaluations for completed runs; later experiments do not inherit it automatically
`list_experiments`	List experiments (replay test runs) for a traced function with status, totals, and delta (fixed/regressed/still passing/still failing)
`get_experiment_traces`	Get individual trace results for an experiment with each replay trace’s verdict compared to the original, plus token usage (input, output, cached, total) for the replay and paired original
`add_graders_to_experiment` / `remove_graders_from_experiment`	Attach or detach active graders on an experiment so they run against its replay traces (idempotent, organization- and function-scoped, 1-100 per call); the effective set at completion is the union with the dataset’s runnable graders; detaching a dataset-overlapping grader from an in-progress run is re-added at completion (still runs) but from a completed run permanently drops it from the finalized snapshot, and attaching to a completed experiment grades existing traces only on the next completion/replay
`get_replay_status`	Read a replay test run’s current status and local replay trace ID to server trace ID mapping while replay is still running

Templates

Tool	Description
`get_template_reference`	Read the Nunjucks template engine reference, render-context schema, and available filters. Call once per session before editing templates
`get_template`	Read the rendering template for a span type, scoped to a trace function key or org-global
`update_template`	Upsert a rendering template for a span type. Controls how span data renders in the Bitfab UI

Instrumentation

Tool	Description
`create_trace_plan`	Post a tracing instrumentation plan and get a URL for user confirmation
`confirm_trace_plan`	Confirm a plan without the browser (the inline “Continue” path) so `setup view`/`setup modify` can find it by key later
`get_trace_plan`	Read a trace plan by ID (after confirmation) or by trace function key (to bootstrap from a prior plan)

Skills

Skill	Description
`$bitfab:setup`	Full setup workflow: authenticate, instrument, create replay scripts
`$bitfab:setup login`	Auth only
`$bitfab:setup analyze-repo`	Non-interactively scan the repo, pick traceable workflows, and upload draft trace plans
`$bitfab:assistant`	Build a dataset from traces, experiment with code changes, improve pass rates or cut token costs
`$bitfab:assistant <key>`	Iterate on a specific trace function
`$bitfab:assistant fix <trace-id>`	Fix one failing trace: diagnose it, make the focused code change, and replay just that trace; once it passes, add it to a dataset, then choose to inspect the before/after in Studio or re-run the full dataset
`$bitfab:update`	Update the plugin to the latest version

Example Workflows

Instrument a new project

$bitfab:setup

The agent detects your project language, finds AI workflows, presents options, and instruments your chosen workflows, all interactively.

Diagnose and fix a failing function

Ask Codex naturally:

"My order-processing traces are failing. What's going wrong and can you fix it?"

The plugin calls search_traces and read_traces to inspect failing traces and suggests code fixes directly. For a specific failing trace, run $bitfab:assistant fix <trace-id>. The agent diagnoses the failure, confirms why the original trace is wrong before editing when the trace or conversation does not already make that clear, makes the focused code fix, and replays only that trace first. Once the fix passes, it adds that trace to a dataset with a validated failing label, then branches: inspect the before/after in Studio, re-run the full dataset (in Studio or terminal-only), keep iterating, or stop. If that full-dataset re-run reveals real regressions (previously-passing traces the fix broke), it reports them and keeps the target trace saved as a red test to revisit. If the replay still fails, it offers to keep iterating or save the trace as a failing test instead.

Iterate on a trace function

$bitfab:assistant memory-search

The agent finds failing traces, walks you through labeling them with expected outcomes, diagnoses the failure patterns in your code, then runs experiments: editing prompts or code, replaying against your labeled dataset, and reporting what improved. You stay in control at every decision point.

Replay after a code change

After updating a function, run your replay script to test against production data:

npx tsx scripts/replay.ts extraction --limit 20

Or ask Codex to do it for you. While the replay runs, Codex runs it in the background and reports progress to you as it goes: one line per trace as it finishes (a pass/fail mark, the running count, and how long that trace took), with any error reason inline, plus a periodic “still running” heartbeat when a slow trace takes a while so the run never looks stuck, then a summary with the total and average time. Full per-item outputs are written under that replay run’s .bitfab/replays/<run-id>/items/ folder and referenced from .bitfab/replays/<run-id>/events.jsonl.

Configuration

Credentials

Credentials are stored in .bitfab/credentials.local.json when that project-local file exists, otherwise in ~/.config/bitfab/credentials.json (created by $bitfab:setup login with owner-readable permissions).

Environment Variables

Variable	Description
`BITFAB_API_KEY`	Override the stored API key

Troubleshooting

Not authenticated

If you see “Not authenticated” on session start:

Run $bitfab:setup login to authenticate via browser
Check that ~/.config/bitfab/credentials.json exists and contains your API key
If using an environment variable, verify BITFAB_API_KEY is set

MCP tools not available

If Codex can’t access the Bitfab tools:

Verify the MCP server configuration in ~/.codex/config.toml or .codex/config.toml
Check that BITFAB_API_KEY is set in your environment
Try restarting Codex

Plugin updates

Run $bitfab:update to install the latest version.

Getting Started

Account

SDKs

Framework Integrations

Integrations

Web Portal

Reference

Changelog

Installation

What the Plugin Does

Automatic Setup

Assistant

Building the Dataset

Running Experiments

Results

MCP Tools

Core

Trace Inspection

Labeling and Datasets

Experiments

Templates

Instrumentation

Skills

Example Workflows

Instrument a new project

Diagnose and fix a failing function

Iterate on a trace function

Replay after a code change

Configuration

Credentials

Environment Variables

Troubleshooting

Not authenticated

MCP tools not available

Plugin updates

​Installation

​What the Plugin Does

​Automatic Setup

​Assistant

​Building the Dataset

​Running Experiments

​Results

​MCP Tools

​Core

​Trace Inspection

​Labeling and Datasets

​Experiments

​Templates

​Instrumentation

​Skills

​Example Workflows

​Instrument a new project

​Diagnose and fix a failing function

​Iterate on a trace function

​Replay after a code change

​Configuration

​Credentials

​Environment Variables

​Troubleshooting

​Not authenticated

​MCP tools not available

​Plugin updates

Installation

What the Plugin Does

Automatic Setup

Assistant

Building the Dataset

Running Experiments

Results

MCP Tools

Core

Trace Inspection

Labeling and Datasets

Experiments

Templates

Instrumentation

Skills

Example Workflows

Instrument a new project

Diagnose and fix a failing function

Iterate on a trace function

Replay after a code change

Configuration

Credentials

Environment Variables

Troubleshooting

Not authenticated

MCP tools not available

Plugin updates