> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bitfab.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Codex Plugin
> Bitfab plugin for OpenAI Codex: trace, diagnose, and iterate on AI workflows directly in your terminal
The Bitfab Codex plugin brings the full evaluation workflow into Codex. It provides MCP tools for trace inspection and diagnostics, skills for authentication and setup, and automatic notifications so you never have to leave your terminal.
## Installation
Run the CLI from your project directory:
```bash theme={null}
npx bitfab-cli init --editor codex
```
This installs the Bitfab plugin, opens your browser to log in, and launches `$bitfab:setup`.
Pass an initial setup request with `--prompt` (or `-p`) to send it straight to the agent:
```bash theme={null}
npx bitfab-cli init --editor codex --prompt "instrument the chat workflow"
```
The CLI checks that Codex is signed in before it launches the Bitfab agent. If needed, run `codex login` first.
Set your API key as an environment variable:
```bash theme={null}
export BITFAB_API_KEY="YOUR_API_KEY"
```
Then add this to `~/.codex/config.toml` (global) or `.codex/config.toml` (project-level):
```toml theme={null}
[mcp_servers.bitfab]
url = "https://bitfab.ai/mcp"
bearer_token_env_var = "BITFAB_API_KEY"
```
Visit the [setup page](https://bitfab.ai/setup) to get your API key and see the full configuration steps.
## What the Plugin Does
### Automatic Setup
The `$bitfab:setup` skill runs a multi-phase workflow:
1. **Login** -- Opens your browser for OAuth authentication, saves credentials securely
2. **Instrument + Replay** (in parallel, per workflow) -- Reads your codebase, finds all AI workflows (LLM calls, agents, AI-driven decisions), and presents them as a numbered list. You choose which to instrument: it adds tracing with minimal diffs and generates a replay script so you can regression-test your trace functions against production data with one command
You can run individual phases:
```
$bitfab:setup explain # Explain what Bitfab is (read-only, no login)
$bitfab:setup login # Auth only
$bitfab:setup instrument # Trace instrumentation only
$bitfab:setup inspect # Diagnose (and offer to fix) your tracing setup
$bitfab:setup replay # Replay script creation only
$bitfab:setup analyze-repo # Scan the repo and upload draft trace plans without prompts
```
The setup is interactive: it presents 2-5 concrete options per decision point with a recommended choice, so you stay in control throughout.
### Assistant
The `$bitfab:assistant` skill turns production traces into code improvements, whether the goal is correctness (improving pass rates) or efficiency (cutting token usage and cost). Your agent will do the mechanical work and collaborate with you on three steps:
1. **Build a dataset** from production traces: search for failures, label them with expected outcomes
2. **Experiment** against that dataset: make isolated code changes, replay, compare results
3. **Hill climb**: repeat until the best change is found, then present results
Run it with an optional trace function key:
```
$bitfab:assistant
$bitfab:assistant order-processing
```
#### Building the Dataset
Your coding agent does the data wrangling: it searches production traces for failures, reads full inputs and outputs, and identifies edge cases. It then presents edge cases for your judgment: is this a failure (and what should the output be), correct, or irrelevant? This labeled dataset becomes the benchmark for all experiments.
The plugin opens a rich UI for navigating and labeling the dataset, then brings you back to your coding agent so you stay in flow. You can label every trace yourself, or label a few and let the agent classify the rest based on the patterns you've established.
#### Running Experiments
The skill reads your code, diagnoses failure patterns, and categorizes proposed changes:
* **Code fixes**: deterministic bugs, bundled into one experiment as a foundation
* **Judgment-based fixes**: prompt changes, search tuning, output formatting, each gets its own experiment
* **Infrastructure proposals**: larger changes noted for future work, not experimented on
Independent experiments run in parallel, each in its own isolated subagent on a separate git worktree. Each subagent edits the code, runs the replay script against your labeled dataset, and compares new outputs to expected outcomes.
#### Results
After each round, you see which traces now match expected outcomes, which still diverge, and whether any regressions occurred. The assistant works through the planned experiments in turn without pausing to ask whether to keep going, then wraps up once the plan is complete. The final summary shows pass rate improvement and all files changed, uncommitted in your working tree for review.
### MCP Tools
The plugin registers MCP tools that Codex can call during conversations. These let you inspect traces, diagnose failures, and improve your code without leaving the terminal.
#### Core
| Tool | Description |
| -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `get_bitfab_api_key` | Retrieve your API key for SDK initialization and environment variable configuration |
| `get_api_key_context` | Returns which Bitfab org the plugin reads/writes to. Call before the first plugin write, or when data you wrote isn't visible in Studio |
| `list_organizations` | List the Bitfab organizations available to the signed-in user, marking the current plugin org |
| `get_database_connection_status` | Report whether the org has connected a database for per-trace replay branching (`none`, `checking`, `connected`, or `failed`) |
#### Trace Inspection
| Tool | Description |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `list_trace_functions` | List all traced functions in your organization |
| `search_traces` | Search and filter traces with keyword search, date ranges, status filters, regex, environment, label filters, dataset, test run, and db-snapshot scoping |
| `read_traces` | Read one or more traces by ID with the trace environment plus summary (truncated) or full span details (input, output, reasoning, context, errors, per-span duration and tokens) |
| `read_trace_labels` | Read just the labels (verdict, annotation, approved flag) for up to 100 traces by ID in one call, no span content, for loading a dataset's verdicts at once |
| `read_span_field` | Fetch the complete, untruncated value of a single span field (input, output, reasoning, content, errors, or contexts) when `read_traces` truncated it |
#### Labeling and Datasets
| Tool | Description |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `update_agent_labels` | Set, skip, or archive agent pass/fail verdicts on traces. Supports confidence levels and annotations for human review |
| `set_human_labels` | Write validated human pass/fail verdicts (with annotations) on traces. Validated immediately with no UI approval step; used by `assistant fix` before adding a trace to a dataset |
| `create_dataset` | Create a labeled dataset for a traced function (named buckets of traces for review and replay) |
| `list_datasets` | List all datasets for a traced function with trace counts and assigned graders |
| `add_traces_to_dataset` | Add traces to a dataset (idempotent, 1-100 per call) |
| `remove_traces_from_dataset` | Remove traces from a dataset without deleting the traces themselves |
| `add_graders_to_dataset` | Assign graders to a dataset (idempotent, organization- and function-scoped, 1-100 per call) |
| `remove_graders_from_dataset` | Remove grader assignments from a dataset without deleting the graders |
| `save_grader` | Create or edit an automated grader (LLM-as-judge pass/fail check) for a traced function: upsert by id or name, rename, clear pass/fail criteria, archive/restore |
| `list_graders` | List automated graders for a traced function with optional name search and cursor pagination (20 by default, 50 maximum; archived hidden by default) |
#### Experiments
| Tool | Description |
| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `create_experiment_group` | Assign or reuse one shared group UUID across selected experiments (test runs) |
| `add_grader_to_experiment_group` | Assign one grader directly to every current experiment in a group and queue missing evaluations for completed runs; later experiments do not inherit it automatically |
| `list_experiments` | List experiments (replay test runs) for a traced function with status, totals, and delta (fixed/regressed/still passing/still failing) |
| `get_experiment_traces` | Get individual trace results for an experiment with each replay trace's verdict compared to the original, plus token usage (input, output, cached, total) for the replay and paired original |
| `add_graders_to_experiment` / `remove_graders_from_experiment` | Attach or detach active graders on an experiment so they run against its replay traces (idempotent, organization- and function-scoped, 1-100 per call); the effective set at completion is the union with the dataset's runnable graders; detaching a dataset-overlapping grader from an in-progress run is re-added at completion (still runs) but from a completed run permanently drops it from the finalized snapshot, and attaching to a completed experiment grades existing traces only on the next completion/replay |
| `get_replay_status` | Read a replay test run's current status and local replay trace ID to server trace ID mapping while replay is still running |
#### Templates
| Tool | Description |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `get_template_reference` | Read the Nunjucks template engine reference, render-context schema, and available filters. Call once per session before editing templates |
| `get_template` | Read the rendering template for a span type, scoped to a trace function key or org-global |
| `update_template` | Upsert a rendering template for a span type. Controls how span data renders in the Bitfab UI |
#### Instrumentation
| Tool | Description |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `create_trace_plan` | Post a tracing instrumentation plan and get a URL for user confirmation |
| `confirm_trace_plan` | Confirm a plan without the browser (the inline "Continue" path) so `setup view`/`setup modify` can find it by key later |
| `get_trace_plan` | Read a trace plan by ID (after confirmation) or by trace function key (to bootstrap from a prior plan) |
### Skills
| Skill | Description |
| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `$bitfab:setup` | Full setup workflow: authenticate, instrument, create replay scripts |
| `$bitfab:setup login` | Auth only |
| `$bitfab:setup analyze-repo` | Non-interactively scan the repo, pick traceable workflows, and upload draft trace plans |
| `$bitfab:assistant` | Build a dataset from traces, experiment with code changes, improve pass rates or cut token costs |
| `$bitfab:assistant ` | Iterate on a specific trace function |
| `$bitfab:assistant fix ` | Fix one failing trace: diagnose it, make the focused code change, and replay just that trace; once it passes, add it to a dataset, then choose to inspect the before/after in Studio or re-run the full dataset |
| `$bitfab:update` | Update the plugin to the latest version |
## Example Workflows
### Instrument a new project
```
$bitfab:setup
```
The agent detects your project language, finds AI workflows, presents options, and instruments your chosen workflows, all interactively.
### Diagnose and fix a failing function
Ask Codex naturally:
```
"My order-processing traces are failing. What's going wrong and can you fix it?"
```
The plugin calls `search_traces` and `read_traces` to inspect failing traces and suggests code fixes directly.
For a specific failing trace, run `$bitfab:assistant fix `. The agent diagnoses the failure, confirms why the original trace is wrong before editing when the trace or conversation does not already make that clear, makes the focused code fix, and replays only that trace first. Once the fix passes, it adds that trace to a dataset with a validated failing label, then branches: inspect the before/after in Studio, re-run the full dataset (in Studio or terminal-only), keep iterating, or stop. If that full-dataset re-run reveals real regressions (previously-passing traces the fix broke), it reports them and keeps the target trace saved as a red test to revisit. If the replay still fails, it offers to keep iterating or save the trace as a failing test instead.
### Iterate on a trace function
```
$bitfab:assistant memory-search
```
The agent finds failing traces, walks you through labeling them with expected outcomes, diagnoses the failure patterns in your code, then runs experiments: editing prompts or code, replaying against your labeled dataset, and reporting what improved. You stay in control at every decision point.
### Replay after a code change
After updating a function, run your replay script to test against production data:
```bash theme={null}
npx tsx scripts/replay.ts extraction --limit 20
```
Or ask Codex to do it for you.
While the replay runs, Codex runs it in the background and reports progress to you as it goes: one line per trace as it finishes (a pass/fail mark, the running count, and how long that trace took), with any error reason inline, plus a periodic "still running" heartbeat when a slow trace takes a while so the run never looks stuck, then a summary with the total and average time. Full per-item outputs are written under that replay run's `.bitfab/replays//items/` folder and referenced from `.bitfab/replays//events.jsonl`.
## Configuration
### Credentials
Credentials are stored in `.bitfab/credentials.local.json` when that project-local file exists, otherwise in `~/.config/bitfab/credentials.json` (created by `$bitfab:setup login` with owner-readable permissions).
### Environment Variables
| Variable | Description |
| ---------------- | --------------------------- |
| `BITFAB_API_KEY` | Override the stored API key |
## Troubleshooting
### Not authenticated
If you see "Not authenticated" on session start:
1. Run `$bitfab:setup login` to authenticate via browser
2. Check that `~/.config/bitfab/credentials.json` exists and contains your API key
3. If using an environment variable, verify `BITFAB_API_KEY` is set
### MCP tools not available
If Codex can't access the Bitfab tools:
1. Verify the MCP server configuration in `~/.codex/config.toml` or `.codex/config.toml`
2. Check that `BITFAB_API_KEY` is set in your environment
3. Try restarting Codex
### Plugin updates
Run `$bitfab:update` to install the latest version.