Skip to main content
This is an unreleased pilot. Use SDK and plugin builds containing the cloud implementation. The direct flow is locally tested. A live customer replay has not yet been verified.
Cloud replay runs your normal replay on your own GitHub Actions. Take the command you already run and add --cloud:
It behaves like the local command:
  • Every replay option works as it does locally, in any order, including --dry-run and --fail-on-error. Your SDK checks the options before anything is uploaded, so an option it does not support is refused right away with the same error the local command gives. With no --trace-ids or --dataset-ids, the runner replays the datasets your registry names, as it does locally.
  • The runner replays a snapshot of your working tree: your changed files, plus new files that git does not ignore. It starts the same bitfab-replay from the same directory with the same arguments.
  • The experiment records your branch and the commit you were on, with your uncommitted changes as its code change, as a local replay does. The runner needs that commit, so the workflow checks out two commits (fetch-depth: 2); --cloud-update adds it to a workflow set up by an older SDK, and to a checkout step you wrote yourself with any <owner>/checkout action, such as useblacksmith/checkout.
  • When the run finishes, the command prints the replay’s progress on stderr and a short summary of its result on stdout, and exits with the replay’s exit code. The summary holds the experiment ID and URL, how many traces were replayed, came out the same, changed, and errored, each errored trace with its error message, and the exit code. Every trace’s input and output are saved in the experiment in Bitfab, where the Bitfab MCP tools read them, so your coding agent works from the same items as after a local replay, and the Actions job log holds no trace data. Add --cloud-full-output to print the replay’s whole JSON result instead, exactly as the local command prints it. That result passes through the job log, so it is refused on a public repository, where anyone can read the log, and on a large run it can exceed what the log holds.
While the run is queued and running, it prints the Actions URL and each step as it starts.

​
Set up once

Ask your coding agent to run bitfab:setup cloud, or run it yourself from the directory where you normally run replay:
It writes one file, .github/workflows/bitfab-replay.yml, and returns what is left to do: the exact GitHub secrets to create and the page to create them on, the push-triggered CI to exclude, and the one merge that registers the workflow. A Node project using pnpm gets:
The workflow holds only your settings. The code that takes snapshots, dispatches, recovers, and runs replay ships inside the SDK, so upgrading the SDK updates it. The runtime, package manager, install command, and replay command are detected from the directory you run it in: Edit the workflow directly for anything detection missed, such as a build step, code generation, a monorepo install filter, or a service. If you normally start replay through a wrapper script that sets environment variables, add them to the replay step’s env. Running --cloud-init again on an existing workflow, or its alias --cloud-update, keeps your edits and only adds lines a newer SDK needs: the execution_id and key_id inputs, the run name that carries the execution, and the id-token: write permission. It also removes the BITFAB_REPLAY_BITFAB_API_KEY line older SDKs wrote, which read a secret by a name built from the run’s inputs and so made GitHub hand the runner every secret in the repository. It then lists what it changed and anything it could not place. It also lists any BITFAB_REPLAY_KEY_* secrets older SDKs left in the repository when a replay could not delete them. Each holds a copy of the Bitfab API key of the person who started that replay, and nothing reads them anymore. Add --remove-old-key-secrets to delete them. Commit the workflow and land it on your default branch once, which registers it with GitHub Actions. After that, each replay dispatches the workflow as it is in that replay’s snapshot, so later edits apply on the next replay without merging. A setup made by an older SDK kept its settings in .bitfab/cloud.json. Cloud replay now stops and asks you to delete that file and the old workflow, then run --cloud-init again.

​
GitHub access

You need git, Python 3.10+, macOS or Linux, and a github.com origin that git can push to. The GitHub CLI is optional. The SDK uses, in order:
  1. The GitHub CLI’s login, when it is installed and logged in.
  2. GH_TOKEN or GITHUB_TOKEN.
  3. The github.com credential git already stores for pushing over https.
The account needs permission to push branches and run Actions. Whichever one it finds, Python makes the requests to GitHub itself. A python.org install of Python on macOS may have no certificate store yet; the error then says to run Install Certificates.command or set SSL_CERT_FILE.

​
Secrets

Each secret the replay reads is an entry in the replay step’s env, such as OPENAI_API_KEY: ${{ secrets.BITFAB_CLOUD_OPENAI_API_KEY }}. That mapping is the only list of secrets: --cloud-secrets sets what it names, and the runner refuses to start when one of them is empty. A secret stored under another name is mapped directly, such as ANTHROPIC_API_KEY: ${{ secrets.EVAL_ANTHROPIC_API_KEY }}, and a non-secret value reads ${{ vars.NAME }}. Store JSON, such as a service account key, on one line: GitHub masks each line of a secret on its own, so a line holding only { hides every brace in the replay’s output. --cloud-secrets stores JSON on one line, and the runner warns when a secret has a line that short.

​
Settings for one replay

To change a setting for one replay without editing the workflow, add --cloud-env NAME=VALUE, once for each variable, such as --cloud-env LOG_LEVEL=debug --cloud-env MODEL=gpt-5-mini. --cloud-env NAME alone copies the value from your shell. The runner sets them for the replay and for the --check command, on top of what the workflow’s env maps. The values are stored in the snapshot commit on GitHub, where anyone who can read the repository can see them, so they are for settings, never secrets. A name that looks like a secret (it contains KEY, SECRET, TOKEN, PASSWORD, or CREDENTIALS as a word), a value holding a URL with a password (such as postgres://user:password@host/db), a name the workflow already reads from a secret, and names that GitHub Actions or the runner set (GITHUB_, RUNNER_, ACTIONS_, BITFAB_REPLAY_, CI) are refused. Put a secret in the workflow’s env and set it with --cloud-secrets instead. BITFAB_API_KEY must belong to the organization that holds the traces you replay. Create a separate key for the runner on the API keys page, named for it, so you can revoke or rotate it on its own. Create the secrets on the page --cloud-init names, or copy them from your local environment files by name:
It reads the names and the Environment from the workflow, creates the Environment when it does not exist, and pipes each value to the GitHub CLI on standard input, so values never appear in arguments, process listings, or shell history. Without the GitHub CLI, which encrypts each value for GitHub, it finishes with createByHand: the secrets your files define and the page to create them on. A blank local value is skipped, because an empty secret replaces a working default with nothing. The runner signs in only with the credentials in its secrets, never as you. Your local gcloud or AWS logins do not travel to it, so a bucket you can read from your laptop can still refuse the runner. Once GitHub has registered the workflow, anyone with write access to the repository can dispatch it on any branch, and that run receives every secret it reads. Keep the secrets in a GitHub Environment (--environment bitfab-replay) with required reviewers, so no run gets credentials until someone approves it. When a setup step needs a credential file, such as a Google service account JSON, write it under $RUNNER_TEMP with umask 077 and export its path through $GITHUB_ENV.

​
Whose API key a replay uses

A cloud replay runs as the person who started it, so the experiment is recorded as yours, not as the owner of a shared key. Before dispatching, bitfab-replay --cloud loads your registry locally to read the key its client resolves, and uses it to ask Bitfab for a one-time token, which it passes as the key_id input. Your key itself never leaves your machine. Bitfab ties the token to the repository, the replay’s branch, and the workflow file. The token is visible to anyone who can read the repository’s Actions runs, but it opens nothing on its own. When the replay step starts, the runner asks GitHub for its own run’s OIDC token (which is why the workflow grants id-token: write), a short-lived statement signed by GitHub naming the repository, branch, workflow, and event of the run. It sends both to Bitfab, which checks GitHub’s signature and refuses the trade unless the run is a workflow_dispatch of that workflow, on that branch, in that repository. Bitfab then answers with a new key under your name that lasts until 30 minutes after the replay’s timeout, or 2 hours when no timeout is set. The runner sets it as BITFAB_REPLAY_BITFAB_API_KEY, which every Bitfab client uses ahead of its configured key, hides it from the log, and turns it off when the replay ends. A token works once and expires after 24 hours if no run uses it. These keys appear under Temporary keys on your API keys page for 7 days, where you can turn one off before or during its run. The run uses the workflow’s BITFAB_API_KEY secret instead, and says why, when your registry’s client resolves no key locally, Bitfab will not issue or trade the token, or the workflow declares no key_id input or does not grant id-token: write. Starting a replay needs no permission to manage the repository’s secrets. A workflow written by an older SDK gets the key_id input and the id-token: write permission, and loses the old BITFAB_REPLAY_BITFAB_API_KEY line, from running bitfab-replay --cloud-update (the same command as --cloud-init). Until then, each run shows a warning in GitHub Actions that names what the workflow is missing.

​
Push-triggered CI

Each replay pushes a temporary bitfab-replay/EXECUTION_UUID branch, which may trigger push-triggered CI and deployments. --cloud-init lists the ones it finds. For a workflow, add bitfab-replay/** to branches-ignore. For Vercel, set git.deploymentEnabled to {"bitfab-replay/**": false} in vercel.json.

​
Databases

The runner needs no database secret when replay uses database branching. Connect your database to your Bitfab organization, set dbBranch on the registry entries that read it, and have your code take its connection from the replay branch URL on each call. A connection pool created once from DATABASE_URL when your code loads never sees the branch. Without branching, the runner needs network access to your database or a replay-safe stand-in.

​
Run replay

See what the snapshot would carry. This pushes nothing:
Before the first replay, check the runner with --dry-run. It dispatches a real run that receives the workflow’s secrets, stops on any that are empty (GitHub passes a missing secret as an empty string), runs your --check command, and resolves the traces with the replay’s own --dry-run:
When a run fails before the replay starts, such as a failed install step, the command prints the job log up to the error. The GitHub run exits with the replay’s exit code, so a replay that finds nothing to run or exits 1 under --fail-on-error shows a failed run in the Actions tab. The exit code also shows as a warning on the run and in its summary, and bitfab-replay --cloud exits with it too. The run also fails when something stops it first, such as an install step, an empty secret, or a --check command. To keep the run green regardless, set continue-on-error: true on the replay step.

​
Follow a replay

Use --cloud-detach to return after dispatch, which suits replays that run for a long time, then follow it by the execution UUID it prints:
--cloud-watch waits, prints the replay’s output, and exits with its exit code. --cloud-status prints the execution’s state and, once it finished, exits with the replay’s exit code. Records live in <worktree-git-directory>/bitfab-cloud/, so follow a replay from the worktree that started it. Ctrl-C detaches without cancelling the run. A detached run’s temporary branch is removed by a later watch, status, or cleanup. If a dispatch response is lost, resubmitting the same arguments with --cloud-request-id EXECUTION_UUID recovers the original execution and never dispatches it twice. Do not use GitHub’s rerun button; the runner refuses reruns. When GitHub refuses a dispatch, such as for a token without Actions write access, no run starts and the temporary branch is removed straight away. When the dispatch fails without an answer, GitHub may still have started the run: --cloud-watch follows it if it did, and --cloud-cleanup removes the branch once --cloud-status shows no run.

​
Replay a branch from CI

A scheduled or triggered job can replay a branch exactly as it is on GitHub, with no snapshot, so the experiment records that branch. This is how a nightly replay of dev feeds reports that count only experiments recorded on dev. From a job with the SDK installed, add --cloud-ref BRANCH to the replay command:
It reads the branch’s current commit from GitHub, dispatches bitfab-replay.yml on that branch, waits, and exits with the replay’s exit code like any --cloud run. Your changes are never uploaded, nothing is pushed, and there is no temporary branch to clean up. --cloud-preview shows the commit and the request without dispatching. The job still runs it from a checkout of the repository, because the command finds the workflow and checks paths such as --registry there and reads the repository from the origin remote. Its GitHub access must read the repository and run Actions. A job’s own token needs both permissions, since naming one permission drops the rest, and must be passed in as GH_TOKEN:
The run’s name carries the execution UUID, which is how the command tells its own run apart from others started on the same branch at the same time. A workflow written before this has no execution_id input and no run-name; add both, as above, before running several replays of one branch at once. A workflow can also dispatch bitfab-replay.yml itself, on any branch other than a bitfab-replay/ snapshot branch, with the replay as JSON in the request input:
Either way, the workflow on that branch must declare the request input, as --cloud-init writes it. A workflow written before this has on: workflow_dispatch alone; replace that with the inputs block above and land it on the branch. The runner’s job exits with the replay’s exit code, so the Actions run fails when the replay does: bad arguments, no matching traces, a replay that could not finish, or, with --fail-on-error, any trace that errored. Changed outputs never fail it. To keep the run green regardless, set continue-on-error: true on the replay step.

​
Limits and safety

Your HEAD, staging area, and working files remain unchanged, and there is no local commit to undo. Ignored files are never included. Credential-like files, such as .env, .npmrc, .pem, and credentials.json, are refused when the snapshot would be the first to put them on GitHub: new or changed files, and files in commits you have not pushed. Files already on GitHub upload nothing new and are left alone, and .env.example, .env.sample, and .env.template are always allowed. When a refused file is safe to upload, such as a test certificate, name it with --cloud-allow-file PATH (repeat it for more); this is refused on a public repository, where anyone can read the snapshot branch. When the file is already on GitHub but was pushed from somewhere else, run git fetch so the check can see it. New files larger than 20 MiB in total are refused. Filename checks are not secret scanning, so review --cloud-preview before the first replay. Submodules and nested repositories are not supported. The runner uses the SDK your snapshot installs. When it is not the SDK you ran locally, the runner stops and says which one to update, so keep your local SDK matched to the lockfile. Replay mocking remains responsible for preventing unsafe side effects; a CI runner is not a substitute for replay-safe code. The generated job sets no timeout, so GitHub’s limit applies: 6 hours on GitHub-hosted runners. Pass --cloud-timeout MINUTES (1 to 7200) to stop one replay sooner; traces that already finished stay saved. Replay arguments are limited to 16,000 characters, as are the --cloud-env names and values together, and with --cloud-full-output the result the replay prints is limited to 16 MiB. If GitHub shows no run for a dispatch within 10 minutes, the command stops and says so.