TypeScript SDK

The Simforge TypeScript SDK captures your AI function calls to automatically generate evaluations. Re-run your prompts with different models, parameters, and inputs to iterate faster.

Installation

# npm
npm install @goharvest/simforge

# pnpm
pnpm add @goharvest/simforge

# yarn
yarn add @goharvest/simforge

Quick Start

import { Simforge } from "@goharvest/simforge"

const simforge = new Simforge({ apiKey: process.env.SIMFORGE_API_KEY })

Need an API key? Get one from the Simforge dashboard or see the API Keys guide for detailed setup instructions.

Coding Agent Prompt (Cursor, Claude Code)

Copy this prompt into your coding agent (tested with Cursor and Claude Code using Sonnet 4.5):

Modify existing TypeScript code to add Simforge tracing.
Do NOT browse or web search. Use ONLY the API described below.

Simforge TypeScript SDK (authoritative excerpt):
- Install: `npm install @goharvest/simforge` or `pnpm add @goharvest/simforge`
- Init:
  import { Simforge } from "@goharvest/simforge"
  const simforge = new Simforge({ apiKey: process.env.SIMFORGE_API_KEY })
- Instrumentation (ONLY allowed form - use getFunction):
  // Declare trace function key once
  const myService = simforge.getFunction("<trace_function_key>")

  // Wrap functions with withSpan
  const tracedFn = myService.withSpan(originalFunction)

  // Or with options:
  const tracedFn = myService.withSpan({ name: "DisplayName", type: "function" }, originalFunction)

  // Span types: "llm", "agent", "function", "guardrail", "handoff", "custom"
- DO NOT modify the original function.
- DO NOT extract helper methods.

Task:
1) Ensure @goharvest/simforge is installed and initialization exists.
2) Read the codebase and identify ALL AI workflows (LLM calls, agent runs, AI-driven decisions).
3) Present me with a numbered list of workflows you found. For each, describe:
   - What it does
   - Why it's worth instrumenting — what visibility tracing gives you into each step
4) After I choose which workflow(s) to instrument:
   - Create a function wrapper with `simforge.getFunction("<trace_function_key>")`
   - Wrap the functions with `myService.withSpan(originalFunction)`
   - Instrument intermediate steps (not just the final output) so each trace has enough context to diagnose issues
   - Replace usages of the original functions with the traced versions
5) Do not change function signature, behavior, or return value. Minimal diff.

Output:
- First: your numbered list of workflows with why each is worth instrumenting
- After my selection: minimal diffs for dependencies, initialization, and the function wrapping

Basic Configuration

new Simforge({ apiKey: string })

// Disable tracing (functions still execute, but no spans are sent)
new Simforge({ apiKey: string, enabled: false })

Missing API key doesn’t crash. If the API key is missing, empty, or whitespace-only, the SDK automatically disables tracing and logs a warning. All wrapped functions still execute normally — no spans are sent, no errors are thrown. You don’t need any conditional logic around the API key.

Tracing

Custom (Recommended)

Using `getFunction()` to Link Spans

Declare the trace function key once and wrap multiple functions:

const orderService = simforge.getFunction("order-processing")

async function processOrder(orderId: string) {
  return { orderId }
}

async function validateOrder(orderId: string) {
  return { valid: true }
}

// Wrap functions - all share the same trace function key
const tracedProcessOrder = orderService.withSpan(processOrder)
const tracedValidateOrder = orderService.withSpan(validateOrder)

Multi-File Projects

For projects with instrumented functions spread across multiple files, create a dedicated file that initializes Simforge and exports the function. Import it wherever you need to instrument.

// lib/simforge.ts — single source of truth
import { Simforge } from "@goharvest/simforge"
const simforge = new Simforge({ apiKey: process.env.SIMFORGE_API_KEY })
export const orderService = simforge.getFunction("order-processing")

// services/processOrder.ts
import { orderService } from "../lib/simforge"

async function processOrder(orderId: string) {
  return { orderId }
}

export const tracedProcessOrder = orderService.withSpan(processOrder)

// services/validateOrder.ts
import { orderService } from "../lib/simforge"

async function validateOrder(orderId: string) {
  return { valid: true }
}

export const tracedValidateOrder = orderService.withSpan(validateOrder)

Spans from different files are automatically linked as parent-child when one wrapped function calls another.

Wrapping Existing Functions Inline

When wrapping a function you didn’t define (e.g. an SDK or library call), pass it directly to withSpan and call the result immediately. This ensures the arguments are captured as span input.

// ✅ GOOD — pass function directly, arguments are captured as span input
const result = await orderService.withSpan(
  { name: "ProcessOrder", type: "function" },
  processOrder,
)(orderId)

// ❌ BAD — anonymous wrapper loses all input capture (span has no input)
const result = await orderService.withSpan(
  { name: "ProcessOrder", type: "function" },
  async () => processOrder(orderId),
)()

Never wrap functions in an anonymous function like async () => fn(args). The SDK captures the wrapper function’s arguments as span input — an anonymous wrapper has no arguments, so the span records nothing.

Using `withSpan()` Directly

For a single span without linking to a function group:

const standaloneTask = simforge.withSpan("one-off-operation", () => {
  return "done"
})

Automatic Nesting

Spans nest automatically based on call stack:

const outer = simforge.withSpan("outer", { type: "agent" }, async () => {
  await inner()  // Becomes a child of "outer"
})

const inner = simforge.withSpan("inner", { type: "function" }, async () => {
  // ...
})

Span Options

Parameters:

traceFunctionKey (required): String identifier for grouping spans
name (optional): Display name. Defaults to function name, then trace function key
type (optional): Span type. Defaults to "custom"

Span Types:

type SpanType =
  | "llm"        // LLM calls
  | "agent"      // Agent workflows
  | "function"   // Function calls
  | "guardrail"  // Safety checks
  | "handoff"    // Human handoffs
  | "custom"     // Default

Examples:

// Function name is automatically captured as span name
async function processOrder(orderId: string) {
  return { orderId }
}
const traced = simforge.withSpan("order-processing", processOrder)
// Span name: "processOrder"

// Override with name option
const traced = simforge.withSpan(
  "order-processing",
  { name: "OrderProcessor" },
  processOrder
)
// Span name: "OrderProcessor"

// Set span type
const checkSafety = simforge.withSpan(
  "safety-check",
  { type: "guardrail" },
  async (content: string) => ({ safe: true })
)

// With getFunction()
const service = simforge.getFunction("order-processing")
const traced = service.withSpan({ name: "CustomName", type: "function" }, processOrder)

Span Context

Use getCurrentSpan() to get a handle to the active span, then call .addContext() to attach contextual key-value pairs from inside a traced function — useful for runtime values like request IDs, computed scores, or dynamic context:

import { getCurrentSpan } from "@goharvest/simforge"

async function processOrder(orderId: string) {
  const userId = await getCurrentUser()
  getCurrentSpan()?.addContext({ user_id: userId, order_id: orderId })
  return { orderId, status: "completed" }
}

const traced = simforge.withSpan("order-processing", { type: "function" }, processOrder)

Each addContext call pushes the entire object as one entry. Multiple calls accumulate entries:

getCurrentSpan()?.addContext({ user_id: "u-123" })
getCurrentSpan()?.addContext({ request_id: "req-789" })
// Result: contexts: [{ user_id: "u-123" }, { request_id: "req-789" }]

Span Trace ID

Access the current trace ID from within a span using getCurrentSpan().traceId. This is useful for capturing trace IDs to use with replay or for logging:

import { getCurrentSpan } from "@goharvest/simforge"

const traced = simforge.withSpan("my-function", async () => {
  const traceId = getCurrentSpan().traceId  // UUID string
  console.log("Current trace:", traceId)
  return { traceId }
})

Outside a span context, getCurrentSpan().traceId returns an empty string.

Span Prompt

Use getCurrentSpan() to set the prompt string on the current span. This is stored in span_data.prompt and is useful for capturing the exact prompt text sent to an LLM:

import { getCurrentSpan } from "@goharvest/simforge"

async function classifyText(text: string) {
  const prompt = `Classify the following text: ${text}`
  getCurrentSpan()?.setPrompt(prompt)
  const result = await llm.complete(prompt)
  return result
}

const traced = simforge.withSpan("classification", { type: "llm" }, classifyText)

The last setPrompt call wins — it overwrites any previously set prompt on the span. Calling setPrompt outside a span context is a no-op (it never crashes).

BAML Auto-Instrumentation

If you use BAML for your LLM calls, wrapBAML automatically captures the rendered prompt and LLM metadata (model, provider, token counts, duration) on the current span — no manual setPrompt or addContext calls needed.

npm install @boundaryml/baml

import { b } from "./baml_client"

// Pass your BAML client to the constructor
const simforge = new Simforge({
  apiKey: process.env.SIMFORGE_API_KEY,
  bamlClient: b,
})

// Wrap a BAML method — prompt and metadata are captured automatically
const tracedClassify = simforge.withSpan(
  "classify",
  { type: "llm" },
  simforge.wrapBAML(b.ClassifyText),
)

const result = await tracedClassify("Hello world")

wrapBAML works by creating a BAML Collector, running the method through a tracked client, then extracting:

Prompt → setPrompt() with the rendered messages (system + user)
Context → addContext() with { model, provider, inputTokens, outputTokens, durationMs }

If @boundaryml/baml is not installed, the method is called directly without instrumentation.

Accessing the BAML Collector

The wrapped function exposes a .collector property containing the BAML Collector instance from the most recent call. This is useful for accessing raw token counts, timing data, or other metadata that BAML captures:

const tracedClassify = simforge.withSpan(
  "classify",
  { type: "llm" },
  simforge.wrapBAML(b.ClassifyText),
)

await tracedClassify("Hello world")

// Access the collector from the last call
const collector = tracedClassify.collector

The .collector is null before the first call or if @boundaryml/baml is not installed.

`onCollector` Callback

For more control, pass an onCollector callback via WrapBAMLOptions. It fires after each BAML invocation with the Collector instance:

const tracedClassify = simforge.withSpan(
  "classify",
  { type: "llm" },
  simforge.wrapBAML(b.ClassifyText, {
    onCollector: (collector) => {
      // Process token usage, timing, or other collector data
      console.log("BAML collector:", collector)
    },
  }),
)

When using the two-argument form (passing bamlClient explicitly), options go in the third argument:

simforge.wrapBAML(b, b.ClassifyText, {
  onCollector: (collector) => { /* ... */ },
})

If the callback throws, the error is silently caught — it never crashes your application.

Trace Context

Use getCurrentTrace() to set context that applies to the entire trace (all spans within a single execution). This is useful for grouping traces by session or attaching trace-level metadata:

import { getCurrentTrace } from "@goharvest/simforge"

const traced = simforge.withSpan("order-processing", { type: "function" }, async () => {
  const trace = getCurrentTrace()

  // Set session ID (stored as database column, filterable in dashboard)
  trace?.setSessionId("session-123")

  // Set trace metadata (stored in raw trace data)
  trace?.setMetadata({ region: "us-west-2", environment: "production" })

  // Add context entries (stored as key-value pairs, accumulates across calls)
  trace?.addContext({ workflow: "checkout-flow", batch_id: "batch-2024-01" })

  return { status: "completed" }
})

setSessionId(id) — Groups traces by user session. Stored as a database column for efficient filtering.
setMetadata(obj) — Arbitrary key-value metadata on the trace. Merges with existing metadata.
addContext(obj) — Key-value context entries. Accumulates across multiple calls.

Error Handling

Errors are captured in the span and re-raised:

const risky = simforge.withSpan("risky-service", () => {
  throw new Error("error")
})

try {
  risky()
} catch (e) {
  // Span records error and timing
}

OpenAI Agents SDK

Attach a trace processor to capture agent runs:

npm install @openai/agents

import { setTraceProcessors } from "@openai/agents"

const processor = simforge.getOpenAiTracingProcessor()
setTraceProcessors([processor])

Native Functions

Simforge’s native functions improve prompt tuning efficacy. The auto-tuning engine has full access to prompts—unlike other Agent SDKs that nest user instructions inside system prompts, making prompts inaccessible to tracing.

const result = await simforge.call("ExtractName", { text: "My name is John Doe" })
// Returns typed result

Advanced Configuration

new Simforge({
  apiKey: string,                    // Required
  serviceUrl?: string,               // Default: https://simforge.goharvest.ai
  timeout?: number,                  // Request timeout in ms (default: 120000)
  envVars?: { OPENAI_API_KEY: string },  // For native function execution
  enabled?: boolean,                 // Default: true
  bamlClient?: unknown               // Generated BAML client (for wrapBAML)
})

timeout: Request timeout in milliseconds for API calls. Defaults to 120000 (2 minutes).
envVars: Pass LLM provider API keys for native function execution via call().
enabled: When false, all tracing is disabled. Wrapped functions still execute normally but no spans are sent.
bamlClient: The generated BAML client instance (e.g., b from @baml). Used by wrapBAML() when no explicit client is passed at the call site.

Replay

Replay historical traces through an updated function version to compare outputs:

const pipeline = simforge.getFunction("my-function")

const updatedFn = pipeline.withSpan(
  { name: "UpdatedPipeline", type: "function" },
  async (input: string) => {
    // New implementation to test
    return { result: input.toUpperCase() }
  },
)

// Replay all traces (up to limit)
const result = await simforge.replay("my-function", updatedFn, {
  limit: 10,
  maxConcurrency: 5,  // Default: 10
})

// Replay specific traces by ID
const result2 = await simforge.replay("my-function", updatedFn, {
  traceIds: ["trace-id-1", "trace-id-2"],
})

// Result structure
console.log(result.testRunId)   // Test run identifier
console.log(result.testRunUrl)  // Dashboard URL
for (const item of result.items) {
  console.log(item.input)           // Original input
  console.log(item.result)          // New output
  console.log(item.originalOutput)  // Original output
  console.log(item.error)           // Error if any
}

Options:

limit — Maximum number of traces to replay (default: all)
traceIds — Specific trace IDs to replay
maxConcurrency — Number of traces to replay in parallel (default: 10)

Replay Script

Create a standalone script to regression-test your trace functions against production data with one command. The script maps pipeline names to their replay functions, accepts CLI flags, and prints a side-by-side comparison with delta summaries.

/**
 * Replay production traces through instrumented functions.
 *
 * Uses simforge.replay() to fetch real traces and re-run them through
 * the current code, creating a test run for side-by-side comparison.
 *
 * Usage:
 *   npx tsx scripts/replay.ts <pipeline>
 *   npx tsx scripts/replay.ts <pipeline> --limit 20
 *   npx tsx scripts/replay.ts <pipeline> --trace-ids id1,id2
 */
import "dotenv/config"
import { simforge } from "../lib/simforge"
import { extractMemories } from "../services/extraction"
import { searchDocuments } from "../services/search"

const FUNCTIONS = {
  extraction: "my-extraction-pipeline",
  search: "my-search-pipeline",
} as const

type Pipeline = keyof typeof FUNCTIONS

const pipeline = process.argv[2] as Pipeline | undefined
const args = process.argv.slice(3)

if (!pipeline || !FUNCTIONS[pipeline]) {
  console.error(
    `Usage: npx tsx scripts/replay.ts <${Object.keys(FUNCTIONS).join("|")}> [--limit N] [--trace-ids id1,id2]`,
  )
  process.exit(1)
}

let limit = 10
let traceIds: string[] | undefined

for (let i = 0; i < args.length; i++) {
  if (args[i] === "--limit" && args[i + 1]) {
    limit = Number.parseInt(args[i + 1], 10)
    i++
  } else if (args[i] === "--trace-ids" && args[i + 1]) {
    traceIds = args[i + 1].split(",").map((id) => id.trim())
    i++
  }
}

// Each pipeline gets its own replay function — replay deserializes
// historical inputs, so if the function signature doesn't match the
// raw input shape, reshape the arguments in a thin wrapper here.

async function replayExtraction() {
  const fn = async (conversation: string, existingItems: unknown[]) => {
    return extractMemories(conversation, existingItems)
  }
  return simforge.replay(FUNCTIONS.extraction, fn, {
    limit,
    ...(traceIds ? { traceIds } : {}),
  })
}

async function replaySearch() {
  const fn = async (query: string, opts: Record<string, unknown>) => {
    return searchDocuments(query, {
      userId: opts.userId as string,
      limit: (opts.limit as number) ?? 10,
    })
  }
  return simforge.replay(FUNCTIONS.search, fn, {
    limit,
    ...(traceIds ? { traceIds } : {}),
  })
}

const REPLAY_FNS: Record<Pipeline, () => ReturnType<typeof simforge.replay>> = {
  extraction: replayExtraction,
  search: replaySearch,
}

async function main() {
  const functionKey = FUNCTIONS[pipeline!]
  console.log(`[replay] Replaying ${traceIds?.length ?? limit} traces from "${functionKey}"...\n`)

  const result = await REPLAY_FNS[pipeline!]()
  console.log(`Test run: ${result.testRunUrl}\n`)

  let changed = 0, same = 0, errors = 0

  for (const item of result.items) {
    const input = item.input as unknown[] | undefined
    const label = typeof input?.[0] === "string"
      ? input[0].slice(0, 80)
      : JSON.stringify(input?.[0] ?? "unknown").slice(0, 80)

    if (item.error) {
      console.log(`  ✗ "${label}"`)
      console.log(`    Error: ${item.error}`)
      errors++
    } else {
      const origCount = Array.isArray(item.originalOutput) ? item.originalOutput.length : 0
      const newCount = Array.isArray(item.result) ? (item.result as unknown[]).length : 0
      const delta = newCount - origCount
      const deltaStr = delta === 0 ? "=" : delta > 0 ? `+${delta}` : `${delta}`

      console.log(`  ${delta === 0 ? "=" : "Δ"} "${label}"`)
      console.log(`    ${origCount} → ${newCount} (${deltaStr})`)
      delta !== 0 ? changed++ : same++
    }
  }

  console.log(`\n─── Summary ───`)
  console.log(`  Pipeline: ${pipeline}`)
  console.log(`  Replayed: ${result.items.length}`)
  console.log(`  Same:     ${same}`)
  console.log(`  Changed:  ${changed}`)
  if (errors > 0) console.log(`  Errors:   ${errors}`)
  console.log(`\n  ${result.testRunUrl}`)
}

main().catch((err) => {
  console.error("[replay] Fatal error:", err)
  process.exit(1)
})

Adapt the imports, pipeline names, and per-pipeline replay functions to match your project’s instrumented workflows.

Sim Runners

Coming soon.

Getting Started

SDKs

Tools

Integrations

Web Portal

Installation

Quick Start

Basic Configuration

Tracing

Custom (Recommended)

Using `getFunction()` to Link Spans

Multi-File Projects

Wrapping Existing Functions Inline

Using `withSpan()` Directly

Automatic Nesting

Span Options

Span Context

Span Trace ID

Span Prompt

BAML Auto-Instrumentation

Accessing the BAML Collector

`onCollector` Callback

Trace Context

Error Handling

OpenAI Agents SDK

Native Functions

Advanced Configuration

Replay

Replay Script

Sim Runners

Getting Started

SDKs

Tools

Integrations

Web Portal

​Installation

​Quick Start

​Basic Configuration

​Tracing

​Custom (Recommended)

​Using getFunction() to Link Spans

​Multi-File Projects

​Wrapping Existing Functions Inline

​Using withSpan() Directly

​Automatic Nesting

​Span Options

​Span Context

​Span Trace ID

​Span Prompt

​BAML Auto-Instrumentation

Accessing the BAML Collector

onCollector Callback

​Trace Context

​Error Handling

​OpenAI Agents SDK

​Native Functions

​Advanced Configuration

​Replay

​Replay Script

​Sim Runners

Installation

Quick Start

Basic Configuration

Tracing

Custom (Recommended)

Using `getFunction()` to Link Spans

Multi-File Projects

Wrapping Existing Functions Inline

Using `withSpan()` Directly

Automatic Nesting

Span Options

Span Context

Span Trace ID

Span Prompt

BAML Auto-Instrumentation

`onCollector` Callback

Trace Context

Error Handling

OpenAI Agents SDK

Native Functions

Advanced Configuration

Replay

Replay Script

Sim Runners