Skip to main content
The Simforge Ruby SDK captures your AI function calls to automatically generate evaluations. Re-run your prompts with different models, parameters, and inputs to iterate faster.

​
Installation

# Bundler
bundle add simforge

# Gem
gem install simforge

​
Quick Start

require "simforge"

Simforge.configure(api_key: ENV.fetch("SIMFORGE_API_KEY"))
Need an API key? Get one from the Simforge dashboard or see the API Keys guide for detailed setup instructions.
Copy this prompt into your coding agent (tested with Cursor and Claude Code using Sonnet 4.5):
Modify existing Ruby code to add Simforge tracing.
Do NOT browse or web search. Use ONLY the API described below.

Simforge Ruby SDK (authoritative excerpt):
- Install: `gem install simforge` or `bundle add simforge`
- Init:
  require "simforge"
  Simforge.configure(api_key: ENV.fetch("SIMFORGE_API_KEY"))
- Instrumentation (ONLY allowed form):
  class MyService
    include Simforge::Traceable
    simforge_function "<trace_function_key>"

    simforge_span :method_name, type: "function"
    def method_name
      # ...
    end
  end
  (simforge_span must be placed immediately ABOVE the `def` it instruments.)
- Span types: "llm", "agent", "function", "guardrail", "handoff", "custom"
- DO NOT use a block form of simforge_span.
- DO NOT extract helper methods.

Task:
1) Ensure the simforge gem is added and initialization exists (Gemfile + initializer).
2) Read the codebase and identify ALL AI workflows (LLM calls, agent runs, AI-driven decisions).
3) Present me with a numbered list of workflows you found. For each, describe:
   - What it does
   - Why it's worth instrumenting — what visibility tracing gives you into each step
4) After I choose which workflow(s) to instrument:
   - Add `simforge_span` directly ABOVE each method's `def`
   - Instrument intermediate steps (not just the final output) so each trace has enough context to diagnose issues
   - Ensure each class includes `Simforge::Traceable` and has `simforge_function` set
5) Do not change method signature, behavior, or return value. Minimal diff.

Output:
- First: your numbered list of workflows with why each is worth instrumenting
- After my selection: minimal diffs for Gemfile, initializer, and the method changes

​
Basic Configuration

Simforge.configure(api_key: String)

# Disable tracing (functions still execute, but no spans are sent)
Simforge.configure(api_key: String, enabled: false)
Missing API key doesn’t crash. If the API key is missing, empty, or whitespace-only, the SDK automatically disables tracing and logs a warning. All instrumented methods still execute normally — no spans are sent, no errors are thrown. You don’t need any conditional logic around the API key.

​
Tracing

Include Simforge::Traceable in a class, declare the trace function key once with simforge_function, then use simforge_span to wrap methods. Three declaration styles are supported: 
class OrderService
  include Simforge::Traceable
  simforge_function "order-processing"

  # Style 1: Before-def (recommended) — declare simforge_span above the method
  simforge_span :process_order, type: "function"
  def process_order(order_id)
    { order_id: order_id }
  end

  # Style 2: Inline — wrap the def directly
  simforge_span def validate_order(order_id)
    { valid: true }
  end, type: "guardrail"

  # Style 3: After-def — declare simforge_span after the method definition
  def enrich_order(order_id)
    { enriched: true }
  end
  simforge_span :enrich_order, type: "function"
end
All three styles are equivalent. The before-def style is recommended for readability.

​
Multi-File Projects

For projects with instrumented methods spread across multiple files, create an initializer that configures Simforge, then include Simforge::Traceable in any class that needs tracing. 
# config/initializers/simforge.rb — single source of truth
require "simforge"
Simforge.configure(api_key: ENV.fetch("SIMFORGE_API_KEY"))

# app/services/process_order_service.rb
class ProcessOrderService
  include Simforge::Traceable
  simforge_function "order-processing"

  simforge_span :process_order, type: "function"
  def process_order(order_id)
    { order_id: order_id }
  end
end

# app/services/validate_order_service.rb
class ValidateOrderService
  include Simforge::Traceable
  simforge_function "order-processing"

  simforge_span :validate_order, type: "guardrail"
  def validate_order(order_id)
    { valid: true }
  end
end
Classes sharing the same simforge_function key are grouped together. Spans from different classes are automatically linked as parent-child when one instrumented method calls another.

​
Using simforge_span with Explicit Key

For a single span with an explicit trace function key: 
class StandaloneService
  include Simforge::Traceable

  simforge_span :standalone_task, trace_function_key: "one-off-operation"
  def standalone_task
    "done"
  end
end

​
Automatic Nesting

Spans nest automatically based on call stack: 
class Pipeline
  include Simforge::Traceable
  simforge_function "pipeline"

  simforge_span :outer, type: "agent"
  def outer
    inner  # Becomes a child of "outer"
  end

  simforge_span :inner, type: "function"
  def inner
    # ...
  end
end

​
Span Options

Parameters:
  • method_name (required): Symbol of the method to wrap
  • trace_function_key (optional): Override class-level simforge_function
  • name (optional): Display name. Defaults to method name
  • type (optional): Span type. Defaults to "custom"
Span Types: 
SPAN_TYPES = %w[
  llm        # LLM calls
  agent      # Agent workflows
  function   # Function calls
  guardrail  # Safety checks
  handoff    # Human handoffs
  custom     # Default
]
Examples: 
class SafetyService
  include Simforge::Traceable
  simforge_function "safety-service"

  # Method name is automatically captured as span name
  simforge_span :check_safety, type: "guardrail"
  def check_safety(content)
    { safe: !content.include?("unsafe") }
  end

  # Override with name option
  simforge_span :validate_input, name: "InputValidator", type: "guardrail"
  def validate_input(input)
    { valid: !input.empty? }
  end
end

​
Span Context

Use Simforge.current_span to get a handle to the active span, then call .add_context() to attach contextual key-value pairs from inside a traced method — useful for runtime values like request IDs, computed scores, or dynamic context: 
class OrderService
  include Simforge::Traceable
  simforge_function "order-processing"

  simforge_span :process_order, type: "function"
  def process_order(order_id)
    user_id = current_user_id
    Simforge.current_span.add_context("user_id" => user_id, "order_id" => order_id)
    { order_id: order_id, status: "completed" }
  end
end
Each add_context call pushes the entire hash as one entry. Multiple calls accumulate entries: 
Simforge.current_span.add_context("user_id" => "u-123")
Simforge.current_span.add_context("request_id" => "req-789")
# Result: contexts: [{ "user_id" => "u-123" }, { "request_id" => "req-789" }]
You can also access the current trace ID via Simforge.current_span.trace_id (returns an empty string outside a span): 
trace_id = Simforge.current_span.trace_id

​
Span Prompt

Use Simforge.current_span to set the prompt string on the current span. This is stored in span_data.prompt and is useful for capturing the exact prompt text sent to an LLM: 
class ClassificationService
  include Simforge::Traceable
  simforge_function "classification"

  simforge_span :classify_text, type: "llm"
  def classify_text(text)
    prompt = "Classify the following text: #{text}"
    Simforge.current_span.set_prompt(prompt)
    llm.complete(prompt)
  end
end
The last set_prompt call wins — it overwrites any previously set prompt on the span. Calling set_prompt outside a span context is a no-op (it never crashes).

​
Trace Context

Use Simforge.current_trace to set context that applies to the entire trace (all spans within a single execution). This is useful for grouping traces by session or attaching trace-level metadata: 
class OrderService
  include Simforge::Traceable
  simforge_function "order-processing"

  simforge_span :process_order, type: "function"
  def process_order(order_id)
    trace = Simforge.current_trace

    # Set session ID (stored as database column, filterable in dashboard)
    trace.set_session_id("session-123")

    # Set trace metadata (stored in raw trace data)
    trace.set_metadata("region" => "us-west-2", "environment" => "production")

    # Add context entries (stored as key-value pairs, accumulates across calls)
    trace.add_context("workflow" => "checkout-flow", "batch_id" => "batch-2024-01")

    { order_id: order_id, status: "completed" }
  end
end
  • set_session_id(id) — Groups traces by user session. Stored as a database column for efficient filtering.
  • set_metadata(hash) — Arbitrary key-value metadata on the trace. Merges with existing metadata.
  • add_context(hash) — Key-value context entries. Accumulates across multiple calls.

​
Error Handling

Errors are captured in the span and re-raised: 
class RiskyService
  include Simforge::Traceable
  simforge_function "risky-service"

  simforge_span :risky
  def risky
    raise "error"
  end
end

begin
  RiskyService.new.risky
rescue => e
  # Span records error and timing
end

​
Flushing Traces

Simforge.flush_traces(timeout: 30)  # Default: 30s
Traces flush automatically on process exit via at_exit hook.

​
Wrapping Third-Party Methods

Use Simforge::Traceable.wrap to trace methods on external classes: 
require "openai"

Simforge::Traceable.wrap(
  OpenAI::Client, :chat,
  trace_function_key: "openai",
  name: "Chat",
  type: "llm"
)

# Now all calls to client.chat are traced
client = OpenAI::Client.new(access_token: ENV["OPENAI_API_KEY"])
client.chat(parameters: { model: "gpt-4", messages: [...] })

​
Replay

Replay historical traces through a method to create test runs. This re-runs past inputs through your updated code and compares the results. 
client = Simforge.client

# Pass an instance for instance methods
service = OrderService.new(api_key: "...", db: db)
result = client.replay(
  service, :process_order,
  trace_function_key: "order-processing",
  limit: 5
)

# Or pass a Class for class methods
result = client.replay(
  OrderService, :process_order,
  trace_function_key: "order-processing",
  limit: 5
)

puts "Test Run URL: #{result[:test_run_url]}"
puts "Test Run ID:  #{result[:test_run_id]}"

result[:items].each do |item|
  puts "Input: #{item[:input]}, Result: #{item[:result]}, Error: #{item[:error]}"
end
Parameters:
  • receiver (required): An instance for instance methods, or a Class for class methods
  • method_name (required): Symbol of the method to replay
  • trace_function_key (required): The trace function key
  • limit (optional): Max traces to replay (default: 5)
  • trace_ids (optional): Array of specific trace IDs to replay
  • max_concurrency (optional): Max threads for parallel replay (default: 10)
Replay specific traces: 
service = OrderService.new(api_key: "...")
result = client.replay(
  service, :process_order,
  trace_function_key: "order-processing",
  trace_ids: ["trace-id-1", "trace-id-2"]
)

​
Replay Script

Create a standalone script to regression-test your trace functions against production data with one command. The script maps pipeline names to their replay functions, accepts CLI flags, and prints a side-by-side comparison with delta summaries. 
#!/usr/bin/env ruby
# Replay production traces through instrumented functions.
#
# Uses Simforge.client.replay to fetch real traces and re-run them
# through the current code, creating a test run for side-by-side comparison.
#
# Usage:
#   ruby scripts/replay.rb <pipeline>
#   ruby scripts/replay.rb <pipeline> --limit 20
#   ruby scripts/replay.rb <pipeline> --trace-ids id1,id2

require "dotenv/load"
require_relative "../config/initializers/simforge"
require_relative "../services/extraction_pipeline"
require_relative "../services/search_pipeline"

FUNCTIONS = {
  "extraction" => "my-extraction-pipeline",
  "search" => "my-search-pipeline",
}.freeze

pipeline = ARGV[0]

unless pipeline && FUNCTIONS.key?(pipeline)
  warn "Usage: ruby scripts/replay.rb <#{FUNCTIONS.keys.join('|')}> [--limit N] [--trace-ids id1,id2]"
  exit 1
end

limit = 10
trace_ids = nil

ARGV[1..].each_with_index do |arg, i|
  case arg
  when "--limit"
    limit = ARGV[i + 2].to_i
  when "--trace-ids"
    trace_ids = ARGV[i + 2].split(",").map(&:strip)
  end
end

# Each pipeline gets its own replay method — replay deserializes
# historical inputs, so if the method signature doesn't match the
# raw input shape, reshape the arguments in a thin wrapper here.

def replay_extraction(limit:, trace_ids:)
  service = ExtractionPipeline.new
  Simforge.client.replay(
    service, :extract_memories,
    trace_function_key: FUNCTIONS["extraction"],
    limit:, trace_ids:
  )
end

def replay_search(limit:, trace_ids:)
  service = SearchPipeline.new
  Simforge.client.replay(
    service, :search_documents,
    trace_function_key: FUNCTIONS["search"],
    limit:, trace_ids:
  )
end

REPLAY_FNS = {
  "extraction" => method(:replay_extraction),
  "search" => method(:replay_search),
}.freeze

function_key = FUNCTIONS[pipeline]
puts "[replay] Replaying #{trace_ids&.length || limit} traces from \"#{function_key}\"...\n"

result = REPLAY_FNS[pipeline].call(limit:, trace_ids:)
puts "Test run: #{result[:test_run_url]}\n"

changed = same = errors = 0
result[:items].each do |item|
  raw_input = item[:input] || []
  label = (raw_input.first || "unknown").to_s[0, 80]

  if item[:error]
    puts "  ✗ \"#{label}\""
    puts "    Error: #{item[:error]}"
    errors += 1
  else
    orig_count = item[:original_output].is_a?(Array) ? item[:original_output].length : 0
    new_count = item[:result].is_a?(Array) ? item[:result].length : 0
    delta = new_count - orig_count
    delta_str = delta == 0 ? "=" : delta > 0 ? "+#{delta}" : delta.to_s

    puts "  #{delta == 0 ? '=' : 'Δ'} \"#{label}\""
    puts "    #{orig_count} → #{new_count} (#{delta_str})"
    delta == 0 ? same += 1 : changed += 1
  end
end

puts "\n─── Summary ───"
puts "  Pipeline: #{pipeline}"
puts "  Replayed: #{result[:items].length}"
puts "  Same:     #{same}"
puts "  Changed:  #{changed}"
puts "  Errors:   #{errors}" if errors > 0
puts "\n  #{result[:test_run_url]}"
Adapt the imports, pipeline names, and per-pipeline replay methods to match your project’s instrumented workflows.

​
Native Functions

Coming soon.

​
Sim Runners

Coming soon.