Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.definable.ai/llms.txt

Use this file to discover all available pages before exploring further.

The replay module lets you look inside completed agent runs, compare two runs side-by-side, and re-execute a past run with different configuration — all without touching trace files manually.

​
Quick Example

from definable.agent import Agent
from definable.model.openai import OpenAIChat

agent = Agent(model=OpenAIChat(id="gpt-4o"))

# Run and inspect
output = agent.run("Summarize the Q4 report.")
replay = agent.replay(run_output=output)

print(replay.model)           # "gpt-4o"
print(replay.tokens.total_tokens)  # 1234
print(replay.cost)            # 0.0042
print(replay.tool_calls)      # [ToolCallRecord(...), ...]
print(replay.status)          # "completed"

​
Inspecting a Run

Build a Replay from any of these sources: 
# From a just-completed run
output = agent.run("Hello")
replay = agent.replay(run_output=output)

​
Replay Fields

FieldTypeDescription
run_idstrRun identifier
session_idstrSession identifier
agent_namestrAgent name
modelstrModel used
inputAnyOriginal input
contentAnyFinal output content
messagesListFull conversation messages
tool_callsList[ToolCallRecord]All tool executions with timing
tokensReplayTokensAggregated token usage
costOptional[float]Total cost in USD
durationOptional[float]Total duration in milliseconds
stepsList[ReplayStep]Step-by-step timeline
knowledge_retrievalsList[KnowledgeRetrievalRecord]RAG retrieval records
memory_recallsList[MemoryRecallRecord]Memory recall records
statusstr"completed", "error", or "cancelled"
errorOptional[str]Error message (if status is "error")

​
ReplayTokens

FieldTypeDescription
input_tokensintPrompt tokens
output_tokensintCompletion tokens
total_tokensintTotal tokens
reasoning_tokensintReasoning/thinking tokens
cache_read_tokensintTokens read from cache
cache_write_tokensintTokens written to cache

​
ToolCallRecord

FieldTypeDescription
tool_namestrTool function name
tool_argsDictArguments passed
resultOptional[str]Tool return value
errorOptional[bool]Whether the call errored
duration_msOptional[float]Execution time

​
Comparing Runs

Compare two runs to see what changed: 
output_a = agent.run("Summarize the report.")
output_b = agent.run("Summarize the report.")

diff = agent.compare(output_a, output_b)

print(diff.token_diff)        # -150  (b used 150 fewer tokens)
print(diff.cost_diff)         # -0.0005
print(diff.content_diff)      # Unified diff string
print(diff.tool_calls_diff.added)    # Tools in b but not a
print(diff.tool_calls_diff.removed)  # Tools in a but not b
print(diff.tool_calls_diff.common)   # Count of matching tools

​
ReplayComparison Fields

FieldTypeDescription
originalReplayFirst run
replayedReplaySecond run
content_diffOptional[str]Unified diff of output content
cost_diffOptional[float]Cost difference (b − a)
token_diffintToken difference (b − a)
duration_diffOptional[float]Duration difference (b − a)
tool_calls_diffToolCallsDiffAdded, removed, and common tool calls
You can also use compare_runs directly: 
from definable.agent.replay import compare_runs

diff = compare_runs(output_a, output_b)  # Accepts Replay or RunOutput

​
Re-Executing with Overrides

Pass override arguments to replay() to re-run the same input with different configuration. This returns a new RunOutput instead of a Replay: 
# Re-execute with a different model
new_output = agent.replay(
  run_output=output,
  model=OpenAIChat(id="gpt-4o-mini"),
)

# Re-execute with different instructions and tools
new_output = agent.replay(
  trace_file="./traces/run.jsonl",
  run_id="abc123",
  instructions="Be more concise.",
  tools=[new_tool],
)

# Compare original vs re-execution
diff = agent.compare(output, new_output)
print(diff.cost_diff)  # How much cheaper was gpt-4o-mini?
Re-execution makes a live API call. The original input is extracted from the replay and sent to the model with your overrides applied.

​
Async API

All replay methods have async equivalents: 
replay = await agent.areplay(run_output=output)
new_output = await agent.areplay(run_output=output, model=new_model)
compare() and compare_runs() are synchronous (no I/O involved).

​
Construction Methods

MethodInputDescription
Replay.from_run_output(run_output)RunOutputBuild from a just-completed run
Replay.from_events(events, run_id=)List[Event]Build from deserialized trace events
Replay.from_trace_file(path, run_id=)str | PathBuild from a JSONL trace file
agent.replay(run_output=)RunOutputConvenience wrapper (returns Replay or RunOutput)
agent.replay(trace_file=)strLoad from trace file
agent.replay(events=)List[Event]Load from events

Tracing

Configure JSONL trace export for replay sources.

Cost Tracking

Understand token metrics and pricing used in comparisons.