Kitaru

What it is

Kitaru is a tool designed to capture and replay agent interactions, specifically turning "agent failures into regression tests." It records every call within a session, including model and tool calls, allowing these real traces to be replayed against updated code. This enables side-by-side comparison of different runs or versions of agents, showing changes in outcomes, costs, or identical outputs.

Best for

Kitaru is best for developers and teams working with AI agents that involve multiple steps, tool calls, or model interactions, and who need to systematically test and compare agent behavior across different configurations or code changes. The source specifically mentions its utility for "typed agents where you want schema validation on every step," "multi-agent runs with handoffs," "graph agents," and "TypeScript agents."

Key strengths

• Detailed Session Recording: Kitaru captures every call within an agent's run, including model and tool calls, enabling detailed inspection and replay.
• Regression Testing: It facilitates turning real agent failures into repeatable regression tests by replaying recorded sessions against new code or configurations.
• Side-by-Side Comparison: The tool provides a direct comparison of agent runs, highlighting differences in failing sessions, costs, or identical outputs between a baseline and a candidate.
• Immutable Cohorts: Sessions can be grouped into "immutable" cohorts, ensuring consistent testing sets where results retain their meaning over time.

Watch-outs

• The source does not explicitly define what constitutes an "agent failure," nor does it detail how Kitaru specifically identifies or flags these failures beyond showing "failing" sessions.
• Certain integrations have noted trade-offs; for instance, the PydanticAI adapter adds a Pydantic dependency and per-call overhead, while the OpenAI Agents adapter is tied to OpenAI-hosted models and their rate limits.
• Replay depth with LangGraph depends on the graph's construction, and some integrations like Mastra and Vercel AI SDK currently support only non-streaming calls.
• The source does not discuss infrastructure requirements for Kitaru itself, only how it wraps existing agent harnesses.