Kullback
A testing tool that rebuilds an AI agent's entire working environment from its past execution logs, so you can replay exactly what it did and check the outcome was actually correct.
🔗 Visit Kullback
Description
Testing an AI agent is harder than testing regular software, because an agent's "correct" behavior often depends on a whole simulated world — the tools it had access to, the data it saw, the rules it was supposed to follow. Kullback's approach is to build that world automatically from a record of what actually happened, rather than asking a developer to hand-write it.
Kullback is an open-source harness that processes AI agent execution logs to automatically reconstruct the tools, data, and operational rules used during past runs. It ingests traces, rebuilds the operational environment, replays the logs for verification, and produces a verdict based on the final state of the data rather than just reading the conversation transcript. It extracts data records, tool functions, policy rules, simulated users, and task verifiers from the traces, with published results based on real historical retail-transaction traces.
💬 Our review
The short version: Kullback's trace-to-environment reconstruction is a genuinely clever approach to a real problem — verifying agents by outcome rather than by transcript — but it's a young, niche tool with retail-flavored examples that may need adaptation for other domains.
Most agent testing today still leans on generic frameworks like Pytest or Hypothesis, which check that code behaves correctly but have no built-in concept of "replay this agent's exact operational world and check the final state." Kullback fills that specific gap: instead of hand-writing mocks for every tool and data source an agent touches, it reconstructs them from real execution traces, which is both more realistic and less work once you have logs to draw from. The published examples center on retail transactions, so teams in other domains will need to validate the approach transfers before betting production QA on it. Being open source and free, the cost of trying it is just engineering time — worth a pilot for teams already struggling to verify agent behavior beyond eyeballing transcripts.
📊 Global score
🤖 AI-enriched data
Open source, gratuit / auto-hébergé.
Pros
Reconstruit automatiquement l'environnement opérationnel complet à partir de traces historiques
Permet le rejeu déterministe des exécutions d'agent pour vérification
Résultats de mesure inclus sur des traces réelles de transactions retail
Cons
Outil relativement niche, adoption communautaire limitée
Documentation centrée sur un cas d'usage retail spécifique