self-bench
Open-source tool that turns any GitHub repo's real pull requests into a benchmark for testing how good an AI coding agent actually is
🔗 Visit self-bench
Description
Most AI coding-agent benchmarks use artificial, hand-picked test problems that don't look like real work. self-bench takes a different approach: it reads through a GitHub repository's actual merged pull requests and reconstructs them as coding tasks — complete with hidden tests and a reference solution — so you can measure whether an AI agent can genuinely do the kind of work your own codebase actually needs.
The generated benchmarks export into the Harbor evaluation framework, letting teams score AI coding models or agents against tasks pulled straight from real-world commit history rather than synthetic puzzles. It's aimed at teams and researchers evaluating AI coding agents who want evaluation data that reflects their own codebase's actual patterns, not a generic leaderboard.
💬 Our review
The short version: self-bench addresses a real weakness in AI coding benchmarks — most are synthetic and don't reflect a specific codebase's real complexity — by generating evaluation tasks directly from a repo's own merged PR history.
Against generic public coding benchmarks (HumanEval, SWE-bench) that test against a fixed, shared problem set, self-bench's differentiator is customizability: you can point it at your own repository and get tasks that actually resemble the work your team does, which is far more relevant for deciding "will this AI agent actually help on our codebase" than a generic leaderboard score. Being open source (MIT) and free removes any cost barrier to trying it. The catch is it depends entirely on the Harbor framework for actually running evaluations, so it's a piece of a larger pipeline rather than a standalone one-click benchmark tool. Worth it for teams seriously evaluating coding agents against their own repo; overkill if you just want a quick, generic quality signal.
💰 Pricing
📊 Global score
🤖 AI-enriched data
Open source sous licence MIT, gratuit.
Pros
Génère des tâches d'évaluation à partir de vraies PR mergées
Tests cachés + solution de référence pour chaque tâche
S'exporte directement vers le framework Harbor
Gratuit et open source
Cons
Dépend entièrement du framework Harbor pour l'exécution
Nécessite un accès à l'historique de PR d'un repo pour être utile
Projet encore jeune (28 étoiles GitHub)