Fixed leaderboards like SWE-bench are useful for comparing models in the abstract, but they don't tell you how an AI coding agent performs on your codebase, with your conventions and your actual tasks. self-bench and Coder Eval both exist to close that gap — but they build the benchmark from opposite starting points.
The short version: self-bench mines your repo's own real, merged pull requests and turns them into evaluation tasks with hidden tests. Coder Eval instead runs the agent against tasks you define, in isolated sandboxes, scored by a weighted mix of file checks, code similarity, and LLM grading — built to run continuously in CI.
self-bench — your merged PRs become the benchmark
self-bench is an open-source tool that turns any GitHub repo's real, already-merged pull requests into a benchmark for testing how good an AI coding agent actually is on that specific codebase. Each generated task ships with hidden tests plus a reference solution (the actual merged PR), and results export directly into the Harbor framework for execution.
Forces: evaluation tasks come from work that actually happened, not synthetic examples; hidden tests plus a real reference solution for each task; free, MIT-licensed.
Limites: depends entirely on the Harbor framework to actually run anything; needs a repo with enough merged PR history to be useful — a young or small repo won't generate much; still an early project (28 GitHub stars).
Coder Eval — define your own tasks, run them in CI
Coder Eval is a Python framework that benchmarks AI coding agents against tasks you define yourself, in sandboxed environments with resource limits — rather than a fixed leaderboard. It scores runs with a continuous weighted score combining file-state verification, code similarity, and LLM-based grading, supports multiple agents (Claude Code, Codex, Gemini, OpenCode) with an A/B experimentation layer, and integrates fully into CI with GitHub Actions and JUnit XML reports.
Forces: sandboxed, resource-limited execution; nuanced scoring instead of pass/fail only; works across multiple agent CLIs with built-in A/B testing; full CI integration with pass/fail gates.
Limites: requires Python 3.13+, a fairly recent constraint; the default sandbox (a tempdir) isn't a real security boundary for untrusted code; the evaluation is specific to the tasks you write — not comparable to a fixed public benchmark. Free, Apache 2.0, but you supply your own model API keys (Anthropic, Bedrock, Gemini).
| self-bench | Coder Eval | |
|---|---|---|
| Task source | Your repo's real merged PRs | Tasks you define by hand |
| Scoring | Hidden tests vs reference solution | Weighted: file checks + similarity + LLM grading |
| Runs where | Via the Harbor framework | Sandboxed, natively in CI |
| Multi-agent support | Whatever Harbor supports | Claude Code, Codex, Gemini, OpenCode + A/B |
| Setup effort | Low — mines existing PR history | Higher — you write the tasks |
| Price | Free (MIT) | Free (Apache 2.0) + your API costs |
Pick self-bench if your repo already has a solid history of merged PRs and you want a benchmark with near-zero setup — it writes the tasks for you from work you've already done. Pick Coder Eval if you want an ongoing, CI-gated regression check across multiple agent CLIs, and you're willing to write the task definitions yourself in exchange for finer-grained scoring and real sandboxing. If you're evaluating one agent once, self-bench is faster to try; if you're comparing agents continuously as your codebase evolves, Coder Eval's CI integration is built for exactly that.