Coder Eval
Python framework that benchmarks AI coding agents against your own tasks in sandboxes, not fixed leaderboards
🔗 Visit Coder Eval
Description
Public benchmarks like SWE-bench tell you how good a coding agent is at someone else's tasks. Coder Eval answers a more useful question for a real team: how good is this agent, this prompt, or this model actually at YOUR tasks — the ones your CI pipeline cares about.
Coder Eval is a Python framework for testing and benchmarking AI coding agents (Claude Code, Codex, Google Gemini, OpenCode) against declarative YAML tasks run in sandboxed environments. It scores the actual files and commands an agent produces using weighted criteria — file checks, code similarity, and LLM-graded rubrics. It integrates into CI via GitHub Actions and a Claude Code plugin, supports A/B experiments for comparing models or prompts, and tracks token counts and cost per tool call. Typical uses include gating CI on agent output quality and validating that a coding-agent skill actually works.
💬 Our review
The short version: if you're building anything on top of a coding agent — a CLI tool, an internal skill, an agent-powered CI gate — Coder Eval answers "does this actually work" with real scoring instead of vibes.
Compared to fixed benchmarks like SWE-bench or SkillsBench, which rank models on a canonical dataset, Coder Eval is task-specific: you write your own YAML tasks and get scored against your own bar, which is more useful for CI gating but doesn't give you a comparable industry-wide score. Its sandbox is a tempdir by default (not a real security boundary — a container driver is needed for untrusted tasks), and it requires Python 3.13+ and your own model API credentials. Free and well-documented; worth adopting if you're shipping agent-powered features and need regression protection in CI.
📊 Global score
🤖 AI-enriched data
Gratuit, licence Apache 2.0. Installation via pip. Nécessite ses propres clés API de modèles (Anthropic, Bedrock, Gemini).
Pros
Exécution en sandbox avec environnements isolés et limites de ressources
Score continu pondéré combinant vérif de fichiers, similarité de code et notation par LLM
Multi-agents (Claude Code, Codex, Gemini, OpenCode) avec couche d'expérimentation A/B
Intégration CI complète : GitHub Actions, rapports JUnit XML, portes pass/fail
Cons
Exige Python 3.13+, une contrainte de version relativement récente
Le sandbox par défaut (tempdir) n'est pas une vraie barrière de sécurité pour du code non fiable
Évaluation spécifique à vos tâches, pas un classement comparable à un benchmark fixe