GamePhanes
A benchmark suite that tests AI coding agents on Godot games instead of static code snippets — agents must inspect, run, observe, and repair playable software, not just generate a function.
🔗 Visit GamePhanes
Description
Most AI coding benchmarks ask a model to write a function and check if the output matches an expected answer — useful, but it doesn't test whether the AI can work with software that actually *runs* and behaves unpredictably. GamePhanes tests that harder skill by having agents fix and modify real, playable games.
Built on Godot, GamePhanes gives agents isolated terminal workspaces and a six-step evaluation loop — inspect, edit, run, observe, repair, verify — across 8 published tasks spanning 2D/3D combat, platformers, physics puzzles, and tower defense, plus 6 playable reference demos. It's Harbor-compatible, ships with Docker-based standardized task structure, reference implementations with test harnesses, and oracle solutions, and is fully open source (MIT) with a community contribution model via GitHub PRs.
💬 Our review
The short version: GamePhanes fills a real gap — most agent benchmarks (SWE-bench, HumanEval) test static code correctness, not an agent's ability to observe runtime behavior and iteratively repair something dynamic, which is closer to how software actually breaks in production.
It's not a general-purpose tool but a research/eval benchmark, so the right comparison isn't a commercial product but other agent-eval suites like SWE-bench, and there it's differentiated by testing genuinely interactive systems rather than isolated functions. At 191 stars and 27 commits it's a young, community-driven project rather than an established standard, so its long-term relevance depends on adoption by model labs and researchers. Worth using if you're evaluating coding agents on dynamic, stateful tasks; not relevant if you just need a general benchmark for text-based code generation, where SWE-bench remains the more recognized reference.
💰 Pricing
📊 Global score
🤖 AI-enriched data
Gratuit, licence MIT, aucune offre commerciale.
Pros
Teste des agents IA sur du logiciel interactif réel, pas juste du code statique
8 tâches Godot publiées + 6 démos jouables de référence
Compatible Harbor, structure Docker standardisée
Open source (MIT), modèle de contribution communautaire
Cons
Projet jeune (191 étoiles), pas encore un standard établi
Benchmark de recherche, pas un outil produit à usage direct
Adoption future dépendante des labs et chercheurs