Coding Evaluation
Deterministic evaluation for software engineering agents and coding assistants. We test whether generated diffs apply cleanly, compile, pass comprehensive test suites, and preserve downstream system stability.
What We Evaluate
We move far beyond simple HumanEval-style function completion. Our testbeds execute realistic repository-level refactoring tasks in Docker sandboxes: multi-file git patch application, abstract syntax tree validation, full test suite execution (Jest, pytest, cargo test), and static type checking.
The Silent Risk of Plausible Syntax
Code models frequently write syntactically pleasing code that fails when executed against real repository import boundaries or creates subtle downstream regressions in unedited modules. We catch these failures by running full unit, integration, and compilation passes.
| Metric | Target Scope | Description |
|---|---|---|
| Clean Patch Application | DIFF RIGOR | Percentage of generated git diffs that apply cleanly without line-offset or merge conflicts. |
| Compilation & Typecheck Pass | BUILD STAGE | Rate of successful builds under strict compilers (tsc --noEmit, rustc, mypy, go build). |
| Pass@1 Full Test Suite | EXECUTION | Deterministic pass rate on target pull request test assertions on the first attempt. |
| Regression Creation Rate | STABILITY | Rate at which fixing a target bug breaks existing, previously passing tests in the repository. |
| Refactoring Task Category | Tasks | Compilation Pass | Pass@1 Suite | Downstream Regression |
|---|---|---|---|---|
| Single-File Bug Fixes | 150 | 96.0% | 74.0% | 2.0% |
| Multi-Module Interface Upgrades | 80 | 62.0% | 44.0% | 14.0% |
| Schema Migration Scripts | 40 | 82.5% | 65.0% | 7.5% |
Benchmark Your AI Coding Assistant
Test coding agents and automated refactoring tools on realistic repository tasks in isolated sandboxes under mutual NDA.