The requirements have consequences
A feature must fit existing behavior, permissions and integrations. We agree completion checks before the run, including what must keep working.
DEVELOPMENT EXPERIENCE, EVALUATION EVIDENCE
Evaluate your AI tool in the work developers actually do.
Acadify AI Lab is part of a software development firm. We scope evaluations where our developers use your tool on approved delivery tasks, then inspect the result, the review effort and the corrections it took to finish.
WHY THE DEVELOPMENT CONTEXT MATTERS
Useful code has to survive the rest of the project.
The correction work is visible
We record meaningful developer interventions: missing context, rejected suggestions, rewritten code and repeated attempts. A plausible first answer is only one part of the evidence.
The handoff can be inspected
We connect findings to task records, permitted artifacts and test outcomes. Tool and model teams can trace a failure and decide what to change.
HOW AN ENGAGEMENT WORKS
Agree the task, run the work, review the evidence.
- 1. Choose an approved task setScope bug fixes, features, tests, refactors or integrations. Agree client authorization, tool access and permitted data before any capture. Availability depends on permission and project fit.
- 2. Define the comparisonRecord the starting revision, environment, acceptance checks and tool version. Agree a baseline and allocation of comparable tasks where a comparison is needed.
- 3. Use the tool with a developerCapture agreed task metadata, prompts, responses and interventions within the permitted boundary. Exclude secrets, personal records and unrelated client material.
- 4. Check the complete outcomeRun acceptance and regression checks, inspect the diff and separate model failures from environment failures. Record developer review and rework effort when timing is in scope.
- 5. Review findings and any reuseDeliver task-level findings and limitations. Dataset preparation or training reuse proceeds only under a separate approved specification and export review.
WHAT YOU RECEIVE
Evidence your product and model teams can use.
An evaluation record
Task definitions, run conditions, acceptance and regression outcomes, intervention notes, reviewer findings and references to the permitted evidence.
Download the record template →A decision-focused report
Failure patterns, review burden, recommended improvements and a rerun plan. When authorized, a separate dataset specification defines provenance, annotations and permitted reuse.
Download the engagement brief →These downloads are blank planning templates, not completed client evaluations. Public sample reports remain clearly labelled demonstrations.
CLIENT WORK IS NOT AN AUTOMATIC TRAINING DATASET
Permission comes before collection. Reuse is a separate decision.
Owning a development engagement does not automatically grant rights to send its code to an AI provider, share logs or train a model. We agree client permission, relevant rights, approved tools, capture scope, access, retention and deletion before the evaluation starts.
Training exports require explicit approval for that use, a rights and sensitive-data review, and an agreed recipient and purpose. Redaction alone does not grant reuse rights. Held-out evaluation tasks stay separate from training data.
If a client task cannot be used, we can scope a representative synthetic, internal or appropriately licensed open-source task instead, labelled by its actual source. No private client records are needed in the initial enquiry.
Read our responsible AI approach →WHAT THIS CAN ESTABLISH
Results belong to the tasks and conditions tested.
Developer-led evaluation complements repeatable benchmarks. It can reveal context, review and delivery problems that a narrow task score misses. It does not establish universal model quality or guarantee a productivity gain. We document sample size, task selection, reviewer judgments and comparison limits.
Related research: SWE-bench’s repository-based evaluation and METR’s study of developers working in familiar repositories. These inform the method; they are not Acadify results or endorsements.
START WITH ONE WORKFLOW
What do you need your tool to do better?
Tell us about the tool, target developers, task types and the product decision you need evidence for. We will discuss feasibility, access, deliverables, timing and commercial terms before work begins.
Discuss your evaluation