Enterprise software creates value when the implementation reflects what the customer actually needs. That depends on hundreds of connected decisions, from discovery and scope to the designs and handoffs that guide delivery.
Building for implementation teams has shown us how easily that context fragments. A revised requirement can leave a design out of date, a delivery story unsupported, or an approval unresolved. Reliable AI has to carry the decision through every affected artifact.
Today, we’re introducing ImplementationBench, developed with General Context Labs. Its 100 synthetic tasks evaluate how agents reason over project history, preserve requirements, and produce consistent, evidence-backed deliverables. We’re releasing the evaluation to make progress on implementation work inspectable and reproducible.
Comparing agent systems
ImplementationBench compares task-specific Auctor workflow configurations with model-based reference agents. Every system receives the same project evidence, instructions, tool interfaces, and resource limits. Select a workflow to inspect the corresponding configuration.
Illustrative results. Scores are invented for this mock-up.
| Rank | Agent system |
|---|
Which workflow configurations are evaluated?
The Auctor suite uses a fixed configuration for each task family. These benchmark configurations organize implementation work across discovery, scoping, change impact, design, and handoffs. They are benchmark entries, and the aggregate includes every task family.
| Workflow | Configuration | Tasks |
|---|---|---|
| Discovery & requirements | Auctor · Discovery & requirements | 25 |
| Scoping & SOWs | Auctor · Scoping & SOWs | 25 |
| Change impact | Auctor · Change impact | 20 |
| Solution design | Auctor · Solution design | 15 |
| Delivery handoffs | Auctor · Delivery handoffs | 15 |
Model-based reference agents use workflow-specific instructions and the same evidence and tools. Human review is represented by explicit approval requirements in the task.
Scoring and reference configurations
Each task has five conversation seeds. Task completion is the share of runs that pass every required check. A task’s rubric score is the mean of its context, policy, records, and communication scores across those runs. Overall scores average across tasks, so workflow groups contribute in proportion to their task counts.
- Context
- Find authoritative decisions and resolve missing or conflicting project evidence.
- Policy
- Respect project scope, approval boundaries, and the supplied implementation playbook.
- Records
- Preserve requirements, source links, artifact versions, and cross-document consistency.
- Communication
- Explain changes, assumptions, open questions, and handoffs with supporting evidence.
The company suite dispatches each task to a workflow-specific configuration through a fixed manifest. Every entrant receives the same workflow label, initiating event, initial state, tools, business policies, and simulated counterparties. Configuration selection is fixed before scoring. Model baselines pair each named model with a reference tool agent and workflow-specific instructions. All agent and deterministic workflow nodes share a per-run budget of 50 environment tool calls and 32,000 generated tokens. Simulator and grader computation is tracked separately. Overall results average the selected configurations across the full task set. These system-level comparisons include model, prompt, routing, and orchestration differences. This track measures project reasoning, artifact production, and approval handling. Configuration execution belongs to a separate platform-sandbox track.
Each result includes model snapshot IDs, agent and workflow versions, the routing manifest, prompts, resource budgets, simulator and grader versions, and execution traces.
Context has to survive the work
A discovery summary can be accurate while the scope built from it is wrong. A design can read well while omitting a critical constraint. ImplementationBench evaluates the chain from evidence to requirements to deliverables, including the relationships between them.
Our research focus is whether an agent can carry context forward as a project changes. The evaluation rewards supported decisions, complete requirements, and consistent outputs that an implementation team can inspect and act on.
Built around a project
Each task runs in an isolated synthetic implementation project with meeting transcripts, messages, versioned requirements, SOWs, solution designs, delivery tickets, and approval rules. A simulated tool layer supports search, artifact editing, and traceability checks.
The initial task set centers on CRM implementation projects. The environment supplies the relevant platform constraints and team playbooks so requirements and deliverables can be checked against an explicit project context.
Inside a task
Consider an approval rule that changes after discovery. The agent needs to understand the new decision, find what it affects, and carry the change through the implementation record.
“Orders above $50,000 now need two approvers. Update the design and delivery stories, and tell us what this changes in scope.”
Signed SOW · latest workshop decision · prior approval design · delivery backlog · change-control policy
Identify the authoritative decision, trace affected requirements, revise the design and stories, and route the scope change for approval.
Inspect the grading criteria
- Resolve the latest decision against the earlier approval rule and cite its source.
- Identify all affected requirements and dependent artifacts.
- Keep the design, delivery stories, and acceptance criteria consistent.
- Flag scope and approval requirements before treating the change as committed.
Make quality inspectable
Deterministic checks test required fields, artifact versions, links, and cross-document consistency. Expert-informed rubrics assess requirement coverage, treatment of ambiguity, and the support for design decisions. Each scored criterion points to evidence in the sources, output artifacts, or action trace.
Rubric score
The mean of context, policy, records, and communication scores within each task, averaged across tasks.
Task completion
The share of runs that satisfy every required criterion, including source grounding, requirement coverage, artifact consistency, and approval handling.
Evaluation protocol and reproducibility
Every system runs each task with five independent seeds and a fresh project state. Workflow manifests, prompts, model snapshots, environment versions, graders, and traces are recorded with the results. Expert reviewers audit rubric agreement and acceptable alternative solutions. A well-supported design can satisfy the rubric through several valid approaches.
The task set measures project reasoning and artifact production. A configuration change in a live CRM requires a separate execution track with a platform sandbox and explicit validation tests.
Open research
ImplementationBench makes project evidence, task specifications, synthetic environments, scoring rubrics, and reference evaluation code available for research. Versioned tasks and execution traces let the community examine where agents preserve context and where the connection between a decision and its deliverables breaks down.
We invite implementation teams to contribute difficult scenarios, researchers to audit the evaluation, and agent builders to reproduce and improve the baselines. Better implementation AI should be supported by evidence that the whole industry can examine.