How to benchmark a local coding agent honestly

A benchmark is a repeatable trial in which an agent receives a task and the team compares the result with agreed criteria and cost. A coding-agent benchmark should answer one question: can the team accept this change into the repository at a known cost and risk? Generation speed and a green public test cannot answer it alone.

For our Qwen3.8-27B evaluation, we wrote work requests like ordinary engineering issues. The agent received a clean, unfamiliar copy of the project, standard tools and enough time to finish or report a genuine blocker. A harness, meaning a program that runs the trial and records its measurements, then checked hidden behaviour, compilation, change scope and the full run.

One complete example: CSV export

The agent saw this exact request:

Customers with several tags are showing up more than once in our CSV order export. Can you fix it? The export code is under src. Please keep the existing columns and add a regression test.

A short README in the repository defined four stable columns and the | separator for multiple tags. The agent could read public tests but not the three hidden checks. After the run, verification executed:

npm test
node --test hidden/export-orders.hidden.test.js

The hidden file lived outside the agent worktree. It checked one row per order, stable columns and the documented separator. low and medium are Qwen Code effort settings; the quality result comes from the public and hidden checks. Each fraction is the number of checks passed out of the total, and time is the elapsed duration of the full run.

Only 5.55 seconds separated the fastest rejected run from the accepted run. Without the hidden contract, OpenCode would look like the winner. This example contains the complete method: an issue-style request, documentation treated as product behaviour, a clean repository, an evaluator-only check and a decision that puts compatibility first.

Five parts of a useful test run

1. A request written like an issue

Write the request as a colleague would write an issue: describe the problem, expected behaviour and constraints without giving away the repair path. Our suite included CSV export, pagination, user import and a frontend repair over a large synthetic dataset.

We did not impose an artificial tool-call limit. A long run could finish or reach a genuine external blocker. Any limits used only for troubleshooting were labelled clearly and excluded from final quality claims.

2. A clean, unfamiliar repository

Every comparison run should start at the same commit in its own worktree. The agent must not benefit from edits or caches left by an earlier run. Record the state before and after, including files changed as a side effect of testing.

That last detail matters. Running Python tests in one example changed tracked __pycache__ files. The implementation could be correct while the workspace remained dirty. That belongs in scope discipline.

3. Public and hidden checks

Public tests tell the agent what the repository already checks. Hidden checks test a contract that the agent should discover in code or documentation. In our CSV export task, every compared result passed three public tests. Qwen Code at low effort and OpenCode both missed the separator contract in the README, passing only one of three hidden checks.

A hidden test should cover user-visible behaviour, backward compatibility, validation or security. It should not reward an evaluator's arbitrary style preference.

4. A quality criteria sheet

Our long acceptance criteria sheet separated correctness, tests, compatibility, scope, verification and documentation. Shorter tasks recorded public and hidden checks plus a written judgment.

The core rule is simple: efficiency becomes a tie-breaker only after quality. A fast run that violates a documented requirement ranks below a slower correct run.

5. Whole-task telemetry

Collect elapsed time, peak VRAM, tool calls, output tokens, context compactions and termination reason. One run does not create a distribution. A single run can select the next configuration to test, but it cannot support a reliability guarantee.

Why the review must be independent

The agent is an interested party. Its final report can omit a defect, treat a textual error as success or describe a property that the code does not provide.

In one long task, local Qwen running through the Codex harness implemented a fallback and described it as deterministic. A separate Codex evaluation tested channels and pointers that the patch's regression tests omitted. Go's %v format included a memory address, so output varied between runs. Public checks passed, but the documentation claim was false. Codex evaluated the recorded result at 87/100 under Syntalith's criteria sheet, deducting 13 points for that defect. The 100 points cover correctness (40), regression tests (20), compatibility (15), scope discipline (10), verification (10) and documentation (5). An external auditor did not review this result.

In the frontend task, Qwen Code's geometry checks found no whole-row overlap. A screenshot still exposed a collision between two elements inside a row. The automated check had asked too narrow a question. An image and a second reviewer asked the right one.

A minimum protocol for a team

Keep failed launches too. A wrong API-key path, false loop detector or fixed 8k output limit is a configuration defect even when the model could have completed the task. Omitting those failed attempts produces a setup that only works in a demo.

A company evaluation is different from a public leaderboard

SWE-bench and similar public results help shortlist candidates. They do not know that your team must preserve a CSV dialect, avoid a broad refactor or operate in a particular Polish regulatory process. A company set should contain examples from the real workflow, anonymised or made synthetic where necessary.

Each example needs an expected result and a human-escalation condition. That set can then compare Qwen with an API model, evaluate a quantized variant and gate the next model upgrade.

What Syntalith delivers at this stage

Syntalith prepares the task list and acceptance criteria, runs the comparison on clean repository copies and delivers a results table with timings, public and hidden checks, changed files, tool logs and rejection reasons. We add a configuration recommendation and a fixed implementation quote. When a team wants to own the method, the AI-Native course uses its repository and tools without tying the curriculum to one vendor.

For a deployment decision, the free process scan first establishes which tasks and risks should be measured. An AI process audit produces a document with the tasks, results, configuration and recommendations that the company may also hand to another supplier.

See the method applied in the Qwen3.8-27B home-PC results.