How to benchmark a local coding agent honestly

A coding-agent benchmark should answer one question: can the team accept this change into the repository at a known cost and risk? Tokens per second and a green public test cannot answer it alone.

For our Qwen3.8-27B evaluation, we wrote ordinary work requests rather than shortened benchmark puzzles. The agent received a clean, unfamiliar copy of the project, normal tools and enough time to finish naturally. A separate process then checked hidden behaviours, compilation, change scope and telemetry.

One complete fixture: CSV export

The agent saw this exact request:

Customers with several tags are showing up more than once in our CSV order export. Can you fix it? The export code is under src. Please keep the existing columns and add a regression test.

A short README in the repository defined four stable columns and the | separator for multiple tags. The agent could read public tests but not the three hidden checks. After the run, verification executed:

npm test
node --test hidden/export-orders.hidden.test.js

The hidden file lived outside the agent worktree. It checked one row per order, stable columns and the documented separator.

Only 5.55 seconds separated the fastest rejected run from the selected run. Without the hidden contract, OpenCode would look like the winner. This fixture contains the complete method: natural request, documentation as product behaviour, clean repository, evaluator-only check and a decision that puts compatibility first.

Five layers of one useful cell

1. A task that resembles work

Write the request as a colleague would write an issue: describe the problem, expected behaviour and constraints without giving away the repair path. Our suite included CSV export, pagination, user import and a frontend repair over a large synthetic dataset.

We did not impose an artificial tool-call limit. A long agent could finish or reach a genuine external blocker. Narrow diagnostic ceilings were labelled as diagnostics and kept out of final quality claims.

2. A clean, unfamiliar repository

Every comparison cell should start at the same commit in its own worktree. The agent must not benefit from edits or caches left by an earlier run. Record the state before and after, including files changed as a side effect of testing.

That last detail matters. Running Python tests in one fixture changed tracked __pycache__ files. The implementation could be correct while the workspace remained dirty. That belongs in scope discipline.

3. Public and hidden checks

Public tests tell the agent what the repository already checks. Hidden checks test a contract that the agent should discover in code or documentation. In our CSV export task, every compared result passed three public tests. Qwen Code at low effort and OpenCode both missed the separator contract in the README, passing only one of three hidden checks.

A hidden test should cover user-visible behaviour, backward compatibility, validation or security. It should not reward an evaluator's arbitrary style preference.

4. A quality rubric

Our long acceptance rubric separated correctness, tests, compatibility, scope, verification and documentation. Shorter tasks recorded public and hidden checks plus a written judgment.

The core rule is simple: efficiency becomes a tie-breaker only after quality. A fast cell that violates a documented contract ranks below a slower correct cell.

5. Whole-task telemetry

Collect wall time, peak VRAM, tool calls, output tokens, context compactions and termination reason. One run does not create a distribution. A single cell can select the next profile to use, but it cannot support a reliability guarantee.

Why the review must be independent

The agent is an interested party. Its final report can omit a defect, treat a textual error as success or describe a property that the code does not provide.

In one long task, Codex implemented a fallback and described it as deterministic. Independent review tested channels and pointers that its regression tests omitted. Go's %v format included a memory address, so output varied between runs. Public checks passed, but the documentation claim was false. The score fell to 87/100.

In the frontend task, Qwen Code's geometry checks found no whole-row overlap. A screenshot still exposed a collision between two elements inside a row. The automated check had asked too narrow a question. An image and a second reviewer asked the right one.

A minimum protocol for a team

Keep failed launches too. A wrong API-key path, false loop detector or fixed 8k output limit is a profile defect even when the model could have completed the task. Removing those cells produces a setup that only works in a demo.

A company evaluation is different from a public leaderboard

SWE-bench and similar public results help shortlist candidates. They do not know that your team must preserve a CSV dialect, avoid a broad refactor or operate in a particular Polish regulatory process. A company set should contain examples from the real workflow, anonymised or made synthetic where necessary.

Each example needs an expected result and a human-escalation condition. That set can then compare Qwen with an API model, evaluate a quantized variant and gate the next model upgrade.

The paid work at this stage

Syntalith can prepare the evaluation set, run the comparison and deliver raw results, an architecture decision and a fixed implementation quote. When a team wants to own the method, the AI-Native course uses its repository and tools without tying the curriculum to one vendor.

For a deployment decision, the free process scan first establishes what should be measured. An AI process audit produces a portable document the company may also hand to another supplier.

See the method applied in the Qwen3.8-27B home-PC results.