Qwen Code in practice: CSV, pagination and import

Local Qwen3.8-27B through Qwen Code handled CSV export and a pagination fix well at medium effort. In the user-import task it passed 12 checks visible in the repository, but only four of five hidden checks. It accepted a row with a blank name or role. In a real import, that missing validation could admit a malformed account while every visible test stays green.

We tested four ordinary changes in clean copies of small projects. The requests were written like issue reports, standard tools were available and no artificial time ceiling was imposed.

Here, a “test” checks one condition. A public test was visible to the agent in the repository; a hidden test checked an additional condition outside its view. We record elapsed time and patch scope because green tests alone do not describe cost or safety. For recorded tasks with a score, OpenAI Codex evaluated the change against Syntalith criteria for correctness, regression tests, compatibility, scope discipline, verification and documentation.

Qwen Code results at medium effort

3/3 means that all three conditions were met. 4/5 means that one condition was missed and does not measure import reliability as 80%. Every row is one run, so the times are measurements of those runs rather than response-time promises or success probabilities.

The requests the agent received

The requests did not prescribe an implementation:

Export: Customers with several tags are showing up more than once in the CSV order export. Fix it under src, keep the existing columns and add a regression test.

Pagination: Filtering works on the first page, but after “load more” entries from other owners appear and matching entries are skipped. Track down the bug in src/activity-store.js and fix it with tests.

Import: Add CSV import for name, email and role. Blank lines are fine, but a bad row or duplicate email must reject the whole file without a partial import. Add tests and update README.

The follow-up changed the contract. An existing email had to update name and role, including when a duplicate appeared earlier in the same file. The operation still had to be atomic and return created and updated counts.

These prompts resemble real tickets. The agent had to find the CSV separator and cursor contract in the repository. From the data model and the phrase “bad rows”, it should have inferred that a blank name or role was invalid. The hidden test checked that missing inference.

CSV export: medium effort changed the answer

The first task was a small export bug. Public tests did not cover the separator specified in the README. Qwen Code at low effort passed all three public tests but only one of three hidden tests. At medium, it found the written contract and passed every test in 38.24 seconds.

This does not show that medium always wins. Low can suit a mechanical change with cheap verification. In this run, shorter reasoning saved 3.47 seconds and left the requested behaviour wrong.

Pagination: compatibility is part of correctness

Qwen Code medium, Codex medium and OpenCode medium passed all available pagination tests. Their patch scope differed.

• Qwen Code medium made the smallest repair that preserved the existing contract.
• Qwen Code low changed the cursor contract despite passing the tests.
• Codex produced a broader rewrite that was backward-incompatible.
• OpenCode made a minimal fix and then inspected unnecessary .git files.

Hidden tests cannot replace a diff review. A change can be logically correct and still cover too much ground to accept safely.

Import: green public tests missed validation

Qwen Code built a working import, documented it and passed the visible suite. The fifth hidden test supplied an empty name or role. The implementation accepted it.

Codex passed 5/5 hidden tests on the same task in 212 seconds. OpenCode also passed 5/5 in 89.04 seconds. Qwen Code did not produce the strongest result on this task. A fair choice of tool needs task-level evidence, patch scope and review effort rather than one brand or an average across unlike work.

The second Qwen Code attempt expanded the import with the requested upsert in the same session, but it did not repair the earlier validation bug. Teams see the same pattern when a later ticket leaves an unstated requirement untouched.

How to read an agent's output

A reviewer should ask five separate questions:

1. Does it build, and do public tests pass?
2. Does it satisfy the contract in documentation?
3. Are difficult inputs validated?
4. Is the change the smallest scope that solves the problem?
5. Does the final report match what was actually verified?

An agent can satisfy four and still produce a change that is not ready to merge. Production use therefore needs repeatable tests and review matched to the impact of failure.

The business metric is a team-accepted change

A coding agent should shorten the path from request to a change the team accepts, while keeping debt and incidents under control. A useful cost model is:

cost per team-accepted change = agent time
  + infrastructure cost
  + human review time
  + expected rework cost

“Agent time” is the run duration, “infrastructure cost” covers hardware and energy, “human review time” is the reviewer's work and “expected rework cost” covers likely repairs after delivery. A slow local model can still pay off on bounded overnight work. A fast model can cost more when a senior engineer must reconstruct every missed contract.

How Syntalith deploys this pattern

We begin with a sample of real tickets, add concrete tests the agent cannot see and run each variant in an isolated repository copy. Then we set tool access, rules for actions that require human approval and escalation rules. The client receives the comparison even when the local model performs worse than the current API.

Teams can learn to build these evaluations in the AI-Native course, using their own repository. For an implementation decision, start with the free process scan.

The full protocol is in how to benchmark a local coding agent.