How we isolated the local Qwen agent experiment

Qwen could write files and run commands, but it worked only in prepared repositories with no secrets or company data. The user's active project remained outside its reach. Every run received a fresh copy. The harness is the layer that drives the task, measures the run and records the result. It retained events, errors, code changes, Git state and a GPU sample every second.

These were real controls from the home-PC experiment. They did not create a complete sandbox. The shell tool had broad permissions, and we did not prove an independent block on outbound traffic. The setup was suitable for synthetic fixtures on a private host. It provides no basis for putting company data in the workspace.

For the next pilot, we use the joint CISA and partner guidance on deploying AI systems securely and OWASP's current prompt-injection guidance as external reference points. Those documents do not certify this experiment. They identify threats and controls to test before company data enters the system.

The agent saw only a small task repository

The daily task set contained four small fixtures:

The agent received prompt.md and the contents of repo/. Verification scripts and expected cases stayed in the parent suite. The model could read the project documentation while the hidden test answers remained outside its directory.

No input contained client data. Candidates, orders and activities were fictional. The frontend fixture created 10,000 synthetic applicants.

Every run started from the same code

A comparison of clients is meaningful only when the second agent does not inherit the first agent's patch. The preparation script restored the task baseline, and longer runs used separate detached worktrees. The user's active checkout stayed untouched.

After each run we recorded:

• git status --short;
• git diff --stat;
• the complete diff;
• process exit code and signal;
• timeout or manual-interruption state;
• the final test-program result.

This record exposed behaviour that a functional test alone can miss. Codex fixed pagination but made a broad, backward-incompatible rewrite. OpenCode produced a minimal patch and then inspected unnecessary Git metadata. In the Qwen Code import fixture, Python modified tracked __pycache__ files. That was a workspace-cleanliness problem; the business-logic change itself remained correct.

The tool list was explicit

The Qwen Code runner passed three tools:

run_shell_command
edit
write_file

The agent tool was excluded, so the model could not start subagents. auto-edit allowed normal work in the fixture because the required result was a working patch with tests. An explanation alone did not satisfy the task.

The central caveat is run_shell_command. It grants broad system access. A tool list in the client did not enforce operating-system boundaries for every file path or the network. Safety came from the prepared host, synthetic data, isolated directory and supervision. A company pilot should put shell inside a disposable container or VM with:

• the task directory as the only writable mount;
• outbound traffic denied except for an explicit allowlist;
• no cloud credentials or SSH keys;
• an unprivileged user;
• CPU, memory, process-count and wall-time limits;
• cache directories outside the repository.

Output and behaviour left evidence

The harness ran Qwen Code with stream-json output and retained the raw event stream. It stored stderr, GPU state and VRAM usage separately.

The summary could then show:

• the exact model, harness, reasoning effort and advertised context;
• permitted tools and approval mode;
• elapsed time, turn count and tool calls;
• provider error, exit state and completion status;
• the final diff and repository state.

This evidence exposed an error in the result recorder: a textual [API Error: ...] was initially counted as a successful result. We corrected result handling. The harness needs its own tests as well.

Raw events may contain source code and user prompts. In a company, they require classification, encryption, access control and a defined retention period. An operational dashboard can usually rely on run ID, version, lengths, timings, error codes and control status.

Hidden checks decided whether the patch was ready

After completion, a separate script ran public and hidden checks. The quality result carried more weight than speed:

60% hidden behaviour
15% regression tests
10% scope and maintainability
10% completion and truthful handoff
5% efficiency after quality requirements were met

That is why OpenCode's 32-second export did not beat Qwen Code medium's 38-second run. OpenCode missed the required separator. Qwen Code's 93-second import also failed the quality bar despite 12/12 public tests because one of five hidden checks found a missing validation.

What stopped a run

Runs intended to finish had no artificial time or tool-call ceiling. We interrupted a process after demonstrating a no-progress loop, a crash or unsafe behaviour, and we stopped it on explicit completion. The stop signal went to the entire process group so no child process remained unsupervised.

This exposed the natural cost, but it required supervision. A company experiment should set a budget and stop condition before launch. Crossing that limit becomes part of the result. Indefinite extra turns would distort cost.

What the experiment leaves behind

A useful experiment should leave the company with:

1. fixture sources and their data classification;
2. the prompt and exact model, client and server versions;
3. permitted tools and the boundaries actually enforced by the environment;
4. hidden tests and the score criteria;
5. events or a redacted summary, telemetry and the diff;
6. failures, exclusions and the resulting decision.

Syntalith can build this pilot around one workflow and teach the team to add cases and distinguish model failures from harness failures. AI-Native courses use the team's repository and rules. An AI process audit also covers security architecture, an API baseline and a defined build scope.