How we isolated the local Qwen agent experiment

Qwen was allowed to write files and run commands, but it worked on prepared repositories rather than a user's active project. Every harness received a fresh copy. Hidden checks lived outside the working directory. The runner retained events, stderr, Git diff, Git status and a GPU sample every second.

Those are real controls from the home-PC experiment. They do not make it a complete sandbox. The shell tool was broad and we did not prove an independent egress block. The setup was appropriate for synthetic fixtures on a private host. It would not justify dropping company data into the workspace and calling the result secure.

For the next pilot, the external control list comes from the joint CISA and partner guidance on deploying AI systems securely and OWASP's current prompt-injection guidance. Those documents do not certify this experiment. They identify additional threats and controls to test before company data enters the system.

The agent saw only a small task repository

The daily-driver suite used four small fixtures:

The agent received prompt.md and the contents of repo/. Verification scripts and expected hidden cases stayed in the parent suite. It could read project documentation without reading the hidden answer.

No client data appeared in the inputs. Candidates, orders and activities were fictional. The frontend stress fixture created 10,000 synthetic applicants.

Every cell had a fresh directory

A harness comparison fails if the second agent inherits the first agent's patch. The preparation script restored the task baseline, while longer cells ran in separate detached worktrees. The user's active checkout remained untouched.

After each run the harness captured:

• git status --short;
• git diff --stat;
• the complete diff;
• process exit code and signal;
• timeout or manual interruption state;
• the harness's final result.

This revealed behaviour that functional tests alone miss. Codex solved pagination but performed a broad incompatible rewrite. OpenCode finished a minimal patch and continued wandering through Git metadata. Running Python in the Qwen Code import fixture modified tracked __pycache__ files. That created workspace-cleanliness debt. The business-logic change itself remained correct.

The tool list was explicit

The Qwen Code runner allowed:

run_shell_command
edit
write_file

It explicitly excluded the agent tool, so there was no subagent fan-out. auto-edit allowed normal work inside the fixture because the result needed a patch and tests rather than an explanation.

The important caveat is that run_shell_command is broad. A CLI allowlist did not isolate every operating-system path or the network. Safety came from the prepared host, synthetic data, isolated directory and supervision. A company pilot should put shell inside a disposable container or VM with:

• the task directory as the only writable mount;
• outbound network denied except for an explicit allowlist;
• no cloud credentials or SSH keys;
• an unprivileged user;
• CPU, memory, process and wall-time limits;
• cache directories outside the repository.

Output and behaviour left evidence

The runner used Qwen Code's stream-json output and retained the raw event stream. Stderr, GPU utilisation and VRAM state were stored separately.

The summary therefore recorded:

• exact model, harness, effort and advertised context;
• allowed tools and approval mode;
• wall time, turns and tool calls;
• provider error, exit state and completion;
• final diff and repository state.

That evidence exposed a runner bug: a textual [API Error: ...] had initially been counted as success. The parser was corrected. Evaluation harnesses need their own tests.

Raw events may contain source code and prompts. In a company they need classification, encryption, access control and retention. An operational dashboard can usually work with run ID, version, lengths, timings, error codes and acceptance status.

Hidden checks outranked the agent's narrative

After completion, a separate script ran public and hidden checks. Quality dominated efficiency:

60% hidden behaviour
15% regression tests
10% scope and maintainability
10% completion and truthful handoff
5% efficiency after quality

That is why OpenCode's 32-second export did not beat the 38-second Qwen Code medium run: it missed the separator contract. Qwen Code's 93-second import was also not accepted despite 12/12 public tests because one hidden validation failed.

What stopped a run

Completion cells had no artificial wall-time or tool-call ceiling. A process was interrupted for a demonstrated non-progress loop, a crash, unsafe behaviour or explicit completion. Signals were forwarded to the entire process group.

This exposed natural cost, but it required supervision. A company experiment should set a budget and stop condition before launch. Crossing it becomes an evaluation result. Indefinite extra turns would distort cost.

The handover package

A useful experiment should deliver:

1. fixture sources and data classification;
2. prompt plus exact model, client and server versions;
3. tool permissions and the restrictions actually enforced;
4. hidden tests and rubric;
5. events or a redacted summary, telemetry and diff;
6. failures, exclusions and the decision.

Syntalith can build this isolated pilot around one workflow and teach the team to add cases and distinguish model failures from harness failures. AI-Native courses use the team's repository and rules. An AI process audit adds the security architecture, an API baseline and a fixed build scope.