Maintaining local Qwen: eight failures that belong in the handover

The most valuable artifact from the Qwen experiment is not the start command. It is the set of failures that now have a regression check or procedure. Without that ledger, the next operator will repeat the same mistakes even with a complete file copy.

These cases come from the after-hours home-PC experiment with an RTX 3090. They are not production incidents or client history.

Failure ledger

Each item belongs to a different layer. Number 1 was client configuration. Numbers 2 and 6 were wrapper issues. Number 3 belonged to the harness. Number 4 was context management. Number 5 was the fixture. Number 8 affects the upgrade process. “The model was weak” diagnoses none of them.

Turn a failure into a test

A useful incident entry has six fields:

id: QWEN-OPS-004
symptom: request exceeds server context and returns HTTP 400
reproduction: resume long harness session until preflight threshold
expected: client compacts or refuses before sending oversized request
artifacts: event stream, request length, server error, profile version
owner: agent-platform
rollback: switch harness/profile; do not enlarge context automatically

Run the regression at the layer that failed. A wrong-key case needs no coding task. Query /health, /v1/models, verify 401 for the bad secret and confirm the correct model for the good secret. The 8k defect needs a long tool write because a tiny smoke test always passes.

An upgrade gate built from observed failures

Before changing the checkpoint, vLLM image, Qwen Code or driver, run three levels.

1. Five minutes: startup and protocol

• endpoint becomes healthy;
• expected model and context are reported;
• incorrect secret is rejected;
• streaming terminates correctly;
• small JSON and a tool call pass their schemas.

2. Thirty minutes: harness regressions

• a write longer than 8k is not truncated;
• a resumed session can reread and edit;
• the client does not submit more context than the server accepts;
• loop detection requires demonstrated non-progress;
• git status contains no cache or unrelated files.

3. Process task: quality and cost

Use the same natural prompt, a fresh worktree, hidden checks, diff review, wall time and telemetry. The retained experiment profile used the json.Marshal error-handling task. Qwen Code completed it in 8:22 and was accepted. Codex on the same model introduced a nondeterministic fallback and therefore failed despite completing.

A newer version does not pass on its release number. It must clear all critical gates and stay within the agreed regression budget.

A handover repository

deployment/
  profiles/
    daily.yaml
    fallback.yaml
  manifests/
    model.json
    server-image.json
    licenses.md
  runbooks/
    start-stop-status.md
    context-overflow.md
    gpu-oom.md
    key-rotation.md
    rollback.md
  eval/
    smoke/
    harness-regressions/
    process-golden-set/
  incidents/
    QWEN-OPS-001.md
    ...
  training/
    operator-lab.md
    reviewer-rubric.md

The manifest records exact checkpoint, format, hash, image, patch commit, driver, tool parser, sampler, context and client. latest is not a reproducible version.

A runbook supplies the command, expected output and first diagnostic step. “Ask the person who set it up” does not pass handover.

Monitoring that separates outages from weak output

Operations records:

• endpoint availability and queue;
• time to first token and total time;
• input/output lengths and compactions;
• provider, tool and parser failures;
• VRAM, GPU utilisation and process exits;
• model, image, harness and profile version.

Quality records:

• hidden-check result;
• acceptance without correction;
• human correction category;
• permission or citation violation;
• diff scope;
• justified and incorrect refusal rates.

A GPU dashboard cannot notice the blank candidate name. “Good answer” feedback cannot notice that the next turn will overflow context.

Rollback drill

Rollback on one GPU should be boring:

1. stop admitting new work;
2. finish or terminate the queue according to policy;
3. select the prior image and profile manifest;
4. verify model ID, context and secret;
5. run smoke plus one critical fixture;
6. restore traffic and retain the failure evidence.

The home runbook switched from vLLM to a 120k llama.cpp fallback. A company also needs a recovery-time target, permission test and user communication. The fallback may be slower. It must be known and safe.

What the team learns

An operator should start a profile, identify a wrong endpoint, read telemetry, rotate a secret and roll back. A reviewer needs a rubric and hidden cases to separate fluent narrative from accepted output. The process owner defines critical errors. Security owns data, network, tools and retention.

A hands-on lab is more useful than prompt slides. One person deliberately introduces a ledger failure, while another diagnoses it with logs and the runbook.

What Syntalith hands over

An implementation includes the agreed profiles, manifests and licenses, integration code, evaluations, dashboard, alerts, rollback, known-failure ledger and training material. The client's team performs start, test and rollback before acceptance.

Syntalith can deliver the system through custom AI applications and train the owning team through AI-Native courses. The free process scan determines whether the next step is a pilot, audit, training or remediation of an existing deployment.