Local Qwen3.8-27B on RTX 3090: a runbook from the working experiment

This runbook is based on the commands that actually operated Qwen3.8-27B on a home PC with an RTX 3090. The selected profile was one Qwen Code session, a 150k context, W4A16 AutoRound weights, FP8 KV cache, MTP-3 and a third-party patched vLLM stack. The fallback was llama.cpp with Dynamic V3 Q4_K_M and a 120k context.

The vLLM serving documentation describes its OpenAI-compatible interfaces, while the official Qwen3.8-27B model card publishes a 262,144-token native context. The retained home-experiment profile used a 150k limit. The vendor does not recommend that exact configuration, and one run cannot establish useful quality for every input of that length.

Paths, username, LAN address and secret have been redacted. This is enough to reproduce the operating pattern of the experiment. It is not a turnkey company production manifest.

What one start command should do

The qwen-engine vllm command performed five operations:

1. stop the llama.cpp user service because both servers cannot occupy one card;
2. start the selected Docker Compose profile;
3. poll /health once per second;
4. stop waiting if the container exits;
5. return success only after the endpoint is ready.

The redacted core is short:

#!/usr/bin/env bash
set -euo pipefail

systemctl --user stop qwen.service
docker compose --profile single up -d --remove-orphans

while true; do
  if curl -fsS http://127.0.0.1:18020/health >/dev/null; then
    echo "Ready: Qwen3.8-27B, 150k, port 18020"
    break
  fi

  if [[ -n "$(docker compose --profile single ps --status exited -q single)" ]]; then
    echo "Server exited before health check passed" >&2
    exit 1
  fi

  sleep 1
done

That is only a small addition to docker compose up -d, but it removes a common ambiguity. A client launched too early produces connection failures that can be mistaken for a model or protocol problem.

A health check does not identify the profile

A healthy endpoint may expose the 120k fallback, a 64k DFlash2 profile or another model. The client wrapper therefore queried /v1/models and checked for the expected 150,000 limit. The secret came from a separate file and never from the repository.

KEY_FILE="/srv/qwen/secrets/api-key"

if [[ ! -s "$KEY_FILE" ]]; then
  echo "Missing API key" >&2
  exit 1
fi

read -r QWEN_API_KEY < "$KEY_FILE"

curl -fsS \
  -H "Authorization: Bearer $QWEN_API_KEY" \
  http://127.0.0.1:18020/v1/models |
  grep -Eq '"max_model_len":[[:space:]]*150000'

A company version should check model ID, image version, context limit and a short generation result. The context field was adequate for this home switcher because only one profile on that port declared 150k. It would be too weak for a shared environment.

The Qwen Code wrapper rejects a bad launch

After secret, health and profile checks, the wrapper selected an isolated configuration directory, removed artificial stream timeouts and launched Qwen Code with an explicit model:

export QWEN_VLLM_API_KEY="$QWEN_API_KEY"
export QWEN_HOME="/srv/qwen/profiles/daily"
export QWEN_STREAM_IDLE_TIMEOUT_MS=0
export QWEN_STREAM_MAX_LIFETIME_MS=0

exec qwen --auth-type openai --model qwen3.8-27b "$@"

The environment-variable name matters. One retained run supplied the key under the wrong name, so the client stopped before sending a token. We excluded it from model evaluation and added the case to wrapper verification.

The client declared the same 150k context as the server, used medium effort, applied the model-card sampler and preserved thinking between turns. We removed a fixed max_tokens: 8192 setting because it disabled Qwen Code's adaptive retry and truncated long tool writes.

Status in one command

The qwen-engine status script answered three questions: which engine is active, whether vLLM is healthy and what the GPU is doing.

systemctl --user is-active qwen.service || true
curl -fsS http://127.0.0.1:18020/health >/dev/null
nvidia-smi \
  --query-gpu=memory.used,utilization.gpu \
  --format=csv,noheader

An extended terminal view sampled GPU and memory utilisation plus process state. That was adequate for the experiment. A company service needs historical metrics, alerts, queue depth, termination reason and the profile version attached to each request.

Fallback has a separate command and contract

Switching to the fallback stopped the container cleanly and then launched the 120k/medium llama.cpp service:

docker compose --profile single down
exec /srv/qwen/bin/qwen-context 120k medium

Both engines are never kept in VRAM together. The profile test runs again after a switch because a 150k client must not assume that a 120k server can resume the same session. The fallback restores useful operation; it does not preserve identical performance.

On one matched task, the complete llama.cpp profile took 24:29 while the selected vLLM profile took 8:22. Weight format, cache and context differed, so this selects the operational profile. It does not isolate an intrinsic engine advantage.

Five minutes of checks after each start

Before a normal workload is admitted, automate these checks:

After an image upgrade, run a real regression task as well. Our profile used the json.Marshal error-handling fixture. A tiny smoke test would not have caught the nondeterministic fallback that later disqualified the Codex result.

What Syntalith turns this into

We convert an experimental runbook into a company-owned deployment: versioned image, weight and license manifest, secret management, private endpoint, queue, dashboard, restart test, previous working profile and a rollback procedure. The package includes evaluation on the buyer's process and training for the people who will own the service.

An AI process audit ends with architecture, a written definition of a correct result and a fixed quote. A full AI app can include the endpoint, control plane and monitoring. The home RTX 3090 supplies operating evidence and useful failure cases; it is not the proposed production infrastructure.