Local Qwen3.8-27B on RTX 3090: a runbook from the working experiment

This runbook describes a profile measured on a home PC with one NVIDIA GeForce RTX 3090 24 GB. Qwen3.8-27B ran through Qwen Code 0.21.13 with one inference slot, a 150,000-token context limit, W4A16 AutoRound weights, an FP8 KV cache, MTP-3 and a patched vLLM stack. The fallback used llama.cpp, Unsloth Dynamic V3 Q4_K_M weights and a 120,000-token limit.

This is a runbook: a short sequence of operational checks. Paths, service names, addresses and secrets in the examples are placeholders. A process answering /health proves only that the server is alive. Before starting an agent, verify the model, context limit and an actual response.

The numbers come from the public benchmark summary and the 150k task metrics. It was an after-hours experiment with one GPU and one slot. It says nothing about safe multi-user throughput.

Describe the profile first

A context limit is the number of tokens accepted by a particular server profile. A token is a piece of text used by the model. The Qwen model card publishes a native limit of 262,144 tokens, while the retained home profile advertised 150,000. Those values describe different configuration layers.

W4A16 roughly means 4-bit weights and 16-bit activations. Lower-precision weights reduce memory use. KV memory stores intermediate values for tokens already in the context, and its format affects the remaining GPU-memory budget. MTP is the recorded speculative-decoding setting; the public artifact does not define what the -3 counts internally.

Start the server and wait for readiness

The following skeleton shows the order of operations. Stopping the other engine releases VRAM, the Compose profile starts the selected container and the loop waits for an HTTP response. Adapt the service name and port to your Compose file.

#!/usr/bin/env bash
set -euo pipefail

systemctl --user stop qwen.service
docker compose --profile single up -d --remove-orphans

while true; do
  if curl -fsS http://127.0.0.1:18020/health >/dev/null; then
    echo "Ready: Qwen3.8-27B, context 150000"
    break
  fi

  if [[ -n "$(docker compose --profile single ps --status exited -q single)" ]]; then
    echo "Server exited before health check passed" >&2
    exit 1
  fi

  sleep 1
done

/health is a process probe. Checking for an exited container as well prevents a startup failure from looking like a client timeout.

Verify what the API is serving

A server can be healthy while exposing a different model or limit. The client wrapper should query /v1/models, compare the model ID and expected max_model_len, then make a short generation request.

KEY_FILE="/path/to/secret/api-key"

if [[ ! -s "$KEY_FILE" ]]; then
  echo "Missing API key" >&2
  exit 1
fi

read -r QWEN_API_KEY < "$KEY_FILE"

curl -fsS \
  -H "Authorization: Bearer $QWEN_API_KEY" \
  http://127.0.0.1:18020/v1/models \
  | grep -Eq '"max_model_len":[[:space:]]*150000'

Keep the secret in a secret store or protected file, rather than in the repository. A shared service also needs image and weight versions plus a record of the profile used for each request.

Launch the client with the same limit

A harness is the program that conducts an agent session, sends instructions and exposes tools. The measurement used Qwen Code 0.21.13 at medium reasoning effort. The client wrapper should select an isolated configuration directory and name the model explicitly:

export QWEN_VLLM_API_KEY="$QWEN_API_KEY"
export QWEN_HOME="/path/to/qwen/profile"
export QWEN_STREAM_IDLE_TIMEOUT_MS=0
export QWEN_STREAM_MAX_LIFETIME_MS=0

exec qwen --auth-type openai --model qwen3.8-27b "$@"

The variable name, endpoint and model must match the server configuration. Keep the 150,000-token limit aligned on both sides and test the client version before removing any fixed response cap that could truncate long tool writes.

Watch the card while it works

nvidia-smi describes hardware state, rather than response correctness. Start with memory use, GPU utilisation and temperature:

nvidia-smi \
  --query-gpu=memory.used,utilization.gpu,temperature.gpu \
  --format=csv,noheader

During the Go task, the selected vLLM profile reached 22,539 MiB of VRAM. With a 24 GB card, that leaves a narrow margin, which is why one slot is a sensible starting point. This measurement provides no multi-user capacity figure.

Switch to the fallback

The fallback has its own weights, limit and engine. Stop vLLM, start llama.cpp and repeat the model, limit and short-response checks:

docker compose --profile single down
exec /path/to/qwen-context 120k medium

A 120k server cannot automatically accept a session prepared for 150k. The switch restores useful operation, while response time and available conversation history can change.

Test after every start

The measured regression task repaired two Go paths that ignored json.Marshal errors. Focused tests and the build passed, and Codex scored the result 100/100 under the Syntalith rubric: 40 points for error-handling correctness, 20 for regression tests, 15 for API compatibility, 10 for scope, 10 for test and build verification, and 5 for documentation. That score belongs to one run and one task.

Where the experiment ends

The home profile shows that this stack can run on one card. A company deployment needs versioned images and weights, secret management, a private endpoint, historical metrics, a queue, a restart test and a verified return to a working version. It also needs evaluation on the buyer's actual process.

An AI process audit can define that test, while a free process scan helps determine whether a local model fits the task, data and required working time.