Local Qwen3.8-27B on RTX 3090: a runbook from the working experiment

This runbook covers the Qwen3.8-27B profile we tested on a home RTX 3090 with 24 GB of VRAM. It used Qwen Code 0.21.13, one active request, a 150,000-token context, W4A16 AutoRound weights, FP8 KV cache, MTP-3 and patched vLLM. The fallback ran llama.cpp with Unsloth Dynamic V3 Q4_K_M weights and a 120,000-token limit.

The commands show the order of checks. Replace paths, service names, addresses and secrets with values from your environment. A /health response confirms the process is running; before starting the agent, also verify the model, limit and a short generated answer.

Configuration data is in the experiment summary and 150k task metrics. The test covered one GPU and one active request.

Describe the profile first

A context limit is the number of tokens accepted by a particular server profile. A token is a piece of text used by the model. The Qwen model card publishes a native limit of 262,144 tokens, while the retained home profile advertised 150,000. Those values describe different configuration layers.

W4A16 roughly means 4-bit weights and 16-bit activations. Lower-precision weights reduce memory use. KV memory stores intermediate values for tokens already in the context, and its format affects the remaining GPU-memory budget. MTP is the recorded speculative-decoding setting; the public artifact does not define what the -3 counts internally.

Start the server and wait for readiness

The following skeleton shows the order of operations. Stopping the other engine releases VRAM, the Compose profile starts the selected container and the loop waits for an HTTP response. Adapt the service name and port to your Compose file.

#!/usr/bin/env bash
set -euo pipefail

systemctl --user stop qwen.service
docker compose --profile single up -d --remove-orphans

while true; do
  if curl -fsS http://127.0.0.1:18020/health >/dev/null; then
    echo "Ready: Qwen3.8-27B, context 150000"
    break
  fi

  if [[ -n "$(docker compose --profile single ps --status exited -q single)" ]]; then
    echo "Server exited before health check passed" >&2
    exit 1
  fi

  sleep 1
done

/health is a process probe. Checking for an exited container as well prevents a startup failure from looking like a client timeout.

Verify what the API is serving

A server can be healthy while exposing a different model or limit. The client wrapper should query /v1/models, compare the model ID and expected max_model_len, then make a short generation request.

KEY_FILE="/path/to/secret/api-key"

if [[ ! -s "$KEY_FILE" ]]; then
  echo "Missing API key" >&2
  exit 1
fi

read -r QWEN_API_KEY < "$KEY_FILE"

curl -fsS \
  -H "Authorization: Bearer $QWEN_API_KEY" \
  http://127.0.0.1:18020/v1/models \
  | grep -Eq '"max_model_len":[[:space:]]*150000'

Keep the secret in a secret store or protected file, rather than in the repository. A shared service also needs image and weight versions plus a record of the profile used for each request.

Launch the client with the same limit

A harness is the program that conducts an agent session, sends instructions and exposes tools. The measurement used Qwen Code 0.21.13 at medium reasoning effort. The client wrapper should select an isolated configuration directory and name the model explicitly:

export QWEN_VLLM_API_KEY="$QWEN_API_KEY"
export QWEN_HOME="/path/to/qwen/profile"
export QWEN_STREAM_IDLE_TIMEOUT_MS=0
export QWEN_STREAM_MAX_LIFETIME_MS=0

exec qwen --auth-type openai --model qwen3.8-27b "$@"

The variable name, endpoint and model must match the server configuration. Keep the 150,000-token limit aligned on both sides and test the client version before removing any fixed response cap that could truncate long tool writes.

Watch the card while it works

nvidia-smi describes hardware state, rather than response correctness. Start with memory use, GPU utilisation and temperature:

nvidia-smi \
  --query-gpu=memory.used,utilization.gpu,temperature.gpu \
  --format=csv,noheader

During the Go task, the selected vLLM profile reached 22,539 MiB of VRAM. With a 24 GB card, that leaves a narrow margin, which is why one slot is a sensible starting point. This measurement provides no multi-user capacity figure.

Switch to the fallback

The fallback has its own weights, limit and engine. Stop vLLM, start llama.cpp and repeat the model, limit and short-response checks:

docker compose --profile single down
exec /path/to/qwen-context 120k medium

A 120k server cannot automatically accept a session prepared for 150k. The switch restores useful operation, while response time and available conversation history can change.

Test after every start

The measured regression task repaired two Go paths that ignored json.Marshal errors. Focused tests and the build passed, and Codex scored the result 100/100 under the Syntalith rubric: 40 points for error-handling correctness, 20 for regression tests, 15 for API compatibility, 10 for scope, 10 for test and build verification, and 5 for documentation. That score belongs to one run and one task.

Where the experiment ends

The home profile shows that this stack can run on one card. A company deployment needs versioned images and weights, secret management, a private endpoint, historical metrics, a queue, a restart test and a verified return to a working version. It also needs evaluation on the buyer's actual process.

Syntalith sets up private models from a first installation to a team server. We can assess your computer, select a configuration, run checks and show you how to operate it. If you do not own suitable hardware, we help compare a purchase with GPU rental.