Qwen3.8-27B quantization: Q4, Q5 or W4A16?

Quantization stores model weights with fewer bits. The model then uses less memory and leaves more room for context, while the approximation can change answer quality. Q4 uses fewer bits than Q5. W4A16 roughly means 4-bit weights and 16-bit activations, and our test required a separate runtime. VRAM is graphics-card memory shared by weights, the context cache and buffers.

On the home RTX 3090, Q5 used more space and constrained long context. Q4 freed room for the KV cache, which stores context. The W4A16 AutoRound configuration received the highest score in one task, while that run also included its server, cache and generation path. It cannot rate the weights in isolation.

When a supplier promises “almost no quality loss”, ask for results on your own tasks and the exact weight-file hash.

A fair comparison uses the same documents and task set for every format. Measure answer quality, memory, completion time and the human correction cost in the target process.

What gets compressed

A 27B model in BF16 needs roughly 55 GB for the weights alone. Quantization stores some values with fewer bits. That reduces memory use and memory traffic while introducing approximation.

A serving configuration can set these independently:

• model-weight precision;
• KV-cache precision for earlier context;
• precision of the additional MTP head used to predict subsequent tokens;
• the format and kernels supported by the server.

The label “4-bit” therefore covers different technologies. GGUF Q4_K_M, AutoRound W4A16 and NVFP4 use different procedures, precision layouts and runtimes.

Official Unsloth file sizes

The Qwen3.8-27B GGUF repository lists:

Our 60k llama.cpp configuration used Q5_K_M with Q8 cache. The 120k and 250k configurations used Q4_K_M. At 250k, cache precision fell to Q4 and batch settings became much smaller. The longer window required several compromises at once.

Four configurations that actually fit in 24 GB

The first three rows came from the same type of long-input fact-retrieval test, with each length run once. The final row came from a different coding task, so it is not a direct memory comparison with the GGUF configurations. It records the peak observed for the configuration selected for further work.

The 250k configuration fit with less than 1 GiB free, a smaller batch and Q4 cache. That margin offers little comfort for several sessions. The server still handled one session at a time.

How W4A16 differs

W4A16 roughly means 4-bit weights and 16-bit activations. Our AutoRound checkpoint was run in a frozen, third-party patched vLLM image with FP8 KV cache and additional RTX 3090 changes. It is not a GGUF file that can simply replace another file in llama.cpp.

On the same Go repair task, Codex awarded 100/100 to the W4A16/vLLM configuration after 502.14 seconds. Dynamic V3 Q4_K_M on llama.cpp received 98/100 after 1,468.76 seconds. The 100-point rubric covers correctness (40), regression tests (20), compatibility (15), scope discipline (10), verification (10) and documentation (5). The server, cache, speculative path and context limit also differed. The result identifies a configuration worth further testing. It does not show that W4A16 weights are inherently more accurate.

How to read vendor accuracy claims

Unsloth describes Dynamic V3 as delivering about 10% higher accuracy at the same size. That is the quantization vendor's claim under its evaluation method. Our experiment did not show a 10% increase in agent quality.

A stronger claim would require:

1. identical prompts and runtime;
2. several weight formats under matched settings;
3. repeated runs with a fixed sampler;
4. a task set representing the target process;
5. independent answer scoring.

The narrower conclusion was enough to choose a configuration for further testing. In this single task, Codex gave both configurations high scores under the same rubric, and the whole-task time difference was large. Codex assigned the 100/100 score under the Syntalith rubric. It describes one run and does not establish repeatability.

KV cache can consume the saving

Long context stores keys and values for earlier tokens. It can occupy several gigabytes for one user and grows with the number of concurrent sequences. A 16.5 GB weight file therefore does not leave a clean 7.5 GB budget on a 24 GB card.

Reducing cache precision from Q8 to Q4 increases capacity but can affect access to long material. Test it with facts at different positions and a synthesis task. A successful server launch reveals very little about this behaviour.

A selection procedure

1. Begin with the highest precision that fits the typical context.
2. Run the acceptance set and retain the outputs.
3. Reduce weight or cache precision separately where possible.
4. Compare quality, time, VRAM and correction cost.
5. Set a required VRAM margin and reject configurations that repeatedly cross it.
6. Record the cryptographic hash of the selected weight file and test results.

A document application may reveal a difference in numbers and citations. A coding agent may reveal it in scope, contracts and tool use. Perplexity, a measure of how well a model predicts the next token, cannot replace tests for both kinds of work.

Syntalith performs this comparison in an AI process audit or as part of an AI application. You receive the selected variant, rejected alternatives, measurement data and an upgrade gate. Book a free scan when it is still unclear whether the process needs a 27B model.