llama.cpp or vLLM for Qwen3.8-27B on an RTX 3090?

In an after-hours home-PC experiment, patched vLLM won: Qwen Code completed one matched task in 502.14 seconds against 1,468.76 seconds on llama.cpp. We retained llama.cpp for later tests as the conservative fallback because GGUF is simpler to launch, move and reproduce.

This compares two complete profiles. It does not show that upstream vLLM always beats llama.cpp, or that W4A16 weights are more accurate than Dynamic V3 GGUF.

Two different operational products

llama.cpp provides a mature GGUF format, a compact server, broad hardware support and direct control over offload and KV cache. It suits pilots, personal workstations and environments where reproducibility matters more than maximum throughput.

vLLM focuses on efficient model serving and exposes interfaces including OpenAI-compatible APIs. The home experiment's build, however, used a third-party Qwen3.8/RTX 3090 patch, frozen vLLM 0.27.1, a separate AutoRound checkpoint, FP8 KV, prefix caching and MTP-3. A stock pip installation does not inherit this result.

Raw 115k recall first

At a 115,074-token prompt, both profiles recovered exact facts at the beginning, middle and end.

The vLLM profile was 11.6% faster while consuming roughly 0.9 GiB more VRAM. This is cleaner inference evidence than an agent session, although weight format and cache still differ.

Then the full Qwen Code task

The coding task measured accepted work:

Both produced usable code. The vLLM cell added a stronger serialization test, avoided compaction and finished much sooner. That justified retaining it for later home-PC runs.

The table cannot separate kernel gains, MTP, 30k more context or the weight layout. Doing that would require a controlled matrix and repeated runs.

Where llama.cpp still wins

• Artifact simplicity. One GGUF file is easier to verify and move.
• Hardware range. CPU, several GPU types and partial offload expand recovery options.
• Memory control. Weight, cache and batch profiles are explicit.
• Reproduction. Fewer out-of-tree patches lower upgrade cost.
• Fallback value. A tested GGUF restores service when a specialised image breaks after a driver change.

The fallback retained in the experiment uses llama.cpp, Dynamic V3 Q4_K_M, Q8 KV, 120k and medium effort through Codex. It is not the fastest profile, but it has retained tests and a simpler diagnostic path.

When patched vLLM is justified

It makes sense when long tasks are frequent, the wall-time margin repeats on the acceptance set, and the team can maintain the image, checkpoint and protocol compatibility. Prefix caching helps when requests share a large prefix. The server also offers a stronger route into later queueing tests.

Maintenance cost belongs in the calculation. An out-of-tree patch can block a security or driver upgrade. At that point, a faster run competes with engineering hours needed to rebuild the image.

Decision matrix

Syntalith deploys the selected profile with a fallback, version manifest, post-restart test and upgrade gate. An AI process audit may recommend the simpler llama.cpp path when scale cannot pay for added complexity. A free process scan identifies the task worth testing.