llama.cpp or vLLM for Qwen3.8-27B on an RTX 3090?

On one home RTX 3090, we compared two ways to serve Qwen3.8-27B. On a matched Go repair, the patched-vLLM profile finished in 502.14 seconds and received 100/100. The llama.cpp profile took 1,468.76 seconds and received 98/100. Each result comes from one run.

This is a comparison of complete configurations. Engine, weight format, KV memory and context limit changed together, so the table selects an operating profile for this experiment. It cannot isolate quantization quality or an intrinsic engine advantage.

The full records are available in the benchmark summary and the 150k task metrics. The experiment ran after hours on a home PC with one GPU and one active inference slot.

What is being compared

llama.cpp is an inference engine. This profile used an Unsloth Dynamic V3 Q4_K_M GGUF file. GGUF keeps weights and their settings in one artifact, while llama.cpp can run across different CPUs and GPUs and offload part of the work between CPU and GPU. That path makes moving and reproducing a service straightforward.

vLLM is an inference server. It accepts requests, manages memory for active contexts and exposes client interfaces, including an OpenAI-compatible interface. Our measurement used a frozen third-party patched stack and a fast W4A16 AutoRound weight variant. That artifact is a different format from GGUF.

W4A16 roughly means 4-bit weights with 16-bit activations. A KV cache stores intermediate attention values for tokens already in the context. Its format and size affect how much text fits in GPU memory. A context limit is the maximum token count for a particular serving profile, separate from the model's advertised capability.

The coding-task result

The task repaired two Go paths that ignored json.Marshal errors. The agent had to preserve successful response formats, add deterministic failure behaviour, add regression tests, build the project and update documentation.

Compaction is the harness, the program that conducts an agent session, shortening the conversation when the context window fills. A 0 means the event did not occur in the recorded run; a 1 means it occurred once.

OpenAI Codex assigned both scores under the Syntalith rubric: 40 points for error-handling correctness, 20 for regression tests, 15 for API compatibility, 10 for scope, 10 for test and build verification, and 5 for documentation. A maximum score means Codex found no deduction under that rubric in this run. It is neither an independent laboratory audit nor a repeatability guarantee.

The vLLM profile reached a recorded maximum of 22,539 MiB of VRAM. The card has 24 GB, so the remaining margin was limited. Both timings include the agent, tool calls and checks, rather than a single decode step.

What the long-context test shows

A separate test planted three exact facts near the beginning, middle and end of a synthetic record, then asked for the three values as JSON. All three checked facts were returned for inputs of 50,059, 115,074 and 230,085 tokens.

Tok/s is generated-token throughput. Wall time covers the recorded run, and peak VRAM is the highest observed GPU-memory use. Each length ran once. The evidence file does not assign these rows to llama.cpp or vLLM separately, so they cannot serve as a direct engine ranking. The test checks retrieval of three planted facts; it does not establish comprehension of arbitrary long documents.

When llama.cpp is the right first choice

This path fits a one-GPU pilot, a personal workstation and a service that must move between machines. One GGUF file is easy to archive. Broad hardware support and partial offload provide another recovery path. Fewer private patches reduce upgrade work.

The experiment retained a llama.cpp fallback with Dynamic V3 Q4_K_M and a 120k limit. After switching engines, check the model, context limit and a short response again. A 150k session cannot automatically resume against a 120k server.

When vLLM earns its complexity

The selected vLLM profile is a candidate when long tasks are frequent, its time margin repeats on a fixed acceptance set and the team can maintain the image, weights and interface compatibility. A server also creates a useful base for later queue and multi-request measurements.

The trade-off is ownership. A frozen out-of-tree stack needs its own version record, a post-driver-update test and a verified return to the last working profile. One GPU and one slot in this experiment provide no team-throughput evidence.

A decision grounded in the evidence

An operational rollout should include the selected profile, weight and image versions, a restart test and a verified recovery path. An AI process audit can define a test around real work, while a free process scan helps identify whether the bottleneck is the model, server or task flow.