llama.cpp or vLLM for Qwen3.8-27B on an RTX 3090?

The same Go repair took local Qwen3.8-27B 502.14 seconds on patched vLLM and 1,468.76 seconds on llama.cpp. The runs scored 100/100 and 98/100 respectively, on one home RTX 3090.

The engine, weight format, KV cache and context limit all changed. These results help select a configuration for further testing; they cannot isolate the speed advantage of either engine. We ran each variant once, with one active model request.

The measurement summary and 150k task metrics contain the data. Below are the differences that matter when operating the server.

What is being compared

llama.cpp is an inference engine. This profile used an Unsloth Dynamic V3 Q4_K_M GGUF file. GGUF keeps weights and their settings in one artifact, while llama.cpp can run across different CPUs and GPUs and offload part of the work between CPU and GPU. That path makes moving and reproducing a service straightforward.

vLLM is an inference server. It accepts requests, manages memory for active contexts and exposes client interfaces, including an OpenAI-compatible interface. Our measurement used a frozen third-party patched stack and a fast W4A16 AutoRound weight variant. That artifact is a different format from GGUF.

W4A16 roughly means 4-bit weights with 16-bit activations. A KV cache stores intermediate attention values for tokens already in the context. Its format and size affect how much text fits in GPU memory. A context limit is the maximum token count for a particular serving profile, separate from the model's advertised capability.

The coding-task result

The task repaired two Go paths that ignored json.Marshal errors. The agent had to preserve successful response formats, add deterministic failure behaviour, add regression tests, build the project and update documentation.

Compaction is the harness, the program that conducts an agent session, shortening the conversation when the context window fills. A 0 means the event did not occur in the recorded run; a 1 means it occurred once.

OpenAI Codex assigned both scores under the Syntalith rubric: 40 points for error-handling correctness, 20 for regression tests, 15 for API compatibility, 10 for scope, 10 for test and build verification, and 5 for documentation. A maximum score means Codex found no deduction under that rubric in this run. It is neither an independent laboratory audit nor a repeatability guarantee.

The vLLM profile reached a recorded maximum of 22,539 MiB of VRAM. The card has 24 GB, so the remaining margin was limited. Both timings include the agent, tool calls and checks, rather than a single decode step.

What the long-context test shows

A separate test planted three exact facts near the beginning, middle and end of a synthetic record, then asked for the three values as JSON. All three checked facts were returned for inputs of 50,059, 115,074 and 230,085 tokens.

Tok/s is generated-token throughput. Wall time covers the recorded run, and peak VRAM is the highest observed GPU-memory use. Each length ran once. The evidence file does not assign these rows to llama.cpp or vLLM separately, so they cannot serve as a direct engine ranking. The test checks retrieval of three planted facts; it does not establish comprehension of arbitrary long documents.

When llama.cpp is the right first choice

This path fits a one-GPU pilot, a personal workstation and a service that must move between machines. One GGUF file is easy to archive. Broad hardware support and partial offload provide another recovery path. Fewer private patches reduce upgrade work.

The experiment retained a llama.cpp fallback with Dynamic V3 Q4_K_M and a 120k limit. After switching engines, check the model, context limit and a short response again. A 150k session cannot automatically resume against a 120k server.

When vLLM earns its complexity

The selected vLLM profile is a candidate when long tasks are frequent, its time margin repeats on a fixed acceptance set and the team can maintain the image, weights and interface compatibility. A server also creates a useful base for later queue and multi-request measurements.

The trade-off is ownership. A frozen out-of-tree stack needs its own version record, a post-driver-update test and a verified return to the last working profile. One GPU and one slot in this experiment provide no team-throughput evidence.

A decision grounded in the evidence

Syntalith configures private LLM hosting on your hardware or rented GPUs. We select the engine, test load and hand over weight versions, server images and recovery instructions. The quote follows the workload and required availability.