MTP and DFlash2 for Qwen3.8-27B: faster decoding, real trade-offs

In the home-PC experiment, an optional DFlash2-7 profile raised median Qwen3.8-27B decoding from 93.3 to 120.9 tokens per second, a 29.6% increase across eight prompts. It reduced available context from 150,000 to 65,536 tokens. We kept it for short sessions and retained the 150k profile as the reference.

Speculative decoding does not make the model “reason faster”. It lets the target verify several proposed tokens in one step when those proposals are accurate enough.

Why one-token generation is expensive

An autoregressive model creates a token, adds it to context and runs another iteration. At single-user load, the GPU is often memory-bound: weights move through the system on every step while some compute capacity remains unused.

Speculation adds a drafting mechanism. It proposes several following tokens and the target model verifies them together. Accepted proposals remain. The first rejection ends the sequence, and the target resumes control.

vLLM documentation describes the technique as a way to reduce inter-token latency, especially at low or medium query rates where inference is memory-bound. The gain depends on proposal acceptance and verification overhead.

MTP is part of Qwen

Qwen3.8-27B was trained with Multi-Token Prediction. Additional heads predict following tokens and can act as an embedded draft without a separate small model.

The experiment's 150k profile used MTP-3. On a warmed set of eight realistic prompts it produced 93.3 decode tok/s and 90.79 end-to-end tok/s. It remained the heavy-work reference because it retained 150k of context.

What DFlash2-7 changed

Three repeats of the local fast profile produced:

Median decoding was 120.9 tok/s. The target model verified proposals; the server did not accept draft output unchecked.

This profile used BF16 KV and a 65,536-token context. Higher speed can pay for short refactors or frequent small questions. In a long repository session, losing 84k tokens may trigger compaction early enough to erase the gain.

More draft tokens did not mean more speed

We also tested DFLASH_TOKENS=15. Its median fell to 101.2 tok/s. The larger verification graph cost more per ordinary generation step, while the prompt set did not produce enough accepted spans to recover that overhead.

The tuning equation is:

gain = accepted tokens × avoided target steps
  - draft cost
  - larger verification cost
  - rejected proposal cost

A setting selected on one prompt can lose on code, Polish prose or structured data because predictable sequences differ.

Testing speed without losing the quality question

1. Use the same target checkpoint and sampler.
2. Freeze the prompts and maximum response.
3. Warm prefix cache or label a cold start.
4. Record decode and end-to-end throughput separately.
5. Measure proposal acceptance per task class.
6. Verify exact output or task success as well as tok/s.
7. Test a long session and its compaction behaviour separately.

Target verification limits the quality risk of draft tokens, while the changed serving stack can still contain parser, cache or protocol defects. The profile must pass acceptance tests.

When this optimisation has commercial value

A 29.6% median decode gain matters when a user waits for short answers repeatedly. It has little value for two overnight tasks or when the profile loses required context. Measure the process bottleneck before selecting speculation.

Syntalith can tune the profile and hand over the comparison script, accepted settings, fallback and upgrade gate as part of a deployment or technical team training. Begin with the free process scan when it is unclear whether the bottleneck is the model, server, queue or working method.