How DFlash2 speeds up Qwen3.8 and shortens context

Speculative decoding prepares several following tokens and asks the target model to check them together. When the proposals fit, one iteration can append several tokens. When the first proposal fails, the target model takes over from that point.

In the home experiment, the 150k MTP-3 baseline reached 93.3 tok/s decoding. An optional DFlash2-7 profile with a 65,536-token limit reached a 120.9 tok/s median across three repeats, a 29.6% increase. The smaller limit removes 84,464 tokens from the available history. The public benchmark contains the records and limitations.

What each part means

An autoregressive model generates one token at a time. A token is a piece of text. After each step, the result enters the context and the model runs another iteration. With one user, moving weights through memory can dominate the arithmetic.

Speculative decoding adds a drafting mechanism. It creates a short proposal, and the target model, the full Qwen3.8-27B, checks it in one pass. Accepted tokens remain in the answer; a rejected span is generated again by the target. Target verification keeps quality control with the main model, while speed depends on proposal acceptance and verification cost.

The vLLM documentation describes the technique as a way to reduce inter-token latency, especially at low or medium load. Tok/s means generated tokens per second. It measures decode throughput, rather than the complete time from prompt arrival to the final answer.

MTP is part of the model

MTP means Multi-Token Prediction, a mechanism for proposing following tokens. A server can use it as an embedded proposal source without adding a separate small model. The public artifact labels the baseline MTP-3; it does not define what the -3 counts internally.

The MTP-3 baseline had a 150,000-token limit and remained the reference for longer work. Decode throughput matters only when the session can retain the history the task needs.

What we measured for DFlash2-7

We ran the local fast profile three times:

The median is the middle value after sorting the runs. Here it is 120.9 tok/s. The public record contains these decode rates and the median. It does not provide a separate prompt-to-final-answer measurement, so the decision here uses decode throughput.

The DFlash2 profile used a 65,536-token context. KV memory stores intermediate values for tokens already in the session. The public record does not state its format, so we do not assign a particular KV precision to this result. The shorter window remains part of the complete configuration with its proposal mechanism, so the result does not describe the algorithm in isolation from its server.

Faster short sessions, less history for long work

DFlash2 can fit short refactors and many small questions, where a user waits for the next answer repeatedly. A repository session can fill a 65,536-token window sooner, trigger history compaction and lose part of the decode gain.

Choose from the task mix. For each work type, measure:

1. prompt preparation time;
2. decode throughput;
3. time to the complete answer;
4. VRAM use;
5. accepted-token count;
6. when history compaction starts;
7. task correctness and tool behaviour.

In the benchmark, each baseline configuration had one recorded run for its test, while DFlash2-7 had the three repeats shown above. This indicates a tuning direction. Reproducibility still needs testing on your own prompts.

A simple cost model

Think of the gain as saved target-model steps minus the extra work:

gain = accepted tokens × avoided target steps
  - proposal cost
  - joint verification cost
  - rejected proposal cost

Code, Polish prose and structured data have different predictability patterns. A setting that works on one prompt can therefore lose on another.

How to test it fairly

1. Use the same target checkpoint and sampling settings.
2. Freeze prompts, response length and temperature.
3. Label a cold start or warm the shared-prefix cache.
4. Record prefill, decode and full-response time separately.
5. Measure proposal acceptance for each task class.
6. Check code correctness, tests and tool calls alongside tok/s.
7. Add a long session to measure history compaction.

Target verification limits the chance of accepting incorrect draft tokens. A changed server can still introduce parser, cache or protocol defects. Treat speed and correctness as separate dimensions.

Deployment takeaway

The 150k MTP-3 profile is the reference for heavier work. DFlash2-7 produced the higher decode median in this short measurement while limiting history to 65,536 tokens. The choice is between per-answer speed and session length.

Syntalith can prepare the measurement script, quality criteria, selected profile settings and recovery path. A free process scan helps establish whether the bottleneck is decoding, context length, the server or the working method.