How DFlash2 speeds up Qwen3.8 and shortens context

DFlash2 raised generation speed in our local Qwen3.8-27B from 93.3 to a median 120.9 tokens/s, a 29.6% increase. Its context limit was 65,536 tokens, compared with 150,000 for the MTP-3 profile. The faster profile gave up 84,464 tokens of available history.

Both use speculative decoding: a helper proposes upcoming tokens and the target model checks them together. Accepting several proposals can add multiple tokens in one iteration. Below are the three DFlash2 measurements and the workloads where shorter context could cancel out the speed gain. The data comes from our home experiment.

What each part means

An autoregressive model generates one token at a time. A token is a piece of text. After each step, the result enters the context and the model runs another iteration. With one user, moving weights through memory can dominate the arithmetic.

Speculative decoding adds a drafting mechanism. It creates a short proposal, and the target model, the full Qwen3.8-27B, checks it in one pass. Accepted tokens remain in the answer; a rejected span is generated again by the target. Target verification keeps quality control with the main model, while speed depends on proposal acceptance and verification cost.

The vLLM documentation describes the technique as a way to reduce inter-token latency, especially at low or medium load. Tok/s means generated tokens per second. It measures decode throughput, rather than the complete time from prompt arrival to the final answer.

MTP is part of the model

MTP means Multi-Token Prediction, a mechanism for proposing following tokens. A server can use it as an embedded proposal source without adding a separate small model. The public artifact labels the baseline MTP-3; it does not define what the -3 counts internally.

The MTP-3 baseline had a 150,000-token limit and remained the reference for longer work. Decode throughput matters only when the session can retain the history the task needs.

What we measured for DFlash2-7

We ran the local fast profile three times:

The median is the middle value after sorting the runs. Here it is 120.9 tok/s. The public record contains these decode rates and the median. It does not provide a separate prompt-to-final-answer measurement, so the decision here uses decode throughput.

The DFlash2 profile used a 65,536-token context. KV memory stores intermediate values for tokens already in the session. The public record does not state its format, so we do not assign a particular KV precision to this result. The shorter window remains part of the complete configuration with its proposal mechanism, so the result does not describe the algorithm in isolation from its server.

Faster short sessions, less history for long work

DFlash2 can fit short refactors and many small questions, where a user waits for the next answer repeatedly. A repository session can fill a 65,536-token window sooner, trigger history compaction and lose part of the decode gain.

Choose from the task mix. For each work type, measure:

1. prompt preparation time;
2. decode throughput;
3. time to the complete answer;
4. VRAM use;
5. accepted-token count;
6. when history compaction starts;
7. task correctness and tool behaviour.

In the benchmark, each baseline configuration had one recorded run for its test, while DFlash2-7 had the three repeats shown above. This indicates a tuning direction. Reproducibility still needs testing on your own prompts.

A simple cost model

Think of the gain as saved target-model steps minus the extra work:

gain = accepted tokens × avoided target steps
  - proposal cost
  - joint verification cost
  - rejected proposal cost

Code, Polish prose and structured data have different predictability patterns. A setting that works on one prompt can therefore lose on another.

How to test it fairly

1. Use the same target checkpoint and sampling settings.
2. Freeze prompts, response length and temperature.
3. Label a cold start or warm the shared-prefix cache.
4. Record prefill, decode and full-response time separately.
5. Measure proposal acceptance for each task class.
6. Check code correctness, tests and tool calls alongside tok/s.
7. Add a long session to measure history compaction.

Target verification limits the chance of accepting incorrect draft tokens. A changed server can still introduce parser, cache or protocol defects. Treat speed and correctness as separate dimensions.

Deployment takeaway

We kept the 150k MTP-3 profile for longer work. DFlash2-7 is worth testing on short sessions, measuring total task time and when history compaction starts. Decoding speed alone cannot establish the business saving.

Syntalith tunes and deploys private models. We can measure time spent loading documents, generating answers, running tools and waiting in the queue, then address the part causing the delay. You receive a before-and-after comparison and a tested fallback profile.