Qwen3-32B, three configs.

Why this matters

A 32-billion-parameter model on a single consumer card decodes at about 75 tokens a second, and that number is a wall. It is set by memory bandwidth, not compute: every token the model produces, it has to stream all 17 GB of its weights through the GPU once. You can quantize harder or buy a second card. Those are the two options everyone talks about.

This run is about a third option that costs nothing. With one flag on the vLLM server, the same model on the same card produced code edits at 350 tokens a second, 4.65 times the baseline, and answered retrieval questions at 181, 2.41 times. No second model was loaded. No VRAM was spent. Output quality on those two workloads was byte for byte identical to the baseline, which I checked rather than assumed. The time to first token got slightly better, not worse.

That is the whole result, and it matters because code editing and quoting from a document are what a local agent does most of the day. An agent that rewrites a file, or answers from a retrieved passage, spends most of its output repeating text that was already in its prompt. This run measures exactly how much speed that repetition is worth on hardware people already own, and what it costs on the one workload where there is nothing to repeat.

How n-gram speculation works

Speculative decoding is the general trick: guess the next several tokens cheaply, then let the big model verify all of them in one forward pass. Verifying six tokens costs about the same as producing one, because the cost is in reading the weights, not in the arithmetic. Every guess that turns out right is a token you got for the price of a fraction of one.

Most speculative setups guess with a second, smaller model, which has to be trained to match the big one and takes its own slice of VRAM. The n-gram method (vLLM calls it ngram; the papers call it prompt lookup decoding) has no model at all. After each token it looks at the last two to four tokens it has generated, searches the prompt for the same sequence, and if it finds one, proposes the five tokens that followed it there. For “rename load_rows to read_csv_rows everywhere”, nearly every line of the output already exists in the prompt, so the guess is almost always right. For “write a story about a lighthouse keeper”, it almost never is, and each wrong guess costs one wasted verification step. That is the 0.95× on creative writing: a 5 percent tax for a 4.65× gain where it counts.

The acceptance rate over the whole run was 51.3 percent (2,635 of 5,140 drafted tokens), averaged across all three workloads. Per prompt, the spread tells the story better: the two pure rename edits ran at 364 and 381 tok/s, edits that insert new text (docstrings, type hints, a new argument) ran at 198 to 228, retrieval answers that quote verbatim ran at 160 to 270, and every creative prompt sat between 70.8 and 73.7.

Decode throughput by workload

Results by config

Workload	baseline (tok/s)	FP8 KV cache (tok/s)	n-gram speculation (tok/s)
Code edit	75.2	76.1	349.8
Creative	75.5	76.7	71.9
RAG	75.3	76.1	181
16-way concurrent, aggregate	635.9	656

The determinism test

A speed number means nothing if the faster config is quietly producing different text. So before comparing anything, the harness had to prove it could detect a change at all. I ran the baseline configuration twice, same 24 prompts, temperature 0, seed 0, and compared the outputs character by character. All 24 matched. That is the control: on this server, the same input produces the same bytes, so any difference in the other configs is real and caused by the config.

Against that control, n-gram speculation matched 18 of 24. All six that differed were creative-writing prompts, and none differed early: the first divergence came at character 189, the others at 274, 496, 684, 1,208 and 1,410. Every code edit and every retrieval answer was byte for byte the baseline’s output. That is the pair of facts that makes the result usable: the speedup lives entirely in the workloads where the output did not change.

Why do the creative ones drift at all, when at temperature 0 the verifier only ever accepts the token the big model would have picked anyway? Because verifying five drafts at once changes the shape of the batch going through the kernels, and with it the order the floating-point sums are done in. In open-ended prose the top two candidates are often within rounding error of each other, so a sum done in a different order can flip the choice, and from that point the text goes its own way. In code and quotation the leading candidate wins by a wide margin and nothing flips. That is my reading of the pattern; I am reporting what was measured, which is 18 of 24, not the theoretical “lossless”.

FP8 KV cache: twice the room, different words

The second config stores the KV cache in 8-bit floats instead of 16. The gain is capacity: the pool went from 33,760 to 65,536 tokens, so the card can hold four full 16K-token conversations at once instead of two, and aggregate throughput at 16-way concurrency rose from 636 to 656 tok/s with zero preemptions in either run. Sequential speed moved by about 1 percent.

The cost showed up in the determinism test. Only 12 of 24 outputs matched the baseline. Creative prompts diverged almost immediately (characters 12, 15, 26, 60, 63, 64), two of the eight code edits diverged, and two of the eight retrieval answers did. Nothing I read looked wrong, but “looked fine” is not the standard this bench holds to, and a config that changes two code edits out of eight is not one I want behind an agent. FP8 KV stays off on my server. The 656 at the top of the page is the honest peak; it is not the config I run.

Timeline, from the log timestamps

Methodology

One server, three configs, same everything else. vLLM 0.20.0 serving Qwen/Qwen3-32B-AWQ (awq_marlin kernels) on a single RTX 5090, with 16K max context, 32 sequence slots, 0.90 GPU memory utilization, prefix caching disabled, seed 0. The configs: baseline; baseline + --kv-cache-dtype fp8; and baseline + --speculative-config '{"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 4, "prompt_lookup_min": 2}'.

The client sends 24 prompts, eight per workload, at temperature 0, seed 0, streaming, thinking disabled, with two unrecorded warm-up requests per run. The code-edit prompts hand the model a 40 to 60 line Python file and ask for a specific change returned as the whole file (rename a function, add docstrings, change a separator, add type hints). The RAG prompts hand it a passage and ask for an extractive answer quoting verbatim. The creative prompts ask for a short piece with nothing to copy from. TTFT is time to first streamed token; decode tok/s is (generated − 1) / decode time; concurrent numbers are 16-way with each prompt duplicated ×4 (64 requests) over the code and RAG sets. vLLM’s /metrics is scraped before and after every run so draft and acceptance counts and preemptions can be attributed to that run.

What I did not capture

Acceptance rate per workload. vLLM exposes drafted and accepted counts as global counters, so 51.3 percent is the blend across all 24 prompts; the per-prompt tok/s is the proxy for how it splits. The n-gram and FP8 configs were each run once sequentially, so the 24/24 control is the only repeated run. No GPU telemetry (power, clocks, temperature) was logged. The creative-writing divergence explanation above is an inference from where the divergences fall, not something I instrumented.

Reproduce it

# three servers in turn, same flags apart from the one under test
vllm serve Qwen/Qwen3-32B-AWQ --port 8199 --served-model-name qwen3-32b-awq \
  --gpu-memory-utilization 0.90 --max-model-len 16384 --max-num-seqs 32 \
  --seed 0 --no-enable-prefix-caching --no-enable-log-requests \
  [--kv-cache-dtype fp8 | --speculative-config '{"method":"ngram","num_speculative_tokens":5,"prompt_lookup_max":4,"prompt_lookup_min":2}']

python bench.py --port 8199 --model qwen3-32b-awq --label baseline_run1 --out results/baseline_run1.json
python bench.py --port 8199 --model qwen3-32b-awq --label baseline_conc16 --out results/baseline_conc16.json \
  --concurrency 16 --dup 4 --workloads code_edit,rag
python analyze.py | tee results/summary.md   # medians, speedups, acceptance, byte diffs

vLLM 0.20.0 · awq_marlin · RTX 5090, 32 GB · 16,384 max context · temperature 0, seed 0 · measured 2026-07-21