← The bench · run of 2026-07-21

Qwen3-32B, three configs.

One vLLM server, three configs, the same 24 prompts. Speculative decoding, FP8 KV cache and 16-way concurrency on a single RTX 5090, with determinism checked rather than assumed.

rig: NVIDIA GeForce RTX 5090 32 GB · AMD Ryzen 9 9950X3D · 64 GB DDR5

At a glance

24
Prompts per config
code, creative, RAG · temp 0
656
Aggregate tok/s, peak
Qwen3-32B AWQ · 16 concurrent
350
Fastest decode, tok/s
n-gram speculation on code edits: 4.65× baseline
51
Draft acceptance, %
n-gram speculative decode (51.3%)

Decode throughput by workload

baselineFP8 KV cachen-gram speculation
0 175 350 Code edit Code edit · baseline: 75.2 tok/s 75.2 Code edit · FP8 KV cache: 76.1 tok/s 76.1 Code edit · n-gram speculation: 349.8 tok/s 349.8 Creative Creative · baseline: 75.5 tok/s 75.5 Creative · FP8 KV cache: 76.7 tok/s 76.7 Creative · n-gram speculation: 71.9 tok/s 71.9 RAG RAG · baseline: 75.3 tok/s 75.3 RAG · FP8 KV cache: 76.1 tok/s 76.1 RAG · n-gram speculation: 181 tok/s 181

Qwen3-32B-AWQ · vLLM 0.20.0 · RTX 5090 · sequential decode, median tok/s per workload · runs of 2026-07-21

Results by config

Workloadbaseline (tok/s)FP8 KV cache (tok/s)n-gram speculation (tok/s)
Code edit75.276.1349.8
Creative75.576.771.9
RAG75.376.1181
16-way concurrent, aggregate635.9656

Methodology

One server, three configs, same everything else. vLLM 0.20.0 serving Qwen/Qwen3-32B-AWQ (awq_marlin kernels) on a single RTX 5090, with 16K max context, prefix caching disabled, seed 0. The configs: baseline; baseline + --kv-cache-dtype fp8; and baseline + n-gram speculative decoding (5 speculative tokens, prompt-lookup window 2 to 4).

The client sends 24 prompts across three workloads (code editing, creative writing, and RAG-style answering) at temperature 0, seed 0, streaming, thinking disabled, with two unrecorded warmup requests per run. TTFT is time to first streamed token; decode tok/s is (generated − 1) / decode time; concurrent numbers are 16-way with each prompt duplicated ×4 (64 requests). vLLM's /metrics is scraped before and after every run so speculation acceptance and preemptions can be attributed to that run.

Determinism is checked, not assumed: two identical baseline runs produced byte-identical output on all 24 prompts. Against that control, n-gram speculation matched 18/24 (all divergences in creative writing) and FP8 KV matched only 12/24, which is why FP8 KV stays off on my server despite the throughput win.