The 5090 receipt run.

A $2,000 gaming card just read an entire novel in one gulp and answered a question about one sentence hidden in the middle of it, in about a minute. Two years ago that was a datacenter demo. The whole trick is that shrinking the model's weights by 10 GB is the difference between the model refusing to load at all and a six-figure context window on hardware people already own.

Every published number for this model's 4-bit quants was measured on datacenter Blackwell. The consumer path (SM120) is real but undocumented: it fails four distinct ways before it works, and none of the error messages name the cause. This page is the missing receipt, with the exact failure modes, the exact fixes, and the eval numbers those fixes produce. To be honest about it, this is not a research breakthrough. It is an access and reproducibility result, and that is exactly what makes it useful.

The model itself is a genuine oddity worth the ink: only 16 of its 64 layers carry a KV cache. The other 48 are linear attention with a small fixed state. That architecture choice, not magic, is why 32 GB holds a novel.

Official Qwen FP8, 30.9 GB: dead on arrival

The most-downloaded quant of this model. The weights alone consumed 28.51 GiB of the card's 31.36 GiB at load, the warm-up forward pass pushed the allocation to 30.35 GiB, and the boot died there.

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 144.00 MiB.
GPU 0 has a total capacity of 31.36 GiB of which 115.75 MiB is free.

QUASAR QAT NVFP4, 20.6 GB: boots at 131,072 context

The weights loaded in 18.48 GiB, and the hybrid architecture does the rest:

GPU KV cache size: 75,264 tokens
Maximum concurrency for 131,072 tokens per request: 2.22x

A 75,264-token pool under a 131,072-token window looks like a contradiction, but it is the hybrid design at work: paged KV for the 16 attention layers plus a fixed per-sequence recurrent state for the 48 linear layers. Empirically, the 103K-token prompt processed without complaint.

Each of these killed the boot or the eval, with an error message that does not name the cause. In the order I hit them:

Wall 1: the CUDA toolkit is too old, and the error will not say so

flashinfer JIT-compiles the SM120 FP4 GEMM kernels at boot and requires nvcc 12.9 or newer. My PATH toolkit was CUDA 12.0. The error is just No supported CUDA architectures found for major versions [12]; the version requirement lives in flashinfer's source, not in the message.

fix: CUDA_HOME=/usr/local/cuda-13.2 · PATH=/usr/local/cuda-13.2/bin:$PATH

Wall 2: a silent 10 GiB tax for an image I will never send

The model has a vision tower, and vLLM budgets memory by profiling the preprocessor's worst case, which this model's config sets to a 16,777,216-pixel image. Datacenter cards never notice. On 32 GB it is the whole margin.

fix: mm_processor_kwargs={"size": {"longest_edge": 262144, "shortest_edge": 65536}}

Wall 3: the hidden per-seat cost of linear attention

The 48 GatedDeltaNet layers carry about 0.3 GiB of recurrent state per sequence slot, charged before the KV pool exists, and vLLM's default slot count is sized for servers. The boot still OOM'd after wall 2 until I capped the slots.

fix: max_num_seqs=8 · max_num_batched_tokens=2048

Wall 4: perplexity evals bring their own memory monster

A rolling loglikelihood pass materializes a prompt-logprobs buffer of roughly 5 GB (an 8,192-token window times a ~150K vocabulary) outside vLLM's profiled budget. At 0.92 utilization the engine died about one second into the first request, twice, with nothing but EngineDeadError to show for it. At 0.75 it sailed through.

fix: gpu_memory_utilization=0.75 for loglikelihood evals (0.92 for generation)

One bonus wall, half height: flashinfer autotunes its FP4 GEMM shapes on first boot, about 70 minutes of tuning at roughly 35 seconds per shape. I skipped it with kernel_config={"enable_flashinfer_autotune": False} and boots dropped to about two minutes.

I put all of Frankenstein (the Project Gutenberg text, 103,459 tokens after templating) into one prompt. Mid-book, splitting a word in Chapter 14, I planted this:

(By the by: the harbour-master logged the vessel's registry number as GUGG-5090, a detail he found oddly memorable.)

The task: quote the one modern insertion exactly, then explain in three sentences what the creature asks Victor for and why the refusal dooms everyone he loves. The model quoted the needle verbatim, punctuation and parentheses intact, and produced a correct synthesis:

The creature asks Victor to make him a female companion, another being as deformed as himself, so that he can have fellowship and will leave humanity in peace. Victor refuses because he fears that a second creature might become even more malignant and bring danger upon the world. By denying the creature the only happiness he can receive, Victor drives him into despair and revenge, so the creature kills or destroys the lives of Victor's loved ones, including William, Justine, Clerval, Elizabeth, and his father.

One disclosure: the 2,048-token output cap caught the model mid-reasoning stream, so the saved response is its working-out with the answer embedded. The needle appears quoted exactly three times and the synthesis is fully formed. Sampling: temperature 0.7, top_p 0.8, top_k 20.

Harness: lm_eval 0.4.12 driven through its Python API, since the CLI cannot pass the nested consumer-card kwargs. gsm8k is the 5-shot default. These are disclosed-n probes, not leaderboard runs; they establish a same-harness baseline, which is their whole job.

I did not log GPU telemetry during the runs. No temperature, power draw, clock, or VRAM-over-time trace exists for the novel test; the runs recorded vLLM's own memory accounting and throughput.

Also disclosed: gsm8k and wikitext are 250-sample and 62-window probes at 8K context, not full leaderboard runs; the wikitext stderr is N/A by construction (rolling perplexity); the 200K ceiling was measured as an OOM, not a near-miss; and vLLM 0.20.0 crashes when images are disabled outright, which is why the fix is a cap rather than a zero.

# every GPU stage runs under this prefix
export CUDA_HOME=/usr/local/cuda-13.2
export PATH=/usr/local/cuda-13.2/bin:$PATH
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
PY=~/.venvs/vllm/bin/python

$PY receipt.py fit Qwen/Qwen3.8-27B-FP8                                         # the OOM
$PY receipt.py fit QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 131072                   # the boot
$PY receipt.py book QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4                         # the novel
$PY probe_receipt.py QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 probe-quasar gsm8k 8
$PY probe_receipt.py QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 probe-quasar wikitext 1 0.75

vLLM 0.20.0 · lm_eval 0.4.12 · CUDA 13.2 · RTX 5090, 31.36 GiB usable · fp8 KV cache · 256K-px image cap · measured 2026-08-31