Gemma 4 31B, thinking off vs on.

Why this matters

Every open-weight model worth running locally now ships with a reasoning switch. Flip it on and the model talks to itself before it answers; flip it off and it answers straight away. The vendor pitch is that thinking makes the model smarter. The practical question for anyone running agents on their own card is narrower and more useful: what does thinking buy on a real task, what does it cost in seconds, and when is it worth paying?

This run answers that for one model on one hard task. Gemma 4 31B, 4-bit, on a single RTX 5090, asked once to write a complete 3D island scene in Three.js from a blank page. No retries, no tools, no feedback, and no way for the model to see what it drew. Same prompt, same seed, same server, the only variable being whether the reasoning phase is on.

The short version, in plain terms:

• Thinking bought diligence, not vision. With reasoning on, the model did the things it already knew it should do. It wrote a real height function and placed every tree on the ground. With reasoning off, it wrote a comment admitting it was skipping that step, and the island came out as a cone of shards.
• The price was tokens, not speed. Decode ran at the same 65 to 67 tokens a second either way. Thinking on simply emitted 3.8 times as many tokens, about 4,800 of them hidden reasoning, so the answer took 124 seconds instead of 31.
• Both ran first try. Two for two in headless Chromium with zero console errors and the same 7 of 8 features. On this task, thinking did not change whether the code worked. It changed how good the result was.
• Neither run could fix what it could not see. Both left the ocean as a square with visible corners and lit “late afternoon” like noon. More reasoning does nothing for a flaw the model would have to look at to notice.

The rest of the page is the evidence: a synchronised split-screen of both runs, every number the harness read out, the lines of generated code that explain the gap, and the exact commands to repeat it.

What I measured

The prompt asks for a self-contained HTML file: a procedural island in late-afternoon light, water, sky, vegetation, shadows, a slow camera orbit, a seeded PRNG so the scene is identical on every load, and a handful of hooks on window so a headless browser can drive the camera and read the renderer. The full prompt is at the bottom of the page. The harness then loads the file, checks the scene for eight features, renders five fixed camera angles and reads the renderer’s own mesh and triangle counts. Nothing on this page was judged by eye except the one-line verdict on the video card.

Results

The feature checks are mechanical: the harness loads the page, waits for __ready, walks the scene graph for a terrain mesh (a big geometry with real height), a flat transparent water plane, vegetation (instanced meshes, or at least ten small props), fog, an enabled shadow map with a light casting into it, a sky (a huge inside-out mesh or a scene background), and a camera that moves on its own, then renders five fixed azimuths and reads renderer.info. “Horizon” asks whether the edge of the water is hidden: it passes when the water plane reaches past the fog’s reach or the camera’s far plane, so the ocean looks endless. Both runs failed it the same way, which is the most interesting line in the table.

What the thinking bought

Diligence, not perception. The thinking-on run spent its reasoning on bookkeeping: it designed a Mulberry32 PRNG before writing it, decided on a getHeight(x, z) function and then used it to place every trunk and every canopy on the ground, scaled the terrain detail up to a 128×128 grid, and placed six times as many trees. The result reads as an island.

The thinking-off run shipped the same architecture with the corners cut. The terrain is there, the fog is there, the shadows are on, but the tree placement is this, verbatim from the generated file:

// Simple height check: only place on "land"
tree.position.y = 0; // We'll raycast or just guess height for simplicity
// For the sake of a single file without complex sampling,
// we'll just place them and let them overlap slightly or use a height map logic
// But here we just use a simple offset based on distance
tree.position.y = Math.max(0, 2 * (1 - d/15));

That comment is the whole benchmark. The model knew the right answer (sample the height map) and wrote down that it was skipping it. With reasoning on, the same model does not skip it. That is what 6,000 extra tokens and 93 extra seconds bought: the model doing the things it already knew it should do.

What neither run bought is an eye. Both put the island on a flat square of water with visible corners and a hard edge against the sky, and both lit “late afternoon” like noon. Those are things you notice by looking at the picture, and the model never looks at the picture. No amount of thinking fixes a problem the model cannot perceive. If I want the horizon fixed, the fix is a feedback loop that shows the model its own render, not a bigger reasoning budget.

What to do with this

If you run a local model as a coding agent, this is the rule of thumb the run supports. Leave thinking off for anything with a feedback loop. When the model gets a compiler error, a failing test or a screenshot back, the loop supplies the rigour and the fast setting gets you more iterations per minute. The thinking-off island was wrong in ways a second pass would have caught.

Switch thinking on for one-shot work the model cannot check. A single file, a migration, a script that runs once, a scene nobody will look at until it ships. There the extra two minutes of reasoning are cheap next to a wrong answer, and this run shows the model using that budget on exactly the bookkeeping that separates a working draft from a good one.

Budget it in seconds, not tokens. On a 5090 at this quant, thinking turned a 31 second answer into a 124 second one. Decode speed did not move, so the cost of reasoning scales with how much the model decides to think, and on this prompt it thought for roughly 4,800 tokens before writing a line of code. On a slower card multiply accordingly.

Do not expect thinking to replace eyes. The one failed check, the horizon, failed identically in both runs. If the task has a visual or empirical judgement in it, the fix is to show the model its own output, not to let it think longer.

Caveats

This is n=1 per setting. Temperature was 0.7 with a fixed seed, so each run is one sample, not an average, and a different seed could flip the smaller numbers. I ran it this way because the point was to record one honest attempt per setting end to end, and the two videos below are those attempts. Treat the direction of the result (thinking buys rigour on things the model can reason about, nothing on things it would need to see) as the finding, and the exact counts as one draw.

The single-run cards, same harness, same orbit:

Reproduce it

Serve the model with vLLM on the OpenAI-compatible port. Thinking is switched per request through the chat template, so one server does both runs. The weights were cyankiwi/gemma-4-31B-it-AWQ-4bit, served as gemma-4-31B on port 11434; context length and the rest of the server flags are yours to pick, the prompt is 305 tokens and the longest answer was 8,136.

# server (one RTX 5090, AWQ 4-bit weights)
vllm serve cyankiwi/gemma-4-31B-it-AWQ-4bit --served-model-name gemma-4-31B --port 11434

# one pass, thinking off
python run_pass.py --host http://127.0.0.1:11434 --model gemma-4-31B \
  --name "Gemma 4 31B" --org google --chips "AWQ 4-bit" "local · RTX 5090" \
  --seed 7 --temperature 0.7 --run-id gemma-4-31b-seed7

# same pass, thinking on (sets chat_template_kwargs.enable_thinking)
python run_pass.py --host http://127.0.0.1:11434 --model gemma-4-31B \
  --name "Gemma 4 31B" --org google --chips "AWQ 4-bit" "local · RTX 5090" \
  --seed 7 --temperature 0.7 --thinking --run-id gemma-4-31b-think-seed7

# split-screen card from the two runs
python build_compare.py runs/gemma-4-31b-seed7 runs/gemma-4-31b-think-seed7 \
  --labels "thinking off" "thinking on" --name "Gemma 4 31B" --org google \
  --chips "AWQ 4-bit" "local · RTX 5090" --out gemma-4-31b-thinking-off-vs-on

run_pass.py sends the prompt, extracts the single ```html block from the reply, opens it in headless Chromium through Playwright, waits for window.__ready, runs the feature checks, renders the five azimuths through __setCamera, writes analysis.json, and records the card. The harness lives with the bench and will go up with the source once the second model is through it.

The prompt, verbatim:

Build a 3D island scene as a single self-contained HTML file.

Rules:
- Use Three.js r169 loaded as an ES module from exactly this URL: https://cdn.jsdelivr.net/npm/[email protected]/build/three.module.js (an importmap is fine). No other external files, textures, models, or libraries.
- Everything is procedural. Do not use Math.random anywhere. Implement a small seeded PRNG and seed it with 7 so the scene is identical on every load.
- The canvas fills the browser window and resizes with it.
- Expose these on window: __renderer (the WebGLRenderer), __scene, __camera, and __setCamera(angleDeg), which places the camera on an orbit around the island at that azimuth in degrees, looking at the island, and renders one frame. Set window.__ready = true after the first frame has rendered.
- The scene keeps a slow camera orbit going on its own when __setCamera is not being called.

The scene: an island in the ocean in late-afternoon light. Terrain rising from the sea with a natural coastline, water, sky, some vegetation, and lighting with shadows. Make it look as good as you can.

Return only the HTML file in one ```html code block, with no explanation before or after.