Fourteen tokens per second but all mine

Published September 8, 2026

This is something out of band. Something I realized and wanted you to know. I’ll keep it short.

The benchmark article says my card does 90 tokens per second. mistral 7B on an RTX 5060 Ti 16 GB, measured and published. One afternoon, the same card gave me 14. Same prompt. Same machine, even.

The card was never the problem.

Two boxes, one card

I keep two desktops. LIGHTS lives at the “Lab”, where I am writing this. DARKNESS lives at the “Factory”, where I am not, for a few more days. Well I also have a “Home”, don’t worry. Same GPU in both: RTX 5060 Ti, 16 GB. I bought it for memory size, not execution speed. It was the highest VRAM I could afford in this country at that time and the models I run care about VRAM before they care about anything else. That is beginning to change in a good way.

Same card in two very different machines. This one is about an afternoon on LIGHTS, and a number that made no sense.

The rules of the afternoon

ollama 0.33.3, driver 616.56, Windows 11 with WSL. One prompt, fixed, asked the same way every time: “Explain what a database index is and when it stops helping. Two short paragraphs.” Every model ran twice, warm, and the numbers came straight from ollama’s API, nanoseconds and all. The Windows desktop eats about 1.2 GB of the 16 before anything starts. That is the tax for having two monitors and doing other stuff when benchmarking. I know, that is not best practice but I have a lot going on, lately.

The table

Read the llama3.1 rows again. Same model, same hardware, same prompt. Five times slower, and the only thing that moved was one setting.

The trap

When a model answers, it carries a scratchpad: the KV cache. An index over everything in view, so the model never rescans the conversation to write the next token. The index exists whether you query it or not, and its size is the size of the view. ollama’s parameter for the view is num_ctx.

Here is the part I did not know before this afternoon. Left to the app’s own settings, llama3.1:8b opened with its full window. ollama show reports “context length 131072”, and the 15.5 GB in the table above is the receipt. A 128K window asks for more cache than this card holds, and ollama does not refuse. It quietly splits the run, part in VRAM, part in system RAM, and every token crosses the PCI express bridge for the model’s working set. VRAM moves around 450 GB per second. The bus, a fraction of that.

Tonight the assistant went back to the machine, on my order. The documentation says a card like mine should default to 4K, and the server log shows it computing exactly that at startup: 16 GB detected, default 4096. Then it asked for the same model with no settings, and the same log shows the runner being launched with 131072, the model’s maximum, anyway. ollama ps says it out loud: full context, 36 percent CPU, 64 percent GPU. The suspect was found later, on the settings screen: the app’s Context length slider sits at 128k, and the runner obeys it, capped by each model’s own maximum. One setting, one open case file, closed. The cure is already below.

Don’t take my word for any of this. Here is the script, the run I did myself, and the machine’s own diary. A reading note first: the table up top is the first bench of the afternoon, and the config lines below are my own run of the same script, a separate session. Run to run the digits wiggle, 79.1 below against 75.2 up top. The story does not: 14 at default, 74 to 79 with the knob at 4096, in both sessions.

function Bench($model, $label, $ctx) {
    $body = @{ model = $model; prompt = $prompt; stream = $false }
    if ($ctx) { $body.options = @{ num_ctx = $ctx } }
    $r = Invoke-RestMethod -Uri 'http://localhost:11434/api/generate' -Method Post -Body ($body | ConvertTo-Json) -ContentType 'application/json'
}
# the default runs pass $null - no options at all:
foreach ($i in 1..2) { $results += Bench 'llama3.1:8b' "llama3.1:8b default / run$i" $null }

config     : llama3.1:8b default / run1      gen_tps : 14.2    vram : 15453 MiB
config     : llama3.1:8b default / run2      gen_tps : 13.8    vram : 15866 MiB
config     : llama3.1:8b ctx4096 / run1      gen_tps : 79.1    vram : 6835 MiB
config     : mistral:7b default / run1       gen_tps : 82.7    vram : 9876 MiB

$ ollama ps
NAME           ID              SIZE    PROCESSOR          CONTEXT    UNTIL
llama3.1:8b    46e0c10c039e    23 GB   36%/64% CPU/GPU    131072     4 minutes from now

$ Select-String -Path $env:LOCALAPPDATA\Ollama\server.log -Pattern "default context"
msg="vram-based default context" total_vram="15.9 GiB" default_num_ctx=4096

$ Select-String -Path $env:LOCALAPPDATA\Ollama\server.log -Pattern "llama-server.exe"
llama-server.exe --model E:\AIModels\.ollama\blobs\... -c 131072 -np 1 ... --context-shift ...

That is the 14. The card was doing its job. The default setting quietly spent it.

The rule

Once you see it, the whole table obeys one rule: the window is whatever num_ctx says or, on this machine that afternoon, the model’s own maximum, since the slider had already said 128k. The only question that matters is whether that window fits the card. The control writes itself: models whose maximum fits the card, run at default, no knob touched.

gemma2:9b caps at 8K, its biggest possible bill fits, so its default is fast. qwen2.5 7B caps at 32K, same story. mistral caps at 32K: 80 and 81. Exactly one model on my list carries a maximum that does not fit this card: llama3.1, at 128K. Exactly one model crawled. Whether ollama’s default is free or catastrophic depends on one question: does the max fit the card. On 16 GB, the 128K models are the trap. The published numbers never run into the trap: mistral’s 32K window fits a 16 GB card, so its default is harmless. The trap needs a model whose maximum does not fit.

What I did not own

Full disclosure, because this is a ledger and not a brochure. The afternoon was not my idea. My AI assistant designed the benchmark, ran it first, and handed me a script to run myself. I executed it. The numbers matched the ones already sitting in the draft of this post; the draft was written before the run happened. It took an outside editor asking one question about this table for me to notice I could not defend a single digit on it. So I went back to understand my own machine, and this section is that going-back. The numbers stand. The measuring is shared. The byline is mine, and so was the not-knowing.

The knob and its price

num_ctx 4096, and the same model does 75. VRAM drops from 15.5 GB to 6.5. You get that with one line in the settings.

Before anyone declares context useless: it is not. My video pipeline runs qwen2.5 14B at 32K on purpose; it reads long transcripts, and long transcripts need the room. That costs 15.4 GB of the 16, and it is worth it there. The knob is real. The mistake was letting a setting I had never noticed turn it for me.

In the end it is one line. Inside the ollama session: /set parameter num_ctx 4096. The session runs right, and the setting dies with the session. For something permanent, a two-line Modelfile plus ollama create builds a model that knows its own size. The default was never a law. It was somebody’s guess about my hardware.

Receipts vs claims

The published benchmark for this card says 90.17 tokens per second on mistral 7B. Mine says 80.4 and 81.1. Different ollama version, different driver, different measurement path, different day. Both honest. This is why I publish my own numbers: theirs are true for their box, not mine.

What’s next

DARKNESS gets the same afternoon. Same prompt, same models, same settings, when I am back at the Factory. Its CPU is one generation newer and its RAM is double, DDR5. The GPU is identical. This time I want to know what I am measuring before I measure it. The question that post will answer: when the card is the same, does the rest of the machine move anything at all? Probably, we will see.

I have a suspicion. Suspicion is not a measurement. The post will have the measurement. It does: Twenty tokens per second, same card.