It looked like a result

Published September 22, 2026

Life is a Game

I like playing video games, a very specific genre, non-real-time strategy. The best example is Civilization. I legally owned every version starting with version 1, since 1991. A copy of the two floppies of Civ1 is tucked neatly on two separate external HDDs. It is deep nostalgia. I never played an FPS in my life, did not own a console and the closest I came to that particular world was playing an hour or so on Wii at a friend’s, I think it was tennis. End of story.

Non-real-time, or turn-based (not the same thing) means I can save the game and shut down. Think about my moves when I am doing something else and when I come back, it is waiting for me in the state I left it. How this is relevant: Some games have very beautiful graphics. I do not need a cutting-edge GPU to play an FPS. I need a mediocre one to enjoy the graphics of a modern-day “No Man’s Sky”.

Oh, by the way, if it can also run some local LLMs, that is a bonus.

A Tale of Two GPUs

My budget allowed for an RTX 5060 Ti with 16 GB VRAM. My mistake was not checking the second-hand GPU market at that time. Looking at the current prices in Turkey, where I live, a second-hand RTX 3090 with 24 GB is just a little bit more expensive than a brand new 5060 Ti with a difference that is inside my budget. That ship had sailed and I am 8 GB short of what could have been.

I am bombarded with all kinds of YouTube videos on running local LLMs. Some are about small PCs with 128 GB unified memory to run AI or Apple silicon. I watch some, just to be on top of things but you will agree, they are not that relevant. Others claim to run very powerful LLMs (local and powerful, that is) on a 3090 or even a 16 GB GPU. My land.

In Fourteen tokens per second but all mine and Twenty tokens per second, same card I wrote about how I tested two boxes with the same model of GPUs but different surrounding hardware (CPU/RAM) that produced very similar results for tokens/sec, one metric. That was for my own mental health. People smarter than me can and did better in that metric, with similar hardware and the details are all over the internet. Enjoy.

Something does not feel quite right, though.

You Can Only Control What You Measure

I ran a fixed prompt with warm runs, two passes each for an LLM, and saw that CPU make and model and RAM did not affect the tokens/sec, if you do not spill over to the rest of the hardware and stay in the 16 GB VRAM. That was “hoped for” and proven to be the case for my two boxes. Granted, ollama bumped a version between my runs on two geo-apart locations and that just happened while I was traveling. As the results more or less matched, I am assuming whatever changed in that version bump did not affect tokens/sec, for the same model.

One finding was about context window setting, which I wrote about in detail. But again, I felt I was being part of the herd (few exceptions on YouTube noted and saluted) and that is the part that does not “feel quite right”. I was just taking it a little bit further. Most of the reviews for local LLMs look at the parameter size, I also checked the context window to see if the model fit in 16 GB. I was not the genius, others did that, too. I am still part of a (smaller) crowd.

I reported results on 5 different LLMs. I had 10 downloaded and ready inside ollama. It could have been 100, for all I care. The problem is the list is very fluid. LLMs will come and go, tokens/sec will change. Reporting on my personal “pick of the day” is just what the field was doing. New today, stale tomorrow.

The “Method”

The hype as of writing this piece is a slew of 27-billion-parameter models that can do amazing things for the cost of your utility bill instead of paying for a cloud frontier model. If you have the GPU, already. The outcome is not equal, of course, but not all of us are after Millennium Prize Problems from the Clay Mathematics Institute, anyway.

Please focus on the words “amazing things”.

The hype reports in terms of tokens per second, and tokens per second measures how the engine runs the model. It says nothing about whether the answer is any use. Two models can run at the same speed and only one of them can do the job in front of you. What follows is a “method” for the question the metric leaves open: for this work, on this card, which model and which settings deliver and how often.

Ten steps, in order. The first four decide what is being tested. The next four decide whether the result means anything. The last two keep it honest. (ok, I know, “The ten … ” is maybe the most common click-bait used in YouTube videos. This is not by design, it happened to be 10. I even thought of combining some to make it less than 10 to escape the cliche.)

Write the task down before you shop for anything. A page, in your own words, describing the work the model has to do. The spec must not name a model, because a specification written after you have seen the field is a ranking of the field. Written first, it is a test the model either passes or fails. This part is mostly about being honest with yourself, and believe me, it is one of the hardest things to do: you are always the most easily deceived, if you decide to deceive somebody.
The fit gate. Decide what is going to hold the thing before you measure it, because that decision caps everything that comes after. The naive rule is the obvious one: if it does not fit in the fast memory, it is out of reach. That is not the rule anymore: there is something called “smart engineering”. AI Labs all around the world are trying to find ways to overcome “the fit” issue. Check MoE models, for example.
The working set may be bigger than your GPU VRAM, but it does not all have to live in the same place. It has to live somewhere and where each part of it lands is just the thing you decide rather than a fact you inherit. The order is not arbitrary at all: whatever gets used every single time belongs in the fastest place you have and whatever gets used occasionally belongs in the place with more room. The question is not whether everything fits, as long as you know where it went. Beware: Get it wrong by accident and you still get a number. Settle it first, because every measurement after it is taken on top of it.
Fix the question, and prove the settings took effect. One prompt, asked the same way every time. Warm runs, two passes each, models evicted between configurations, numbers taken from the API rather than read off a screen. Better yet, multiple prompts for all the things you plan to do with the local models. Expand. And write the settings down with the question, not just the question. The settings move the result more than the thing you think you are comparing, and a figure without them is not a measurement, it is a mood. Then prove the setting did something, because nothing else will tell you. A setting can be accepted and quietly do nothing. A wrong result can look exactly like a right one. The thing you measured can be replaced under its own name. None of that raises an error. So change it deliberately, once, and confirm the output moved with it. If nothing moves, you are not measuring what you think you are measuring.
Judge before you time. Run the candidates in a randomised order, repeat every run, and do not know which model produced which answer until the scores are written down. For that, “hire” a frontier model to write the code for you and keep you at arm’s length. Close your eyes (metaphorically) and try to stay in the dark. Be careful what you compare against. The honest baseline is not the perfect version that nobody uses. It is the ordinary one you would otherwise have used. Measured against the ideal, anything looks like a miracle. Compare against your real alternative, or the comparison is decoration.
Score against your own failure modes, one question per failure. Not is it correct: that is several judgments hidden inside one question, and the answer averages them into nothing. Ask instead whether the answer commits each specific fault you already know a bad answer commits. The list comes from you. A question taken from your own words beats a question written for you.
The output is binary. Acceptable or not. No partial credit, because in real life you either use the answer or you ask again.
Minutes per acceptable output. The only number here that survives a hardware change, because it counts the runs you threw away. A fast model that passes two times in five is slower than a slow one that passes five times in five. Count everything the run spent, including the part that produced nothing you kept. A run can look fast and still be slow, if the work is happening somewhere you are not counting. The flattering number hides it. Minutes do not.
Report the worst run, not the best. The best run is the one you got lucky on. Luck is a rare find in real life. Do not depend on it. Ever. State the conditions, every time: what you set, what you assumed, and what you left at its default. Almost every impressive figure in circulation is a best case measured with those conditions cropped off the end of the sentence. Yours will be one of them unless you write them down next to it.
Find the boundary, not the winner. What you want is where the model stops being usable and which setting moves it. Find it on your own setup, because improvements do not travel. Something that transforms someone else’s result can do nothing at all for yours, and the result somebody is selling may depend on conditions you do not have. The boundary is a property of your setup, not of the thing you are measuring.
Write down when you would re-test. The list of models is fluid. New today, stale tomorrow. A result with no expiry date is a claim, not a measurement. Pin exactly what you ran, because the thing you measured may not be the thing that is there next month. It happened to me: A careful published result can become unreproducible in weeks, not through anyone lying, just through the ground moving underneath it. I understand that is something we cannot control. How can you know when the next amazing model will drop? Try your best to remain on top of the news. YouTube will provide enough hype, don’t worry. Unashamed self-promotion: Check Winnow in Generalized, Sanitized.

The Judge

The method says the output has to be judged. That is the expensive part. My AI agent made me read the output of 20 runs, and label them as “acceptable” or “not quite right”. It takes time and focus. So the obvious question is whether a machine can do it instead of me.

Five models, four runs each, all of them on the prompt I have used since Fourteen tokens per second but all mine: “Explain what a database index is and when it stops helping, in two short paragraphs”. Eighteen of the twenty came back as two paragraphs and two did not, which matters in a moment. I judged all twenty by hand with my reasons written down. Boring after the third. Reading the same explanation with subtle wording differences and trying to spot missing or conflicting information was hard. One of them said “higher selectivity in an index is bad”. That was the only statement that was outright wrong, the others needed that focus I mentioned, fully.

Then I ran the same twenty past a judge model and counted how often we agreed. Before I ran it, I wrote the decision rule down: Eighteen out of twenty or better and the instrument is usable. Fourteen to seventeen and it is good enough to screen with, not to decide with. Thirteen or worse and either the rubric is wrong or the judge.

A threshold chosen after you have seen the number is not a threshold, and the method says to write the question down before you shop.

I used TypeSafe’s Jev. It is a different kind of model. It does not write prose, it returns probabilities for typed questions. You give it state, you ask it a question with a yes or a no, and you get a number back. When I was writing this piece, it was “the hype”, I got carried away, I confess.

Sixty requests cost me $0.001875. Three thousandths of a cent an answer.

The first thing I checked was whether it holds still. Three passes over the same twenty inputs moved by 0.007 to 0.042 on average, and 0.09 at worst. The models it was judging do not hold still like that. So the instrument was reproducible, which was more than I could say for the thing I was pointing it at.

Then my rubric fell over.

I wrote four questions: Is it technically correct? Does it answer the question? Does it hold the format? Is it usable as it stands? The format question scored both single-paragraph answers at 0.78 and all eighteen two-paragraph answers between 0.07 and 0.27. I had asked it to compare a paragraph count against a number. Jev cannot count. The vendor says so in its own documentation and I had read that and written it into my notes before I built the question.

The question about answering the question scored between 0.87 and 0.97 on all twenty, including the ones I had already thrown out. It measured nothing at all.

Two of my four questions were broken. Sixteen out of twenty is what that run gave me, and two broken questions are enough on their own to explain it. By the rule I wrote down before I started, sixteen is screening only.

So I threw the rubric away and built a new one out of my own written reasons. Four questions, each one asking whether an answer commits one specific false claim, and each claim taken from a criticism I had already written by hand. Nineteen of twenty agreed.

That number is a fit and not a measurement, and the difference matters. I learned it the hard way when I was working on my long-duration Trading Methods project: You can always find a method that fits your goal, if you tweak enough parameters, enough times, most of the time out of pure luck. That is the part again for you: Deceiving yourself. I derived the questions by reading my reasons for four specific answers, then scored them against the same twenty. They fit the set they were built from. Nobody has tested whether they catch the twenty-first answer, and when my AI agent suggested I read twenty more index definitions to find out, I declined, because it was boring. In a piece about honest measurement, I should say that plainly. The last experiment was called off for being dull.

What it could not do is the better finding.

My standard turned out to have two halves. Nothing false is a property of the text. A claim is untrue or it is not. Nothing missing is a property of the text relative to what it was trying to do. The judge could not tell an answer that attempted a list and got it wrong from an answer that never attempted one. That same 19 of 20 is the first half. The second did not reproduce at all.

And specificity beat generality every time I tested it. One question covering two identical errors scored 0.93 on one answer and 0.13 on the other. Two separate questions scored 0.97 and 0.91. A question asking whether an answer contains a false claim scored 0.27 on one worth keeping and 0.29 on one worth throwing away.

Every question I took from my own words beat every question written for me. Which is the same lesson as the rest of this piece, arriving from the other direction. The instrument was never the hard part.