A 27-billion-parameter model is not supposed to fit on an 8 GB card, and the interesting part is that it does. Bonsai at Q1_0 sits at 4.3 GB resident of 8,192 MiB with the whole model on the GPU, nothing offloaded to CPU, generating at 30.9 tokens per second warm. One-bit quantisation of a model that size sounds like it should arrive lobotomised, and on the task I care about it does not.
The build is what makes or breaks this, more than the parameter count. The BF16 releases use a custom architecture that stock Ollama refuses outright, so the natural move — reach for the highest-quality file — returns a 500 and teaches you the wrong lesson about your hardware. The Q1_0 is standard LLaMA and loads without argument. Context is cheaper than I assumed too: 32K costs about 1 GB over 4K, measured at each size rather than extrapolated from one, so the 4,096 default leaves real capability unused for a saving not worth having.
On twenty system descriptions, scored the same way by the same grader, the 2.6B house model reached 49% node-label F1 and Bonsai reached 81%. Twenty rows resolve about eight points, so thirty-two clears the bar comfortably. That is a genuine finding about quantisation rather than about my card: 27B-class knowledge survives compression to one bit on this task, and the small model’s ceiling is not a hardware problem.
Two things point the other way and belong here rather than in a footnote, because both are inconvenient for the result I was pleased with. Bonsai failed schema validation on 1 of 20 where the small model passed all twenty — small n, and it is the wrong direction for a model being considered precisely because you want structured output from it. And it has no num_predict cap at all; ollama show --parameters comes back empty. It spent 719 tokens answering a two-sentence question, which is why the honest cost is 107 seconds per row and 2,144 seconds for the set, rather than anything you would predict from 30.9 tokens per second.
That second one is a configuration fault wearing a performance result’s clothes, and separating them matters. At 30.9 tok/s, 107 seconds is roughly 3,300 tokens spent on a task whose answer is a short JSON object — the model reasoning until it decides to stop, on a question that needed none of it. The same unbounded shape wedged this card a day earlier with an entirely different model, which is what makes me think it is the default rather than the model. A cap sized to the longest legitimate answer would cut most of those 2,144 seconds without touching a single one of the 81 points. A seventeen-fold cost gap that is partly self-inflicted is a different fact from one that is intrinsic, and should not be reported as the same thing.
That cap has since been set and tested, and it came back with a correction to the paragraph above: on this model a cap does not shorten the answer, it deletes it. The budget is spent inside a thinking block the adapter strips, so a limit that expires before the block closes returns an empty string rather than a truncated one — the cap that returned nothing. The cost figures here are unaffected; what changed is that “sized to the longest legitimate answer” turns out to mean sized to the longest legitimate deliberation, which is a larger and much less obvious number.
And then the comparison that decides anything went the other way entirely. The 2.6B model scored 80% on those same twenty rows once its instruction included one sentence naming what to attend to — statistically level with the 27B, at a seventeenth of the wall clock. The thirty-two point gap is real, reproducible, and irrelevant, because it closes for free with a line of prompt.
I spent a morning getting a large model onto a small card, which was fiddly and satisfying and is the kind of work that feels like progress while you are doing it. The comparison it was meant to settle was won by editing a string. So the small model stays the house model and Bonsai stays installed as the reference, which is a real job rather than a consolation: it earns its place where no worked instruction exists for a task yet, and as the honest check on whether the cheap prompt is still holding up. Keeping a 27B around to audit a 2.6B is a reasonable use of eight gigabytes.
The general version, which I am fairly confident about and intend to keep testing: on a task a small model can nearly do, a much larger model and a better prompt buy about the same thing, and only one of them costs anything per call. Which means the model comparison is the wrong experiment to run first. Until you have found the sentence, every benchmark you run between two models is mostly measuring your prompt, and the bigger model is just the more expensive way to paper over an instruction you have not written yet.