The advice for training on a consumer card is always about the model: pick a smaller one, quantise it, attach a low-rank adapter. All three help. None touches the two terms that actually decided whether my runs fit, and I found that out by reading the estimator’s arithmetic rather than its verdict.
The budget has six terms: base weights, gradients, optimizer states, activations, the logits buffer, and fixed overhead — a CUDA context that exists before a single weight loads, plus whatever the desktop compositor holds. The first three are what everyone talks about, and they are the three a smaller model and an adapter genuinely shrink. Gradients and optimizer states are computed over trainable parameters only, so an adapter reduces them to a rounding error.
Activations are the term the estimator’s own documentation calls the one nobody budgets for, and its note is blunt: LoRA shrinks gradients and optimizer states, it does not shrink activations, and that is the mistake that makes people believe an adapter is free. An adapter changes what you update, not how much intermediate state a forward pass holds — the forward pass is over the whole model either way.
The logits buffer is stranger, and it is the one I want to put on a page. It is computed as sequence length times batch times vocabulary times four bytes. Read that list again for what is missing: no hidden size, no layer count, no parameter count. The largest single term in a small-card training budget is invariant to the size of the model being trained. Swapping a 7-billion-parameter base for a 1.5-billion one leaves it exactly where it was.
What it does depend on is the tokenizer, and the numbers are not subtle. At a coding model’s vocabulary of about 152,000 tokens, with a batch of one, the buffer is 1.16 GiB at sequence 2,048, 2.32 at 4,096, and 4.64 at 8,192. Against the roughly 6.5 GiB this card actually makes available, that last figure is 71 per cent of the entire budget spent before a single weight, gradient or activation is accounted for. Run the same arithmetic for a small model’s 49,000-token vocabulary and 8,192 costs 1.50 GiB instead — the same sequence length, the same card, a third of the memory, because the vocabulary is a third of the size.
Two practical consequences follow, and they are the opposite of the usual advice.
The first is that when a run does not fit, the lever with the most travel is sequence length, not model size. Both of the terms above scale with it — the logits buffer linearly, activations worse than linearly, since the attention part of the formula carries a sequence-squared factor. Halving the sequence halves one and does better than halve the other. Halving the parameter count leaves both roughly alone.
The second is that gradient checkpointing stops being an optimisation. With it off, every layer’s activations are live at once; with it on, the estimator assumes one layer live plus the stored per-layer inputs, at a cost of about 30 per cent more time per step. On a card this size that is not a trade you evaluate. It is a precondition, and a configuration where it is off is not a slower version of the same run, it is a different run that does not happen.
There is a limit on all of this worth stating in the same breath as the numbers, because it bounds them precisely. The activation formula was derived for GPT-style multi-head attention and is approximate for the SwiGLU and grouped-query architectures most current models actually use — the estimator says so itself in the assumptions it returns. The logits term has no such caveat, since it is four multiplications and no architecture, which is part of why I trust it more than the term it sits beside.
The estimator also refuses rather than guesses. If it cannot read a model’s geometry — hidden size, head count, vocabulary — it returns nothing for both terms and says in its assumptions that the largest ones were not computed, rather than falling back to a default and handing you a total. A fabricated total would look identical to a real one right up until the run died.
How wrong is it? The shape of the miss matters more than its size
None of the above says how close the estimator lands. Two runs measured against their own predictions missed by 0.22 and 0.24 GiB while the ratio between prediction and measurement moved from 1.43 to 1.62 — and a difference that holds still while the ratio moves is the signature of a term omitted entirely, not a percentage got wrong. The two readings disagree by more than the headroom either one reports, so a run was designed to separate them and its predictions written down first: the run that was designed before its answer.
What I took from reading this is smaller than a rule and more useful than one. The question will this fit sounds like a question about the model, and the model is the thing you have been choosing all along, so that is where attention goes. On a card like mine, most of the answer is set by two numbers you chose without thinking about memory at all: how long your sequences are, and how large your tokenizer is.