ML Harness — Max Naidenov

ML Harness exists because the difficult part of training a model, for someone who has not done it before, is almost never the training. It is knowing what to train, finding or building data worth training on, choosing between the three things people mean when they say “train”, and finding out — before you spend a night on it — whether the machine in front of you can hold the thing you just described.

It runs locally, on one consumer GPU, with a model you attach yourself.

The loop it has to pass

Someone arrives wanting a model. Say they want a 14-billion-parameter model that can use tools. The harness should:

Propose an approach, and say plainly which of the three it is — pretraining, fine-tuning, or a low-rank adaptation. They are not interchangeable and the difference is most of the cost.
Get data. Download an existing dataset, or generate one from a description of the behaviour wanted. The second is the interesting case and is roughly what Adaption Labs shipped as Invent a Dataset.
Check the machine honestly, and refuse when it cannot fit, with the number that made it refuse.
Run it, and produce something usable at the end.

Point four is the whole thing. The output I care about is a model I can actually use and put on Hugging Face, not a score in a table.

Why it is built the way it is

Every number carries where it came from — measured, stated, inferred or defaulted — and the interface shows which. This sounds fussy until you have been burned by a number that was a default wearing the clothes of a measurement.

Generated rows are treated as the most dangerous thing in the system. They cannot open a gate and cannot be trained on unverified. What is explicitly not claimed is that the harness can tell a good generated row from a bad one — it cannot, which is exactly why a person reads a sample.

An asserted number opens nothing. Three ledgers declare gates, and a gated outcome is reachable only through its own ledger’s gates, enforced by tests rather than by intention.

The thing that scores a model is the environment that made it. A LoRA adapter is neither an HTTP endpoint nor an Ollama model, so nothing outside the training sandbox can load one. The recipe that trains an adapter also evaluates it, in the same pinned virtualenv — and grading stays in the harness, so the baseline and the adapter meet the same instrument. Two graders would be two instruments, and two scores from two instruments are two facts rather than a comparison.

What it has caught

The most useful output so far has not been a model. It has been the harness catching its own instruments being wrong.

A judge kept 19 of 72 rewrites that had nothing wrong with them, and explained every one with a deletion that never happened. The judge that invented deletions.

A test suite printed Ran 3315 tests ... OK on a tree where discovery found 3,822. A runner reports what it ran, never what it was supposed to run. Running what nothing ran.

A statistic rewarded exactly the failure it was meant to detect. The statistic that rewarded failure.

A corpus count that never said “distinct” was not a size at all — 95 was 60. The count that never said distinct.

There is a pattern in those and it is the reason I keep writing them down: the instrument is wrong more often than the result is. A check that returns something true and useless, in the right format, at the right time, is the defect I meet most.

Where it is now

The training path is real — LoRA and DPO recipes in their own pinned environments, with a preflight that refuses a CPU-only PyTorch rather than running twenty times slower in silence and telling you nothing. The suite stands at 4,594 tests.

What it has not done yet is produce a verified win on held-out data. The held-out set for the first honest attempt — a small model taught to emit consistent architecture structure — is written and committed before its training data exists, which is the only order that makes the result mean anything.