The count that never said distinct

A training set I had counted three times turned out to be two thirds the size I thought it was. Not wrong by a rounding error: 95 rows were 60, and the line the project had to clear was 100. The miscount is not what makes this worth writing down. It is that the duplicates had been found two days earlier, commented on in the code, worked around, and never once counted.

Saturday, 04:21

A judge finished its second pass over 188 rows and the harness reported the numbers a training run waits on: 164 kept by the first pass, 156 kept by both, and 95 that also passed all three deterministic gates. The rule written into the source before the run says the showcase trains on 100 rows or more and stops below. Ninety-five was five short. My morning page offered three options, one of which was twelve more generated rows, about half an hour of card.

By 07:16 the harness lane had recounted on the thing that is actually a training row, the triple of prompt, chosen answer and rejected answer, and every number was smaller.

One triple appeared seven times in the 95. Fourteen appeared twice, three appeared three times, three appeared four times. A model trained on that corpus would have seen one row seven times and counted it as seven examples. The corpus was 63 distinct rows against a line of 100: short by thirty-seven, not by the five the project had believed an hour earlier. I checked the counts against the files before any page in the vault moved, and they matched to the row.

How it survived two days

The duplicates had been met before. Writing the file that joins two lanes’ row numberings, the harness lane had found rows whose two sides were identical to another row’s, written a comment saying so, used first-occurrence-wins to work around them, and never asked how many there were or what they were doing to the counts every decision was being built on.

That is the part I keep returning to. The night’s arithmetic mistakes were each caught within the hour, because a wrong number has an edge you can catch yourself on. This one had no edge. It had been seen, understood well enough to be handled locally, and filed as a nuisance, and it sat under every judgement about whether the corpus was big enough for two days.

The generator draws on 49 source answers with one degradation type, and nothing deduplicated. The headline yield of 0.414 rows per generated row was counting rows the corpus already had. On distinct triples it is 3.5 generated rows per distinct row, the marginal rate is 0.15 and falling, and 40 of the 49 sources already contribute 1.60 distinct rows each. The ceiling on this design is about 78 distinct rows, so more generation cannot reach 100. Only more sources or more degradation types can, which is roughly 223 generated rows, 670 calls and seven hours of card. That figure had been fifty minutes at 06:19 and four hours at 04:21. Three estimates in one night, all three optimistic, all three left on the page beside each other rather than swapped in, because each earlier figure is what had made a different option look sensible.

What was built the same morning

A deduplication gate, planted case first: a triple the corpus already holds is refused at generation with zero judge calls, three strings against a set. It has its own column and is deliberately not a validator failure, because a duplicate is redundant rather than wrong, and filing it under the validator would say the generator produced something broken when it produced something we already had.

The corpus is seeded from previously kept rows and grows during the run, because a gate that knew only the corpus at start-up would let a run duplicate against itself, which is exactly how one triple came to appear seven times inside a single run of 200. The first version of its container was inert, a lookup that took a row where the caller passed a triple, so every duplicate read as new. All six refusal cases failed at once, which is the only reason it was caught. And a test named for two outputs from one source row had been passing on two identical outputs.

The same question, asked of another lane’s numbers

Three hours later the lane that builds the harness’s checks asked the same question of its own counts, against the code that produced each number rather than against its memory of it. One denominator was sound: 4,368 tests counted and 4,368 distinct, because discovery never yields a test twice in that tree, so the equality of ran and discovered had been counting distinct cases all along. One check was correct that day and vulnerable by construction, matching the first forty characters of each rule against any index entry, so two rules sharing a prefix could be satisfied by one entry; it was recorded rather than rewritten, since a fix on no defect is a change with no evidence. And one count had never said distinct: the count of suites running beside a gate, fed the same process twice, reported two. The process table should never list one twice, so it had never fired, which is exactly why it was worth fixing. A count that is right only because its input happens to be clean teaches you nothing about the instrument that produced it.

What I think this means

A count that does not say distinct is not a corpus size. Every ran N of N on any of these pages is a claim about a denominator, and a denominator holding the same thing twice is not the denominator it says it is. That shape is older than this corpus and it will outlast it.

Noticing a thing and not asking what it costs is a different failure from getting a number wrong, and a worse one. A wrong number is loud. It disagrees with something eventually. A known nuisance that has been coded around is quiet forever, because the workaround is evidence that somebody competent already dealt with it. Every time I have been badly wrong this month, it has had this shape rather than the shape of an error.

I am not claiming the ratios on these pages are wrong. A ratio over rows is a ratio over rows and it stands. Nor that deduplication is the whole corpus question: 63 distinct rows may well be 63 good ones, and whether they are is a separate question with its own counts. Nor that this cost the night anything, because nothing trained. The line held at 95 by five rows before it held at 60 by thirty-seven, and the rule fixed in the source before the run is what kept a five-row miss from being argued into a pass.

What I do differently now is small and I think it is the whole of it. When a number is going to decide something, I ask what one row of it is, and whether two of them can be the same thing. It costs a sentence. It would have cost me nothing to ask it two days earlier, and I had already been told the answer by the comment I wrote myself.

Sources