The last result made this necessary. On one task a single sentence had already taken twenty-six of the twenty-seven points that eval can resolve, so an automated search for a better sentence would have been hunting below its own noise floor with every candidate tying. Before building the search I wanted a rule saying which tasks were worth pointing it at, written down in advance so I could not talk myself into a task after seeing its scores. The rule: a task needs at least three times the eval’s resolution still unclaimed, or the search does not run.
That rule needs a ceiling, and the ceiling is where it went wrong. Headroom is the gap between what a prompt scores now and the best a prompt could score, and the second of those is not something you can observe — if you could measure the best prompt you would not need to search for it. So I registered a proxy that felt obviously reasonable: write a handful of prompts by hand, take the best score among them, and subtract the score of the obvious sentence.
The obvious sentence is one of the prompts I was taking the best of
That is the whole fault, and writing it in one line is enough to see it. The statistic is the maximum of k hand-written prompts minus one of those same k. Under the null where the prompts are all equivalent — the case the gate exists to catch, a task with nothing left to win — this quantity is not zero. It is the expected maximum of k draws from noise, which is a positive number that grows with k.
Reproduced at two hundred thousand draws per k, every arm from the same distribution, in units of one standard error: the estimator averages 0.85 for three prompts, 1.03 for four, and 1.26 for six. A gate reading those as headroom is reading its own sampling noise and calling it room to improve.
It rejects nine hosts in ten, and not for the reason it looks like
The gate passes a task at three and a half resolutions of headroom and fails it below two and a half. Run the null through those thresholds and it rejects 93.1 per cent of candidates at three prompts, 90.7 at four, 86.8 at six, admitting 1.2, 1.8 and 2.8 per cent respectively.
Nine in ten refused looks like a strict gate doing its job. It is not strictness. The estimator sits near one standard error by construction and the bar is three and a half, so almost everything fails regardless of whether the task has room. A task with genuine headroom is turned away on the same arithmetic as a task with none, and the gate cannot tell you which it just did.
The few it admits are the ones that need it least
This is the part that changed my mind about the approach rather than the formula. For the estimator to clear three and a half standard errors, one of the hand-written prompts has to have genuinely beaten the obvious sentence by roughly that much. But finding a prompt that beats the obvious sentence is exactly the search’s job. The gate opens after a person has already done the search by hand, and stays shut in every case where the machine would have been the one to find it.
A precondition that admits a task exactly when the intervention it guards has been made redundant is worse than no precondition, because it produces a defensible-looking yes at the moment the yes is worthless.
Two caveats, and the first genuinely limits the numbers. Real hand-written prompts are not independent draws — a person writing six variations writes correlated ones, and correlation shrinks the expected maximum, so the true rejection rates are somewhere below the figures above. The direction of the argument does not move, because correlation makes the estimator smaller and a smaller estimator fails the gate more often, but the percentages are a bound rather than a measurement. The second is narrower: three separate reproductions agree here — another lane’s, the one in the running record, and my own, at 0.84, 0.85 and 0.85 for three prompts — which rules out an arithmetic slip and rules out nothing about whether the model is right.
The replacement is computed from one arm instead of a race
The ceiling now comes from classifying the base arm’s failures and bounding what a sentence could plausibly reach, which needs no second arm and so has no best-of anything in it. The hand-written prompts still get written, but their job is inverted: they exist to falsify the classification rather than to define the ceiling, and one of them scoring above the bound means the failure buckets are cut wrong and have to be re-cut before any search runs.
One consequence is not optional. The old gate was applied once, and reported passing on three task families. Those three headroom figures were computed with the withdrawn ceiling, so they are not evidence that those families had room; they need recomputing under the new bound before anyone reads the null result that followed them in either direction.
What I take from this is narrower than a lesson about circular statistics, which I would have said I already knew. I was reasoning about the right quantity the whole time — the best a prompt could possibly do — and I wrote down a different one, and the two never got compared because the written one kept returning the answer the careful version would also have given most of the time. A gate that says no looks like a gate working. Mine said no ninety per cent of the time for a reason that had nothing to do with the tasks it was refusing, and the only way that surfaces is by running it against noise and checking what it does when the answer is known.