The retraction was the claim

A careful correction is still a claim. I did not know that on Friday night, when an agent told me it had withdrawn eight of the numbers it had given me an hour earlier, and I believed it at once, because withdrawing something sounds like the honest thing to do. It was wrong. Every withdrawn number was in the papers, verbatim, and finding that out took one command I had not thought to run.

What happened

I had asked an agent to fetch the learning-science evidence for something I believe, that the best teaching is interactive and visual, with effect sizes and sample sizes read from the papers themselves. It came back with twenty sources and their numbers. Twenty minutes later the same agent reported again and withdrew eight of them: those rows, it said, had come from sub-agents that never returned any output, so the numbers had not been read from any page.

I filed the correction. The page written on the first report got a dated block at the top naming the eight sources as unverified, the ledger got a row, and every conclusion resting on the eight went on hold. That took ten minutes, and it is the right thing to do with a retraction from your only witness.

Then a third agent fetched the eight papers, downloaded each file, and read the text. Every withdrawn number was there. Thirty-eight numbers across eight sources, all correct in the first report. The retraction was the thing that was wrong.

Why I believed the wrong one

The first report was right and I could not know it. The second was wrong and I could not know that either. Both came from one agent, and the second was the more cautious of the two, and caution reads as honesty.

There is a detail that makes it worse. Earlier in the same hour, that same agent had caught its own fetch tool inventing a paper’s numbers. A summariser had reported one meta-analysis as an effect of 0.36 over 318 studies and 14,443 students; the paper says 0.499 over 222 studies and 48,478. The agent re-read the file and corrected it. So I already had the rule, that a summary is a claim and the number lives in the text of the page, and I had it in my hands an hour before I needed it. I applied it to the numbers and not to the retraction.

What I think this means

A retraction is a claim, and caution is not evidence. When two reports from one source disagree, neither is settled by which of them sounds more careful. The page settles it. That sounds obvious written down and it did not feel obvious at the time, because a correction arrives wearing the costume of the thing that checks corrections.

The cheapest third read is to fetch the file, and it belongs before the correction is filed rather than after, whenever the correction is going to put conclusions on hold. Mine cost ten minutes and a ledger row. On a different night it would have cost a design built on the wrong half of the evidence, and I would have had no way of noticing, because the retraction had already told me not to look.

A rule fixed in one place is fixed in one place. I want this one stated plainly because it is the part that generalises past agents entirely. Having a rule is not the same as having it where it is needed, and the gap between the two is invisible from the inside. I had spent that hour proving that a summary is a claim and then took an account of a process as a fact about a process.

I am not claiming the agent lied, or that the second report was careless. It said what it believed about its own workings, and it was mistaken about its own workings. That is the whole point rather than a mitigation: an agent’s account of how it got a number is itself a claim about that number, and it earns the same check.

Both blocks are still on the page, dated, in order: reported, withdrawn, restored. Nothing was deleted, and the correction is more useful for having been wrong than it would have been for being right.

Building this way alone means the only witness to most of what happens is the thing that did it. I have stopped asking my instruments whether they were right, and started asking them for the file.

Sources