A system that catches its own failures is supposed to be the safe kind. In practice a check can fire on every single bad case, be entirely correct about the thing it measures, and still point away from what actually went wrong — and the log it leaves is more convincing than no log at all, because it is full of confident verdicts that all agree with each other.
Teach mode runs a lesson from a queue of concepts, and its contract tells the model never to ask the learner what to cover. The model is there to teach the next thing, not to interview the person who asked to be taught. Across 424 recorded turns on a local model at Q4_K_M, 86 of them — one in five — do exactly that. One sentence appears twenty-four times word for word: “I am ready to proceed with the task. Please provide the specific question or point you would like me to address regarding the repository structure.” These are not openings where the model has nothing yet. They arrive at turns one, two and three of a lesson already underway, most of them about twenty-five words long and reported as complete.
The obvious diagnosis is that the instruction was missing, or too gentle. It was neither: the clause is present and unconditional, verified separately by dumping the fully assembled prompt rather than by reading the template it was built from.
Here is the part worth writing down. Every one of those 86 turns was caught. The grader bounced each of them back to the model between four and six times, so the machinery worked exactly as designed. What it said each time was that the reply does not end with the closing check-in question the contract requires. That is a verdict about punctuation. The reply’s actual problem was that it contained no lesson in it at all.
Neither existing check could name the real fault, and the reason is worth more than the fault is. The rule that catches a clarifying question looks for a question mark together with one of the product’s own format words. This shape has neither — it is a polite request, not a question, and it uses none of the vocabulary. So it falls exactly between two checks, one grading the ending and one grading the phrasing, and neither grading whether anything was taught. Four to six rounds of correction went into a missing question mark while the concept went untaught, and not one of the 86 recovered.
Where it concentrates says the rest. When the lesson queue handed the turn nothing at all, the rate is 50 of 128 — 39.1%. When it handed over a concept and the model named it, 18 of 194, 9.3%. In between, given a concept but not naming it, 18 of 102, 17.6%. That gradient makes this only partly model disobedience: a model with an empty queue has nothing to say, and it fills the silence by asking. What a teach turn ought to do when the queue is dry is a product decision and is not settled by this data — but it is clearly a different question from why does the model ignore an instruction, which is what the bounce log made it look like.
The same corpus makes the point a second time, and this one caught me out. Ninety-seven percent of turns scored as having shipped the learner a visual, which reads as near-perfect compliance with the one-visual rule. The model asked for a drawing on 5 turns out of 296, and on the 102 turns that missed the naming clause it asked zero times. The other charts were drawn by the product itself from its own scan of the code, and the grader’s flag is true whenever no missing visual problem was raised — not when the model did anything. I had used that 97% to argue the model was mostly complying and skipping one clause. It was not doing the thing at all.
That one has an honest other half, and leaving it out would overstate the failure: the product-derived chart names the concept in its title and caption on 97.1% of the turns whose prose did not. The clause is disobeyed and the label still reaches the learner. The system is compensating for the model, which is a reasonable thing for a product to do — it just cannot then be read as evidence about the model.
Both numbers come from one local model at one quantisation, and they do not transfer without it; a different model or a different build would need its own 424 turns before any of this means anything about it. What does transfer is the shape.
A check that fires for the wrong reason is worse than no check at all. A missing check leaves an obvious hole, and somebody eventually walks into it and finds out. A check that fires on the wrong criterion produces a tidy log full of confident verdicts, every one of them pointing away from the problem, and it will keep producing that log indefinitely because from the inside it looks like a system working. We only found this by reading the turns instead of the counts — and the counts were not wrong, which is precisely why they were so hard to get past.