I asked a model to sort 273 of my own notes into four buckets. It did the job. Then I asked it how sure it was about each one, and 271 of the 273 answers came back at 0.90 or higher.
That is a constant wearing a decimal point.
The notes are a store my systems read from: facts I told them once and they have believed ever since. That works until a fact stops being true, because nothing in the record marks the moment it stopped. The store had no way to tell what was permanent from what went off like milk.
So the job was to sort by how fast a thing decays. Immutable: true once, true forever. Slow: changes on the order of seasons. Seasonal: expected to move within a fortnight. Live: never store it; keep a pointer and fetch it every time.
The final split was 78 immutable, 177 slow, 16 seasonal, 2 live.
The number that wasn’t a measurement
I had planned to use the model’s own confidence as a gate. Anything it wasn’t sure about would default to live, the safe class, the one that forces a fetch instead of trusting the record. One line of rule. It could never fire, because a gate set below 0.90 does nothing when 271 of 273 answers sit above it.
The tell wasn’t that the confidence was high. It was that it came back as two values. No curve, no tail to set a threshold inside, just two constants repeated 273 times. The model was filling in a field.
I don’t think a better prompt fixes this, and I don’t think it’s peculiar to the model I used. Asking a system to grade its own certainty is asking it to do a second task with the same machinery and nothing to calibrate against. You get a plausible number. Plausible is the dangerous property, because a plausible number survives a skim.
What did work
I ran the corpus twice, both at temperature zero, with two different prompts. Then I looked at which answers moved.
Fifty-seven of the 273 changed class between the two runs.
That signal cost exactly as much as the useless one, and it doesn’t depend on the model having any insight into itself. It’s an observation from outside. Where two framings of the same question disagree, something about that item is genuinely underdetermined. Where they agree, the answer at least holds up under rewording.
The rule I took from it: ask whether the system has ever given a different answer, not whether it feels sure. Confidence is a claim. Disagreement is evidence.
Where I had it wrong
The same question showed up one level higher, and this time the mistake was mine.
I set up the human review to send live and seasonal answers to me and auto-accept immutable and slow. My reasoning: those two classes change how the system behaves, so those are the ones a person should look at.
I had it backwards. Work out what each mistake costs. A fact wrongly marked live costs a lookup I didn’t need. It’s wasteful, it corrects itself, and it shows up immediately. A live fact wrongly marked slow gets stored and served as true for months, with no error anywhere, to anything that asks.
So the expensive mistakes could only land in the bucket I had chosen not to read. It was also the biggest bucket: 177 of 273, about two thirds of the corpus, waved through because it looked boring.
I aimed the review at where the interesting answers were. It needed to be aimed at where the costly errors were. Those are different questions, and here they had opposite answers. Nothing had shipped yet. I added a spot-check of twenty slow rows before accepting anything.
Nothing was written
The pass changed nothing in the store. It produced a map, a proposal about 273 notes, which I reviewed in a browser and accepted. The store itself still carries no decay field.
That was deliberate, and it is why my mis-aimed review rule became a design note instead of 177 bad records, discovered months later when something hands you a fact from a world that no longer exists.
The whole exercise was about facts that go stale without announcing it. It would have been a poor showing to create some.