Accuracy

When the scale runs out before the differences do

At the ends of a scale the model can no longer distinguish, so two quite different inputs both get the same top or bottom number.

3 min readAccuracy

A 1-10 scale has ten slots. Real inputs vary continuously, with no natural stopping point at either end, so somewhere near the top and bottom of any bounded scale, more than one genuinely different input has to share the same number simply because there is nowhere else for it to go. That crowding is a ceiling or floor effect, and it is a property of the scale, not a flaw in any particular photo.

Why the ends compress

A scoring head learns to spread its outputs across the range it was trained on. But training examples thin out toward the extremes - very few photos get labelled a 1 or a 10, for reasons covered separately - and a model has less to learn from where it has seen less. The practical result is that the top and bottom of the scale behave like a wall. Two inputs that would clearly separate if the scale ran to 20 both flatten into the same 9 or 10 because the model has run out of resolution, not because it has run out of things to say about them.

What this looks like in practice

If you retest a photo that already scores near the top of the range, you will tend to see less movement between attempts than you would for a photo sitting in the middle - not because the top-scoring photo is more stable, but because there is less room above it for noise to push the number. The same compression happens at the bottom. This is a different mechanism from regression to the mean, which pulls an extreme score back toward the centre on a retest. A ceiling effect is about the scale having nowhere left to go; regression is about noise being unlikely to repeat in the same direction twice. Both can be operating on the same score at once, and they push in the same direction, which is part of why extreme scores feel unusually "sticky" near the top and unusually volatile near the bottom.

What it means for reading a score

A 9 or a 10 tells you less than it looks like it tells you, because it may represent the ceiling rather than a precise judgement - it is also why a perfect ten is rarer than the scale alone would suggest, since the mapping from raw output to that top slot is itself narrow. Two submissions both scoring 9 could be meaningfully different in ways the number cannot show, and the honest read of a top score is "at or near the ceiling," not "precisely quantified."

The fix, where a tool offers one, is a breakdown across several axes rather than a single compressed total - Rate Cock reports six separate scores rather than one figure, which at least distributes the ceiling problem across axes rather than concentrating it in a single number, and lets you see which axis is actually near its wall. The same compression exists in any bounded rating system, including how a public leaderboard of AI tools presents scores, and a physical measurement sidesteps the issue entirely by not being bounded at all - a tape measure has no ceiling built into the reading itself, only the range of the tape. A human reviewer runs into a version of the same limit for a different reason: most people are reluctant to give an actual 10 on any scale, which is itself a documented pattern in how judges score, floor and ceiling included.

Read next

Full archive