Accuracy

When the scale runs out before the differences do

At the ends of a scale the model can no longer distinguish, so two quite different inputs both get the same top or bottom number.

By Updated 3 min readAccuracy

Guides on Accuracy: There is no true score to be accurate against, Change one thing, hold the rest, repeat, Exposure that suits one skin tone hides another

A ceiling or floor effect is what happens when genuinely different inputs share the top or bottom number because the scale has run out of room. A 1-10 scale has ten slots, real inputs vary continuously, and near either end the crowding is a property of the scale, not a flaw in any photo.

Why the ends compress

A scoring head learns to spread its outputs across the range it was trained on. But training examples thin out toward the extremes - very few photos get labelled a 1 or a 10, for reasons covered separately - and a model has less to learn from where it has seen less. The practical result is that the top and bottom of the scale behave like a wall. Two inputs that would clearly separate if the scale ran to 20 both flatten into the same 9 or 10 because the model has run out of resolution, not because it has run out of things to say about them.

What this looks like in practice

If you retest a photo that already scores near the top of the range, you will tend to see less movement between attempts than you would for a photo sitting in the middle - not because the top-scoring photo is more stable, but because there is less room above it for noise to push the number. The same compression happens at the bottom. This is a different mechanism from regression to the mean, which pulls an extreme score back toward the centre on a retest. A ceiling effect is about the scale having nowhere left to go; regression is about noise being unlikely to repeat in the same direction twice. Both can operate on the same score at once.

What it means for reading a score

A 9 or a 10 tells you less than it looks like it tells you, because it may represent the ceiling rather than a precise judgement - it is also why a perfect ten is rarer than the scale alone would suggest, since the mapping from raw output to that top slot is itself narrow. Two submissions both scoring 9 could be meaningfully different in ways the number cannot show, and the honest read of a top score is "at or near the ceiling," not "precisely quantified." Measurement research treats this as a testable defect: the widely used Terwee et al. (2007) criterion, as applied by Wamper et al. (2010), counts more than 15% of respondents at the top or bottom score as unacceptable, and their review of a clinical hip score found a pooled ceiling effect of 20%.

The fix, where a tool offers one, is a breakdown across several axes rather than a single compressed total - Rate Cock reports six separate scores rather than one figure, which at least distributes the ceiling problem across axes rather than concentrating it in a single number, and lets you see which axis is actually near its wall. The same compression exists in any bounded rating system, including how a public leaderboard of AI tools presents scores, and a physical measurement sidesteps the issue entirely by not being bounded at all - a tape measure has no ceiling built into the reading itself, only the range of the tape. A human reviewer runs into a version of the same limit for a different reason: most people are reluctant to give an actual 10 on any scale, which is itself a documented pattern in how judges score, floor and ceiling included.

Read next

Full archive