What it means for a scorer to agree with itself

Test-retest reliability asks one narrow question: if you give a scorer the same input twice, does it give the same answer. Nothing about whether the answer is right, or what the number means. Just whether the number holds still, which is a lower bar than accuracy and worth separating from it.

The idea in one comparison

Imagine two bathroom scales. The first reads 82.0kg, 82.0kg and 82.0kg across three weigh-ins a minute apart. The second reads 79kg, 84kg and 81kg for the same three weigh-ins. You cannot yet say which scale is closer to your true weight - that needs a separate calibration check against a known reference. But you can already say the first scale is more reliable, because reliability is a property of the spread, not of where the spread sits.

An AI scorer works the same way. Feed it the identical photo, unchanged, several times in a row. A reliable model returns numbers clustered tightly together. An unreliable one scatters, and the scatter is the whole finding - it tells you nothing about whether either number was correct, only that the instrument itself is loose.

What actually produces reliability, or the lack of it

A model's output on one input is not always a single deterministic value. Some scoring heads are pure regression and genuinely deterministic given identical bytes. Others route through a language model that samples its output, which means repeat runs draw from a distribution rather than reading off a fixed point. Preprocessing can add noise too - a resize algorithm, a batching order, a floating-point rounding difference between GPU runs. None of this is a bug specific to any one tool; it is a property of how these systems are built, and it is why "the same photo" does not always mean "the same input" by the time it reaches the model.

The statistic researchers use to summarise this is called an intraclass correlation, or ICC, and the intuition behind it matters more than the formula. It compares how much a set of scores vary within the same subject scored repeatedly against how much scores vary between different subjects. An ICC near 1 means repeat scores on one photo sit close together relative to how far apart different photos land - a tight, well-behaved instrument. An ICC near 0 means the wobble from repeating the same photo is about as large as the gap between genuinely different photos, which makes any single score close to meaningless on its own. The most cited reporting guideline, Koo and Li (2016, Journal of Chiropractic Medicine), treats values below 0.5 as poor, 0.5 to 0.75 as moderate, 0.75 to 0.9 as good and above 0.90 as excellent, judged on the 95% confidence interval rather than the point estimate alone.

Reliability is not the whole story

A perfectly reliable model can still be wrong about everything it measures. Picture a scale that reads 82.0kg every single time, on every person, regardless of who steps on it. That is a maximally reliable instrument and a completely useless one. Why that gap matters, worked through with a rating example, is its own subject, because conflating the two is one of the most common misreadings of a good number.

What reliability does buy you is the ability to trust that a change in score reflects a change in something real, rather than instrument noise. That is the precondition for every other kind of analysis - comparing your own submissions over time, testing whether a lighting change matters, checking whether a tool has drifted. Without it, you cannot tell signal from static.

What to actually do with this

You cannot know a tool's reliability from a single upload; you learn it by repeating the same input and looking at the spread, which is the practical method worked through separately. The unglamorous version - run it a handful of times, note how far the numbers travel - beats trusting a single result, however confident that result looks.

This kind of check is not unique to AI. Measure My Cock deals with the equivalent question for a tape and a method, where the instrument is physical rather than statistical, and the same logic about repeat measurement applies there too. On the human side, a commissioned review carries a different kind of reliability question, since the same judge scoring the same submission twice will not agree with themselves as tightly as a model does - that inconsistency is a documented property of human raters, not a flaw specific to any one reviewer. Rate Cock reports a breakdown across several axes rather than one blended figure, which makes a reliability check more legible: you can see whether the wobble is concentrated in one axis or spread evenly across all of them, and reading a score against that breakdown is worth doing before treating any single result as final.