Reliability and validity are different failures

A model that gives the identical score to the identical photo every time has proven exactly one thing: that it is consistent with itself. It has proven nothing about whether that score is correct. Reliability and validity are separate properties, they fail in different ways, and mistaking the first for the second is one of the most common errors in how people read an AI score.

Two different questions

Reliability asks: does this instrument agree with itself on repeated measurement. Validity asks: does this instrument measure the thing it claims to measure. A stopped clock is perfectly reliable - it gives the same reading every time you check it - and wrong twice a day at best. A clock running fast by a fixed, unchanging amount is also perfectly reliable, and it is wrong all day, every day, in a way you would never catch just by checking it twice.

Test-retest reliability is the property covered elsewhere on this site: does a scorer land close to the same number on the same unchanged input. That is checkable from the outside, cheaply, by repeating an upload. Validity is a much harder claim, because checking it requires a correct answer to compare against, and for a subjective rating there usually is not a clean one available. What "accurate" would even mean here, and why ground truth is the hard part, is a question in its own right.

A worked example

Suppose a scoring model was trained mostly on photos taken under strong, direct light, and it learned - without anyone intending this - to associate strong contrast with higher scores, because contrast happened to correlate with the training labels for reasons that had nothing to do with what the axis was supposed to capture. Feed that model the same photo twice and it will return the same number both times. It is behaving reliably. It is also, in this scenario, scoring lighting rather than the thing the axis claims to score, which is a validity failure that reliability alone can never reveal, because reliability only ever compares the model to itself.

The only way to catch a validity problem is to compare the model's output against something outside the model - a different measurement, a panel of independent judges, a known reference case where the right answer is not in doubt. A tool can pass every reliability check you throw at it and still be validated against nothing.

Why this gets mixed up

Reliability is easy: repeat an upload, look at the numbers, done. Validity is hard: it needs an external reference, a defined standard, and usually more effort than most people are willing to spend checking a single score. Because reliability is the cheap check, it is the one people actually run, and a tight, consistent result quietly gets treated as proof of correctness even though it was never designed to demonstrate that. The comparative claim that AI is "more consistent than a human rater" is true and is also a reliability claim, not an accuracy claim, and the two get conflated constantly in exactly this way.

Reading a score with both in mind

A useful mental checklist: first, is this reliable - does it hold roughly still on repeat runs. Second, and separately, is there any reason to think it is valid - has anyone checked it against something outside itself, and if so, against what. A tool that reports a tight, repeatable breakdown across several axes rather than one blended figure is at least giving you the material to ask the second question, even if it cannot fully answer it - Rate Cock is one example of a scan built that way, showing the axes rather than only the total. The equivalent question shows up for a tape measure too: a measurement can be perfectly repeatable and still wrong if the method itself is flawed, which is why Measure My Cock's method write-up spends as much time on technique as on the number. Reading a result with this distinction in mind changes what you take away from it, and a human reviewer sidesteps some of this differently again, since a considered written opinion is being asked a different kind of question than a number ever is.