Inter-annotator agreement is the accuracy ceiling

No, a scoring model cannot be more consistent than the people who labelled its training data: their agreement with each other sets its ceiling. That measure, inter-annotator agreement, is rarely discussed outside machine learning papers, yet it is the most revealing question to ask of any training set.

What agreement actually measures

Give the same set of images to several labellers, working from the same rubric, and compare their scores. If they mostly land on the same or nearby values, agreement is high. If their scores for the same image scatter widely, agreement is low. Researchers often summarise this with statistics like Cohen's or Fleiss' kappa, which adjust raw agreement for the amount you would expect from chance alone - two labellers can agree fairly often purely by coincidence if the scale is short. McHugh's 2012 review of the kappa statistic in Biochemia Medica is blunt about low values: below 0.60, "one can surmise that about half the data may be incorrect."

The exact statistic matters less than the concept: agreement is a measurable property of the labelling process, sitting there in the data before a single model has been trained on it.

Why it becomes a ceiling

A supervised model is trained to predict the label it was shown, and when multiple labellers scored the same image differently, the model typically ends up learning something close to their average, because averaging is what minimises its training error across the disagreement. That average is a genuinely useful thing to learn - it is the most representative single number available - but it is not the same as learning a sharp, singular truth, because no such sharp truth existed in the training data to begin with.

Here is the part that is easy to miss: the model cannot become more internally consistent than the data it learned from allowed it to be, for the same image, scored by different people. If five labellers looking at one photo produced scores of 5, 6, 6, 7, and 8, no amount of additional training data elsewhere in the dataset fixes that specific image's inherent ambiguity. A model trained on data like this ends up with an irreducible amount of scatter baked in, not because the model is poorly built, but because the target it was asked to approximate was itself a soft, disagreed-upon quantity.

The worked intuition

Picture two datasets, both used to train an otherwise identical model. In the first, five labellers scoring the same image typically land within one point of each other - agreement is tight. In the second, five labellers scoring the same image typically span four or five points - agreement is loose. Both datasets can be used to train a model, and both models will report scores. The first model's scores can plausibly track something close to a shared, stable judgement, because that judgement existed among its labellers. The second model's scores are still an average of something, and the average is still meaningful in a coarse sense, but nobody should expect a model trained on it to reliably distinguish a 6 from a 7, because its own labellers frequently could not.

A model, in effect, learns the average labeller. It cannot learn a sharper answer than the average labeller's collective sharpness allowed, no matter how large the network or how much compute went into training it.

Why this rarely gets reported

Agreement figures require deliberately having multiple people score the same images and comparing them, which costs more than having each labeller score a unique set of images and move on. Many labelling pipelines skip this step for exactly that reason, which means many deployed scoring models have never had their ceiling measured at all - not because it would look bad, necessarily, but because nobody paid to find out. A model without a known agreement ceiling is not automatically less reliable than one with a documented ceiling; it is simply a model whose reliability nobody has quantified, which is a different and less comfortable state than the reliability question a repeat test can at least partially answer from outside.

What this does not tell you

Agreement bounds the achievable consistency of a model relative to its own labellers; it does not, on its own, say anything about where those labels came from in the first place - crowd, expert, or engagement proxy - which is a separate question covered at length elsewhere. A model can have high internal agreement among its labellers and still be trained on a narrow or biased sample of images, which is a different failure mode entirely.

The practical takeaway

A published model card that reports inter-annotator agreement, even roughly, is telling you more about how much to trust a specific point difference than almost anything else the tool could say. Its absence is not proof of a problem, but it does mean the ceiling is unknown rather than reassuringly high. Rate Cock treats the six-axis breakdown partly as a hedge against exactly this uncertainty - a single blended number hides which axis carries how much labeller disagreement, while separate axes at least let a reader see where the numbers move together and where they do not. None of this touches a physical reading; Measure My Cock's approach has no labeller-agreement question because it has no labellers, only a tape and a repeatable protocol. A single human reviewer sidesteps the averaging problem in one sense - there is only one opinion to report - while introducing a different one, since a lone judgement has no built-in check against another person's read of the same photo, a tension Penis Rater discusses when comparing AI and human results side by side. A score is only ever as sharp as the agreement behind it, and most interfaces never say what that agreement was.