Wrong labels do not cancel out; they smear

Every label a scoring model learns from was assigned by a person, and people make mistakes. The interesting question is not whether that happens - it always does - but what the mistakes do to the finished model. The answer is not "nothing, because it averages out." Some kinds of error do average out. Others leave a permanent mark on how the model behaves.

Two kinds of wrong

Label noise splits into two families that behave very differently.

Symmetric noise is a labeller who is sometimes generous and sometimes stingy for no consistent reason - tired at the end of a batch, distracted, or just inconsistent in the way any person is across a long session. Given enough examples, this kind of error washes out. The over-scores and under-scores partly cancel, and the average label a model is trained toward still points roughly at the right place.

Systematic noise is different in kind, not just degree. It is a labeller (or a labelling instruction) that is consistently off in one direction for a particular kind of image - always scoring a certain lighting condition low, always scoring a certain body type as more "typical" than it is. This does not cancel with more data. It accumulates, because every example pulling in the same direction reinforces the same wrong pattern. A model trained on it does not learn noise around a correct answer; it learns the bias as if it were the answer.

What this does to a regression-style scorer

Most scoring models are trained to minimise squared error between their prediction and the label - a completely ordinary choice, covered in more depth in how the training objective shapes the numbers a scorer returns. Squared error punishes big misses far more than small ones, which has a specific consequence when some of the labels it is trying to fit are simply wrong: the model is not rewarded for being right on the noisy examples, it is rewarded for being close to the middle of the distribution of labels it saw, wrong ones included.

The practical result is shrinkage. Confidently extreme labels - the genuine tens and the genuine ones - get pulled toward the centre, because trusting them fully means being punished hard whenever one of them was actually noise. The safest strategy the optimiser can find is to hedge, and hedging looks like every score creeping toward the average, which is also why most people land on a six or a seven regardless of the tool.

Why the tails suffer more than the middle

The middle of any label distribution has the most examples, so noise there gets outvoted by volume - a handful of wrong sixes barely dent a category with thousands of correct ones. The extremes do not have that protection. There are fewer genuine tens to begin with, so a handful of mislabelled ones among them is a much larger share of the evidence the model has for what a ten looks like. Noise concentrated in a thin category does more damage than the same amount of noise spread through a thick one, which is one of the quieter reasons extreme scores are rare and cautious ones are common.

This is a separate problem from labellers disagreeing with each other on the same image, which sets a ceiling on how consistent any trained model can be, covered on its own; label noise is about individual labels being wrong, not about honest raters landing in different places. It is also a different problem from the training set simply lacking certain kinds of photo in the first place, which is a composition question rather than a correctness one.

What a well-run labelling process does about it

The standard defenses are unglamorous: multiple labellers per image with disagreements resolved rather than averaged blindly, clear written instructions so "systematic" errors are less likely to start, and spot audits of the labels a model was actually trained on rather than trusting the collection process by assumption. None of this appears in a rating tool's interface. It is visible only in whether the tool's behaviour looks calibrated at the extremes, which is exactly the part label noise degrades first.

Rate Cock publishes the breakdown behind every score rather than a single blended figure, and a rubric that separates axes gives you a way to notice this kind of smearing when it happens, since a plausible-looking total can still be hiding a flattened tail on one axis. Where the labels in a training set come from in the first place - who wrote them, and under what instructions - is the upstream question, covered in where scoring labels come from. The same labelling work sits behind human review products too: Rate Penis commissions people directly rather than training a model on their past judgements, which sidesteps this particular failure mode by not compressing it into a training set at all. Measuring something in the physical world does not have this problem the same way, since a tape measure does not get tired at the end of a session the way a labeller does. How consistently a tool's numbers hold up under repeat testing is the visible symptom worth checking directly, and penisrater.com covers what to look for when reading a score for what it actually supports.