What was over-represented in training becomes 'normal

A model has no independent idea of what a typical photo looks like. It only has the training set it was shown, and whatever that set contained in abundance becomes, in effect, the model's definition of normal. Everything else is judged relative to that, whether the designers intended it or not. Age is one property a training set can skew on without anyone deliberately choosing it, and how a vision model ends up using visual proxies for age at all is worth reading as a specific case of this same composition problem.

Composition sets the reference point

Training data is collected somehow - scraped, submitted by users, licensed, sourced from a particular platform or community - and every collection method has a shape. Certain cameras, certain lighting setups, certain body types and certain photo styles end up more common in the resulting set than others, not because anyone decided they should matter more, but because that is simply what was available and easiest to gather.

A model trained on that set learns statistical patterns from what it actually saw, in the proportions it saw them. It does not correct for the imbalance, because nothing in ordinary training tells it that the imbalance exists - from the model's perspective, the training distribution simply is the world. The over-represented categories become the categories the model has seen thousands of times and has a confident, well-calibrated sense of. The under-represented ones become categories it has seen rarely, and its judgement there is built on a much thinner sample.

Why a thin sample produces a worse score, not a random one

This is a distinct failure from a model simply being unsure - a rare category does not usually make the model say so. It still returns a confident-looking number, because nothing about the output format changes when the underlying evidence was scarce. What actually happens is that the score for a thinly represented category is disproportionately influenced by whichever few examples of it happened to be in the training set, since there is not enough volume for their idiosyncrasies to average out the way they would in a well-represented category. A handful of unusual training examples can end up defining what the model thinks an entire category looks like, simply because there was nothing else to weigh them against.

Where this comes from in the pipeline, specifically

This sits upstream of the model architecture and the training objective entirely. A model trained with a well-chosen loss function and a carefully designed head still inherits whatever the training set was built from, and no amount of tuning either of those corrects for a category the data simply did not contain much of. It is also different from noisy or wrong individual labels, which is its own problem with its own mechanism; dataset bias is about which images exist in the training set in the first place, not about whether the labels attached to them were correct.

A fine-tuned model compounds this in a specific way worth knowing: it inherits the composition biases of its pretrained backbone's original training data, and then adds whatever biases its own smaller fine-tuning set carries on top, so the effect is layered rather than singular.

What this looks like from outside

A tool cannot easily tell you where its own blind spots are, because the blind spot is precisely the region it has the least evidence to describe accurately - the model does not know what it does not know. The visible symptom, when it shows up at all, is scores that feel less consistent or less trustworthy for certain kinds of photo than others, without any obvious reason in the image itself.

The fairness consequences of this - which specific groups end up scored from thinner evidence, and what that does in practice - are a live subject with real stakes, and they get their own dedicated treatment on this site's accuracy hub rather than compressed into a paragraph here. What belongs here is the mechanism: composition in, calibration out, with nothing in between to correct for the gap.

Rate Cock reports scores across six separate axes rather than a single number precisely because a breakdown makes an axis that behaves oddly for a given photo easier to notice than a total ever would. How a model comes to have a sense of "normal" at all - the pretraining and fine-tuning chain that produces the final scorer - is covered here, and the separate question of what individually wrong labels do, as opposed to what a skewed collection of correct ones does, is its own subject. Physical measurement sidesteps this entirely, since a tape does not have a training set to be biased by, and Measure My Cock covers that method directly. A human judge brings their own biases rather than a dataset's, which is a different and separately interesting comparison covered at Rate Penis. Reading a score with this in mind, rather than taking it as a flat verdict, is covered practically at penisrater.com.