Out-of-distribution inputs get out-of-distribution scores

An unusual photo still gets a confident-looking score, but the score means less. A model is calibrated only on the range of lighting, framing and subject its training photos covered. Inside that range - "in-distribution" - the number has genuine grounds.

A photo well outside it - "out-of-distribution" - gets a number too, produced by exactly the same procedure, and that similarity in output hides a real difference in how much the number is worth.

Why the model does not just say so

Nothing about a standard scoring model includes a step where it checks whether an input resembles its training data before answering. The architecture takes an image in and produces a score out, and it will do this for literally any image handed to it, including ones that share almost nothing with what it was trained on. There is no built-in mechanism forcing the model to notice the gap, because noticing requires a separate kind of check that most scoring pipelines simply do not include.

This means an out-of-distribution photo does not come back looking uncertain in any obvious way. It comes back with a number formatted exactly like every other number the tool produces, with no flag attached, because flagging is not something the underlying architecture does on its own.

Why the number means less

Inside its trained range, the model's score reflects patterns learned from thousands of comparable examples, which is a reasonable basis for a judgement even given everything else covered on this site about what that judgement actually is. Outside that range, the model is extrapolating from patterns learned elsewhere and applying them to something they were never verified against. The output is still a number, produced by the same weights doing the same arithmetic, but the arithmetic is now operating on an input the model has comparatively little grounding for, and the result is closer to a guess dressed in the same format as everything else.

This compounds with a related but separate issue: categories that were thin in the training set to begin with produce shakier scores even for photos that are otherwise fairly ordinary, because the model had little evidence to build a reliable pattern from in the first place. Distribution shift is the sharper version of the same underlying gap - not just thin evidence, but evidence of a genuinely different kind.

How a tool could actually detect this

Techniques exist to estimate whether an input sits near or far from a model's training distribution - comparing the input's internal representation against the spread of representations seen during training, or checking how much the model's confidence changes under small input perturbations. The simplest baseline is old and cheap: Hendrycks and Gimpel (2016) found that "correctly classified examples tend to have greater maximum softmax probabilities than erroneously classified and out-of-distribution examples," so even the model's own confidence carries a usable warning signal. It is not a fix on its own - a large-scale evaluation of uncertainty under dataset shift (Ovadia et al., 2019) found that "traditional post-hoc calibration does indeed fall short" once inputs drift away from the training data. None of this is exotic or unavailable; it is simply an extra step that most rating tools do not build in, because it adds complexity and a scoring product built around a clean single number is not naturally inclined to also show "and here is how much to trust it." A tool that did include this would, in effect, be able to say "this photo is unlike anything I was trained on, treat this score with more caution," which almost none currently do.

What this is not

This is a different problem from overfitting, where a model has memorised its training examples too closely; distribution shift is about the training set's overall coverage rather than about how tightly the model fit to it, and the two can occur independently of each other. It is also a mechanism rather than a verdict on any one tool - every model trained on a finite set has some boundary beyond which its calibration weakens, and the honest position is that the boundary exists rather than that any particular tool is unusually bad at handling it.

Rate Cock scoring across six separate axes at least makes an unusual result more legible, since an axis wildly out of step with the others on an unusual photo is a visible clue worth noticing rather than a single total quietly absorbing the discrepancy. What sets the boundary of a model's trained range in the first place is training-set composition, covered here, and the training objective that shapes how the model hedges near that boundary is covered here. A human reviewer notices an unusual photo the way a person notices anything unfamiliar, which is part of what Rate Penis covers about the difference a person makes. Physical measurement has no training distribution to fall outside of, since Measure My Cock's method is a repeatable procedure rather than a learned pattern. What an unusual photo actually looks like on the practical side, framing and all, is covered at penisrater.com.