Every score is a memory of somebody's opinion

A scoring model has never seen the thing it scores in any sense a person would recognise. What it has seen is a large number of photographs, each attached to a number that a person once assigned, and its entire training process consisted of adjusting itself until its own outputs matched those numbers as closely as possible. Everything the model appears to know is downstream of that dataset, which means the people who produced the labels, and the instructions they were given, are doing more of the work than the phrase "AI scoring" suggests.

The basic supervised setup

Training a scoring model requires pairs: an image, and a target value for that image. The target does not come from the world in any direct way - it is not measured off the photo by an instrument. It comes from a person, or a small group of people, looking at the image and producing a number or a set of numbers according to some instructions. The model's job during training is purely to reduce the gap between its own prediction and that target, averaged across every example in the dataset.

Once training is complete and the model is deployed, it has no way to consult anything beyond what it learned from that process. A finished model is, in a very literal sense, a compressed record of how a particular group of people, under a particular set of instructions, tended to score a particular set of images. Ask it to score something meaningfully outside that record, and it will still return a number - the interface always shows a number - but the number is an extrapolation from a pattern rather than a fresh judgement.

Where labels actually come from

Crowd labelling. Large platforms recruit many people, often paid per task, to score a large volume of images according to a written rubric. This scales cheaply and produces broad coverage, at the cost of variable attention and inconsistent adherence to the rubric across hundreds or thousands of individual labellers who may interpret the same instruction differently.

Expert labelling. A smaller number of people with specific relevant background score a smaller, more carefully chosen set of images. This tends to produce more consistent, more defensible labels, at the cost of volume - training a modern vision model well typically wants more examples than a small expert team can realistically produce.

Engagement proxies. Rather than asking anyone to score anything directly, some systems use existing behavioural signals - likes, views, click-through - as a stand-in for quality, on the reasoning that popular content is content people judged favourably with their attention. This is cheap and scales without limit, and it measures something meaningfully different from a deliberate quality judgement: it measures what an audience engaged with, filtered through everything that shapes what gets shown to that audience in the first place.

Most production systems blend more than one of these, often starting from a large engagement-derived or crowd-labelled set and refining with a smaller expert-labelled set for calibration. How these two source types actually compare as a design tradeoff, rather than just as categories, is covered directly in engagement labels vs expert labels.

What the instructions actually determine

A rubric handed to labellers is rarely just "rate this 1 to 10." It typically defines what each end of the scale is meant to represent, sometimes with example images anchoring specific points on the scale, and it draws boundaries around what the labeller should and should not factor in - whether presentation counts, whether the labeller should imagine a specific context, how to handle an image that does not clearly fit the rubric at all.

Two teams training on the same raw images but different rubrics will produce models that behave differently on the same input, because the ground truth the models were fitted to is different, not because one is technically better than the other. The rubric is not incidental to the model. It is closer to being the model's actual specification, with the network doing the work of generalising that specification to images nobody wrote instructions for.

Agreement, and why it matters before anything else does

If you show the same image to several labellers working from the same rubric and they give noticeably different scores, that disagreement is not noise the model can train past. It is a ceiling. A model fitted to a dataset where labellers agreed loosely can, at best, learn the average tendency across those labellers - it cannot reconstruct a single sharp "truth" that individual labellers themselves did not converge on. Why disagreement among labellers bounds what the model can ever do is worth reading as a direct continuation of this point, because it is easy to treat model inconsistency and labeller inconsistency as separate problems when the second one usually causes a portion of the first.

What label noise does, mechanically

Not every disagreement is principled difference of opinion; some of it is plain error - a labeller misreading the rubric, clicking the wrong value, rushing through a batch late in a shift. This kind of noise does not average out cleanly the way people sometimes assume. Its actual effect on a trained model tends to flatten confidence and pull the model's response to unusual inputs toward the safe middle of the scale, rather than simply adding harmless random scatter around the correct answer.

Two ways around scoring an absolute number directly

Because people are often more reliable comparing two things than assigning an absolute number to one, some training pipelines sidestep absolute labelling entirely and instead ask labellers "which of these two is better," turning the result into a ranking that gets converted to a scale afterward. How pairwise comparison training works and why it tends to produce steadier models than direct scoring is a specific design choice within this larger picture, distinct from the labelling-source question this piece is about.

How many labellers a single image actually needs

A single labeller's score for one image is a sample size of one, and treating it as ground truth means treating that one person's mood, attention and interpretation of the rubric as fact. Most careful pipelines score each image several times, with different labellers, and combine the results - typically an average, sometimes a median if the pipeline wants to be less sensitive to one outlier score. More labellers per image narrows the noise around the combined value, in the same way that averaging several measurements narrows the noise around any estimate, but it also multiplies the cost of building the dataset, so there is a real budget tradeoff between labelling each image once and labelling fewer images several times each.

Teams that publish anything about their process tend to report a number here - three, five, sometimes more per image for a subset used to establish agreement - though many teams report nothing at all, and a dataset that scored each image only once has no way to tell you how much any individual label should be trusted relative to another labeller who might have scored the same photo differently.

What happens when the rubric itself changes

Labelling is rarely a single event. A model is retrained periodically, sometimes with a revised rubric, updated instructions, or a different labelling vendor entirely, and each of those changes shifts what the resulting scores actually represent, even when the axis keeps the same name on the screen. An axis called "shape" trained under one rubric and an axis called "shape" trained under a later, slightly different rubric are not guaranteed to be measuring the same thing, and nothing about the interface distinguishes one version from the other unless the product explicitly says so. This is a slower-moving cousin of the labelling question above: it is not about who labelled the current dataset, but about the fact that the current dataset is itself one snapshot in an ongoing, occasionally revised process, and a score from an old model version sits on a scale that may no longer match today's.

What this does not decide

Where the labels came from and how consistent the labellers were is a question about the training data and the ceiling it sets. It is a related but separate question from dataset bias - whether the images themselves, independent of their labels, represent the range of subjects the model will later be asked to score - which is its own subject and not covered here. It is also separate from the ranking-versus-rating design choice above; a pipeline can be well labelled and still choose either an absolute or comparative training approach.

Reading a score with this in mind

None of this means a trained scoring model is arbitrary. A well-run labelling process, with a clear rubric, enough labellers per image to average out individual noise, and measured agreement above a sensible threshold, produces a model whose outputs track something real: a stable, socially shared sense of how a given rubric applies to a given kind of image. The honest caveat is that "real" here means real relative to that group of people and that rubric, not real in the sense of an independent fact the labels were merely reporting.

Rate Cock publishes its axis definitions rather than leaving the rubric implicit, on the theory that a score is more useful once you know roughly what question it was trained to answer. This labelling process has nothing to do with how Measure My Cock arrives at a number - a tape measure does not need labelled training examples, which is exactly why its figures answer a different question than any trained scorer's do. A human reviewer, by contrast, is closer to being a single fresh labeller than a trained model at all; what a commissioned human judgement actually involves is a different process with a different relationship to consistency. Reading any AI score with a rough sense of where its labels came from, rather than treating the number as self-evidently objective, is the habit Penis Rater tries to build in its own coverage of how to read a result. A score, in the end, is a compressed opinion, held by however many people wrote the rubric and did the labelling, run through a process built to generalise that opinion to a new photograph. Knowing whose opinion, and how carefully it was gathered, is most of what separates trusting a number and understanding one.