Sampling randomness, and why it should be off for scoring

A language model does not simply produce the most likely next word every time. It samples from a distribution over possible next words, and a setting called temperature controls how random that sampling is. When a scoring tool is built on a language model and leaves temperature above zero, the number it returns is partly a dice roll, and most users never learn that the dial exists.

What temperature actually controls

At each step, the model computes a probability for every possible next token - "7" might get 40%, "6" might get 25%, "8" might get 15%, and so on down a long tail. At temperature zero, sometimes called greedy decoding, the model always takes the single highest-probability token. Given identical input, it gives the identical output, every time.

Raise the temperature and the distribution flattens: lower-probability tokens become more likely to be picked, and the choice becomes genuinely stochastic. A temperature of one leaves the raw probabilities largely intact; higher values push toward more surprising, more varied output; values between are a sliding scale of how much randomness is injected before a token is chosen. Top-p, a related setting, works by narrowing the pool of candidate tokens the model is allowed to sample from before applying temperature, rather than changing the shape of the distribution itself.

None of this is specific to scoring. Temperature exists because varied, less predictable text is desirable for creative writing, conversation and brainstorming, where the same prompt producing the same reply every time would feel robotic. A scoring task has the opposite requirement, and inherits the setting anyway if nobody changes it.

Why a nonzero temperature is the wrong default for a number

When the output is a number rather than a sentence, the token being sampled is a digit, and sampling a digit is sampling the answer. If "7" and "6" are close in probability, a temperature above zero means some fraction of runs on the exact same photo will return a 6 and some will return a 7, purely from where the sampling landed that time, with no change to the image, the prompt or the model's underlying belief.

This is a distinct source of variance from the kind that comes from photo conditions - lighting, angle, distance - which genuinely changes the input the model sees. It is also distinct from hardware-level nondeterminism in the numerical operations themselves, a separate question with its own explanation. Temperature-driven variance is different from both: identical input, identical model, and the output still moves, because the generation step includes a deliberate random draw.

A well-built scoring pipeline sets temperature to zero, or as close to it as the provider allows, for exactly this reason. Anything else means part of the number a person reads as a judgement about their photo is, in a literal and checkable sense, a coin flip the tool chose to include.

How to notice it from outside

You cannot see a tool's temperature setting from the outside, but you can infer it. Submit the same file to the same tool several times in quick succession and watch the numbers. If they never move, temperature is at or near zero, or the tool has some other mechanism smoothing the output - running the same input several times and combining the results is one such mechanism, and it produces a stabler number precisely by absorbing the randomness this article describes rather than eliminating it upstream. If they move by a point or two with nothing else changed, temperature is doing some of that.

Rate Cock runs its scoring at a fixed low temperature and states as much, which is the kind of detail that only matters once you know the setting exists. It is a small thing to check for in any tool whose numbers you plan to compare over time.

Where this sits next to other sources of change

Wording changes in the rubric are a separate cause of score movement from sampling randomness - the rubric-as-prompt problem covers that side. Photo-to-photo variance from real conditions is a third, larger source, covered in why the same subject scores differently across a session. Temperature is the narrowest of the three: it moves the number even when everything else, including the exact bytes of the file, is held constant, which makes it the cleanest thing to test for and the easiest for a tool to simply turn off. Measure My Cock's method coverage exists partly because a physical measurement has no equivalent dial to worry about, which is one of the tradeoffs between an inferred score and a tape. Penis Rater's guide to reading scores touches on repeat testing from the user side, and a human reviewer sidesteps the question entirely, since nothing about a person's judgement is a sampling temperature - their variance comes from somewhere else altogether.