Where the wobble in a score comes from

An AI score varies for four separable reasons: how the photo was captured, how the pipeline preprocesses it, which model version and weights score it, and how the output is sampled. A variance budget assigns the wobble to those four, so you know which one a given change in a number actually came from.

Why a budget, and not just "noise"

When two scores for what feels like the same photo differ, the usual reaction is to call it noise and move on. That word hides a great deal. Some of the difference is real, because you changed the photo. Some is a property of the tool's plumbing, which you did not change at all. Some is chance, built into how the model produces its answer.

Treating all of it as one lump makes two mistakes possible. You can read a meaningless fluctuation as a real change, and you can dismiss a real change as noise because you have seen the number jump around before.

A variance budget is the habit of itemising. It borrows from accounting: a total has to be explained by its parts, and a part you cannot account for is a finding in itself. The parts here are not arbitrary; they follow the order in which a photo moves through a scorer.

The four lines of the budget

Think of the path as four stages, each of which can add spread.

1. Capture. Everything about how the photo came to exist: light, angle, distance, lens, camera settings, the moment.
2. Preprocessing. Everything the pipeline does to the file before the model sees it: resizing, cropping or padding, colour handling, normalisation, quality checks.
3. Model. Which network, trained how, at which precision, possibly several combined.
4. Sampling. Any randomness at output time: the temperature of a language model, repeated runs, averaged views.

Spread at each stage compounds into the final score. If the sources are roughly independent, their variances add, which means the standard deviations combine as the square root of the sum of squares. One large source dominates the total, and shrinking a small source barely helps.

That arithmetic is the most useful consequence of the budget. It tells you where effort pays off, and where it is wasted.

Line one: capture

Capture variance is the spread between photos of the same subject taken at slightly different moments. Hold a phone, take five shots in a row, and you get five different photos, even if you feel you did the same thing. Light shifts, the camera refocuses, the angle drifts by a few degrees.

This is often the biggest line in the budget and the one people most want to blame on the model. A model with zero internal noise would still give five different scores to five different photos, because they are five different inputs.

Other pieces cover individual capture factors in depth, such as how framing shifts the number and how context leaks in from the background, so this piece treats capture as a single source and moves on. The thing to know is how to isolate it: re-shoot, and compare the spread between photos with the spread you get from re-submitting one photo. The gap between those two spreads is capture.

Line two: preprocessing

Preprocessing variance comes from the pipeline's choices rather than from chance. It is deterministic for a given file and pipeline, but it differs between pipelines and between versions of one.

The steps include shrinking the photo, which happens before the model looks, the choice of resize kernel as described in bilinear versus bicubic, the handling of non-square shapes, and the normalisation constants. Each moves the numbers a little.

Because it is deterministic, preprocessing does not show up when you re-submit the same file to the same tool. It shows up when you compare tools, or when a tool is updated. That is why a gap between two services is partly a statement about their plumbing rather than their models.

To isolate it, feed the same file through two pipelines that differ only in the preprocessing step, which you can do by pre-resizing the file yourself in two ways. If you cannot control the pipeline, you can still bound the effect by comparing the same photo at a few resolutions you produce.

Line three: the model

Model variance is the difference between two models that were meant to do the same job. It has several sources.

Training itself is stochastic. Dodge and colleagues (2020) showed, for fine-tuned language models, that random seeds controlling weight initialisation and data order account for substantial variation in results, even with everything else fixed. The same logic applies to vision models: retrain on identical data with a different seed and you get a slightly different scorer.

Beyond seeds, there is version drift, as a tool swaps in a retrained or updated model. There is serving precision, where quantised models give slightly different scores. And there is composition, where a tool combines several models in an ensemble, so that the identity of the scorer is itself a mix.

Model variance is frozen during a session: the same model answers the same way each time, so you cannot see it by re-submitting. You see it only across time, across tools, or across versions. That is why a score that was stable all month can shift overnight when nothing about your photo changed.

To isolate it, hold the photo and preprocessing fixed and change only the model, which in practice means comparing two tools over many photos or watching one tool across an update.

Line four: sampling

Sampling variance is randomness at output time, and it is the line most people think of when they say "noise".

A language-model scorer chooses its words probabilistically, so the same photo can get a seven on one run and a six on the next, a mechanism explored in the temperature setting. A pipeline that scores several views and averages them has randomness in which views it picks, as in test-time augmentation. Some serving stacks add small non-determinism of their own from parallel computation.

This is the cleanest line to measure. Submit the identical file repeatedly and look at the spread. Everything upstream is frozen, so what varies is sampling and serving noise alone. The method is the subject of test-retest reliability.

It is also the line that can be reduced most cheaply. Repeating the run and taking the median, as in asking five times, shrinks sampling variance directly, though it does nothing for the other three lines.

Putting the lines in a table

The pattern in the table is that each source is invisible to the experiments that freeze it. Re-submitting one file measures sampling only. That is why a tool can look rock-solid under a repeat test and still give you a very different number when you re-shoot.

A nested way to run it

A budget is easiest to build as nested comparisons, each adding one source.

Start with one file, uploaded many times. The spread is sampling. Then take several different photos of the same scene, each uploaded several times. The extra spread between photos, beyond what each photo shows alone, is capture. Then run the same files through a second pipeline or the same tool at a later date. The systematic shift between the two sets is preprocessing and model, which you cannot separate without being able to change one of them.

Enough repeats are needed to see past chance in the estimate itself. With only two or three runs, a spread is a guess. How to run a controlled comparison covers the discipline of changing one thing at a time and repeating enough.

You do not need special tooling. A spreadsheet with a column per condition and a row per upload is enough to see which spread is large.

Reading a difference against the budget

Once you have rough sizes for each line, a change in a score has something to be measured against. A difference smaller than the sampling spread is indistinguishable from re-submitting. A difference smaller than the capture spread is indistinguishable from re-shooting. Only a difference larger than both is a candidate for a real change.

That discipline matters most for before-and-after comparisons of your own photos, where it is tempting to read a small improvement as a result of something you did. It also guards against the opposite error: waving away a real effect because the numbers always move a bit.

The budget does not tell you whether the score measures what you think it measures. That is a question about validity, and the ground-truth problem is where it is worked through. A tool can have a tiny variance budget and still be answering the wrong question with great consistency.

What the budget cannot see

Three limits are worth stating.

Independence is an assumption. Capture and preprocessing interact, because a resize kernel affects a sharp photo differently from a soft one. When sources interact, the variances do not simply add, and the table above is a first approximation.

Magnitudes depend on the tool. Nothing here says how big any line is for a specific service, and nobody outside the pipeline can say in general. The direction of each source is knowable; the size has to be measured per tool.

Finally, a budget describes spread, not bias. A systematic error that shifts every score in the same direction adds no variance at all, and every line above can be tiny while the number is still off.

Where this leaves a single score

A single number from a single upload is one draw from a distribution whose width you do not know. Sampling and capture are the parts you can estimate yourself; preprocessing and model are the parts you can only compare across tools.

Rate Cock, as one example, describes the stages of its own scan on its how-it-works page, and the budget above is the general frame for deciding which of those stages could be moving a number. For capture in practical terms, the human-side guidance on penisrater.com about photos covers what changes between shots from the user's chair. Where a question is about a physical figure rather than a score, the method belongs to measuremycock.com, where repeat measurement has its own conventions. And when what you want is a second reading from a person rather than a second run of a model, ratepenis.com on judges covers that alternative.

The most useful single habit is simple: before you believe a change, find out how much the number moves when nothing changed.