Change one thing, hold the rest, repeat

Most people who conclude something about an AI score have run one photo through once, or two different photos through once each, and treated the result as a finding. Neither is a test. A comparison that means something isolates one variable, holds everything else steady, repeats enough times to see past ordinary noise, and looks at the spread of results before it looks at any single number or even the average. This post is the method; the reasons the noise exists in the first place are covered elsewhere and are worth reading first if you have not.

Why a single pair of scores tells you almost nothing

The spread across a single afternoon of otherwise-identical attempts is wider than almost anyone expects, and that post catalogues the sources of that spread in detail: lighting, angle, distance, crop, blur, colour, even the state of the subject itself. Every one of those is a variable that can differ between two photos you think of as "the same," and if you have not controlled for them, a difference in score between two attempts could be caused by any of them, or several at once, in combinations you cannot untangle after the fact.

This matters because the conclusions people draw from an uncontrolled pair are usually causal ones - "the tool likes photos taken this way," "the light hurt my score," "retaking it made it worse" - and a causal claim needs a comparison that actually isolates the cause. Two photos that differ in framing, lighting, and time of day all at once cannot support a claim about any one of those three, however tempting it is to pick the explanation that feels most plausible.

Step one: decide on the one variable

Pick a single thing you want to learn about - angle, crop tightness, lighting, time of day, whether a mirror shot behaves differently from a direct one, whatever the actual question is. Everything else in the setup then has to be held as close to identical as you can manage across every photo in the test. That is a longer list than it sounds like, and most of it has its own explanation elsewhere on this site if you want the mechanism behind why each item matters:

• Distance and crop. How much of the frame the subject fills changes what the model's resize actually contains, so if crop is not your test variable, keep your framing distance consistent between shots.
• Angle. Camera angle changes apparent shape and silhouette on its own, independent of anything else in the frame, so hold your camera position fixed unless angle is the thing you are testing.
• Lighting direction and type. Shading gradients are a large part of how shape gets inferred from a flat image, and both the direction of the light and whether it is direct flash or ambient light change those gradients; keep the light source and its position fixed between attempts.
• White balance and colour. Auto white balance recolours the whole frame based on a guess about the light source, and that guess can shift between otherwise-identical shots taken moments apart; shooting in the same spot under the same light reduces but does not eliminate this, so it is worth checking that consecutive shots look colour-consistent before treating them as controlled.
• Background and clutter. Everything in frame contributes to the embedding the score comes from, so the backdrop, its colour, and anything else visible should stay the same across the set unless background is what you are testing.
• Blur type. Motion blur and defocus blur are different signals to a model even when they look similarly soft to the eye, so stability and focus should be as consistent as you can manage shot to shot.
• Capture method. A frame extracted from video is not the same input as a still photo, carrying heavier compression and different artefacts, so do not mix the two within one comparison.
• Resolution, format and file handling. Keep the same camera, the same export settings, and the same upload path each time, since resizing, re-encoding or converting between formats introduces its own small shifts.
• Subject state. Temperature, time of day and general physiological state change surface appearance in ways a model reads as part of the image; running your whole comparison set in one sitting, rather than spread across days, removes a variable you otherwise cannot control for.

That is a long list to hold fixed, and the honest reading of it is that a genuinely controlled comparison takes more setup than most people expect before you have even taken a photo. The list is also the reason single, casual comparisons are so unreliable: almost nobody controls for all of it, so almost every casual pair of scores has several uncontrolled variables baked in.

Step two: repeat enough times

One trial in each condition proves nothing, because you cannot distinguish "this condition scores higher" from "the noise happened to land higher this time." The number of repeats you actually need depends on how wide the underlying spread is for the tool and subject in question, and you cannot know that spread until you have taken several measurements to begin with - which is a bit circular, and the practical way through it is to start with a reasonable batch, look at how much the scores vary within a single condition, and add more repeats if that spread looks wide relative to the difference you are trying to detect. As a starting point, five to ten repeats per condition is enough to see whether a spread exists at all; if the effect you are looking for is subtle, you will need more than that to be confident it is not just noise, and there is a fuller treatment of exactly how many repeats a given spread requires elsewhere.

Every repeat has to be a genuinely separate attempt - a new shot, not the same file uploaded twice, and not a cached result standing in for a fresh one.

Step three: randomise and avoid caching the result

If you are comparing two conditions - say, direct light against angled light - do not run all of condition A first and all of condition B second. Anything that drifts over the course of a session, including your own attention to detail or subtle changes in the room, will then correlate with condition rather than with time, and you will not be able to tell the two apart. Interleave the conditions instead: A, B, A, B, or a shuffled order, so that any drift affects both conditions roughly equally rather than stacking onto one of them.

Some tools recognise a previously-uploaded file by its hash and return a stored result rather than running inference again, which means re-uploading the same file is not a repeat trial at all - it returns the same cached number every time and tells you nothing about the tool's actual variance. Running a handful of unrelated or random images through the same protocol as a control group, the way a placebo test does, is a useful sanity check that your setup is not producing an effect out of nothing. Take a genuinely new photo for every trial, even if the framing is meant to be identical, so that each one is a fresh pass through the pipeline rather than a lookup.

Step four: record everything, not just the number

Write down the score for each trial alongside a note of the condition, the order it was taken in, and the time. This sounds like overkill for what might be ten photos taken in an afternoon, and it is the single most common thing people skip, and the reason most informal comparisons cannot be checked or trusted afterward, including by the person who ran them. A simple table - trial number, condition, score, any note about something unusual that happened during that shot - is enough. The point of the record is not analysis; it is being able to look back and see the actual pattern rather than remembering it, which people are bad at, especially when they had a hypothesis going in.

Step five: look at the spread before you look at the average

Once you have your set of numbers for each condition, resist the pull toward computing an average and comparing the two averages first. Look at the range within each condition before anything else - the lowest and highest score you got under identical conditions. If that range is nearly as wide as the difference you were hoping to find between conditions, you do not have a finding yet, no matter what the two averages say, because the averages of two overlapping distributions can differ by chance alone. Reading the spread rather than the score is worth its own habit, independent of any specific comparison you are running, and it is the single biggest upgrade available to anyone testing a scoring tool informally.

This post deliberately does not go into the statistics of how much overlap is too much, or how to compute a formal reliability figure from your results - that is a separate, more technical subject, and the eyeball version above (does the range within each condition swallow the difference between conditions) gets you most of the practical value without it.

What a controlled comparison is not for

None of this method turns an AI score into a measurement. It tells you, with reasonable confidence, whether one condition produces a different distribution of scores than another for a given tool, which is a genuinely useful thing to know and is not the same claim as "this photo is objectively better." If what you actually want is a number in physical units rather than a comparison between conditions, Measure My Cock's method is built for exactly that and does not depend on any of the above, because a tape measure does not have the variance a vision model does.

A controlled comparison run against Rate Cock or any other scoring tool is also only ever a comparison of that tool's own behaviour - the same method run against a different tool, or a human reviewer, can turn up a different pattern entirely, since what a human reads and what a model reads are not the same input in the first place. Penis Rater's own coverage of reading scores is a shorter, more practical companion to this post for someone who wants the everyday version rather than the full protocol: what a result supports and what it does not, without running a formal comparison at all. Blinding the comparison is the specific discipline that keeps a controlled comparison honest once it is actually running. Whichever version you use, the underlying discipline is the same one this whole site keeps returning to: a score is a description of the photograph you gave the model, not of anything it measured, and a controlled comparison is simply the way to find out how much of a given result is signal and how much is the photograph you happened to take that day.