What 8-bit weights do to a number out of ten

A trained model's weights are, by default, stored as 32-bit or 16-bit floating point numbers. Quantisation converts them to a lower-precision format - 8-bit integers is a common target - before the model is served to real traffic. The model architecture does not change and no retraining is required for the simplest forms of quantisation; the same weights are just represented with fewer bits each.

Why services do it

Lower precision means smaller weight files, less memory used per loaded model, and faster arithmetic on the hardware actually running inference, since integer operations at 8 bits are cheaper than floating-point operations at 32. At the scale of a service handling continuous traffic, this translates directly into lower infrastructure cost and lower latency per request, both of which matter more to the business running the tool than a small amount of numeric precision. It is a well-established technique across production machine learning generally, not something specific to scoring models, and it is usually applied after training is complete, as a separate serving-time optimisation.

Where the shift comes from

Reducing precision means each weight is now an approximation of its original trained value, rounded to the nearest representable number in the lower-precision format. Individually these rounding errors are tiny. Propagated through every layer of a model and every one of the many multiplications that make up a forward pass, they accumulate into a small but real shift in the final output compared to the full-precision version of the same model given the same input.

For a task like image classification, where the output is a coarse category, this shift is usually irrelevant - the answer does not change even if the confidence moves slightly. For a task like scoring, where the output is a continuous or near-continuous number that later gets rounded to something like one decimal place or a whole number out of ten, a shift too small to matter for classification is sometimes exactly large enough to move a score across a rounding boundary. A photo that would render as 7.4 from the full-precision model can come back as 7.5, and round to a different displayed number, purely from the serving format rather than from any change to the photo or the model's trained knowledge.

What it does and does not explain

Quantisation explains small, boundary-crossing shifts, not large swings. It is a fixed, deterministic property of which serving format a request happened to hit, not a source of randomness that varies run to run on its own - that separate question, about whether identical input produces identical output at all, has its own causes worth reading if the variance seen is larger than a rounding step. It is also unrelated to how physically distant the camera was, or how the photo was cropped - those affect what the model is looking at; quantisation affects only how precisely it computes on whatever it is given.

Which precision a service runs is not something a user can see or control, and Rate Cock, like most tools operating at any real traffic volume, uses a quantised model for at least some of its inference for cost reasons. None of this is the kind of variance worth chasing on its own; reading the spread across several repeats rather than a single number absorbs it along with everything else, and it has no bearing on an actual physical measurement or on what a human judge's number does not owe to floating-point arithmetic at all.