What 8-bit weights do to a number out of ten

Serving a model at lower numeric precision makes inference cheaper and faster, and shifts its outputs by small amounts - sometimes enough to flip a rounded score. Quantisation converts weights stored as 32-bit or 16-bit floats to a lower-precision format, commonly 8-bit integers, before the model meets real traffic.

Why services do it

Lower precision means smaller weight files, less memory used per loaded model, and faster arithmetic on the hardware actually running inference, since integer operations at 8 bits are cheaper than floating-point operations at 32. At the scale of a service handling continuous traffic, this translates directly into lower infrastructure cost and lower latency per request, both of which matter more to the business running the tool than a small amount of numeric precision. The architecture does not change, and the simplest forms need no retraining; the same weights are just represented with fewer bits each. It is a well-established technique across production machine learning, not something specific to scoring models: Jacob and colleagues at Google (2017) described integer-only inference as implementable more efficiently than floating point, and framed the choice as a tradeoff between accuracy and on-device latency.

Where the shift comes from

Reducing precision means each weight is now an approximation of its original trained value, rounded to the nearest representable number in the lower-precision format. Individually these rounding errors are tiny. Propagated through every layer of a model and every one of the many multiplications that make up a forward pass, they accumulate into a small but real shift in the final output compared to the full-precision version of the same model given the same input.

For a task like image classification, where the output is a coarse category, this shift is usually irrelevant - the answer does not change even if the confidence moves slightly. For a task like scoring, where the output is a continuous or near-continuous number that later gets rounded to something like one decimal place or a whole number out of ten, a shift too small to matter for classification is sometimes exactly large enough to move a score across a rounding boundary. A photo that would render as 7.4 from the full-precision model can come back as 7.5, and round to a different displayed number, purely from the serving format rather than from any change to the photo or the model's trained knowledge.

What it does and does not explain

Quantisation explains small, boundary-crossing shifts, not large swings. It is a fixed, deterministic property of which serving format a request happened to hit, not a source of randomness that varies run to run on its own - that separate question, about whether identical input produces identical output at all, has its own causes worth reading if the variance seen is larger than a rounding step. It is also unrelated to how physically distant the camera was, or how the photo was cropped - those affect what the model is looking at; quantisation affects only how precisely it computes on whatever it is given.

Which precision a service runs is not something a user can see or control, and Rate Cock, like most tools operating at any real traffic volume, uses a quantised model for at least some of its inference for cost reasons. None of this is the kind of variance worth chasing on its own; reading the spread across several repeats rather than a single number absorbs it along with everything else, and it has no bearing on an actual physical measurement or on what a human judge's number does not owe to floating-point arithmetic at all.