Retraining changes the ruler you did not have

A deployed scoring model is not a fixed object. It is a snapshot of weights that a team can, and routinely does, replace with a different snapshot, and nothing about the interface has to change when they do.

Why models get replaced

Teams retrain for ordinary reasons: new labelled data has come in, a bug in the labelling pipeline got fixed, a better architecture became available, or the old model was drifting on inputs it was not seeing enough of before. Each of those is a reasonable thing to want to fix.

None of them requires telling the user. The endpoint keeps the same name, the interface keeps the same 1-to-10 scale, and the number that comes back can shift for reasons that have nothing to do with the photo.

From the outside, a retrained model and the original model are indistinguishable in every way except their output. There is no version number stamped on a result by default, no changelog entry most users would ever see, and no requirement anywhere that a tool announce when the thing producing its numbers has changed underneath the same familiar interface.

Why the scale does not carry over automatically

A new model trained on a different or expanded dataset does not inherit the old model's calibration. Its scoring head was fit to a fresh set of labels, and there is no mechanism that forces the fresh model's "7" to mean the same thing as the old model's "7" unless someone deliberately checks and adjusts for it.

In practice this means two scores taken months apart can differ because the subject changed, because normal variance did what variance always does, or because the ruler itself was replaced in between - and from outside the tool, those three cases are indistinguishable. A version bump is not usually visible to a user at all, unless the tool goes out of its way to expose one.

What honest disclosure would look like

A changelog naming the date of a retrain, even without technical detail, would let a user separate "the tool changed" from "the subject changed." A version identifier attached to each result would make it possible to notice, after the fact, that two scores came from different models rather than assuming they are comparable.

Re-running a small fixed reference set after every deployment and publishing whether the average moved is the more rigorous version of the same idea, though few tools do this because it requires keeping a benchmark around specifically for the purpose, rather than because the benchmark is used for anything else. None of these steps is technically hard; what they cost is the willingness to admit, in public, that the tool changed, which is a harder thing for a team to commit to than the engineering itself. Where that kind of tracking exists, it belongs to a separate practice - watching a tool's output drift over months using a fixed reference set is worth reading on its own, since it is about detecting the consequence from outside rather than the cause from inside.

What this means for comparing your own results

If you are comparing a score from several months ago to one from today, the honest baseline assumption is that you cannot tell whether the model changed underneath you, unless the tool has told you otherwise. That is not a reason to distrust either individual score. It is a reason to treat a long-gap comparison as weaker evidence than a same-day comparison, where at least the model is guaranteed to be the same one.

This is a general property of any deployed machine-learning system, not something specific to rating tools, which is why measuremycock.com's coverage of method treats "which version produced this number" as a basic methodological question rather than trivia. Reading a single historical score with appropriate scepticism is part of reading any result honestly, and it is one more way a human reviewer differs structurally, since a person's judgement does not get silently swapped for a different person's between one review and the next the way a model can. Rate Cock is one tool among many where this applies, and the practical habit is the same regardless of which one you use: treat "same scale" as a claim to verify, not an assumption to make.