Fatigue and order affect people, versioning affects models

Human raters drift within a single session, through fatigue and the order photos arrive in; a deployed model drifts only when a new version replaces it. Both are drift, on different clocks, and knowing which clock you face changes what a comparison across time can tell you.

The human clock

A person judging a run of submissions is affected by what they just saw. In a 2013 PLOS ONE study, Kramer, Jones and Sharma found that people rating a series of faces for attractiveness had each judgement drawn toward the rating they had given the face before it. Pegors and colleagues (2015) found two pulls at once: ratings shifted toward the previous rating and away from the previous face's attractiveness. Either way, the ordering effect has nothing to do with the photo itself. Fatigue can move standards too, within a single sitting, without the rater necessarily noticing it.

The model clock

A deployed model does not get tired and has no session to be affected by order within. Its scores move on a completely different schedule: when the tool retrains and redeploys, often without announcing it, and a score from last month's version is not on the same scale as one from today's, even if nothing about your input changed.

Between deployments, a frozen model is about as fixed as an instrument gets. The catch is that you cannot always tell when the version underneath you has changed, which turns machine drift into a hidden step-change rather than the gradual creep a tired human produces.

Why the distinction matters

Comparing two of your own scores, taken an hour apart from a model, is safe from the human failure mode entirely - there is no session, no order, no fatigue for a machine to have. Comparing two scores taken months apart carries the opposite risk: the model itself may no longer be the same model, and a widened gap could be entirely due to versioning rather than anything about the input. Tracking whether a tool's numbers have shifted over a longer period is the direct way to check, and it is a different exercise from noticing a single afternoon's inconsistency in a person.

Neither clock makes one method categorically better; they just fail at different timescales. A commissioned human reviewer works within a session in ways worth understanding on their own terms, and a repeatable score you can compare or an actual measurement taken with a tape each sidestep one kind of drift while remaining exposed to the other. Rate Cock versions its model like any other tool in this category, which is worth knowing before reading too much into a score from six months ago.