Accuracy

Small edits that move the number a lot

Vision models can be pushed by perturbations invisible to people, which says something about what the score was tracking all along.

4 min readAccuracy

An adversarial example is an input changed just enough to shift a model's output, without changing enough for a person to notice anything different. The classic demonstrations are in general image classification - a picture of a panda, altered by a pattern of pixel-level noise invisible to the eye, gets confidently classified as something else entirely. The same underlying vulnerability exists in scoring models, and it is worth understanding not because it is a practical threat most people will encounter, but because of what it reveals about how these systems actually work.

Why this is possible at all

A vision model's decision boundary - the line, in its internal representation, that separates one output from another - is not aligned with anything a human would consider a meaningful visual difference. It is a mathematical surface shaped by training, and it can be close to the actual input in directions a person cannot see, because the model's notion of "close" is defined in a high-dimensional numerical space, not in terms of what looks different to an eye. A carefully constructed nudge in exactly the right direction, in that space, can cross the boundary while barely moving the pixels a person would look at. This is possible precisely because the model is not doing anything like human perception - it never had access to the original scene, only to a vector encoding of it, and a vector can be nudged in ways an eye has no equivalent for.

What this implies about the score

The existence of adversarial examples is evidence that a model's score is a function of its specific internal representation, not a direct readout of some property that obviously exists in the photo. If the score were tracking something robust and simple - overall brightness, say - it would be very hard to move without a person noticing the change too. The fact that it can, in principle, be moved by an imperceptible perturbation means the score is sensitive to fine structure in the model's learned representation that has no obvious correspondence to anything a viewer would call "quality." This does not mean the score is meaningless on ordinary, unmanipulated photos - adversarial vulnerability is a worst-case property, deliberately engineered, and it says little about how the model behaves on typical inputs it was never specifically attacked with.

Why this is a narrow, engineered problem in practice

Constructing an effective adversarial example generally requires access to the model's internals - its gradients, specifically, which tell an attacker which direction in pixel space will move the output the most. A model reachable only through an upload box, with no visibility into its weights or gradients, is far harder to attack this way, though not perfectly immune - some techniques work by querying a model repeatedly and inferring its behaviour from the outputs alone, without ever seeing its internals directly. This is a meaningfully different threat model from casual photo editing, and it is worth being precise about the distinction: cropping tighter, changing the lighting, or picking a more flattering angle are not adversarial attacks in this technical sense, even though they also move the score. Those are ordinary changes to a normal-looking photo that happen to influence what a model sees, which is a much broader and more mundane category, closer to the everyday version of optimising toward a score rather than toward what a person would notice. None of this touches a measured figure either way, since a tape measure has no gradient to attack - the entire category of vulnerability only exists for a system that infers rather than reads a value directly.

What the research is actually useful for

Outside the scoring context, adversarial robustness research exists mainly to stress-test models used in higher-stakes settings - content moderation, medical imaging, autonomous systems - where a small, deliberately engineered failure could have real consequences. For a consumer rating tool, the practical takeaway is smaller and more diagnostic than protective: auditing a black-box scorer with paired inputs is a related but distinct technique, using deliberate, visible changes to find bias rather than invisible ones to find a break, and it is the more useful tool for most people who want to understand a scorer's behaviour rather than attack it.

The honest summary

Adversarial vulnerability is real, well documented in the general vision-model literature, and mostly irrelevant to how an ordinary user experiences a rating tool day to day. Its value here is diagnostic rather than practical: it demonstrates, cleanly, that a score is a property of the model's representation rather than a direct readout of the world, which is the same lesson the ground-truth problem makes from a completely different angle. Rate Cock is not immune to this category of research finding any more than any other vision-based scorer, though ordinary use never approaches the kind of deliberate, engineered input this research is about. None of this has an equivalent on the human-judging side - a person reading a photo cannot be fooled by a pixel-level perturbation the way a model's decision boundary can, which is one of the structural differences between the two approaches rather than a simple advantage for either. Reading your own results as a trend across ordinary retakes, the way penisrater.com frames the practical use of a score, is unaffected by any of this, since nothing about typical use resembles an adversarial attack.

Read next

Full archive