Two architectures, and what differs for a rating task

A rating tool built on a convolutional network and one built on a vision transformer can produce very similar scores from very different internal routes. The architecture is not usually the thing that decides whether a tool is any good; the training data and the scoring head sitting on top of it matter more, and it is worth being clear about why before assuming the fancier-sounding option wins.

Two ways of looking at an image

A convolutional network, or CNN, processes an image with small filters that slide across it, each one looking at a local neighbourhood of pixels at a time. Early layers see only tiny patches; deeper layers see progressively larger effective regions, built up by stacking many local operations. This gives a CNN a built-in bias toward local structure - edges, textures, small repeated patterns - and it has to work, layer by layer, to combine local information into anything more global.

A vision transformer, by contrast, splits the image into a grid of patches from the start and lets every patch attend to every other patch directly, from the first layer onward. There is no built-in locality bias - a transformer can, in principle, relate a patch in one corner to a patch in the opposite corner immediately, rather than needing several layers to build up to it. How that patch-and-attention mechanism actually works is its own explanation, covered in patches and attention: how a transformer reads a photo; this post stays at the comparison level rather than repeating that mechanism here.

Texture bias versus shape bias

Researchers comparing the two families - notably work following Geirhos and colleagues on texture versus shape bias in CNNs, published around 2019 - have found that CNNs trained the ordinary way lean toward classifying images by texture cues over global shape, sometimes surprisingly strongly. Vision transformers, with their whole-image attention from the start, tend to weigh global arrangement more heavily relative to local texture, though the difference is a matter of degree rather than an absolute split, and can be shifted by training choices either way.

For a rating task that includes axes about both texture and about proportion or shape, this is not an idle academic point. A texture-leaning backbone may be more sensitive to surface condition and less decisive about overall form; a shape-leaning one may do the opposite. Neither is simply better - they are differently shaped tools, and which one suits a given axis is an empirical question a tool's designers have to actually check rather than assume from the architecture's reputation.

Data appetite

Vision transformers, in their original form, generally need more training data to reach the same performance as a CNN, because they lack the locality assumptions a CNN gets for free and have to learn spatial structure from examples instead of having it built in. This was the headline finding when the architecture was introduced - Dosovitskiy and colleagues' 2020 paper on the original Vision Transformer noted it underperformed comparable CNNs on mid-sized datasets and only pulled ahead with far larger training sets. In practice, most modern vision transformers used in products are pretrained on very large general-purpose datasets first, which sidesteps the problem, but it is a real historical difference and a reason the two families are not simply interchangeable at every scale.

What actually decides the score

For a scoring task specifically, the backbone - CNN or transformer - sets what kind of information is available to work with; it does not decide what the model does with it. That is the job of the scoring head trained on top, and the labels it was trained against, both of which are separate design choices covered in regression head or class buckets: how the number comes out and in the training-data posts elsewhere on this site. A well-trained head on a modest CNN backbone can outperform a poorly trained one on a state-of-the-art transformer, because the ceiling on any scoring system is set by its labels and its head at least as much as by its backbone - a point covered in general terms in how image models score anatomy.

None of this is something a user of a rating tool can inspect from outside, and no tool advertises its backbone as a selling point worth trusting on faith. Rate Cock does not make backbone choice part of its pitch, for the same reason a camera brand is not the thing that decides whether a photograph is good - the parts further down the pipeline matter more. If your interest is in a number that does not depend on either architecture's biases, a tape measure and a method sidesteps the whole comparison, and if you want a second opinion that reasons about shape and texture the way a person does rather than either kind of network, Rate Penis's human-judge option is the alternative worth knowing about. Reading a resulting score for what it actually supports, regardless of which architecture produced it, is covered from the user's side by Penis Rater.