The photo is cut into tiles, and the tiles vote

A vision transformer does not look at a photo as a whole image, and it does not scan it the way older networks do either. It cuts the resized photo into a grid of small square tiles - patches, typically 16 by 16 pixels each - and treats each patch as a separate unit from the very first layer.

Turning tiles into tokens

Each patch is flattened into a list of numbers and passed through a simple linear transformation, producing what the model treats as a token - conceptually the same kind of object a language model uses for a word. A photo resized to a few hundred pixels square ends up as a few hundred of these tokens, all handed to the network together, none of them privileged over any other at the start.

Because a flattened patch on its own carries no information about where it sat in the photo, a position embedding is added to each token - a learned or fixed pattern of numbers that encodes location and gets summed into the patch's own values. Without this step, the network would have no way to tell a patch from the top-left corner apart from an identical-looking patch from the bottom-right, so position embeddings are not a minor detail; they are what makes spatial arrangement legible at all.

What attention actually does

Once every patch is a token with a position baked in, the network's core operation - self-attention - lets every token look at every other token and decide how much to weigh it. Concretely, each token produces a query, a key and a value; the query from one patch is compared against the keys from all patches, producing a weight for each, and the token's updated representation is a weighted sum of everyone's values using those weights. This mechanism, introduced by Vaswani and colleagues in their 2017 paper "Attention Is All You Need" for language and later adapted to images, is what lets a transformer relate distant regions of a photo directly, in one step, rather than needing to build up to it through many local layers the way a convolutional network does.

The practical effect is that a patch covering, say, part of the subject can have its representation shaped by a patch covering the background, the edge of the frame, or anything else in the photo, in a single layer. Stack several such layers and the amount of cross-talk between distant regions of the image compounds. Eventually these patch-level tokens have to collapse into one description the scoring head can read, and how that pooling step blurs exactly where in the frame a feature sat is a related consequence of the same architecture.

Why this makes context leak further

This is the mechanistic reason a transformer-based scorer can be more sensitive to what surrounds the subject than an older architecture with a stronger locality bias. It is not that the model is deliberately reading the background as part of a judgement; it is that attention has no structural reason to ignore it, and unless a pipeline explicitly masks or crops the subject out first, every patch in frame gets a vote on every other patch's final representation. Background and context leaking into scores covers the observable consequence of this in more detail; this post is about the mechanism that makes it possible in the first place.

It also means that changes far from the subject - lighting in a corner of the frame, clutter near an edge - are not automatically discounted the way a person's focused attention would discount them. A person naturally ignores irrelevant background when forming a judgement; a transformer has to learn to downweight it from examples, and how well it has learned that is untested unless someone specifically checks.

How this differs from the alternative

A convolutional network builds up its sense of the whole image gradually, through many layers of local operations, and the comparison between the two approaches at a broader level - including where each tends to perform better on a scoring task - is covered in CNN or vision transformer: does it change the score?. This post has stayed inside the transformer case specifically, because the patch-and-attention mechanism is detailed enough to deserve its own explanation rather than a summary paragraph inside a comparison piece.

What this means practically

If a tool you are using is transformer-based, tighter framing that excludes irrelevant background gives the attention mechanism less to work with beyond the subject itself, which is one part of a broader practical picture Rate Cock's photo guide covers properly, rather than as a side note here. None of this changes what the embedding produced at the end of the process actually is or how it gets used downstream - that step is covered in what an image becomes before it is scored. Where a system reports several axis scores rather than one, framing-driven noise from an inattentive crop tends to hit presentation-sensitive axes hardest, a distinction Penis Rater covers from the practical side, while Measure My Cock's gear notes touch on the camera-and-framing side of the same problem, and Rate Penis covers the entirely separate question of what a human reviewer needs from a submitted photo instead.