Layer by layer, from contrast to concept

A vision model builds its judgement in layers: early ones respond to edges and colour, middle ones to textures, later ones to parts and whole objects. By the time a score is produced, the model is reasoning several steps removed from raw pixels.

Every one of those earlier steps still runs, though, and whatever entered at the bottom is still in there somewhere.

The stack, roughly

Input pixels
   |
Layer 1-2:  edges, colour blobs, contrast boundaries
   |
Layer 3-5:  textures, simple repeated patterns
   |
Layer 6-9:  parts - curves, contours, recognisable shapes
   |
Layer 10+:  whole-object concepts, the things a scoring head reads

The earliest layers in a convolutional network, or the earliest attention operations in a transformer, respond to the simplest things an image contains: a boundary between light and dark, a patch of saturated colour, a gradient in one direction. Nothing here is specific to any subject matter. Yosinski, Clune, Bengio and Lipson (2014) found first-layer features "similar to Gabor filters and color blobs" that "appear not to be specific to a particular dataset or task." The same early layer that responds to an edge in a photo of a hand responds identically to an edge anywhere else.

Layers in the middle of the stack combine those primitives into textures - repeated local patterns like the weave of fabric, the graininess of skin, the regular structure of hair. This is where the model starts to distinguish surfaces from each other, still without any notion of what object it is looking at.

Layers near the top combine textures and local shapes into parts: curves that form an outline, junctions that suggest where one region ends and another begins, arrangements that recur often enough in training data to be worth encoding as a distinct pattern. The final layers, or the scoring head sitting on top of the backbone, read from this part-level representation to produce whatever the model outputs - a score, a class, a caption.

Why this matters for a rating model

The score comes from the top of the stack, but the top of the stack is built entirely out of what the bottom layers passed up. Nothing at layer nine can attend to information that layer two threw away. If the earliest layers responded to a compression artefact, a sensor noise pattern, or a lighting gradient as though it were meaningful contrast, that response propagates upward and can influence which higher-level pattern gets activated, even though nothing about it was ever a real edge in the subject.

This is the mechanical reason low-level image quality problems - the kind covered in detail by posts on JPEG compression and lighting - are not cleanly separable from the "real" judgement a model makes. There is no clean handoff where quality issues get filtered out and only content-relevant information proceeds. Everything proceeds, weighted by how strongly each layer responded to it.

What this is not

This is a different claim from what a "feature" is when the model's own vocabulary talks about one - that question, and why the model's learned patterns rarely match the anatomical parts a person would list, has its own answer. This post is about the vertical structure - which layer sees what - rather than the content of any individual feature.

The practical shape of it

You cannot point to a single layer and say "this is where the score gets decided," because the decision is distributed across the whole stack, with each layer contributing a transformation rather than a verdict. What you can say is that low-level noise entering early has more opportunities to matter than a single clean signal entering late, simply because it participates in more of the pipeline. This is one reason a controlled, repeatable capture setup produces steadier results than a haphazard one - Measure My Cock's approach to method is built around exactly that principle, even though its purpose is a tape measure rather than a model. Rate Cock reports separate axis scores partly because a single blended number obscures which part of this stack - surface-level texture reading or higher-level shape reading - actually moved between two photos of the same subject, a distinction explored further in why a rubric beats one number. A human reviewer does not have layers in this sense at all; what a person notices that this stack cannot is a genuinely different kind of process, not a faster version of the same one. Comparing scores across tools without knowing anything about either one's stack is, in this light, closer to comparing two different cameras than two readings of the same ruler - a point Penis Rater returns to often.