An Architecture Borrowed From Language
For most of the 2010s, computer vision meant convolutional neural networks. Convolutions build in two assumptions about images: nearby pixels are related, and a pattern means the same thing wherever it appears. Those assumptions are correct often enough that CNNs learned well from modest amounts of data, and every major vision benchmark was led by one.
The Transformer came from machine translation and has neither assumption. It treats its input as a set of tokens and lets every token look at every other token through self-attention. In October 2020, a team at Google Research led by Alexey Dosovitskiy posted a paper titled «An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale,» later published at ICLR 2021. It asked what would happen if a standard Transformer were applied to images with as few changes as possible. The model was called the Vision Transformer, or ViT.
From Pixels to Tokens
A Transformer needs a sequence, and an image is a grid. Treating every pixel as a token is not practical, because self-attention compares every token with every other one, so its cost grows with the square of the sequence length. A 224 by 224 image has more than 50,000 pixels.
ViT cuts the image into square patches instead, typically 16 by 16 pixels. A 224 by 224 image gives 196 patches. Each patch is flattened into a vector and passed through one linear layer to produce a patch embedding. These 196 vectors are the «words» of the title.
Two more pieces are added. A sequence of patches has no inherent order, so a learned position embedding is added to each patch to tell the model where it came from. And an extra learnable token, the class token, is placed at the start of the sequence. After the Transformer layers, the output at that position is used as the representation of the whole image and fed to a classifier.
Everything after that is a standard Transformer encoder: alternating layers of multi-head self-attention and small feed-forward networks, with residual connections and layer normalization. Nothing in it is specific to images.
What Attention Sees in an Image
In a CNN, a neuron in an early layer sees only a small neighborhood, and the network needs many layers before information from opposite corners of the image can meet. In a ViT, every patch can attend to every other patch in the very first layer.
The original paper measured how far attention reaches in each layer. Some attention heads in the earliest layers already covered most of the image, while others stayed local, behaving a lot like convolutions. Deeper layers attended globally. Raghu and colleagues compared the internal representations of ViTs and ResNets in 2021 and found that ViTs have more uniform representations across layers and carry spatial information further into the network.
The Need for Data
The most important result in the ViT paper was about data. Trained on ImageNet alone, with about 1.3 million images, ViT was worse than a comparable ResNet. Without the built-in assumptions of convolution, the model had to learn from scratch that nearby patches are related, and 1.3 million images were not enough for that.
With more data, the result reversed. Pre-trained on ImageNet-21k (14 million images), ViT matched the best CNNs. Pre-trained on JFT-300M, Google’s internal dataset of 300 million images, the largest model (ViT-H/14) reached 88.55% top-1 accuracy on ImageNet after fine-tuning, ahead of the best convolutional models of the time. The authors summarized it by saying that large-scale training trumps inductive bias.
That left a practical problem, since almost nobody outside Google had 300 million labeled images.
DeiT and Training on ImageNet Alone
A few months later, Hugo Touvron and colleagues at Facebook AI showed that the data requirement was partly a matter of training recipe. Their Data-efficient image Transformers (DeiT), published at ICML 2021, trained a ViT on ImageNet only and reached 81.8% top-1 accuracy (83.1% at a higher input resolution), competitive with CNNs of similar size. The changes were heavy data augmentation, strong regularization and a longer, carefully tuned schedule.
DeiT also introduced a distillation token. Alongside the class token, a second special token is trained to reproduce the prediction of a teacher network, and the best teachers turned out to be CNNs. The student Transformer picks up some of the convolutional network’s inductive bias through the teacher’s outputs.
Swin and Hierarchical Designs
Plain ViT has two limits for tasks beyond classification. It produces features at a single, coarse resolution, while object detection and segmentation need features at several scales. And the quadratic cost of global attention becomes too high for large images.
Swin Transformer, from Ze Liu and colleagues at Microsoft Research Asia, addressed both. It computes self-attention only inside small local windows, so cost grows linearly with image size. Between consecutive layers, the windows are shifted so that information can flow across window borders. Patches are merged as the network gets deeper, which produces a pyramid of feature maps at decreasing resolution, like a CNN. Swin set records in object detection on COCO and semantic segmentation on ADE20K and received the Marr Prize for best paper at ICCV 2021.
Self-Supervised Pre-Training: MAE and DINO
The other way around the labeled data problem is to not use labels. Masked Autoencoders (MAE), from Kaiming He and colleagues at CVPR 2022, hide 75% of the patches of an image and train a ViT to reconstruct the missing pixels from the remaining quarter. The encoder only processes the visible patches, so pre-training is fast, and a ViT-H trained this way on ImageNet-1k alone reached 87.8% accuracy after fine-tuning.
DINO, from Mathilde Caron and colleagues in 2021, trains a student network to match the output of a slowly updated teacher on different crops of the same image. A side effect got a lot of attention: the attention maps of the class token outlined objects in the image, although the model had never seen a segmentation label. DINOv2, released by Meta in 2023, scaled the approach to a curated set of 142 million images and produced features that work for classification, segmentation and depth estimation with no fine-tuning.
The CNN Response
ViT’s success raised the question of whether attention was the reason, or whether the improvements came from everything around it: bigger models, better augmentation, new training schedules. In «A ConvNet for the 2020s» (CVPR 2022), Zhuang Liu and colleagues took a standard ResNet and modernized it step by step, borrowing design choices from Swin such as larger kernels, fewer activation functions and layer normalization. The resulting ConvNeXt matched or beat Swin at similar size on ImageNet, detection and segmentation, using only convolutions.
The conclusion most researchers took from this is that the two families are closer than the early results suggested. Convolutions remain efficient on small data and limited hardware. Transformers scale better with data and combine naturally with text and other modalities, since everything becomes tokens.
ViT as a Backbone
That last property explains where ViTs ended up. CLIP, from OpenAI in 2021, trains an image encoder (in its best versions a ViT) together with a text encoder so that matching images and captions get similar embeddings. Vision-language models that answer questions about pictures typically feed ViT patch embeddings into a language model. The Segment Anything Model (Kirillov and colleagues, 2023) uses a ViT pre-trained with MAE as its image encoder and was trained on 11 million images with over a billion masks. Diffusion models for image generation have also moved from convolutional U-Nets to Transformer backbones operating on patches.
Closing Observation
ViT is a useful case for a general question in machine learning: how much knowledge should the designer build into a model, and how much should be left for the data to teach it. Convolutions build in knowledge about images, and that helps when data is scarce. ViT removes most of it, does worse with little data, and does better once there is enough. Neither choice is right in general. The better one depends on how much data and compute are available, and for image models that amount grew very quickly in the years after 2020.
By: Max Johnson B.
Deja una respuesta