{"id":113,"date":"2026-10-09T14:22:40","date_gmt":"2026-10-09T14:22:40","guid":{"rendered":"https:\/\/evoxia.online\/?p=113"},"modified":"2026-10-09T14:22:40","modified_gmt":"2026-10-09T14:22:40","slug":"clip-and-contrastive-multimodal-models-teaching-images-and-text-to-share-one-space","status":"publish","type":"post","link":"https:\/\/evoxia.online\/?p=113","title":{"rendered":"CLIP and Contrastive Multimodal Models: Teaching Images and Text to Share One Space"},"content":{"rendered":"\n

A Thousand Labels Is a Small Vocabulary<\/strong><\/p>\n\n\n\n

For most of the history of computer vision, a classifier knew exactly the categories it was trained on and nothing else. A model trained on ImageNet’s 1,000 classes could tell a Siberian husky from an Alaskan malamute, but if you asked it to find \u00aba dog wearing a raincoat\u00bb it had no way to even represent the request. Adding a new category meant collecting new labeled images and retraining.<\/p>\n\n\n\n

The internet, meanwhile, is full of images that already come with text: captions, alt attributes, titles, surrounding paragraphs. That text is noisy and inconsistent, but it describes far more than 1,000 concepts. Contrastive multimodal models are built on a simple bet: if a network learns which caption goes with which image across hundreds of millions of examples, it will learn a visual vocabulary as wide as the language people use to describe pictures.<\/p>\n\n\n\n

Two Encoders and a Matching Game<\/strong><\/p>\n\n\n\n

CLIP (Contrastive Language-Image Pre-training) was published by Alec Radford, Jong Wook Kim and colleagues at OpenAI in 2021 and presented at ICML that year. The architecture has two separate parts. An image encoder (a ResNet or a Vision Transformer) turns a picture into a vector. A text encoder (a Transformer) turns a caption into a vector of the same size. Both vectors are normalized, so comparing them is just a cosine similarity.<\/p>\n\n\n\n

Training uses a batch of N image-caption pairs. The model computes the similarity between every image and every caption, which gives an N by N matrix. The N correct pairs sit on the diagonal; everything else is a mismatch. The loss is a cross-entropy in both directions: each image has to pick its own caption out of the N candidates, and each caption has to pick its own image. A learned temperature parameter scales the similarities before the softmax.<\/p>\n\n\n\n

The batch is where the negatives come from, so batch size matters a lot. CLIP used batches of 32,768 pairs, which means every caption competes against more than 32,000 wrong images at each step. The training set, called WebImageText, held 400 million pairs collected from public sources on the internet.<\/p>\n\n\n\n

Classification Written as a Sentence<\/strong><\/p>\n\n\n\n

The payoff is zero-shot classification. To classify an image into a set of categories CLIP has never been trained on as labels, you write each category as a sentence (\u00aba photo of a cat\u00bb, \u00aba photo of a truck\u00bb), encode all the sentences, and pick the one closest to the image embedding. The class list is just text, so you can change it at inference time.<\/p>\n\n\n\n

The paper reported that its best model, a ViT-L\/14 at 336 pixel resolution, reached about 76 percent top-1 accuracy on ImageNet this way, roughly the same as the original ResNet-50 trained on ImageNet’s 1.28 million labeled images. CLIP saw none of those labels. The wording of the prompt turned out to matter: \u00aba photo of a {label}\u00bb worked better than the bare class name, and averaging over dozens of templates helped further. People started calling this prompt engineering for vision.<\/p>\n\n\n\n

A second result got less attention at the time but may matter more. On variants of ImageNet with sketches, renditions and adversarially filtered photos, a standard ImageNet model loses much of its accuracy. Zero-shot CLIP lost far less. The authors argued that a supervised model learns shortcuts specific to one dataset, while a model trained on a broad distribution of web images and text has fewer of them to exploit.<\/p>\n\n\n\n

Scale Over Cleanliness<\/strong><\/p>\n\n\n\n

Within weeks of CLIP, Google described ALIGN (Chao Jia and colleagues, also ICML 2021), which used the same dual-encoder idea with an EfficientNet image encoder and a BERT text encoder. Its dataset was 1.8 billion image and alt-text pairs with only minimal filtering. The message of ALIGN was that you can skip most of the expensive cleaning if you have enough data, because the contrastive objective averages out a lot of the noise.<\/p>\n\n\n\n

Neither OpenAI’s nor Google’s dataset was released. The LAION project filled that gap with LAION-5B (Schuhmann et al., NeurIPS 2022 Datasets and Benchmarks track), 5.85 billion image-text pairs filtered by CLIP similarity itself. The OpenCLIP project trained open models on it, which made it possible to study scaling laws for CLIP-style training in public. Later work such as DataComp (Gadre et al., 2023) turned the question around: hold the model and training recipe fixed, and compete only on how you filter the data. Filtering choices alone moved results by large margins.<\/p>\n\n\n\n

Replacing the Softmax With a Sigmoid<\/strong><\/p>\n\n\n\n

The softmax in CLIP’s loss has a practical cost. Every image’s probability depends on every caption in the batch, so devices have to share all embeddings to compute it, and the memory grows with the square of the batch. In 2023 Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov and Lucas Beyer at Google proposed SigLIP, which treats every image-caption pair as an independent binary question: do these two belong together, yes or no? Each cell of the similarity matrix gets its own sigmoid loss.<\/p>\n\n\n\n

That change removes the global normalization. SigLIP trained well with smaller batches than CLIP needed, and it scaled to larger ones too. In one setup that kept a pre-trained image encoder frozen and trained only the text side, the authors reached 84.5 percent zero-shot ImageNet accuracy using four TPUv4 chips for two days. SigLIP encoders have since become a common choice as the vision component of open multimodal language models.<\/p>\n\n\n\n

An Embedding Space Other Systems Reuse<\/strong><\/p>\n\n\n\n

The shared embedding space ended up being more useful than zero-shot classification. Image search is the direct application: encode a text query, find the nearest image vectors, no tags required. Content moderation and dataset filtering use the same trick, and LAION itself was built by keeping only pairs whose CLIP similarity passed a threshold.<\/p>\n\n\n\n

Generative models borrowed the text side. Stable Diffusion’s first versions conditioned image generation on the output of CLIP’s ViT-L\/14 text encoder, and OpenAI’s DALL-E 2 generated images from CLIP image embeddings. Going the other way, LLaVA (Liu et al., 2023) connected a frozen CLIP image encoder to a language model through a small projection layer and trained the result to answer questions about images. Meta’s ImageBind (Girdhar et al., CVPR 2023) pushed the idea further, aligning audio, depth, thermal and motion data to the same space by pairing each one with images, so that modalities never seen together during training could still be compared.<\/p>\n\n\n\n

Case: An Apple Labeled \u00abiPod\u00bb<\/strong><\/p>\n\n\n\n

In March 2021, OpenAI researchers studying CLIP’s internal units found what they called multimodal neurons: single units that responded to a concept whether it appeared as a photo, a drawing or written text. One responded to spiders, to the word \u00abspider\u00bb and to Spider-Man. The same property produced a strange weakness. When they stuck a handwritten paper note saying \u00abiPod\u00bb on a Granny Smith apple, CLIP classified the photo as an iPod. They called these typographic attacks. Unlike most adversarial examples, they need no gradients and no computer, only a pen.<\/p>\n\n\n\n

The failure follows from the training signal. Web captions often repeat the text that appears inside an image, so reading the text is a reliable shortcut for matching captions. CLIP learned to read because reading helped, and it has no separate notion of \u00abthe object\u00bb versus \u00abwords written near the object.\u00bb<\/p>\n\n\n\n

What a Bag of Words Cannot Tell Apart<\/strong><\/p>\n\n\n\n

Other limits show up in careful benchmarks. Yuksekgonul and colleagues (ICLR 2023) tested whether vision-language models understand word order and relations, for example whether they can distinguish \u00abthe horse is eating the grass\u00bb from \u00abthe grass is eating the horse.\u00bb CLIP-style models often could not. Their explanation was that the contrastive task rarely requires order: in a batch of random web pairs, getting the nouns right is almost always enough to find the matching caption, so the model has little reason to learn more. CLIP also struggled with counting objects and, in the original paper, scored poorly on handwritten digits (MNIST) despite its broad training.<\/p>\n\n\n\n

Bias came along with the web data. An audit by Agarwal and colleagues in 2021, using images from the FairFace dataset with extra labels added, found that photos of Black people were classified into non-human categories such as \u00abgorilla\u00bb at a rate of about 14 percent, against under 8 percent for every other group. Changing the label set changed the outcome, which is the other side of zero-shot flexibility: whoever writes the class list also shapes the errors.<\/p>\n\n\n\n

Closing Thought<\/strong><\/p>\n\n\n\n

CLIP’s objective never asks the model to understand an image. It asks only that the right caption score higher than the other captions in the batch. Everything the model learns, from recognizing dog breeds to reading text on an apple to ignoring word order, is whatever happened to be useful for winning that comparison against random internet captions. That makes the choice of negatives, and of data, the real specification of what these models know. A harder batch teaches a finer distinction; a lazier one lets the shortcut stand.<\/p>\n\n\n\n

By: Max Johnson B.<\/p>\n","protected":false},"excerpt":{"rendered":"

A Thousand Labels Is a Small Vocabulary For most of the history of computer vision, a classifier knew exactly the categories it was trained on and nothing else. A model trained on ImageNet’s 1,000 classes could tell a Siberian husky from an Alaskan malamute, but if you asked it to find \u00aba dog wearing a […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-113","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/posts\/113","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=113"}],"version-history":[{"count":0,"href":"https:\/\/evoxia.online\/index.php?rest_route=\/wp\/v2\/posts\/113\/revisions"}],"wp:attachment":[{"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=113"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=113"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/evoxia.online\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=113"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}