Lecture 30 Vision Language Models
Multimodality in Communication Theory
Multimodality refers to transmitting information about one medium through multiple modes. Think of it as using different ways to communicate the same thing.
In lectures, the medium is the content being taught. The modes are the different methods the teacher uses to get the content across, such as slides, their speech, and their body language. This helps students understand the material better because they're getting the information in multiple formats.
Multimodal Systems in AI
These are AI systems that combine multiple types of input data to make predictions or take actions. Instead of just looking at text, they might also look at images or listen to audio.
Input modes include:
Sequential discrete data (text): This is written language, which is sequential because words come in a specific order, and discrete because each word is a distinct unit.
Sequential continuous data (audio): This is sound, like speech or music, which is sequential because it unfolds over time, and continuous because it's a wave that flows without breaks.
Spatial data (images): This is visual information, like a photograph, which is spatial because it exists in two dimensions.
Sequential and spatial data (videos): This combines both images and audio, so it has both spatial and sequential characteristics.
Any valid form of representing information can be a mode. If it can be used to convey data, it can be a mode.
Humans learn best when information is transmitted in a multimodal fashion (e.g., technical books with text and diagrams). Think about textbooks. They're not just walls of text; they have pictures, charts, and graphs to help you understand the concepts.
Multimodal systems that accept multiple forms of information may perform better. The more information an AI system has, the better it can understand and respond to the world.
Vision and Language
The two most common modalities in multimodal systems are vision and language. These are popular because they mimic how humans perceive and interact with the world.
These systems accept visual and language input and can produce visual and language output. For example, an AI could look at a picture and describe it in words, or read a description and generate a matching image.
Universal Nature of Multimodal Systems
The idea of unifying different modalities into one system is universal and elegant. It suggests that all types of information can be processed in a similar way.
Humans process information using abstract thoughts, regardless of the input mode. When you see a picture or read a word, your brain turns it into an abstract concept.
Multimodal systems use "representations" as abstract thoughts. These are ways of turning data into a format that the AI can understand, no matter the original form.
Representations are vectors in embedding space, with similar concepts clustered together. Imagine a map where ideas that are alike are located close to each other.
Images are split into patches processed by a vision transformer, with each patch's embedding vector representing its meaning. This is like breaking an image into pieces and understanding each piece individually.
Embedding vectors or representations are synonymous and represent abstract thoughts for computers.
Different dimensions of the embedding vector can represent different things (e.g., number of animals or color of fur). Just like a coordinate system, each dimension can describe a different aspect of the data.
Language and images can be turned into representations. This allows the AI to compare and combine information from different sources.
Almost all data in modern deep learning are turned into representations for processing. It's a universal way to handle information.
Everything becomes an array of numbers that can be converted into different types of output. This is how computers can work with images, text, and audio in a unified way.
Auto-regressive models generate language from vectors. These models can produce text that sounds natural by predicting the next word in a sequence.
Diffusion models generate images from a concise representation. These models can create detailed images from a simple set of instructions.
Machines process information in an abstract form and attach modules to process or produce different forms of information. Think of it as a modular system where different parts can be plugged in to handle different tasks.
Contrastive Language Image Pre-training (CLIP)
A fundamental piece of work for multimodal systems, originally studied by OpenAI. CLIP is a big step forward in teaching AI to understand both images and text together.
CLIP differs from conventional image classification by matching abstract representations from images and language. Instead of just labeling images, it understands the meaning behind them.
Conventional image classification uses predefined classes with integer labels, without understanding the label's meaning. It's like memorizing vocabulary without knowing what the words mean.
CLIP uses abstract representations to match images and language, enabling dynamic classification without predefined classes. This means it can recognize new things without being specifically trained on them.
CLIP analyzes the meaning of an image and matches it to the numerical representation of language labels. This allows it to connect what it sees with what it knows in a flexible way.
CLIP Architecture
Uses a vision encoder (e.g., ViT or CNN) to encode images as representations. This part of the system turns images into a format that the AI can understand.
Matches the image representation to candidate labels processed by a separate language encoder. It compares the image to different text descriptions to find the best match.
Selects the label with the closest match for prediction. This is how the AI decides what the image is most likely to be.
The key is the alignment of representations from the vision and language encoders, ensuring that the same positions in the array mean the same thing. It's like making sure that both encoders are speaking the same language.
CLIP can predict novel labels without fine-tuning because there are no prescribed language labels. This is a big advantage because it can recognize things it hasn't seen before.
In the old setup, we will need to do fine tuning to update that.
Contrastive Learning
A self-supervised learning approach that derives supervision signals from differentiating positive examples from negative examples. This is a way for the AI to learn without needing someone to tell it what everything is.
Self-supervised learning uses the data itself for supervision instead of explicit labels provided by humans. It's like learning by doing, rather than by being told.
Contrastive learning is a self-supervised method that algorithmically develops labels and training signals from the data. The AI figures out what's similar and what's different on its own.
CLIP collects image-text pairs from online sources (e.g., Wikipedia) and constructs positive and negative pairs. It finds pictures and descriptions that go together, and also ones that don't.
For example, given three pairs (apple, banana, orange), a 3x3 matrix is created where diagonal pairs are positive, and off-diagonal pairs are negative. This is a way of organizing the data so the AI can learn from it.
The self-supervision signal is not always correct, but the incorrect signals are rare and tolerable. Sometimes the AI makes mistakes, but it learns to ignore them.
With these pairs, we can mechanically construct positive and negative training examples to train the model. This is how the AI learns to match images and text.
CLIP Training
Image and text encoders are used to digest images and text into embedding vectors. These encoders turn the raw data into a format the AI can understand.
Representations of images and text are matched, with the diagonal of the matrix representing positive instances. This is how the AI learns to associate images and their descriptions.
At every iteration, a batch of image-text pairs is sampled. The AI looks at many different examples to learn the connections between images and text.
Text and image encoders embed each text and image into a vector, respectively. This turns the data into a mathematical representation.
The goal is to have T and I in the same space, with the encoders speaking the same language. This ensures that the AI can compare images and text directly.
Linear projections are used to align representations from encoders with different dimensionalities (e.g., 512 and 768). This is like translating between different languages.
The entire architecture maximizes cosine similarity between positive pairs and minimizes it between negative pairs. The AI tries to make sure that related images and text are close together, and unrelated ones are far apart.
It is feasible to start with pre-trained text and image encoders or train them from scratch. You can use encoders that already know something, or start from zero.
Pseudo Code
Embed images and text, then apply projection and normalization. This prepares the data for comparison.
Compute the cosine similarities.
Calculate the cosine similarity
Compute the loss using asymmetrical cross-entropy, matching image to text and text to image. This measures how well the AI is doing at matching images and text.
Labels are the, for the first, so this is saying, because this label, a range, it's just an array of zero, one, two, three, four, five, and so on.
Minimize the loss to ensure high similarity along the diagonal and low similarity off the diagonal. The AI tries to reduce its mistakes and improve its accuracy.
CLIP Prediction
During image classification, CLIP predicts the label with vector similarity. It chooses the label that is most similar to the image.
The image and text labels are embedded into vectors, and the image vector is matched to all text vectors. The AI compares the image to all possible labels.
The match with the highest similarity is chosen as CLIP's prediction. This is how the AI decides what the image is most likely to be.
CLIP can classify novel classes, but its performance depends on the relevance of its contrastive learning knowledge to the novel classification task. It can recognize new things, but it's better at things it has seen something similar to before.
Novel classification behavior has limits based on the encoders' knowledge. The AI can only recognize things that are within its understanding.
In cases of high relevance, CLIP can outperform neural networks trained directly on the target dataset, leveraging its large-scale contrastive learning. Because it has seen so much data, it can sometimes do better than systems trained on specific tasks.
Zero Shot Transfer
In zero-shot transfer, there are no additional instance to learn from. This is when the AI can recognize something without any specific training.
It may not really mean it in the most technique, most strict technical sense
Limitations of CLIP
CLIP requires a much larger dataset for training compared to strictly supervised approaches. It needs a lot of data to learn effectively.
The advantage is that it does not require explicit labels, allowing the use of images from online sources. It can learn from unlabeled data, which is easier to get.
Prediction Process Illustration
An image and a set of labels are transformed into complete sentences for the encoder. This prepares the data for the AI to understand.
The image is embedded by the image encoder, and the text is embedded by the text encoder. The similarity between each text and the image is evaluated, and the text with the highest similarity is selected. This is how the AI decides what the image is most likely to be.
Intuitive Understanding
CLIP works because the vector representations of both encoders are aligned by contrastive learning. The AI learns to connect images and text in a meaningful way.
If the training data includes car and trailer instances but not truck instances:
A dimension (alpha) representing four wheels is activated by car instances in both the image and language encoders.
A dimension (beta) representing an open-top container is activated by trailer instances in both encoders.
During transfer learning with a novel class like pickup truck, both encoders activate alpha and beta, even though pickup truck was not in the training data. This is how the AI can recognize new things based on what it already knows.