Multi-Modal Systems: Cross-Attention, CLIP, and BLIP for Product Intelligence

Comparison of Self-Attention and Cross-Attention

• Embeddings and Multi-modal Systems: Embeddings serve as the fundamental core concept for discussing and building multi-modal systems which integrate different types of data (e.g., text and images).

• Self-Attention:

  • Definition: An algorithm that allows a sentence or sequence to understand the semantic relations within itself.

  • Example: In the sentence "Ramit eats pizza, and it is good," self-attention determines the attention of each word relative to every other word in that same sentence (e.g., connecting "it" to "pizza").

• Cross-Attention:

  • Definition: A mechanism used to find the context of a word or data point relative to a different structure or vector space. It involves an "intermixing" of information.

  • Mechanism: Use the same internal logic as self-attention—$Query$, $Key$, and $Value$ —but the vectors involved come from two different spaces rather than a single space.

  • Analogy: It acts as a "bridge" or a "translator." If an English-speaking person wants to talk to a French-speaking person, they need a translator; cross-attention is that translator for different data types.

• Use Cases of Cross-Attention:

  • Language Conversion: Converting Hindi to English, English to French, or French to German. It helps understand relations between words in different languages.

  • Multi-modal Mapping: Connecting word embeddings to image embeddings. This is essential for Generative AI (e.g., ChatGPT or Gemini) to provide descriptions for images or generate images from text.

  • Code Generation: Mapping English language space to coding syntaxes, which are often not purely English.

• Syntax vs. Semantics:

  • Syntax: The specific rules followed to perform a task.

  • Semantics: The logical or emotional meaning of something.

CLIP (Contrastive Language-Image Pre-training)

• Origin: Developed by OpenAI, CLIP represented a revolution in connecting images to text at an initial level.

• Functionality: It is a categorization model. Given an image and a set of text options, it identifies which option best describes the image based on probability.

• Training Process:

  • The model is provided with an image (e.g., a dog) and a list of options (e.g., dog, cat, horse, t-shirt).

  • An Image Encoder converts the image into an image embedding.

  • A Text Encoder converts the text options into word embeddings.

  • The model uses internal parameters (denoted as $\lambda$) to project both subsets of embeddings into a common embedding space.

• Closeness and Similarity:

  • The model calculates the distance between the image point and the text points in the common vector space.

  • This distance is typically measured using Cosine Similarity or Euclidean Distance.

  • Points that are closer in space receive a higher probability percentage.

• Optimization:

  • Initially, an untrained model might misclassify (e.g., giving "cat" an $80\%$ probability for a dog image).

  • The model is "punished" during training, causing it to update its parameters ($\lambda$) until the correct text point (dog) moves closer to the image point.

BLIP (Bootstrapping Language-Image Pre-training)

• Origin: A model developed by Salesforce.

• Functionality: It is a generative model primarily used for captioning. Unlike CLIP, which chooses from options, BLIP generates new text from scratch.

• Generative Process:

  • An image is converted into an image embedding.

  • Cross-attention is applied to map the image embedding vector to a text embedding space.

  • This result, known as a BLIP embedding, is fed into a Feed-Forward Neural Network.

  • The network functions as a language model, predicting the next word in a sequence to generate a caption (e.g., "dog with a long tail").

• Comparison with CLIP:

  • CLIP is for categorization/classification.

  • BLIP is for generation.

  • BLIP directly uses cross-attention to provide results, whereas in CLIP, cross-attention-like components are used internally to bring embeddings to a common space for distance calculations.

Practical Implementation and E-commerce Workflows

• E-commerce Metadata Generation: To generate titles, categories, and descriptions for products (like a t-shirt), three methods are common:

  1. BLIP to LLM: Use BLIP to get a short caption, then pass that caption to an LLM/SLM (Small Language Model) with a system prompt to flesh out a full description and tags.

  2. CLIP + BLIP: Use CLIP to determine the category and BLIP for the description.

  3. Multi-Classifier Approach: Train multiple specialized ML models (Logistic Regression, Decision Trees, or Neural Networks) to identify separate attributes like color, category, and texture, then feed all attributes to an LLM.

• Similarity Search in Retail:

  • Platforms like Amazon and Flipkart recommend "similar products" by mapping all inventory into a vector space.

  • When a user selects a product, the system performs a Nearest Neighbors search in the vector space.

  • The algorithm retrieves the top $k$ products (where $k=5$, for example) with the highest similarity scores.

• Technical Tools for Implementation:

  • Kaggle: A platform for AI development offering free GPU access (e.g., T4 or P100) for approximately $30$ hours per week. It allows direct access to large datasets without manual downloading.

  • FAISS (Facebook AI Similarity Search): A vector database used to store large volumes of embeddings and perform high-speed similarity searches.

  • Qwen 2.5 3B Instruct: An example of a medium-sized language model used for processing text prompts.

  • Parameters:

    • Temperature: Set to an exact value (e.g., $0.1$) to ensure precision and reduce randomness.

    • do_sample = False: Used to ensure reproducibility, so the model gives the same answer for the same query every time.

Advanced Product Intelligence Assignment Details

• Task 1: Complimentary Recommendations: Move beyond visual similarity to recommend products purchased together (e.g., running shoes with sports socks or water bottles). This requires filtering similarity searches by category.

• Task 2: Catalog Creation: Filtering a massive dataset (e.g., $45,000$ products) down to a representative subset (e.g., $500$ products) while maintaining variety. This involves analyzing the distribution of points across the entire vector space.

• Numerical Specifics Captured:

  • Vector similarity score examples: $0.81$ (high similarity).

  • Max new tokens for metadata generation: $150$.

  • Qwen model parameter size: $3\text{ billion}$.

  • Kaggle GPU free tier: $30\text{ hours/week}$.

  • Dataset size mentioned: $45,000\text{ products}$.

Questions & Discussion

• Question: What is the difference between an SLM and an LLM?

• Response: The primary difference is the number of parameters. Small Language Models (SLM) have fewer parameters compared to Large Language Models (LLM).

• Question: Can self-attention alone do translation?

• Response: No, self-attention alone cannot do translation; you need cross-attention to bridge the two different language spaces.

• Question: What if you upload an image of a t-shirt with a dog on it in CLIP?

• Response: The probabilities would likely split (e.g., $30\%$ for t-shirt and $30\%$ for dog), with other categories receiving lower scores.

• Question: Why use Kaggle instead of local environments?

• Response: Multi-modal models are extremely heavy. Kaggle provides necessary GPU accelerators (T4/P100) and pre-hosted datasets to avoid heavy local processing and manual data management.