Contextualized Embeddings Notes

Contextualized Embeddings

Introduction

• Distributional word representations (word embeddings), like Word2Vec, are trained on large corpora.
• Traditional word embeddings assign a single global representation to each word, disregarding context.
• Contextual embeddings enhance global word representations by considering context.
• Each token's representation is a function of the entire input sequence.
• Context-dependent representations capture syntactic and semantic properties in diverse linguistic contexts.

Word Embeddings

• A text corpus is a sequence S of tokens ($t<em>1, t</em>2, … t_N$).
• Distributed representations associate each token $t<em>i$ with a dense feature vector $h</em>i$.
• Traditional methods learn a global word embedding matrix $E ∈ R^{V×D}$, where V is vocabulary size and D is the number of dimensions.
• Each row $e_i$ of E corresponds to the global embedding of word i in vocabulary V.

Contextualized Word Embeddings

• Contextualized Word Embeddings associate each token $t<em>i$ with a representation $h</em>i$ that depends on the entire input sequence S.
• $h<em>i = f(e</em>1, e<em>2, …, e</em>i)$, where $t<em>i$ is initially mapped to its non-contextualized representation $e</em>i$ before aggregation function $f$ is applied.
• There are many model architectures for $f$.

Unsupervised Pre-training via Language Modeling

• Language models learn distributed token embeddings by estimating a probability distribution over token sequences.

• Given a sequence of N tokens $(t<em>1, t</em>2, …, t_N)$, a language model factorizes the probability as:

  $p(t<em>1, t</em>2, …, t<em>N) = \prod</em>{i=1}^{N} p(t<em>i | t</em>1, t<em>2, …, t</em>{i-1})$

• Language modeling uses maximum likelihood estimation (MLE) to estimate model parameters:

  $L(\theta) = \prod<em>{i=1}^{N} f(x</em>i | \theta)$

• A left-to-right language model considers the left context $(t<em>1, t</em>2, …, t<em>{i-1})$ of $t</em>i$ for conditional probability estimation.

• Pre-training contextual embeddings can be unsupervised (e.g., language modeling) or supervised (e.g., machine translation).

ELMO

• ELMO stands for Embeddings from Language Models.
• ELMO generalizes traditional word embeddings to extract context-dependent representations from a bidirectional language model.
• Forward L-layer LSTM and backward L-layer LSTM encode left and right contexts, respectively.

ELMO: Bidirectional Language Model

• Bidirectional Language Model (BiLM) consists of forward and backward language models.

ELMO

• At each layer j, contextualized representations are concatenations of left-to-right and right-to-left representations, obtaining N hidden representations $(h<em>{1,j}, h</em>{2,j}, …, h_{N,j})$ for a sequence of length N.

ELMO: Forward and Backward

• Forward Language Model:
  • Given a sequence of N tokens $(t<em>1, t</em>2, …, t<em>N)$, it computes the probability of the sequence by modeling the probability of token $t</em>k$ given the history $(t<em>1, …, t</em>{k-1})$:
  • At each position k, each LSTM layer outputs a context-dependent representation.
• The top layer LSTM output is used to predict the next token $t_{k+1}$.

ELMO: Log-likelihood

• A biLM combines both a forward and backward LM.
• The formulation jointly maximizes the log likelihood of the forward and backward directions.

ELMO: Task Specific

• To use ELMo in downstream tasks, the (L + 1)-layer representations (including the global word embedding) for each token k are aggregated as:

  $ELMO<em>{k}^{task} = \gamma^{task} \sum</em>{j=0}^{L} s<em>{j}^{task} h</em>{k,j}$

  • $s^{task}$ are layer-wise weights normalized by softmax to linearly combine the (L + 1)-layer representations of token k and $\gamma^{task}$ is a task-specific constant.

  • Given a pre-trained ELMo, incorporate it into a task-specific architecture to improve performance.

  • As supervised models use global word representations $x<em>k$ in their lowest layers, these can be concatenated with corresponding context-dependent representations $ELMO</em>{k}^{task}$.

ELMO: Results

• ELMo shows improvements on tasks like Q&A (SQUAD), textual entailment (SNLI), semantic role labeling (SRL), coreference resolution (Coref), named entity recognition (NER), and sentiment analysis (SST-5).

ELMO: Availability

• Pre-trained ELMo models are available at https://allenai.org/allennlp/software/elmo
• AllenNLP is a deep learning library on top of PyTorch.
• ELMo can process character-level inputs.
• Japanese, Chinese, Korean ELMo models are possible.

BERT: Introduction

• BERT (Bidirectional Encoder Representations from Transformers).
• Released in late 2018 by Google AI researchers.
• BERT dynamically informs word representations based on context.
• Word2Vec has a fixed representation of each word regardless of the context.
• BERT produces different word embeddings for "bank" depending on the sentence (e.g., "robbing a bank" vs. "bank of the river").

BERT: Core Idea

• BERT's core idea is to "fill in the blank" based on context.
• BERT produces labeled data for originally unlabeled data, using a technique called Self-Supervised Learning.

BERT: Masked Language Model

• BERT uses Masked Language Model (MLM), randomly masking words in the sentence and predicting them.
• Masking allows the model to use the full context of the sentence (both left and right surroundings).
• Unlike ELMo, it considers both previous and next tokens simultaneously.
• BERT can also perform next-sentence prediction.
• BERT uses encoders only.

BERT: Main Goals

• BERT uses the encoder part of the transformer architecture to understand semantic and syntactic language information.
• BERT's output consists of embeddings, not predicted next words.
• To leverage these embeddings, add layers on top for tasks like text classification or question answering.
• Fine-tuning is used.

BERT: How it Works

• BERT is based on a Transformer.
• Transformers consist of an encoder to read the text input and a decoder to produce a prediction for the task.
• Self-attention learns the weight of the relationship between each word and other words in the input sequence.

BERT: Input

• The input to the encoder for BERT is a sequence of tokens, converted into vectors and processed in the neural network.
• BERT input is decorated with extra metadata:
  1. Token embeddings: Add a [CLS] token at the beginning of the first sentence and a [SEP] token at the end of each sentence.
  2. Segment embeddings: Add a marker indicating Sentence A or Sentence B to each token, distinguishing sentences.
  3. Positional embeddings: Add a positional embedding to each token to indicate its position in the sentence.

BERT: Input - Special Tokens

• [SEP]: Append at the end of every sentence for two-sentence tasks (e.g., answering questions).
• [CLS]: Prepend to the beginning of every sentence for classification tasks.
• BERT consists of 12 Transformer layers. Each transformer takes in a list of token embeddings and produces the same number of embeddings on the output.
• On the output of the final (12th) transformer, only [CLS] is used by the classifier for the downstream task!

BERT: Input - Constraints

• BERT has two constraints:
  • Sentences must be padded or truncated to a fixed length.
  • Maximum sentence length: 512 tokens.
• Padding:
  • Adds a special padding token to ensure shorter sequences have the same length as the longest sequence in a batch or the maximum length accepted by the model.
• Truncation:
  • Truncates long sequences.

BERT: Masking - Original Idea

• Replace 15% of the words in the input with a [MASK] token.
• Run the entire sequence through the BERT attention-based encoder.
• Predict only the masked words based on the context provided by the non-masked words.
• Problem: the model only tries to predict when the [MASK] token is present in the input.

BERT: Masking - Solution

• Out of the 15% of tokens selected for masking:
  • 80% are replaced with the [MASK] token.
  • 10% are replaced with a random token.
  • 10% are left unchanged.

BERT: Subwords

• BERT uses subwords: a compromise between meaningful embeddings and acceptable memory consumption; falls between a character-level (small vocabulary) and a word-level tokenizer (large vocabulary).
• Frequently used words should not be split, but rare words should be decomposed into meaningful subwords.
• For example, "annoyingly" could be decomposed into "annoying" and "ly."

BERT: Subwords - How to Obtain a Single Representation per Word

• Taking the first subword’s embedding (common approach).
• Mean pooling: Averaging all subword embeddings.
• Max pooling: Taking the maximum value across subwords.
• Attention-based pooling: Learning a weighted combination of subword embeddings.

BERT: Subwords - Attention-Based Pooling

• Attention-based pooling:
  • Step 1: Obtain Subword Embeddings
  • Step 2: Compute Attention Weights
  • Step 3: Compute Weighted Sum of Subword Embeddings

BERT: Available Models

• Four pre-trained versions of BERT exist:
  • BERT-Base, Cased: 12-layers, 768-hidden, 12-attention-heads, 110M parameters, case sensitive
  • BERT-Base, Uncased: 12-layers, 768-hidden, 12-attention-heads, 110M parameters, case insensitive
  • BERT-Large, Cased: 24-layers, 1024-hidden, 16-attention-heads, 340M parameters, case sensitive
  • BERT-Large, Uncased: 24-layers, 1024-hidden, 16-attention-heads, 340M parameters, case insensitive
• BERT is open-source: https://github.com/google-research/bert

BERT: Datasets for Pre-training

• BERT was pre-trained on an unlabeled, plain text corpus (English Wikipedia and the Brown Corpus).
• BERT was trained on 3TB of data.

BERT: Tasks

• Tasks can be performed with BERT:
  • Sequence-to-sequence-based language generation:
    • Question answering
    • Abstract summarization
    • Sentence prediction
    • Conversational response generation
  • Natural language understanding:
    • Polysemy and Coreference resolution
    • Word sense disambiguation
    • Natural language inference
    • Sentiment classification

BERT: Results

• BERT is used on multiple downstream tasks with the use of the GLUE (General Language Understanding Evaluation) Benchmark.

GPT: Introduction

• Generative Pre-trained Transformer (GPT).
• Released by OpenAI.

GPT: Core Idea

• GPT uses decoders only: it is designed to generate text as output!

• Semi-supervised learning (unsupervised pre-training followed by supervised fine-tuning) for NLP tasks has two components:

  • Unsupervised Language Modelling (Pre-training)

    • $\max<em>{ \theta } \sum</em>{i}^{ } log P(t<em>i | t</em>1, …, t_{i-1}; \theta)$

  • Supervised Fine-Tuning where T was the set of tokens in unsupervised data $\begin{Bmatrix} t<em>1,…,t</em>n \end{Bmatrix}$, k was size of context window, $\, \theta$ were the parameters of neural network trained using stochastic gradient descent.

GPT: Decode Only

• Decoders only are used to generate next words, for example when translating text or generating stories.
• The outputs are words with probabilities.

GPT: Versions and Datasets

• Dataset:
  • GPT-1: BooksCorpus with 4.5 GB of text from 7000 unpublished books, 117 million parameters, 512 input tokens
  • GPT-2: BookCorpus + 40 GB of text, 8 million documents (Redditt), 1.5 billion parameters, 1024 input tokens
  • GPT-3 570 GB plaintext, 300 billion tokens of CommonCrawl, WebText, English Wikipedia, and two books corpora, 175 billion parameters, 2048 input tokens

Differences between GPT and BERT

• The two models focus on different scenarios:
  • BERT (encoder): classification (e.g., sentiment), questions and answers, summarization, named entity recognition
  • GPT (decoder): translation, generation (e.g., stories)
• The outputs of the core models are different:
  • BERT (encoder): Embeddings representing words with attention information in a certain context
  • GPT (decoder): Next words with probabilities
• The way they perform training is different:
  • BERT is an encoder-only model trained with the masked language-modelling objective and operates non-autoregressively.
  • It produces all of its outputs in parallel.
  • GPT is a decoder-only model trained using the left-to-right language objective and operates autoregressively.
  • Each output of the network is generated based on previously generated output

LLAMA: Introduction

• Large Language Model Meta AI is a family of large language models released by Meta AI.
• First version in January 2023: four model sizes were trained using 7, 13, 33, and 65 billion parameters.
• July 2023: Meta released several models as Llama 2, using 7, 13, and 70 billion parameters.

LLAMA: Main Characteristics

• LLaMA uses the transformer architecture, but with a few differences compared to GPT:
  • Uses SwiGLU activation function instead of ReLU.
  • Uses rotary positional embeddings instead of absolute positional embedding.
  • Uses root-mean-squared layer-normalization instead of standard layer-normalization.
  • Increases context length from 2K (Llama 1) tokens to 4K (Llama 2) tokens between.

LLAMA: SwGLU Activation Functions

• SwGLU activation functions = non-linearity into the output of a neuron
• Swish is a non-monotonic activation function
• GLU is a linear function gated by a sigmoid activation function
• SwGLU = Swhish + GLU

LLAMA: SwiGLU Advantages

• Smoothness: Smoother than ReLU, leading to better optimization and faster convergence.
• Non-monotonicity: Captures complex non-linear relationships.
• Gating: Selectively activates neurons based on input, reducing overfitting and improving generalization.
• Performance: Outperforms other activation functions like Swish and GLU.

LLAMA: Rotary Positional Embeddings (RoPE)

• Encodes absolute positional information using a rotation matrix.
• The matrix tells the model where each token is in relation to the others (absolute and relative position).
• RoPE can capture relative position dependency in the self-attention formulation.
• RoPE decays inter-token dependency with increasing relative distances and can be used to incorporate relative position encoding in linear self-attention.

LLAMA: Rotary Positional Embeddings (RoPE) - Absolute vs Relative

• Instead of encoding the absolute position, RoPE encodes the distance between words.

LLAMA: ROtary Positional Embeddings (RoPE) - Advantages

• Encodes Relative Positions Instead of Absolute Positions

• Enables Distance-Aware Attention Scores

• Introduces Rotational Invariance

• Reduces Parameter Overhead

LLAMA: ROtary Positional Embeddings (RoPE) - Flexibility and Inter-token dependency

• Flexibility: Unlike traditional position embeddings that are limited to a specific sequence length, RoPE can be expanded to work with any sequence length.

• Inter-token dependency: RoPE help to decay inter-token dependency with increasing relative distances. This means that the influence of each token on the others decreases as the distance between them increases. This is important for long sequences as it helps to reduce computational complexity while still maintaining accurate predictions.

LLAMA: Pre-normalization Using RMSNorm

• LLaMA normalizes the input of each transformer sub-layer, instead of normalizing the output

• LayerNorm, It has two properties:

  • re-centring : It make model insensitive to noises on both inputs and weights.

  • re-scaling: It keeps the output representations intact when both inputs and weights are randomly scaled.