Contextual Language Models

Polysemous Words

words with different meanings

Naive Polysemy Handling

● Perform word sense disambiguation on the entire corpus

● Create embeddings of the disambiguated words

Issues with Polysemy Naive Approach

● Error propagation (word sense disambiguation is hard)

● Requires labeled training data (unlike the embedding methods)

● The vocabulary size explodes

● We also need word sense information when using the embeddings

Machine Translation History

Sequence to Sequence Learning

RNN

Sequence to sequence learning with an RNN:

● At each time step t, the neural network A looks at some input xt

and outputs a value ht.

● Information from step t is passed to step t+1 in a loop.

● The number of iterations of the loop depend on the length of the sequence

● The loop notation can be unrolled into individual steps.

● The more iterations, the more likely the model will forget previously seen information

RNN Architecture

Long Short-Term Memory Networks (LSTM)

Intermediate Context Vectors

Sequence to sequence learning for translation

● Words in the input sentence are encoded as word embeddings

● Word embeddings are fed sequentially into a RNN / LSTM model

● In each step, the input is used to update an internal context vector

● A memory mechanism is used to keep track of relevant prior information

● The combination criteria for input, memory, and context vector are learned

● To generate the translation, the steps are “reversed” by a decoder

Sentence representations from translations

● The context vector that is passed from encoder to decoder lies in a latent space

● The context vector contains the semantic information of the input sentence

● A translation LSTM without a decoder is a sentence embedder

Embedding words Machine Translation

The context vector and memory in the final generation step retain the semantic information for the options of the next word

Cons of Memory

● Forgetting is still a problem, even for LSTMs

● We cannot “remember” the future...

● sentence ordering is flexible in many languages

Downsides of LSTMs / RNNs:

● Sentences are read sequentially

○ Good for dealing with varying sentence lengths

○ Bad for parallelization (for example in a GPU)

● Processing a sentence word by word without the ability of backtracking is different from natural human reading behavior

Attention

● We process all words of a sentence concurrently

(instead of sequentially)

● Instead of learning to remember, we learn how to pay attention to the parts of a sentence that are important to a given word

● This is called self-attention since we focus on other words in the same input sentence.

● Grounded in syntactic / semantic relations in natural language

Self-Attention

● All words in a sentence are encoded as vectors

● All words are fed into the model in parallel, each word in its own “lane”.

● For each word, the model learns weights that are used to determine how important other words in the sentence are.

Transformer Model

● Originally designed for machine translation

● Input: sentence in language A

● Output: sentence in language B

● Encoder:

○ Processes all words of the input sentence in parallel

○ Uses attention to determine word relations

○ Outputs one vector per input word

● Decoder:

○ Sequentially processes vectors created by encoder (similar to LSTM decoder) but has access to all vectors.

○ Predicts output words one at a time

BERT Bidirectional Encoder Representations from Transformers

BERT is a contextual language model, based on transformers:

● It uses only encoders (12 transformer encoder layers)

● Bidirectional links connecting the “lanes” between layers

● Trained on ≈ 3.3 Billion words (Wikipedia & Google Books corpus)

● Created with transfer learning in mind:

○ Trained on 2 simple unsupervised tasks (for which lots of data is available)

○ Can be fine-tuned for numerous complex tasks (for which few labeled data are available)

● Due to the transformer’s attention mechanism, We get contextualization “for free”

BERT Training Tasks

● Masked language modeling (MLM)

○ This is the cloze task we already know:Random words in the input are replaced by a [MASK] token and the model is trained to predict them

○ Unlike word2vec, BERT uses the signal from all other words in the context simultaneously for this prediction (using attention to determine weights)

● Next sentence prediction (NSP)

○ Given a sentence, the model has to predict the next sentence.

○ Modeled as a classification problem: Given sentences A and B, does B follow A?

BERT Input Rep

BERT accepts input in a fixed-width window of at most 512 tokens:

● Input starts with a special [CLS] token

● Input may contain two separate sentences, which are separated by a [SEP] token.

● The second sentence is optional

BERT Training

Contextualized Embeddings

feature vectors from any layer can be used as embeddings, depending on the task at hand!

BERT produces feature vectors for each token in each of its 12 transformer layers.

Due to the attention mechanism, they capture different aspects of the input tokens.

BERT Tasks via Transfer Learning

Word piece Tokenization

Most language have millions of words

○ The English Wikipedia has ≈ 600,000 headwords alone...

○ ...but 1.3 million definitions

● Most words are very rare (recall Zipf’s law)

● Using a large vocabulary increases the size of the model

In practice, contextual language models use word piece tokenization:

● Think of it like morphological decomposition…

● ...but driven by statistics, not linguistic insight!

Keep this in mind when extracting word embeddings from BERT

GPT-2 (only decoder):

● Training objective: predict next word in the sentence

● Trained on "Common Crawl" (a massive dataset of web pages)and Book Corpus, a dataset of over 7,000 unpublished fiction books from various genres

● Learns only from context before

● Used for natural language generation tasks

● Generates contextual word and sentence representations

Transfer Learning

The idea:

1. Take a pre-trained model …

2. … and adapt its weights for a …

a. domain (e.g., biology, law)

b. or task (e.g., sentiment classification)

We do NOT train the model from scratch!

We learn on top of the existing weights from the pre-trained model (e.g., BERT)

Domain-adaptive pre-training

an unsupervised fine-tuning approach based on a masked language modeling task on text from a specific target domain

Task-specific fine-tuning

deals with adaptingan LM to a particular output label distribution

○ We need labeled data! But much lessthan when training the model from scratch!