LLM and Transformers Detailed Notes

Large Language Models (LLMs) are a recent breakthrough in AI, enabling more intelligent systems with a richer understanding of language.
LLMs vastly extend the capabilities of systems to process text. Two major categories are generative language models (like GPT2 and GPT3) and representation language models (like BERT).
Trained on large amounts of raw text in a self-supervised fashion, where the objective is automatically computed from the inputs, eliminating the need for human labeling.

Masked Language Models (e.g., BERT):
- Randomly mask words in the input sequence.
- The model predicts the masked words based on the context.
- Utilizes a Transformer encoder.
- Example: BERT base has 12 encoders with 768 embedding vector size and 12 attention heads
- Bidirectional attention.
Causal Language Models (e.g., GPT):
- Predict the next word in a sequence.
- Utilizes a Transformer decoder.
- Autoregressive models.

The general strategy to achieve better performance is by increasing the models’ sizes and the amount of data they are pretrained on.
Training large models requires a large amount of data, which is costly regarding time, compute resources, and environmental impact.

The original Transformer model is an encoder-decoder architecture suitable for translation tasks.
BERT is an encoder-only architecture focusing on representing language.
Encoder blocks consist of self-attention followed by feedforward neural networks.
The input contains a classification token ([CLS]) used as the representation for the entire input.
Trained on large datasets like Wikipedia to understand the semantic and contextual nature of text.
Fine-tuned for specific tasks like text classification, clustering tasks, and semantic search.
Examples:
- BERT base: 12 encoders, 768 embedding vector size, 12 attention heads
- BERT Large: 24 encoders, 1024 embedding vector size, 16 attention heads
- ALBERT, DistilBERT, ELECTRA, RoBERTa

Decoder models use only the decoder of a Transformer model.
At each stage, attention layers can only access words positioned before it in the sentence.
These models are often called auto-regressive models

GPT is a decoder-only architecture targeting generative tasks, uses stacked decoder blocks.
Trained on a corpus of books and web pages, GPT-1 consisted of 117 million parameters which capture the model’s understanding of language.
Generative LLMs take in text and attempt to autocomplete it; fine-tuning allows training as chatbots to answer questions.
Completion models rely on context length or context window, representing the maximum number of tokens the model can process.
Examples: CTRL, GPT, GPT-2, Transformer XL

Encoder-decoder models (sequence-to-sequence models) use both parts of the Transformer architecture.
Encoder attention layers access all words in the initial sentence, while decoder attention layers can only access words positioned before a given word in the input.
Best suited for tasks like summarization, translation, or generative question answering.
Examples: BART, mBART, Marian, T5

Tokenization: Encode text into numeric representations.
Token Embeddings: Put words with similar meanings close in vector space.
Embedding Functions: Use a pre-trained model.
Pre-Trained Transformer Model: Billions of parameters.
Decoding: Predict the next word.
Uses books, Wikipedia, crawled data from the Internet, and custom curated datasets.