* popular in machine translation
* 2017: google brain introduced the transformer language model
* model is based solely on attention mechanisms, w/o recurrence
* can outperform recurrent and convolutional models
* appliesn a self-attention mechanism → directly picks up rship bw all words in a sentence and choose the best meaning
* neural networks typically contain an encoder reading the input sentence sentence and generating a representation of it. a decoder then generates the output sentence word by word while consulting representation generated by encoder
* Transformer starts by generating initial representations, or embedding for each word. these r represented by unfilled circles. then, using self-attention, it aggregates info from all the other words → generating new representation per word informed by the entire context → this step is repeated multiple times for all words → suggests fully generating new represenations
* decoder works similarly, but generates one word at a time . attends to both previously generated words and final representations generated by encoder
* design of Transformers allows dvpt of pretrained models (google;s BERT and OpenAI’s GPT), that can be fine-tuned w smaller datasets for more specific tasks
* fine tuning of general-purpose models → purpose-specific models = TRANSFER LEARNING
* training of BERT involves MASKING some words randomly n asking model to predict masked word based only on context