Notes on Computational Linguistics and Language Models

Computational Linguistics

• Definition: Field of study focused on the statistical description of natural languages.
• Key Components: Often involves the creation and evaluation of language models.

Language Models - Motivation

• Task: Create a program to add diacritics to emails where they are omitted.
• Solution Approach:
  • Use a table of pairs (word, wordwithoutdiacritics).
  • Condition replacements based on surrounding words:
  • Previous word context (e.g., "she had" vs. "she had ?")
  • Following word context (e.g., "Peter went" vs. "Peter went ?")
  • Working with a larger context is ideal, but in practice often limited to N-last words.

Applications of Language Models

• Noisy Channel: Framework to understand the role of noise in various applications:
  • Diacritic Addition: Process of restoring diacritics in text.
  • Speech Recognition: Translation of spoken language into text via vocal cords and microphones.
  • Optical Character Recognition (OCR): Conversion of scanned text images to machine-encoded text, facing issues with added noise.
  • Machine Translation: Translating text from one language to another while retaining semantic meaning.

Bayesian Solutions - Speech Recognition Example

• Example Task: Dictating words to a computer.
  • Utilizes a dictionary of 600,000 words.
  • For each spoken word, the system calculates the probability of recognition.
• Mathematical Representation:
  • Mark the digitized signal as D, and the set of classes (words) as Tr.
  • Probability of recognizing a word can be expressed as: $P(Tr | D)$.

Bayes' Rule in Recognition

• Components:
  • Prior Probability: $P(Tr)$ - likelihood of a word before observing D.
  • Likelihood: $P(D | Tr)$ - probability that class Tr generated D.
  • Marginal: $P(D)$ is constant across classes, allowing simplifications.

Importance of a Prior Probability

• Example Scenario: Disambiguating words that sound similar, e.g., "were" vs "were".
• Models like unigram and bigram adjust the recognition result based on language patterns to improve accuracy, where without a model, success rates drop significantly.

Bayesian Solutions - OCR Example

• Character Recognition:
  • Similar to speech, where images are divided into parts corresponding to characters.
  • Recognizes characters by determining the probability $P(Tr | D)$ for each character.

Bayesian Solutions - Language Detection Example

• Task: Identify if text belongs to English or Slovak without diacritics.
• Utilize character language models similar to the previous examples for the language classification.

Derivation of Bayes Rule

• Formula:
  • $P(A ext{ and } B) = P(B ext{ and } A) = P(A|B) imes P(B) = P(B|A) imes P(A)$
  • Important for natural language processing tasks.
• Deciding Factor:
  • A{best} = ext{argmax}A ig{ P(B|A) imes P(A) ig}

The Ideal Language Model

• A model should predict a sequence of words $W = { w<em>1, w</em>2, w<em>3,… w</em>n }$ and calculate $p(W)$ effectively using chain rules, although impractical for large sets due to excessive parameters.

n-gram Language Models

• Types of models:
  • 0-gram: Uniform distribution over words (1 parameter).
  • 1-gram (Unigram): Individual word probabilities.
  • 2-gram (Bigram): Conditional probabilities based on previous word.
  • 3-gram (Trigram): Conditional probabilities based on two previous words.
• Model Size: Choose the size n based on practical considerations:
  • Ideally as large as possible to approximate the ideal model, with limits based on language and task specificity.

Language Model Parameter Estimation

• Parameters $p(w | h)$ are estimated from a corpus, and preprocessing is crucial to ensure data quality:
  • Normalize, remove typos, foreign words, and handle special cases (e.g., hyphens).

Maximum Likelihood Estimate Example

• Example for Trigrams:
  • $p(w<em>i | w</em>{i-1}, w<em>{i-2}) = \frac{c(w</em>{i-2}, w<em>{i-1}, w</em>i)}{c(w<em>{i-2}, w</em>{i-1})}$

Language Model Calculation Example

• Example Sentence: " today or maybe today I'll see you at the cinema"
• Uniform Model: All words have equal probabilities.
• Unigram Model: Specific probabilities derived from frequency counts.
• Bigram and Trigram Models further refine estimates based on preceding words or pairs.

Sign Language Models

• Similar strategies can be employed to analyze sign language, focusing on characters instead of words, often resulting in effective language comparison and identification using n-gram models appropriate for task specifics.