Computational Linguistics: Overview

  • Computational Linguistics (CL) is the branch of linguistics where techniques from computer science are applied to analysis and synthesis of language and speech.
  • CL focuses on computational aspects of the human language faculty and requires knowledge of both linguistics and computer science.
  • CL sits at the intersection of cognitive sciences and artificial intelligence (AI), aiming at computational models of human cognition.
  • CL has applied and theoretical components:
    • Theoretical CL tackles issues in theoretical linguistics and cognitive science, dealing with formal theories about linguistic knowledge needed to generate and understand language.
    • Computers now manage the complexity of these theories via formal models implemented as computer programs.
  • Findings from cognitive psychology, especially psycholinguistics, inform computational modelling of linguistic competence and lead to subdisciplines like computational psycholinguistics.
  • The discipline attracts people from humanities, natural and behavioural sciences, and engineering; teaching occurs in computer science, linguistics, or cognitive science programs.
  • There is a growing number of programs devoted solely to computational linguistics.

Key Concepts and Foundations

  • Interdisciplinarity: CL combines linguistics, computer science, psychology, mathematics, and cognitive science.
  • Formal modelling vs. empirical verification: CL develops formal models of language, implements them, and empirically verifies predictions via practicable programs.
  • Cognitive grounding: CL aligns with psycholinguistics to examine cognitive processes underlying language use.
  • Uniform grammar formalisms vs. diversity of languages: Research questions include whether a single grammar formalism can model many languages efficiently, and whether such formalisms are cognitively adequate.
  • Empirical bias: CL emphasizes empirical validation of linguistic models through data and experiments.

What CL Produces: Applications and Outputs

  • Practical outputs span translation, information retrieval, summarisation, e-science, and more:
    • Machine Translation (MT) and automatic translation between natural languages
    • Information retrieval and automatic summarisation from large language-based databases
    • Location of information in heterogeneous sources (Internet, databases, corporate portals)
    • Automatic question-answering over large databases or the Web
    • Language-learning and correction software (vocabulary trainers, spelling/grammar correction)
    • Linguistic interaction with AI systems in robotics, virtual worlds, or computer-aided medical care
  • Research issues arise from both universal, cross-language properties and language-specific divergences; MT and NLP systems must handle these differences while reflecting commonalities.
  • The field emphasizes empirical verification of linguistic models via real-world language data and software implementations.

Machine Translation (MT): Scope, History, and Challenges

  • MT is the formal domain of converting text from one natural language to another using computers; it may involve human-assisted processes (MAHT/HAMT) or be fully automated (MT).
  • Early MT aimed for word-by-word translation via bilingual dictionaries; this failed due to lack of deep understanding of meaning.
  • A grander goal emerged: meaning-based translation via an intermediate representation, then rendering in the target language, ideally with minimal post-editing.
  • Modern MT reality:
    • Large-scale systems exist, ranging from research prototypes to commercial products.
    • Translation quality is generally lower than high-quality human translation, and post-editing is common.
    • Some domains (e.g., weather reports, highly formal documents) yield better results than others.
  • Famous language translation pitfalls and lessons:
    • One early famous mistranslation: The spirit is willing, but the flesh is weak → translated as The vodka is strong, but the meat is rotten.
    • Other examples include header mistranslations (Plans → takhTiiT in Arabic) or phrases like dur Faranca fi al-harb (France’s role in the war vs. France’s turn in the war).
  • MT success is often domain-specific: high-quality automatic translations can be produced in restricted subject domains with appropriate controls on input language and terminology.
  • MT is closely tied to linguistic research and AI; MT serves as a test-bed for theories and techniques in linguistics and AI.
  • Major obstacles in MT are linguistic, not computational: lexical ambiguity, syntactic complexity, cross-language vocabulary, ellipsis, and implicit information needing explicit expression in the target language.
  • Statistical approaches can aid MT (e.g., lexical ambiguity resolution) and MITRE-like groups have shown success using statistical techniques.
  • MT is fundamentally applied research, drawing on linguistics, CS, AI, translation theory, and related fields.

MT Boundaries and Related Concepts

  • Definitions:
    • MT: Computerized systems that produce translations from one natural language to another, possibly with human assistance.
    • MAHT (Machine-Aided Human Translation) / HAMT (Human-Aided Machine Translation): Systems where humans collaborate with computers in translation.
    • CAT (Computer-Aided Translation) / Computer-Assisted Translation: Tools that aid humans, not fully autonomous; often include translation memories and term bases.
  • MT vs. CAT:
    • MT aims to automate translation; CAT assists humans and often saves time by segmenting text and reusing previous translations.
    • CAT tools rely on translation memories (TMs) and term bases; MT focuses on automatic generation and may require post-editing.
  • Translation Memory (TM): A database of previous translations that learns from every translation and suggests matches for new texts (exact or fuzzy).
  • Fuzzy matches vs. exact matches:
    • Fuzzy matches approximate similarity (e.g., 55%–99% in a given system), useful as starting points for human refinement.
    • Exact matches are 100% matches of source and target text segments.
  • Term Base (Glossary): Centralized collection of industry- or company-specific terms with approved translations, context, and notes on terms not to translate (e.g., brand names).
  • Pre-editing / post-editing: Pre-editing tightens input to reduce ambiguity; post-editing revises MT output to meet quality standards.
  • Controlled language: Writing input in a restricted style/lexicon to reduce ambiguity and improve translation quality.
  • Interactive MT: Involves human translators resolving ambiguities during translation.

Computer-Aided Translation Tools (CAT): Focus and Features

  • CAT tools assist translators by segmenting text, storing translations, and managing terminology.
  • Key benefits:
    • Reducing translation time through segment-based work and reuse of previous translations.
    • Maintaining terminology consistency via translation memories and term bases.
    • Providing a reusable translation database for training MT systems.
  • Core components:
    • Translation Memory (TM)
    • Terminology Management / Term Bases
    • Search and retrieval features for previous translations
    • Compatibility with various document formats and translation workflows
  • Examples of popular CAT tools (illustrative only; not exhaustive):
    • SDL Trados Studio
    • memoQ
    • Wordfast (Classic & Pro)
    • Déjà Vu
    • Across
    • CafeTran Espresso (free up to TM size)
    • OmegaT (free/open source)
    • Similis, Wordfast Anywhere, MateCat, SmartCAT, Virtaal, Pootle, Lokalize, GlobalSight, gtranslator, etc.
  • Market landscape notes:
    • No single perfect CAT tool; selection depends on agency requirements, budget, user tech skill, and needed features.
    • CAT tools support multiple formats and can be cloud-based or software-based.

Information Retrieval (IR) and Question Answering (QA)

  • Information Retrieval (IR) systems search large documentary databases for relevant information.
  • Basic IR concepts:
    • Keys: keywords attached to documents, OR/AND query strategies, ranking by relevance.
    • Recall vs. precision:
    • Recall: Recall=number of relevant documents foundtotal number of relevant documents in the databaseRecall = \frac{\text{number of relevant documents found}}{\text{total number of relevant documents in the database}}
    • Precision: Precision=number of relevant documents foundtotal number of documents foundPrecision = \frac{\text{number of relevant documents found}}{\text{total number of documents found}}
    • Trade-off: generally increasing recall reduces precision and vice versa; QA often prioritizes recall to ensure candidate answers are not missed.
  • Thesaurus and keyword expansion: automated indexing may enrich keywords using hierarchical term systems (thesaurus) and expert knowledge bases.
  • Keyword extraction for QA: an 8-step heuristic is described (often only the first 6 are used):
    • 1) Include non-stop words in quoted expressions
    • 2) Proper nouns (named entities)
    • 3) Complex nominals and their adjectives
    • 4) All other complex nominals
    • 5) All nouns and their adjectival modifiers
    • 6) All other nouns
    • 7) All verbs
    • 8) The question focus
  • WordNet as a supplementary resource for synonym expansion of keywords.
  • QA architecture overview (open-domain):
    • Question Processing Module: determine question type, expected answer type, and question focus; pass keywords to Paragraph Indexing
    • Question Type Classification: various taxonomies (Lehnert 13 classes; Li & Roth 6 coarse classes and 50 fine classes; Moldovan 200 questions in TREC)
    • Answer Type Classification: map question type to potential answer types; allow multiple answer types due to ambiguity
    • Question Focus: pinpoint the information requested; example: "What is the longest river in New South Wales?" focus: "longest river"
    • Keywords extraction and expansion: via heuristics and WordNet
    • Paragraph Indexing Module: retrieves and ranks paragraphs likely to contain the answer; may issue new queries if needed
    • Information Retrieval (IR) Engine choice: standard IR vs. QA-oriented retrieval; aim to retrieve documents containing all keywords
    • Information retrieval metrics: precision and recall as above; QA prioritizes recall
    • Paragraph Filtering: reduces candidate text for faster processing; rank by plausibility of contained answer
  • Open-domain QA architecture details:
    • Use web search engines (e.g., Google) to supply broad recall; apply Boolean AND to ensure documents contain all keywords
    • Use specialized resources/databases for domain-specific accuracy (e.g., Amazon, CIA World Fact Book); these may offer higher precision/recall within limited corpora
    • Predictive annotation: offline indexing of documents with concepts to improve retrieval, guided by answer type taxonomy
    • Paragraph filtering to manage the number of candidate documents and text length

Corpus Linguistics: Foundations and Definitions

  • Corpus linguistics is the study of language using large, structured collections of real language data (corpora) to observe patterns across usage, genres, and contexts.
  • Core idea: corpus-based analysis uses machine-readable text samples to study language in authentic use (synchronic and diachronic perspectives).
  • Salient features of corpora (CORPUS: CAPable of Representing Potentially Unlimited Selections of text):
    • Quantity: large size, spanning many texts
    • Quality: authenticity of texts from real-world language use
    • Representation: balanced coverage across domains and language use
    • Simplicity: plain text with minimal markup to begin with
    • Equality: samples of even size (though this is debated)
    • Retreavability: easy data retrieval and reusability
    • Verifiability: open to empirical verification
    • Augmentation: growth over time to capture linguistic changes
    • Documentation: separate metadata/documentation from the text itself

Corpus Classification and Types

  • Genre of text:
    • Written corpus (e.g., TDIL Corpus)
    • Speech corpus (spoken language, including formal and informal discourse; often transcribed)
    • Spoken corpus (an extension of speech corpus; often annotated with phonetic transcription)
  • Nature of data:
    • General corpus: broad coverage across disciplines and registers; large, diverse, and representative
    • Special corpus: focused on particular varieties or topics; may be unbalanced and not suitable for full language description
    • Corpus of language of children, non-native speakers, dialects, etc.
  • Sublanguage corpus: homogeneous text variety with specialized lexicon; small but highly representative for a domain
  • Sample corpus: enriched subset to boost frequency of phenomena of interest in a given study
  • Type of text:
    • Monolingual corpus: texts in a single language, spanning a period or multiple periods
    • Bilingual corpus: texts in two languages; parallel corpus aligns texts by translation
    • Multilingual corpus: representative collections from more than two languages
  • Purpose of design:
    • Un-annotated corpus: raw plain texts without annotation
    • Annotated corpora: rich with linguistic information (POS tagging, syntax, semantics) for deeper analysis (not detailed in the excerpt, but implied)

Corpus Processing and Analysis

  • Corpus processing enables quantitative and qualitative analysis:
    • Quantitative: counts, statistics, and models to identify genuine linguistic patterns vs. chance occurrences
    • Qualitative: detailed description of phenomena observed in the corpus; complements quantitative findings
  • Statistical approaches and multivariate techniques used in corpus analysis:
    • Descriptive statistics, inferential statistics, evaluative statistics
    • Multivariate analyses (e.g., Factor Analysis, Multidimensional Scaling, Cluster Analysis, Log-linear Models)
  • Concordance: indexing words by their textual environments to study patterns and usage; useful for lexicography, language teaching, and stylistics
  • Lexical collocation: study of word co-occurrence patterns to understand word roles and distribution in text

Utility and Limitations of Corpora

  • Corpus usefulness spans language description, syntax, phonetics, prosody, morphology, lexicography, discourse, pragmatics, language teaching, lexicography, etc.
  • Potential users include language specialists (lexicographers, terminologists), content specialists (historians, literary critics), and media specialists (information retrievers, QA researchers).
  • Limitations and challenges:
    • Lack of linguistic generativity: corpora reflect performances, not underlying competence; classic criticisms from Chomsky about corpora failing to capture innateness and competence
    • Technical challenges: large-scale data collection, annotation, processing; requires training and coordination between linguists and computer scientists
    • Lack of dialogue data: corpora may underrepresent spontaneous, dialogic interactions; context and pragmatics can be underrepresented
    • Absence of visual elements: corpora mostly text; graphs, tables, figures, images may be missing but are often important in understanding meaning
    • Overreliance on written texts: speech representations and social variation may be underrepresented; corpora may not capture diachronic social dynamics or emotion in language
    • Contextual limitations: corpora may fail to reveal social, cultural, and pragmatic aspects and how these affect language use

Concordance, Collocation, and Lexical Studies in Corpora

  • Concordance: creating an index of word occurrences with textual environments; provides access to patterns not easily observed by intuition; software examples include MonoConc, ParaConc, Conc, Free Text
  • Lexical collocation: analysis of word co-occurrence to understand typical contexts and permissions for word usage; helps test and refine linguistic hypotheses against empirical data

Practical Implications and Takeaways

  • Corpus linguistics integrates empirical data with computational methods to study language, enabling robust language technology development (MT, CAT, IR, QA, etc.).
  • The success of language technologies depends on quality corpora, appropriate annotation, and careful model selection to reflect real-world language use.
  • Limitations of corpora must be acknowledged when drawing conclusions about language competence, variation, and change.

Summary of Key Points and Concepts

  • CL blends linguistics and computer science to analyze and synthesize language; it spans theory and application.

  • MT and CAT are central to applied CL, with MT focusing on automated translation and CAT on human-in-the-loop translation aided by translation memories and term bases.

  • QA architectures combine question processing, classification, focus extraction, information retrieval, paragraph indexing, and answer verification; recall is often prioritized over precision for effective QA.

  • Corpora are large, structured reserves of language data; their design criteria include quantity, quality, representation, and documentation. They are categorized by genre, data type, language, and purpose, and they are processed using both quantitative and qualitative methods.

  • Limitations of corpora include representativeness, lack of dialogue and visual data, and the gap between performance (corpus data) and competence (theoretical language knowledge).

  • Important numerical references:

    • Large-scale search engines indexing on the order of 8×1098 \times 10^9 documents.
    • Translation memory usefulness described in terms of exact vs. fuzzy matches (e.g., 55%–99% for fuzzy matches in some systems).

  • Notable examples:

    • MT error example: "The spirit is willing, but the flesh is weak" mistranslated to suggest a drinking context in one early system.
    • Arabic-English translation issues (e.g., header mis-translation, idiomatic expressions) illustrating the need for deep linguistic analysis and context.

  • Core equations:

    • Recall: Recall=relevant documents foundtotal relevant documents in the databaseRecall = \frac{\text{relevant documents found}}{\text{total relevant documents in the database}}
    • Precision: Precision=relevant documents foundtotal documents foundPrecision = \frac{\text{relevant documents found}}{\text{total documents found}}

  • Key dates and currents:

    • MT emerged as a long-standing dream; progress has shifted from word-by-word translation toward model-based and statistically guided approaches.
    • The field continually treads between theory (linguistics, cognition) and applied engineering (IR, QA, MT, CAT).

Note: This set of notes reflects the provided transcript and aims to consolidate major and minor points, definitions, examples, and implications for a cohesive study resource. It preserves the breadth of topics—from foundational concepts in CL to practical concerns in MT, QA, and corpus linguistics—while highlighting essential formulas and real-world examples referenced in the material.