Text Processing: Parsing and Transformation

Text Processing

Transforms raw document content into a consistent representation for search.

Goal: preserve information useful for retrieval while reducing irrelevant formatting and textual variation.

What raw content do web documents contain?

HTML containing information we want to search and markup, metadata, links, scripts, and other non-content elements.

What are the major steps of text processing?

1. Parsing and Transformation

Parsing

Recognizes the document structure and extracts useful content

Transformation

Cleans and normalizes the extracted text

What does HTML structure consist of?

HTML documents consist of elements defined by tags. These elements contain content and attributes

DOM (Document Object Model)

How browsers represent an HTML document. Organizes a document as a tree of elements with relationships.

RegEx (Regular Expressions)

Describes a pattern of characters that can be used to identify patterns within text or attribute values.

What can RegEx be used for?

It can match patterns for URLs, emails, numbers, formatted dates, whitespace, etc.

Common RegEx patterns