CHAPTER 9: DESIGN A WEB CRAWLER

Web crawler purposes

1. Search engine indexing

2. Web archiving

3. Web mining

4. Web monitoring

Basic steps of web crawler

1. Download web pages for given list of URLS

2. Extra URLs from web pages

3. Repeat

Requirement questions

1. What is the purpose

2. How many pages per month

3. What content types? HTML or PDF, etc

4. Should we store HTML contents

Characteristics of good web crawler

1. Scalable, parallelization

2. Robust to handle bad HTML, unresponsive server, crashes, etc

3. Polite: not make too many request to a server

4. Extensible: easy to support new features

Estimation

1. Pages to crawl per second

2. Storage needed: page size x page craw per second x years of storage

Diagram components

1. Seed URLs

2. URL frontier

3. DNS resolver

4. HTML downloader

5. Content parser

6. Content storage

7. URL extractor

8. URL filter

9. URL storage

What makes good seed URL

Popular websites by countries or topics

Why we need a content parser

1. Validate web page format

2. Reduce storage space

What does URL extractor do

1. Get URL from content

2. Convert relative URL to absolute ones

What does URL filter do?

1. Exclude certain content types

2. Exclude error links

3. Exclude blacklisted sites

Requirements of URL frontier

1. Prioritize more important URLs

2. Rate limit requests to each domain

URL frontier components

1. Prioritizer: compute prioritization of URL

2. Front queue each with different priority

3. Front queue selector: choose a queue to process with bias towards high priority

4. Mapping table: map domain to queue

5. Back queue router: enqueue URL to corresponding queue based on domain

6. Back queue: contains URL from list of domains

7. Back queue router: router queue to workers

When to recrawl pages

1. Based on historical update frequency

2. Based on importance

How does storage for URL queue work

1. Storage in disk because it won’t all fit in memory

2. Write to memory buffer first and bulk write to disk

3. Bulk fetch from disk to memory for data to dequeue

What is Robots.txt

1. Robots exclusion protocol

2. Specifies what pages crawlers are allowed to download

How to speed up HTML downloading

1. Scale up downloader

2. Cache DNS records

3. Deploy downloader closer to website hosts

4. Specify max wait time

How to make the system more robust

1. Have multiple downloaders

2. Save craw states and data snapshot

3. Exception handling to prevent crash

4. Data validation

Where to add more modules such as download PNG files?

After content parser

How to check if web page is duplicate?

Check hash

What is spider trap

A webpage that causes crawler in an infinite loop. For example, infinite deep directory structure

Wrap up talking points

1. Server side render SPAs

2. Filter out spam webpages

3. Increase database availability

4. Collect analytics