Web Crawler – Non-Functional Requirements

Key non-functional requirements for designing a web crawler, including durability, latency, and security considerations.

What are the 4 key non-functional requirements of a web crawler?

1. No CAP theorem (not applied here)

2. Fault tolerance / durability

3. Latency of the entire operation (x days)

4. Security: avoid bots/loops, respect robots.txt, rate limiting

Why is the CAP theorem not applied in this web crawler design?

It is not directly relevant for crawler functionality, so it’s excluded.

What does fault tolerance/durability mean for a web crawler?

The crawler must survive failures without data loss and continue reliably.

What does latency of the entire operation mean?

The total time it takes to complete a full crawl (often measured in days).

What security and politeness measures must a web crawler follow?

Avoid getting stuck in bot traps/loops, respect robots.txt, and apply rate limiting.