Web Crawler – Monitoring

Key metrics and signals to monitor when running a large-scale web crawler.

What should be monitored in a web crawler?

• Rate of new pages being added in the metadata directory

• Queue sizes and backlog

• Error rates and retries

• Fetch latency and success rate

• Storage growth and IOPS

Why monitor the rate of new pages being added in the metadata directory?

It measures crawler progress and coverage — how many new pages are discovered and added per time interval.

Why monitor queue sizes and backlog?

Large or growing queues indicate the crawler is falling behind and may need scaling or optimization.

Why monitor error rates and retries?

High error rates may signal network issues, blocked domains, or crawler misconfiguration.

Why monitor fetch latency and success rate?

Tracking average fetch times and success percentage helps ensure performance and reliability of page downloads.

Why monitor storage growth and IOPS?

Ensures blob storage and databases can handle incoming data and input/output operations without bottlenecks.