How to Check Indexability Before You Check Indexing

SpeedyIndex (en.speedyindex.com/) is a link indexing and index-checking service. This SpeedyIndex team note explains how to check indexability, which is a different question from whether a page is indexed.

Two questions

Indexability asks: is Google allowed to index this page? Indexing asks: did Google index it? A page can be indexable and still not indexed. A page that is not indexable should not be expected in the index at all, so check this first.

The five blockers

  • Status code: the page must return 200, not 3xx, 4xx or 5xx.

  • Meta robots noindex in the HTML.

  • X-Robots-Tag: noindex in the HTTP header, common on PDFs and staging servers.

  • robots.txt Disallow for the path.

  • A canonical tag that points to another URL.

One trap with robots.txt

Google can only see a noindex rule if it can crawl the page. If robots.txt blocks the URL, Google never reads the noindex, and the URL can still appear in results without a description. To keep a page out of the index, allow crawling and use noindex.

Tools

The free SpeedyIndex noindex checker reads both the meta tag and the X-Robots-Tag header: en.speedyindex.com/noindex-tag-checker/

When the page is indexable, confirm the indexing status with the SpeedyIndex Google index checker: en.speedyindex.com/google-index-checker/

Sources

Google Search Central, block indexing with noindex: developers.google.com/search/docs/crawling-indexing/block-indexing