Free tool

Robots.txt Checker

This tool downloads your robots.txt, parses it the way search engine crawlers do (RFC 9309: most specific rule wins, Allow wins ties) and tests it against the pages linked from your homepage.

Checks performed

What robots.txt does (and does not do)

robots.txt tells crawlers which URLs they may request. It does not hide a page from search results: a blocked URL can still be indexed if other sites link to it. To keep a page out of Google, use a noindex tag and let it be crawled.

The mistake that costs the most

A "User-agent: * / Disallow: /" copied from a development server blocks the entire site. Rankings then decline over the following days and weeks. This checker flags it as critical.

Good practice

Keep robots.txt short, block only genuinely useless areas (admin, cart, internal search results) and add a Sitemap line so every crawler finds your sitemap.

FAQ

Where must robots.txt be located?

At the root of the host: https://www.example.com/robots.txt. A file in a sub-folder is ignored, and each subdomain needs its own.

Is an empty Disallow line a problem?

No. "Disallow:" with no value means "allow everything".

Should I block CSS and JavaScript?

No. Google renders pages like a browser and needs those files to understand the layout and mobile-friendliness.

Check everything at once

The full audit crawls up to 10 pages and runs 63 checks across SEO, performance, accessibility and technical health.

Run a free full audit

Other tools: Meta Tag Checker · Sitemap Checker · Heading Checker · Open Graph Checker · Image Alt Text Checker · Broken Link Checker · Website Speed Checker