About the Robots.txt Tester
Enter a URL and, optionally, a crawler name such as Googlebot or GPTBot. The tester fetches robots.txt, picks the group that applies to that crawler and finds the longest matching Allow or Disallow rule, as RFC 9309 and Google specify.
It also reports syntax problems, the declared sitemaps and whether the file blocks the whole site.
How it works
- We fetch the URL from our server like a crawler: redirects are followed (up to five), private and internal addresses are refused, and large pages are capped.
- The checks run on the live HTML and response headers. Nothing is guessed: every finding shows what we observed.
- Results are not stored unless you are signed in, in which case they appear in your dashboard history.
Frequently asked questions
Does Disallow remove a page from Google?
No. It stops crawling, but a blocked URL can still be indexed from links. Use noindex on a crawlable page to keep it out of results.