About the AI Crawler Accessibility Checker
AI companies publish the user-agent tokens of their crawlers, and most separate training crawlers from search or user-triggered fetchers. The checker evaluates your robots.txt for each of them, for the exact URL you enter, and shows the rule that decides.
Blocking is a legitimate choice. The point is to make it a deliberate one.
How it works
- We fetch the URL from our server like a crawler: redirects are followed (up to five), private and internal addresses are refused, and large pages are capped.
- The checks run on the live HTML and response headers. Nothing is guessed: every finding shows what we observed.
- Results are not stored unless you are signed in, in which case they appear in your dashboard history.
Frequently asked questions
Does robots.txt bind AI companies?
It is a voluntary standard. The major operators state they honour it for their declared crawlers.