robots.txt check
Paste the content of your robots.txt file and see which AI crawlers can access your website. The check runs in your browser: nothing is sent to our server.
Who has access
For example / for the whole site or /blog/.
Create a robots.txt file
Whether to let in crawlers that train AI models is your decision. Both variants keep access for search crawlers.
Cost of variant B: citations with a link remain, because they come from search crawlers. Models trained after this date may know less about your brand when they answer without searching. Nobody has measured the size of this effect.
Good to know
- robots.txt is a request, not a security measure. Crawlers whose operator does not commit to following the file are marked in the results.
- The tool assesses the content of the file. It does not check how the server delivers it: the response code, redirects or cache lifetime.
- Crawler classes and behaviour are based on the operators' own documents. Each crawler links to its source.