Free tool
robots.txt Tester
Check whether GPTBot, ClaudeBot or Googlebot can reach a page.
What this tool does
Checks whether the main search and AI crawlers can reach a page on your site under its robots.txt: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Googlebot, Google-Extended and Bingbot.
If a crawler is blocked, you cannot be found or cited by the product behind it. The tool shows the robots.txt it read, which crawlers are allowed or blocked, and a suggested fix where one is needed.
Frequently asked questions
- Which crawlers should I allow for AI search?
To appear in ChatGPT search, allow OAI-SearchBot; for Claude, Claude-SearchBot; for Perplexity, PerplexityBot; and for Google, including AI Overviews, Googlebot. GPTBot and Google-Extended relate to training rather than search results.
- Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended controls some uses of your content for Gemini. AI Overviews draw on pages Googlebot crawls, so they are controlled by Googlebot and your snippet settings.
- Can a crawler be blocked even if robots.txt allows it?
Yes. Firewalls, bot protection and CDN rules can block crawlers before robots.txt is ever read. This tool checks robots.txt only, so if a crawler is allowed here but still cannot reach you, check those too.
- Is it free, and do you store anything?
Yes it's free. We do not save the page or the result. We fetch the address you give, check it, and show you what we found. We keep a record that the tool ran (when, and your IP address to prevent abuse).
Start getting found on AI, Google and Reddit
- 15 articles a month optimised for AI & search citation
- Get backlinks on autopilot from other members
- Real time Reddit monitoring with reply drafts to drive traffic and AI mentions
- Weekly deep site audits with a full report card, tracked over time
- All built to maximise AI mentions and search ranking