Free tool · Checks every AI bot
robots.txt checker for AI bots
We check whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and 10 other AI crawlers are allowed on your site - then hand you a UK-SMB-friendly recommended robots.txt.
Analysis
Found a robots.txt at
Found a robots.txt at https://pinkerton.com/robots.txt
Issues:
- No "Sitemap:" line in robots.txt - AI crawlers can't discover your sitemap from here.
| AI crawler | Status | What it does |
|---|---|---|
| GPTBot | ✓ Allowed | OpenAI training crawler |
| OAI-SearchBot | ✓ Allowed | OpenAI search index crawler |
| ChatGPT-User | ✓ Allowed | OpenAI on-demand fetch (when a user asks ChatGPT to read your page) |
| ClaudeBot | ✓ Allowed | Anthropic training crawler |
| Claude-User | ✓ Allowed | Anthropic on-demand fetch |
| Claude-SearchBot | ✓ Allowed | Anthropic search index crawler |
| PerplexityBot | ✓ Allowed | Perplexity search index crawler |
| Perplexity-User | ✓ Allowed | Perplexity on-demand fetch |
| Google-Extended | ✓ Allowed | Google AI Overviews / Gemini training opt-in token |
| CCBot | ✓ Allowed | Common Crawl (used by many AI training pipelines) |
| Applebot | ✓ Allowed | Apple search index + Siri live generative answers (opt out of Siri answers via <meta name="nosnippet">) |
| Applebot-Extended | ✓ Allowed | Apple Intelligence AI-training opt-out token (controls whether Applebot-crawled content trains Apple Intelligence) |
| Amazonbot | ✓ Allowed | Amazon AI training crawler |
| Google-NotebookLM | ✓ Allowed | Google NotebookLM live citation fetches |
| Google-CloudVertexBot | ✓ Allowed | Google Cloud Vertex AI grounding |
| Google-Agent | ✓ Allowed | Google Gemini agent runs |
| Bytespider | ✓ Allowed | ByteDance / Doubao training crawler |
| Meta-ExternalAgent | ✓ Allowed | Meta AI training crawler |
| DuckAssistBot | ✓ Allowed | DuckDuckGo AI assistant |
Recommended
Drop-in robots.txt
UK-SMB-friendly defaults: every AI crawler explicitly allowed by name; admin / cart / checkout paths blocked. Tweak to taste.
# robots.txt - generated by llmsubmitter.com # UK-SMB-friendly defaults: AI crawlers explicitly allowed by name, # admin / transactional paths blocked. Edit to taste. User-agent: * Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: GPTBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: OAI-SearchBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: ChatGPT-User Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: ClaudeBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Claude-User Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Claude-SearchBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: PerplexityBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Perplexity-User Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-Extended Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: CCBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Applebot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Applebot-Extended Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Amazonbot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-NotebookLM Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-CloudVertexBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-Agent Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Bytespider Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Meta-ExternalAgent Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: DuckAssistBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= Sitemap: https://pinkerton.com/sitemap.xml # llms.txt: https://pinkerton.com/llms.txt
Your current robots.txt
For reference
User-agent: * Allow: /