Free tool · Checks every AI bot
robots.txt checker for AI bots
We check whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and 10 other AI crawlers are allowed on your site - then hand you a UK-SMB-friendly recommended robots.txt.
Analysis
Found a robots.txt at
Found a robots.txt at https://autoid.ro/robots.txt
| AI crawler | Status | What it does |
|---|---|---|
| GPTBot | ✓ Allowed | OpenAI training crawler |
| OAI-SearchBot | ✓ Allowed | OpenAI search index crawler |
| ChatGPT-User | ✓ Allowed | OpenAI on-demand fetch (when a user asks ChatGPT to read your page) |
| ClaudeBot | ✓ Allowed | Anthropic training crawler |
| Claude-User | ✓ Allowed | Anthropic on-demand fetch |
| Claude-SearchBot | ✓ Allowed | Anthropic search index crawler |
| PerplexityBot | ✓ Allowed | Perplexity search index crawler |
| Perplexity-User | ✓ Allowed | Perplexity on-demand fetch |
| Google-Extended | ✓ Allowed | Google AI Overviews / Gemini training opt-in token |
| CCBot | ✓ Allowed | Common Crawl (used by many AI training pipelines) |
| Applebot | ✓ Allowed | Apple search index + Siri live generative answers (opt out of Siri answers via <meta name="nosnippet">) |
| Applebot-Extended | ✓ Allowed | Apple Intelligence AI-training opt-out token (controls whether Applebot-crawled content trains Apple Intelligence) |
| Amazonbot | ✓ Allowed | Amazon AI training crawler |
| Google-NotebookLM | ✓ Allowed | Google NotebookLM live citation fetches |
| Google-CloudVertexBot | ✓ Allowed | Google Cloud Vertex AI grounding |
| Google-Agent | ✓ Allowed | Google Gemini agent runs |
| Bytespider | ✓ Allowed | ByteDance / Doubao training crawler |
| Meta-ExternalAgent | ✓ Allowed | Meta AI training crawler |
| DuckAssistBot | ✓ Allowed | DuckDuckGo AI assistant |
Sitemap declarations found:
- https://www.autoid.ro/sitemap_index.xml
- https://autoid.ro/sitemap.xml
Recommended
Drop-in robots.txt
UK-SMB-friendly defaults: every AI crawler explicitly allowed by name; admin / cart / checkout paths blocked. Tweak to taste.
# robots.txt - generated by llmsubmitter.com # UK-SMB-friendly defaults: AI crawlers explicitly allowed by name, # admin / transactional paths blocked. Edit to taste. User-agent: * Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: GPTBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: OAI-SearchBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: ChatGPT-User Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: ClaudeBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Claude-User Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Claude-SearchBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: PerplexityBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Perplexity-User Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-Extended Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: CCBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Applebot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Applebot-Extended Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Amazonbot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-NotebookLM Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-CloudVertexBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-Agent Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Bytespider Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Meta-ExternalAgent Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: DuckAssistBot Disallow: /cart/ Disallow: /checkout/ Disallow: /account/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= Sitemap: https://autoid.ro/sitemap.xml # llms.txt: https://autoid.ro/llms.txt
Your current robots.txt
For reference
# This file is automatically added by Rank Math SEO plugin to help a website index better # More info: https://rankmath.com/?utm_source=Plugin&utm_medium=Robots&utm_campaign=WP User-Agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://www.autoid.ro/sitemap_index.xml User-agent: * Disallow: /wp-content/uploads/wc-logs/ Disallow: /wp-content/uploads/woocommerce_transient_files/ Disallow: /wp-content/uploads/woocommerce_uploads/ Disallow: /*?add-to-cart= Disallow: /*&add-to-cart= Disallow: /*?remove_item= Disallow: /*&remove_item= Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ # robots.txt - generated by llmsubmitter.com # UK-SMB-friendly defaults: AI crawlers explicitly allowed by name, # admin / transactional paths blocked. Edit to taste. User-agent: * Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: GPTBot Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: OAI-SearchBot Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: ChatGPT-User Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: ClaudeBot Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Claude-User Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Claude-SearchBot Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: PerplexityBot Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Perplexity-User Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-Extended Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: CCBot Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Applebot Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Applebot-Extended Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Amazonbot Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-NotebookLM Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-CloudVertexBot Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Google-Agent Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Bytespider Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: Meta-ExternalAgent Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= User-agent: DuckAssistBot Disallow: /cos/ Disallow: /finalizare-comanda/ Disallow: /contul-meu/ Disallow: /api/ Disallow: /admin/ Disallow: /wp-admin/ Disallow: /login Disallow: /logout Disallow: /search? Disallow: /*?utm_ Disallow: /*?ref= Sitemap: https://autoid.ro/sitemap.xml # llms.txt: https://autoid.ro/llms.txt