AI protection
AI crawler auditor.
Put in your address and see which AI crawlers your robots.txt allows right now — GPTBot, ClaudeBot, Google-Extended, Perplexity and the rest. Then change what you want and take the corrected file away.
Crawler rule builder
Check a live site
Reads the site’s real robots.txt and reports which AI crawlers it allows today. Most sites have never been checked, and a single line written years ago is the usual reason a business never appears in an AI answer.
Protection Level
Strong
Bots Blocked
5/14
High Risk Blocked
5/5
Custom Rules
0
AI crawlers database
Additional Settings
Custom Rules
No custom rules added yet
# ═══════════════════════════════════════════════════════ # robots.txt — AI Crawler Policy # Generated by Solvey AI Crawler Auditor # ═══════════════════════════════════════════════════════ # ─── AI Bot Restrictions ────────────────────────────── User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Bytespider Disallow: / # ─── General Rules ──────────────────────────────────── User-agent: * Allow: / Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /admin/ Disallow: /cgi-bin/ # ─── Sitemap ────────────────────────────────────────── Sitemap: https://example.com/sitemap-index.xml
How to Deploy
- Click "Copy robots.txt" above
- Save as
robots.txtin your website's root directory - It must be accessible at
https://example.com/robots.txt - Check it in Search Console's robots.txt report
How the auditor works
14 AI bots tracked
Comprehensive database of known AI training crawlers from OpenAI, Google, Anthropic, Meta, ByteDance, and more — with risk level ratings for each.
Risk-based blocking
Each crawler is rated High, Medium, or Low risk. Quickly select all high-risk bots, block everything, or customise your policy per crawler.
Production ready
Generated robots.txt includes properly formatted rules, sitemap references, and admin path blocking. Copy and deploy directly to your web root.
How to use it
Getting a useful result.
- Decide what you want first. Most small businesses want to be found by AI assistants and are less worried about their text training a model — those are two different crawlers and can be answered differently.
- Allow the crawlers that send people back to you, and block the ones that only take. The generated file separates them, so you are not forced into all-or-nothing.
- Save the result as
robots.txtin your site's root, so it answers atyourdomain.com/robots.txt. - Remember that robots.txt is a request, not a lock. Well-behaved crawlers honour it; nothing stops one that does not.
There is a real decision buried here. Blocking every AI crawler protects your writing from being used as training data — and also removes you from the answers those assistants give when somebody asks for a web designer in your city. For most local businesses the visibility is worth more than the text, but it is your call, and it should be a decision rather than a default.
The two categories are worth separating. A search crawler indexes your page so an assistant can cite and link it, which sends you traffic. A training crawler copies your content into a dataset, which does not. Several providers use different agent names for each, and that is what makes a considered answer possible.
Blocking a crawler in robots.txt does not remove anything already collected, and it does not affect Google's normal search indexing unless you block Googlebot itself — which almost nobody should do.
Common questions
Will blocking AI crawlers hurt my Google ranking?
No, provided you do not block Googlebot. Google-Extended is a separate control that governs training for Gemini and related products; declining it does not affect how Google Search crawls, indexes or ranks your pages. Those are deliberately independent switches.
Should I block them or not?
If you want to be recommended when someone asks an assistant for a business like yours, allow the crawlers that index for answers and citations. If your value is in original writing you do not want absorbed into a model, block the training agents specifically. Doing both is entirely reasonable.
Does robots.txt actually stop anyone?
It stops the operators who choose to respect it, which includes all the major named ones. It is a published request, not an enforcement mechanism — there is no technical barrier behind it. Anything that must not be public needs a login, not a robots rule.
Being readable to AI answer engines is part of how we get clients named when someone asks an assistant instead of searching — digital marketing and GEO.