AI bots can’t recommend a site they can’t crawl

ChatGPT, Claude and Perplexity all send their own crawlers to your pages before any of them will cite you or recommend you.

If those crawlers are getting blocked, and on plenty of sites they are, you’re invisible to AI search before anyone has even judged your content.

AI bots can’t recommend a site they can’t crawl because ChatGPT, Claude and Perplexity each send their own crawlers to fetch pages before citing them in answers, and if those crawlers are blocked by your robots.txt, firewall or CDN settings, your content becomes invisible to AI search regardless of its quality.

Most site owners have never checked whether this is already happening to them. So let’s check.

Robots.txt blocks AI crawlers by default on more sites than you’d think

Open yours and look for disallow rules against GPTBot, ClaudeBot, PerplexityBot and Google-Extended.

If you don’t recognise a rule blocking one of these, you probably didn’t put it there. A security plugin or a “block AI scrapers” toggle did, usually during an update, and nobody told you.

This happens more often than people realise. Security plugins love adding bot blocks as default behaviour. They market it as protecting your content from AI training, which sounds reasonable until you realise they’ve also blocked the crawlers that would have cited your site in AI search results.

Don’t just read the file and assume it’s working. Test your actual pages against each bot’s user agent. Screaming Frog lets you set a custom user agent for exactly this. Your server logs will also tell you which bots have actually turned up and which ones gave up at the front door.

Here’s what to look for in your robots.txt file:

• User-agent: GPTBot followed by Disallow: /
• User-agent: ClaudeBot followed by Disallow: /
• User-agent: PerplexityBot followed by Disallow: /
• User-agent: Google-Extended followed by Disallow: /
• Any wildcard rules that might catch AI user agents

If you find any of these and you didn’t consciously add them, trace back which plugin or update introduced them. Then decide if that’s actually what you want.

Why is the difference between training bots and citation bots important?

This is where most robots.txt advice gets sloppy, and it’s the bit worth getting right.

OpenAI, Anthropic and Perplexity each run more than one crawler, and they don’t all do the same job.

The training crawlers

GPTBot, ClaudeBot and Google-Extended feed model training data. Block these and your content stays out of the next model’s training set. It has no effect on whether that AI tool can find and cite your page right now.

This is a legitimate choice. You might not want your content used to train models. That’s your call to make.

The search and citation crawlers

OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages live to answer a question in the moment. Block these and you disappear from AI Overviews, ChatGPT Search and Perplexity answers today, not in some future training run.

So blocking Google-Extended stops Gemini training on your content. It does nothing to your chances of being cited in an AI Overview, because that pulls from the live index.

These are two different decisions, and your robots.txt should treat them as two different decisions.

Decide separately whether you want your content used for training and whether you want it cited in answers, then write the rules to match. Write down which is which, because the next plugin update won’t know your reasoning and will happily undo it.

If you’re not sure where to start with this distinction, our AI optimisation checklist walks through the key access decisions you need to make.

Can your CDN or firewall block bots your robots.txt allows?

Yes. And this is the sneaky one.

Robots.txt is a request, not a lock. It’s standardised in RFC 9309, and well-behaved bots honour it. Not all of them are well-behaved. Perplexity has been caught running crawlers that ignore it entirely, so a permissive robots.txt file doesn’t guarantee a visit.

The bigger risk runs the other way. Your firewall or bot-management service can block a crawler your robots.txt happily allows, and that block won’t show up in a normal SEO audit.

Check for this directly:

• Look at your CDN or WAF, Cloudflare or Sucuri included, for bot-management rules that catch AI user agents by default
• Check for JavaScript challenges or CAPTCHA walls. These stop anything that isn’t a browser rendering a page, AI bots included
• Allowlist verified AI crawler IP ranges the same way you already allowlist Googlebot

Cloudflare’s own bot management has an “AI Scrapers and Crawlers” category built in. If it’s switched on with default settings, it may well be catching bots you actually want visiting.

The frustrating part is that everything can look fine from your end. Your site loads normally. Your SEO tools report no issues. But when ChatGPT or Perplexity tries to fetch your page to answer a question, they hit a JavaScript challenge and give up. You never see the failed request. They just cite someone else instead.

What happens when AI bots can’t reach your content?

The outcome is straightforward. You don’t get cited.

When someone asks ChatGPT or Perplexity a question your content could answer, those tools send their crawlers to fetch relevant pages. If your page returns a block, a challenge or an error, they move on to the next result. There’s no retry. There’s no flag for manual review. Your content simply doesn’t exist in that answer.

This compounds over time. The more questions get answered without your content, the more citation opportunities you miss. And unlike traditional SEO where you can track rankings, you often can’t see this happening. There’s no report showing you “your page was blocked from 47 potential citations this week.”

Understanding the AI search ranking factors that influence visibility helps, but none of those factors matter if the crawlers can’t reach your pages in the first place.

Recheck this every time your site changes

One security plugin update can add new bot blocks overnight, and nobody reviews robots.txt when they approve a plugin update.

This isn’t paranoia. I’ve seen it happen on client sites where everything was configured correctly, then a routine update changed default settings and suddenly AI crawlers were blocked. Nobody noticed for months because traditional search traffic looked normal.

Set a recurring quarterly check of your robots.txt file and your firewall’s bot rules. Save a baseline copy of both and compare your current setup against it. That’s how you catch a change that got made without a decision behind it.

Here’s a simple process:

• Export your robots.txt and save it with the date
• Screenshot your firewall’s bot management settings
• Note which AI user agents are currently allowed
• Set a calendar reminder to compare against this baseline every quarter
• Check after any security plugin updates, not just at scheduled intervals

Treat AI crawler access as a technical SEO metric you check on a schedule, not a setting you configure once and forget.

How do you fix blocked AI crawlers?

Start by confirming what’s actually happening. Check your robots.txt for explicit blocks. Check your server logs for AI crawler activity. Check your CDN and firewall settings for bot management rules.

Then make deliberate decisions:

• Which training crawlers do you want to block? Document why.
• Which search and citation crawlers do you want to allow? Document why.
• Does your firewall need updated allowlists for AI crawler IP ranges?
• Are there JavaScript challenges that need exceptions for verified bots?

Write these decisions down somewhere your team can find them. The person who makes the next plugin update needs to know why those rules exist.

If you want this audited properly against a live site, our search visibility framework includes a full crawl access review as part of the technical foundation work.

Frequently asked questions about AI bots and site crawling

How do I know if AI bots are being blocked from my site?

Check your robots.txt file for disallow rules against GPTBot, ClaudeBot, PerplexityBot and their search variants. Then check your server logs for actual visits from these user agents. If you see no AI crawler activity but your robots.txt allows them, your firewall or CDN may be blocking them at a different layer.

Will blocking AI training bots stop me appearing in ChatGPT answers?

No. Blocking GPTBot or Google-Extended stops your content being used to train future models, but it doesn’t affect citation crawlers like OAI-SearchBot. These are separate crawlers with different purposes. Block training if you want, but keep citation crawlers allowed if you want to appear in AI search results.

Why would a security plugin block AI crawlers without telling me?

Security plugins often add bot blocks as default settings or through feature updates. They market this as protection against AI scraping. The problem is they don’t distinguish between training crawlers and citation crawlers, so they block everything. And they rarely announce this in update notes you’d actually read.

Does Cloudflare block AI crawlers by default?

Cloudflare has an “AI Scrapers and Crawlers” category in its bot management settings. If this is enabled with default settings, it may block AI crawlers that you’d actually want to allow. Check your settings and add exceptions for verified AI search crawlers if needed.

How often should I check my AI crawler access settings?

Quarterly at minimum, plus after any security plugin updates. Save baseline copies of your robots.txt and firewall settings so you can spot changes that were made without a conscious decision. One plugin update can undo months of careful configuration.

Can I allow citation crawlers but block training crawlers?

Yes. This is exactly the distinction worth making. In your robots.txt, you can block GPTBot and ClaudeBot while allowing OAI-SearchBot and Claude-SearchBot. This keeps your content out of training datasets while still letting AI search tools cite your pages in real-time answers.