For most B2B software companies, yes blocking AI crawlers removes you from the layer where buyers now form shortlists, and B2B SaaS does not monetise the pageviews it would be protecting.
For publishers and ad-supported businesses the calculation genuinely differs, and blocking is defensible.
The decision comes down to one question: Does your business make money from people visiting your pages, or from people knowing your product exists?
If it is the first, extraction without clicks is a direct loss. If it is the second, being absent from AI answers is a much larger loss than any traffic you might protect.
What do AI crawlers actually do?
Different crawlers serve different purposes, and blocking them has different consequences:
- Crawler Operator What it does Effect of blocking ?
- GPT Bot OpenAI Collects data for model training Removed from future trained knowledge
- OAI-Search Bot OpenAI Indexes for ChatGPT search Removed from ChatGPT’s live search results
- ChatGPT-User OpenAI Fetches a page when a user’s query triggers browsing Cannot be retrieved during live browsing
- PerplexityBot Perplexity Indexes for Perplexity answers Removed from Perplexity citations
- Claude Bot Anthropic Collects data for training Removed from future trained knowledge
- Google-Extended Google Controls use in Gemini and AI training Excluded from Gemini grounding
- CCBot Common Crawl Open web crawl used by many datasets Removed from a widely used training source
- Byte spider` ByteDance Data collection Limited relevance for most B2B
An important distinction: Google-Extended does not affect normal Google Search ranking. Blocking it excludes you from Gemini and AI training uses while leaving Search unaffected. Blocking Googlebot , by contrast, removes you from Search entirely these are different things and are sometimes confused.
Note also that AI Overviews in Google Search are governed by standard Googlebot access, not by Google-Extended. You cannot currently opt out of AI Overviews while remaining in Search.
What happens if you block them?
Blocking prevents retrieval, which removes you from any answer that depends on current web access. It does not erase what a model already learned.
Two mechanisms, affected differently:
Live retrieval: When a model searches the web to answer a question. Blocking eliminates this entirely. Your new content cannot be found, and you cannot be cited in answers that depend on current information.
Trained knowledge: What the model learned during training. If your brand was in the training data, it remains. But blocking training crawlers means future model versions will not learn about you, and the effect compounds over successive releases.
The practical consequence for B2B SaaS: Blocking removes you from the half of the mechanism that responds fastest and that you can most influence. Any content you publish becomes invisible to these systems.
The case for blocking:
Three legitimate reasons, worth stating fairly:
You monetise pageviews: Media, ad-supported publishing, content businesses. A model that answers a question using your article and sends no click has extracted the value and left the cost. This is a real economic problem, and blocking is a rational response.
You have genuinely proprietary content: Original research you sell, subscriber-only analysis, methodology you consider a competitive asset. Restricting these specifically is reasonable.
Legal or contractual obligations: Some client contracts or data agreements constrain how content may be used. Check before deciding.
The case for allowing:
For B2B SaaS, three reasons that generally outweigh the above :
You do not sell pageviews: Your revenue comes from software subscriptions. A buyer who learns your product exists from an AI answer and then searches your brand name is more valuable than one who read a blog post and left.
Absence is total: In a generated answer naming four vendors, there is no position five. Blocking guarantees, you are not one of the four.
Your competitors are visible: If three competitors allow crawlers and you do not, the answer names them. This is the version of the trade-off most companies have not thought about the choice is not between extraction and protection, but between being in the shortlist and not.
The middle position:
Allow crawlers on the content you want quoted and restrict what you genuinely want to protect.
This is available and rarely used. Most companies treat it as a binary and it is not
1. Allow
2. Product pages
3. Pricing pages
4. Documentation
5. Comparison and alternatives pages
6. Blog content that supports the product
Consider restricting
- Gated original research you sell or use for lead generation
- Customer-specific content
- Internal or partner-only sections
The syntax path-level control:
- User-agent: GPT Bot
- Allow: /product/
- Allow: /pricing/
- Allow: /docs/
- Allow: /compare/
- Disallow: /research/
- Disallow: /customers/
How to check and change your settings?
To check: open `yourdomain.com/robots.txt` in a browser and search for the crawler names above.
To allow all AI crawlers: the appropriate default for most B2B SaaS. If there are no `Disallow` rules for these agents, they are already allowed. Explicit permission is not required, but being explicit prevents accidental future blocking:
- User-agent: GPT Bot
- Allow: /
- User-agent: OAI-SearchBot
- Allow: /
- User-agent: ChatGPT-User
- Allow: /
- User-agent: PerplexityBot
- Allow: /
- User-agent: Claude Bot
- Allow: /
- User-agent: Google-Extended
- Allow:
To block all AI crawlers
- User-agent: GPT Bot
- Disallow: /
- User-agent: Claude Bot
- Disallow: /
- User-agent: PerplexityBot
- Disallow: /
- User-agent: Google-Extended
- Disallow: /
- User-agent: CCBot
- Disallow: /
Where to edit it: WordPress: a plugin such as Yoast or Rank Math, or the file at your web root. Webflow: Project Settings → SEO → robots.txt. HubSpot: Settings → Website → Pages → robots.txt. Custom builds: the file at your web root, or a route serving it
Check this before assuming you allow access:
A meaningful number of companies have blocked AI crawlers without a deliberate decision.
The usual sequence: a security review, a legal review, or an IT audit in 2023 or 2024 recommended blocking AI crawlers as a precaution. Someone implemented it. Nobody revisited it. Two years later the marketing team is asking why the company does not appear in ChatGPT.
Some hosting platforms and CDNs also began blocking AI crawlers by default. Cloudflare in particular has offered one-click AI crawler blocking, and it has been enabled on a lot of sites by people who were not thinking about marketing implications.
Check three places:
1. Your `robots.txt`
2. Your CDN settings Cloudflare, Fastly, Akamai bot management
3. Any WAF or bot-protection rules
This is a ten-minute check, and it occasionally explains an entire visibility problem.
Does blocking protect your content?
Blocking reduces direct crawling but does not prevent your content being described through other sources.
If a third party writes about your product, that article can be crawled and used regardless of your own settings. Your competitor’s comparison page about you is on their domain. Review platform listings are on theirs.
So, blocking does not achieve invisibility it achieves absence from the sources you control while leaving you described entirely by sources you do not. For most B2B companies that is the worst of both outcomes.
Frequently Asked Questions:
Does blocking GPTBot affect Google Search rankings?
No. GPT Bot is OpenAI’s crawler and has no relationship to Google Search. Blocking Google-Extended` also leaves normal Search unaffected it controls Gemini and AI training uses only.
Can I block AI training but allow AI search?
Partially. GPT Bot and Claude Bot are training-oriented; OAI-Search Bot, ChatGPT-User and `PerplexityBot are search and retrieval oriented. You can allow the second group and block the first, though the distinction is imperfect and evolving.
Do AI crawlers respect robots.txt?
The major operators state that they do, and evidence broadly supports this for the crawlers listed above. Smaller or unnamed scrapers may not.
Will allowing crawlers hurt my traffic?
Possibly a small amount, if users get answers without clicking. For B2B SaaS this is generally outweighed by being present in the shortlist. For ad-supported publishers the calculation is different.
Should I opt out of Google AI Overviews?
You currently cannot do so while remaining in Google Search AI Overviews use standard Googlebot access. Google-Extended controls Gemini and training, not AI Overviews.\
What if I blocked crawlers a year ago?
Unblock and expect four to eight weeks before live retrieval reflects your content. Trained knowledge will update on future model releases.