robots.txt for AI Crawlers: GPTBot, ClaudeBot, PerplexityBot and Google-Extended

AI companies run different crawlers for different jobs: training, powering AI search results, and fetching a page when a user asks. You control them in robots.txt. To appear in AI search answers, do not block the search and user-fetch crawlers. To opt out of training only, block the training crawlers separately.

Key takeaways

• One company often runs several crawlers with different purposes. Blocking the wrong one can remove you from AI search by accident.
• robots.txt is a request that reputable crawlers honour. It is not security. Do not rely on it to protect private data.
• Many sites block AI crawlers by default through a CDN, firewall or plugin without the owner knowing. Test your site.
• Blocking Google-Extended does not remove you from Google Search or AI Overviews, according to Google.
• Decide on purpose: visibility in AI answers, or protecting content from training, or both.

The main AI crawlers (check each vendor's docs for updates)

Company	User agent	Purpose
OpenAI	GPTBot	Collecting content that may be used to train models
OpenAI	OAI-SearchBot	Finding and surfacing pages in ChatGPT search results
OpenAI	ChatGPT-User	Fetching a page when a user asks ChatGPT to look at it
Anthropic	ClaudeBot	Collecting content that may be used to train models
Anthropic	Claude-User and Claude-SearchBot	Fetching pages on a user's request and supporting search results
Perplexity	PerplexityBot	Indexing pages for Perplexity search results
Perplexity	Perplexity-User	Fetching a page when a user asks about it
Google	Google-Extended	A control token for some Google AI products. Not used for Search ranking
Google	Googlebot	Normal crawling for Search, including AI features in Search
Common Crawl	CCBot	Builds an open web archive that AI developers sometimes use

Treat this table as a starting map, not a source of truth. Companies add and rename crawlers. The vendor documentation pages are the authority.

Three common setups

1. Open: I want to appear in AI answers and do not mind training

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

This is the simplest approach and fits most businesses that depend on being found.

2. Middle: appear in AI search, opt out of training

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

This blocks training-focused crawlers while leaving search and user-request crawlers alone. Be aware that the line between "training" and "search" is the company's definition, and it can change.

3. Closed: block AI crawlers entirely

Add a Disallow: / for every AI user agent you want to exclude, including the search and user-fetch ones. You will not appear in those products' answers from live search. Publishers with paid content often choose this, and then negotiate licences.

Check what is actually blocking you

A very common problem: your robots.txt is fine, but something else turns crawlers away.

• CDN or firewall bot rules. Some services block AI user agents by default. Check Cloudflare, Sucuri, Wordfence and similar settings.
• Hosting security plugins that rate-limit unknown bots.
• Login walls and JavaScript-only content. If the text appears only after scripts run, many crawlers will see an empty page. Server-render the important text.
• noindex or nosnippet tags that were left from a staging site.

Test from outside: fetch your page with a command like curl -A "OAI-SearchBot" https://www.example.com/ and confirm you get the real content, not an error or challenge page. Also read your server logs to see which of these crawlers actually visit.

Where robots.txt falls short

• It is voluntary. Well-behaved crawlers follow it, bad actors ignore it.
• It does not remove content already collected. It only affects future crawling.
• It cannot protect private information. Put sensitive content behind a login.
• Syntax errors can silently open or close the whole site. Test after every edit.

Deciding with a simple rule

Your priority	Choice
Lead generation and brand awareness	Stay open. Being cited is the goal.
Original paid content or research	Consider blocking training crawlers, and allow search crawlers if you want traffic
Private or regulated data	Do not publish it. Use authentication, not robots.txt
Unsure	Start open, monitor for a few months, tighten if needed

Two details most guides miss

User-initiated fetchers may ignore robots.txt

OpenAI's documentation says that because ChatGPT-User actions are started by a user, robots.txt rules may not apply to them. Perplexity says the same of Perplexity-User: because a user requested the fetch, it generally ignores robots.txt. That is different from the search and training crawlers. If you need to stop a page being fetched in this way, use authentication or server-side rules, not robots.txt.

Changes are not instant

OpenAI states it can take around 24 hours from a robots.txt update for its search systems to adjust, and Perplexity gives a similar window. Do not judge a change on the day you make it.

Verify who is really visiting

User agents can be faked, so verify the important ones by IP. OpenAI publishes ranges for OAI-SearchBot, GPTBot and ChatGPT-User, and Perplexity publishes ranges for PerplexityBot and Perplexity-User. Filter your logs by user agent first, then compare IPs with the published lists.

# count requests per AI user agent in an access log
grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|PerplexityBot" access.log \
  | awk '{print $1}' | sort | uniq -c | sort -rn | head

How the Growmerz robots.txt is built

Our robots.txt is generated from code. It allows all crawlers, disallows only our internal /api/ paths, and then lists the AI and search crawlers by name: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot and Bingbot. The named group repeats the /api/ rule on purpose. Under the Robots Exclusion Protocol (RFC 9309), a crawler follows the most specific group that matches its name and ignores the generic * group, so any rule you want to keep must be repeated. It is a common source of accidental exposure. We chose to stay open because visibility in AI answers matters more to us than blocking training, and you may choose differently.

Test before and after you deploy

1. Open /robots.txt in a browser and read it as a crawler would.
2. Use Google Search Console's robots.txt report for Googlebot.
3. Run curl -A "OAI-SearchBot" https://www.example.com/ and confirm the real page comes back.
4. Check CDN and firewall bot settings, which can block AI user agents even when robots.txt allows them.
5. Re-check logs after 48 hours.

Related guides

• llms.txt Explained: Should You Add One to Your Website?
• Google AI Overviews: How to Get Your Content Cited
• Answer-First Writing: How to Structure Pages So AI Can Quote Them
• Entity SEO for AI: How Search and AI Decide Who Your Business Is
• Service: AI Chatbots and Knowledge Assistants

Frequently asked questions

Should I block GPTBot?

Only if you want to opt out of training. Blocking GPTBot alone does not stop ChatGPT search, which uses a different crawler. Check OpenAI's documentation for the current setup.

Does blocking Google-Extended hurt my SEO?

Google states it does not affect inclusion or ranking in Google Search. It limits some use of your content in certain Google AI products.

Will blocking AI crawlers stop my content appearing in AI answers?

Blocking search and user-fetch crawlers reduces your chances in live answers. Content already in a model's training data stays there.

How do I know which bots visit my site?

Check your server or CDN logs and filter by user agent. Remember that user agents can be spoofed. Providers publish IP ranges for verification.

Do I need llms.txt as well?

No. llms.txt is optional and does not control access. See our llms.txt explainer.

Does Applebot-Extended matter?

It is a robots.txt control token Apple uses for opting content out of training its AI models. It does not change whether Apple's search features can index your site.

Can I allow one folder and block the rest?

Yes. Use Allow and Disallow rules under a named user agent. Test carefully, because rule precedence trips people up.

Sources and further reading

Primary documentation and research behind this guide.

• OpenAI: Overview of OpenAI crawlers
• Perplexity: Bots and crawlers guide
• RFC 9309: Robots Exclusion Protocol
• Google Search Central: Introduction to robots.txt

If you are unsure what your firewall, CDN and robots.txt are doing to AI crawlers, we can check all three. Start with a free audit or see our AEO and GEO service.