AI crawlers and robots.txt: which bots to allow or block

AI companies send three different kinds of bots to your website, and they do three different jobs. Training crawlers collect content that may be used to train AI models. Search crawlers build the index an assistant uses to find and link to pages. User-triggered fetchers load a page because a person asked the assistant about it. The vendors give each one its own name, called a user-agent token, so you can allow or block them separately in your robots.txt file.

For most businesses that want to be found, the sensible default is to allow the search crawlers and user-triggered fetchers, and to make a deliberate choice about training crawlers. Blocking a search crawler has a documented cost. OpenAI, for example, says sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers." Blocking a training crawler is a policy decision about your content. OpenAI and Perplexity both say each of their settings works independently, so a training opt-out does not remove you from search.

The tables below list the crawlers each vendor documents as of October 2026, by exact token.

Three kinds of AI bots

The names look alike, so it is easy to block a search crawler when you only meant to opt out of training.

The AI assistants' crawlers

These are the tokens documented by OpenAI, Anthropic and Perplexity. Use the token exactly as written.

OpenAI

OpenAI's crawler documentation gives the clearest statement of the split: "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training." OpenAI recommends allowing OAI-SearchBot and allowing requests from its published IP ranges. After you change robots.txt, it says search results can take about 24 hours to adjust.

Anthropic

Anthropic's crawler article says its bots honor robots.txt, and it states no exception for its user-triggered agent. It asks you to set the opt-out on every subdomain you want covered. It advises against blocking by IP address, because that stops its bots from reading your robots.txt at all.

Perplexity

Perplexity's bot documentation says changes may take up to 24 hours to be reflected, and that sites behind a web application firewall may need to explicitly allow its bots.

Google works differently

Google does not have a separate crawler for AI Overviews or AI Mode. Its documentation says: "AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search." If you block Googlebot, you are blocking Google Search.

That leaves four separate controls, each with a different job.

Two points catch people out. First, Google-Extended is not a bot you will see in your logs. Google's crawler documentation says it "doesn't have a separate HTTP request user agent string" and that the token "is used in a control capacity." Second, blocking it does not take you out of AI Overviews or AI Mode. It does opt you out of grounding in the Gemini app, which is a visibility cost to weigh before you add the line.

Google also runs user-triggered fetchers, including Google-Agent, which agents hosted on Google infrastructure use to browse and act on a user's request. Google says these fetchers "generally ignore robots.txt rules." The Search Console control and the reporting that goes with it are covered in our guide to Google AI Mode.

Microsoft, Apple, Meta, Amazon and Common Crawl

Microsoft is the outlier. Because Copilot draws on the Bing index, the way to limit Copilot is with page-level meta tags, not a robots token. Bing's webmaster guidelines say NOARCHIVE "prevents content from being used in Copilot responses and grounding results," and NOCACHE limits Copilot to the URL, title and snippet. We could not find a Microsoft-documented robots.txt token that blocks Copilot or AI training without also blocking Bing search. Our guide to Bing Webmaster Tools and IndexNow covers the Microsoft side.

Apple's Applebot page adds one more control: the nosnippet meta tag keeps content out of Apple's AI-generated answers.

Crawlers that are not documented

You will see other bot names in block lists, including Bytespider, generally attributed to ByteDance. We could not find working first-party documentation for it as of October 2026, so its token, purpose and robots.txt compliance are unverified and it is left out of the tables above. For a bot with no operator documentation, a robots.txt line is a guess, and a firewall rule is the more reliable tool.

Sample robots.txt rules

This is an example, not a recommendation for every site. If you are new to the file itself, start with our robots.txt guide.

Allow AI search, opt out of model training

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

This leaves OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot untouched, provided no other rule in your file blocks them. CCBot is on the list because Common Crawl publishes an open dataset that anyone can use. Three choices are left out on purpose because each has a trade-off: Google-Extended also governs grounding in the Gemini app, meta-externalagent covers indexing as well as training, and Amazonbot is used to improve Amazon's products as well as for possible training.

Remember that robots.txt is public and is not a security measure. Anything private belongs behind a login.

Robots.txt is a request

Every rule above depends on the bot choosing to follow it. Google's introduction to robots.txt puts it this way: "The instructions in robots.txt files cannot enforce crawler behavior to your site; it's up to the crawler to obey them."

Most of the companies in this guide either state that their automated crawlers honor the file or name it as the way to control them. Meta's page is less explicit: it tells you to add a disallow line to block its crawlers. Three limits still apply.

• User-triggered fetchers are often exempt. OpenAI, Perplexity, Google, Meta and Amazon (whose agent is Amzn-User) each say their user-triggered agents may not follow robots.txt.
• User-agent strings can be faked. Common Crawl warns that it is "aware of crawlers falsely identifying themselves as CCBot." OpenAI, Anthropic, Perplexity, Google, Bing, Apple, Amazon and Common Crawl publish IP ranges so you can verify real requests.
• Changes are not instant. OpenAI, Perplexity, Amazon and Meta each cite about 24 hours. Amazon also says its crawlers may use a cached copy of your file from the last 30 days.

If you need a hard block, do it at the server, CDN or firewall.

Check your CDN and firewall too

A clean robots.txt does not help if another layer is turning bots away. OpenAI's eligibility guidance asks you to confirm that your host or content delivery network allows traffic from its published search bot IP addresses. Perplexity gives the same advice for web application firewalls. Google lists crawl access "by any CDN or hosting infrastructure" among its best practices for AI features.

1. Open yourdomain.com/robots.txt and read every group, including any wildcard block.
2. Check your CDN or security plugin for a bot-blocking or "AI bots" setting someone may have switched on.
3. Look in your server logs for the tokens in the tables above and note the response codes they receive.

This check is part of any proper technical audit for GEO.

Conclusion

A robots rule decides who may fetch your pages; it does not earn a citation, and it is not the same thing as an llms.txt file. Treat AI crawlers as three groups. Allow the search crawlers if you want to be found, expect most user-triggered fetchers to arrive regardless, and make a deliberate choice about training crawlers using each vendor's own token. Handle Google separately, because its AI features run on Googlebot and are managed through snippet controls and Search Console, and handle Copilot through Bing's meta tags. If you are not sure what your site is telling these bots today, request a free GEO audit. More guides are in our GEO and AEO Guide.

Frequently asked questions