List of AI crawlers in 2026: which ones to allow

The AI crawlers a website sees in 2026 come in three kinds: training crawlers such as GPTBot, ClaudeBot and CCBot, search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, and live fetchers such as ChatGPT-User that open a page while someone chats. A business that wants AI answers to cite it should allow the search crawlers and live fetchers, then decide about training separately. Google-Extended and Applebot-Extended are not crawlers at all: they are robots.txt tokens that control how Google and Apple may use your pages for AI.

Below: the list, robots.txt lines for each policy, and Cloudflare's 2026 changes. For how AI search picks its sources, read how to rank in ChatGPT and AI search.

List of AI crawlers in 2026, with robots.txt tokens

The list of AI crawlers below covers the AI bots the major companies document as of October 2026, taken from the crawler pages of OpenAI, Anthropic, Perplexity, Google, Apple and the others named. The token is the name that goes after User-agent: in robots.txt. The last column reports what each company says, not what anyone has measured.

ByteDance, TikTok's owner, is missing: it publishes no crawler documentation we could find. In October 2024, Fortune reported research by the bot management firm Kasada finding that its crawler, Bytespider, ignored robots.txt.

Training crawlers, search crawlers and live fetchers: what's the difference?

AI crawlers differ by what happens to your page after the visit. A training crawler collects pages that may end up inside a future AI model. A search crawler builds the index an assistant searches when it answers, which is how a page gets cited with a link. A live fetcher opens one page because a person asked, right now.

Each kind has its own switch. OpenAI says "each setting is independent of the others", so a site can disallow GPTBot and still appear in ChatGPT search. Anthropic, Perplexity, Meta, Amazon and Mistral split their bots the same way.

Live fetchers are the awkward group: OpenAI, Perplexity, Google, Meta and Amazon say theirs may not follow robots.txt, because a person started the request. Anthropic says Claude-User follows it.

What Google-Extended and Applebot-Extended actually block

Google-Extended is a robots.txt token, not a crawler. Google says it "doesn't have a separate HTTP request user agent string": Googlebot does the crawling, and the token only tells Google how it may use what Googlebot fetched.

Disallowing Google-Extended keeps your pages out of training for future Gemini models and out of grounding, the live lookups behind answers in the Gemini app and Vertex AI, Google's cloud AI platform. Google says it does not affect inclusion in Search and is not a ranking signal.

Google-Extended also does not keep you out of AI Overviews or AI Mode, which draw on the Search index. Those have their own controls: the nosnippet tag, and a generative AI setting in Search Console that Google said reached every site on August 31, 2026. See our guide to appearing in Google AI Overviews.

Applebot-Extended works the same way for Apple. Apple says it "does not crawl webpages" and only decides whether pages Applebot collected may train Apple's foundation models. Pages that disallow it can still appear in Siri, Spotlight and Safari search, and the rule is not used in ranking. Blocking Applebot itself is different: that stops the crawling behind Apple's search features.

Do AI crawlers respect robots.txt?

Most AI crawlers from the major companies say they respect robots.txt, but robots.txt is a request, not a lock. Nothing technical stops a crawler that ignores it, and any scraper can borrow a well-known name. Treat the table's last column as stated policy, then check your server logs, matching each visit's IP address against the company's published list.

Those policies have been tested in public. In August 2025, Cloudflare accused Perplexity of using undeclared crawlers posing as a Chrome browser to reach sites that had blocked it, and removed Perplexity from its verified bots. Perplexity called the analysis flawed, saying Cloudflare had confused user-requested fetches with crawling. In the same post, Cloudflare named OpenAI as an example of good practice: ChatGPT-User stopped when disallowed.

robots.txt also only works forward: Anthropic says a ClaudeBot block excludes a site's "future materials" from training.

Should you block AI crawlers?

Most business websites should not block AI search crawlers, because that removes them from answers that could send customers. Blocking training crawlers is a separate, smaller decision. OpenAI and Apple say it keeps pages out of future models without hiding them from search, and Anthropic runs a separate bot for each job.

The case for blocking training is about value. Cloudflare reported that 52% of the crawler requests it saw in June 2026 were for AI training, up from 22% in spring 2025, and training sends no visitors back. Publishers that sell or license their writing have good reasons to say no.

The case for allowing it is unproven: a model that learned about your business may describe it correctly without searching. No AI company documents that effect, so treat it as a bet.

Two traps catch sites that block. Google-Extended also controls grounding in the Gemini app, so blocking it can cost Gemini app answers. And where live fetchers obey a block, assistants can no longer open your page when someone asks about you by name. Cloudflare said in September 2026 that fewer than 1% of sites on its network block search crawlers, while 17% restrict AI training.

robots.txt examples for AI crawlers

robots.txt is a plain text file at the root of a domain, such as yoursite.com/robots.txt, and each subdomain needs its own. One rule trips people up: a crawler follows only the group that names it most specifically and ignores the User-agent: * group, as Google's robots.txt documentation explains. If you name a bot, repeat any Disallow lines it should still obey.

Allow every AI crawler

QuQi's own robots.txt allows all. After User-agent: * and Allow: /, it names eleven AI tokens in one group, repeating its two private paths:

• User-agent: GPTBot
• User-agent: OAI-SearchBot
• User-agent: ChatGPT-User
• User-agent: ClaudeBot
• User-agent: Claude-SearchBot
• User-agent: Claude-User
• User-agent: PerplexityBot
• User-agent: Perplexity-User
• User-agent: Google-Extended
• User-agent: Applebot-Extended
• User-agent: CCBot
• Allow: /
• Disallow: /admin
• Disallow: /api/

Block AI training, stay in AI search

Add a group that disallows only the training tokens, and leave everything else alone:

• User-agent: GPTBot
• User-agent: ClaudeBot
• User-agent: Google-Extended
• User-agent: Applebot-Extended
• User-agent: meta-externalagent
• User-agent: MistralAI-Training
• User-agent: CCBot
• Disallow: /

Amazonbot is a judgement call: Amazon says it may train Amazon's AI models but also serves other Amazon features, while Amzn-SearchBot handles search. Bing has no robots.txt training token. Its control is the noarchive page tag, which since 2023 keeps a page out of Bing's chat answers, now Copilot, and out of Microsoft's model training. Cloudflare reported that a robots.txt option from Microsoft is planned for early 2027.

Block every AI crawler

Put every token from the table except Googlebot, Bingbot and Applebot in one group ending in Disallow: /. That still leaves three things open:

• The AI features of Google, Bing and Apple, which use their search crawlers. Those answer to Search Console's setting, nosnippet and noarchive, not robots.txt.
• Live fetchers that may ignore robots.txt. Only a firewall or CDN rule stops them.
• Anything collected before the rule went up.

Does Cloudflare block AI crawlers by default?

Cloudflare blocks some AI crawlers by default on new sites. Since September 15, 2026, new domains get a default that blocks AI training and AI agent bots on pages that display ads, while search crawlers stay allowed. Sites without ads are offered a preset that allows all three.

• July 1, 2025: Cloudflare began blocking AI training crawlers by default on new domains.
• July 1, 2026: AI traffic split into Search, Agent and Training, each set separately by every customer, free plan included.
• September 15, 2026: the ad-aware defaults took effect and Disallow AI Training launched. The old Block AI bots switch gave way to the three controls, and managed robots.txt to Bot Preference Sync, announced in August.

The trap is choosing Block for Training. Googlebot, Bingbot and Applebot crawl for both search and training, and Cloudflare applies the strictest rule, so that choice blocks them and can take a site out of Google. Disallow AI Training avoids it by publishing a no-training preference in robots.txt, Google-Extended and Applebot-Extended included, while search crawling stays open. Bot Preference Sync writes such lines above your own rules, so check your live robots.txt.

How QuQi helps

Letting search crawlers in makes a site eligible, not named. QuQi's AI Visibility teammate checks every week whether Google's AI Overview, ChatGPT, Perplexity and Gemini name your business for the questions your buyers ask, records who is named instead, and turns each gap into a brief for the Copywriter. The SEO Auditor audits the site and puts what costs the most traffic first. Every batch runs in a chat you can open and step into. See pricing.

Questions people ask