AI crawler

An AI crawler is a bot that visits websites on behalf of an AI company, either to collect content for training language models or to fetch pages in real time so an AI assistant can answer a question.

AI crawlers are automated programs that read web pages for AI systems. They work much like search engine crawlers such as Googlebot, but the content they collect ends up in language models or AI answers instead of, or as well as, a classic search index.

Types of AI crawlers

It helps to separate three kinds, because they have different effects on your site:

  • Training crawlers collect content to train future models. Examples are GPTBot (OpenAI), ClaudeBot (Anthropic) and CCBot (Common Crawl, whose archive is widely used for training).
  • Search crawlers build or refresh an index that AI assistants use to find and cite sources. Examples are OAI-SearchBot for ChatGPT search and PerplexityBot for Perplexity.
  • User-triggered fetchers visit a specific page because a user asked the assistant to read it or the assistant needs it during a conversation. Examples include ChatGPT-User and Perplexity-User.

Google is a special case. AI Overviews and AI Mode use the regular Googlebot index, so you cannot block them without leaving Google Search. The separate Google-Extended token in robots.txt only controls whether your content may be used for Gemini model training and grounding outside Search.

How to manage AI crawlers

Most AI companies respect robots.txt. You can allow or block each bot by its user agent name. Many sites choose to allow search and user-triggered bots, because those lead to citations and visits, while deciding case by case on training bots. Check your CDN or firewall too: some security settings block AI bots by default without the site owner knowing.

Server logs show which bots visit, how often and which pages they request. That is the most reliable way to see whether AI crawlers can actually reach your content.

Why it matters

If AI crawlers cannot access your pages, AI assistants cannot cite them. Many AI crawlers also do not execute JavaScript the way Googlebot does, so content that only appears after scripts run may be invisible to them.

Practical tips

  • Review robots.txt and CDN bot settings at least once a year, as new bots appear regularly.
  • Serve important content in the initial HTML, not only through client-side JavaScript.
  • Watch for fake bots that use AI crawler names. Verify by IP ranges that the providers publish.

Common mistakes

Blocking every AI bot with one rule and then wondering why the site never appears in ChatGPT or Perplexity, or assuming that blocking a training bot removes content from models that were already trained.

Questions about AI crawler

Should I block AI crawlers?

It depends on your goals. Blocking search and user-triggered bots stops AI assistants from citing you, while blocking training bots only affects future model training.

How do I block GPTBot?

Add a robots.txt rule with User-agent: GPTBot followed by Disallow: /. This stops OpenAI from using your pages for training, but does not affect OAI-SearchBot.

Can I block Google AI Overviews?

Not separately. AI Overviews use the normal Googlebot index. Google-Extended only controls use of your content for Gemini training and grounding outside Search.

Related terms