robots.txt

robots.txt is a plain text file in the root of a website that tells crawlers which paths they may or may not fetch. It controls crawling, not indexing, and is also where you allow or block specific AI crawlers.

The robots.txt file lives at the root of a host, for example example.com/robots.txt. Before crawling, well-behaved bots read it to see which parts of the site they are allowed to request. It follows the Robots Exclusion Protocol, which was formalised as an internet standard in 2022.

A file is built from groups. Each group starts with a User-agent line naming a crawler (or * for all), followed by Allow and Disallow rules. You can also add one or more Sitemap lines pointing to your XML sitemaps. For example, a group with User-agent: * and Disallow: /search/ tells all bots to stay out of internal search results.

What robots.txt does not do:

  • It does not remove pages from the index. A blocked URL can still be indexed (usually without a description) if other sites link to it.
  • It does not hide content. The file is public, so never use it to "hide" private directories.
  • Google does not support noindex inside robots.txt and ignores the crawl-delay directive.
  • It is a request, not a lock. Bad bots simply ignore it.

If you want a page out of Google, let it be crawled and add a noindex tag. Blocking it in robots.txt at the same time means Google never sees the noindex.

robots.txt and AI crawlers. Since 2023 this file has become the main place to decide how AI companies may use your content. Most large AI providers publish separate user agents for training and for live retrieval. Google-Extended, for instance, is a token that controls whether your content is used for Gemini models, without affecting Google Search. Blocking a training crawler while allowing a search or retrieval crawler is a common middle ground for sites that want to appear as a source in AI answers without donating content to training. Check each provider's documentation, because names and behaviour change.

Common mistakes:

  • A leftover Disallow: / from a staging site that blocks the whole live site.
  • Blocking CSS and JavaScript files, which stops Google from rendering the page properly.
  • Blocking a URL that carries a noindex or a canonical you want Google to see.
  • Forgetting that each subdomain and protocol needs its own robots.txt.
  • Rules that are too broad, because paths are matched as prefixes: Disallow: /blog also blocks /blog-archive.

Google reads up to 500 KiB of the file and caches it for up to a day. Use the robots.txt report in Google Search Console to see which version Google fetched and whether it found errors.

Questions about robots.txt

Does blocking a page in robots.txt remove it from Google?

No. It only stops crawling. To remove a page from the index, allow crawling and add a noindex tag, or use the Removals tool for a temporary hide.

How do I block AI crawlers with robots.txt?

Add a User-agent group for each AI bot you want to block, with Disallow: /. Use each provider's documented user agent name and remember that training and search bots often have different names.

Where should robots.txt be placed?

In the root of each host, for example https://www.example.com/robots.txt. A file in a subfolder is ignored, and every subdomain needs its own.

Related terms