Indexing

Indexing is the step where a search engine processes a crawled page, understands its content and stores it in its database. Only indexed pages can appear in search results and be picked up by AI features built on that index.

After a page is crawled, Google analyses it: it reads the text, title, headings, images, structured data and links, works out the language and main topic, and checks whether it duplicates another URL. If the page passes, it is added to the index, a huge database of documents that Google searches through when someone types a query. That process is indexing.

Indexing is not guaranteed. Google indexes only what it thinks is worth storing. Typical reasons a page stays out:

  • Noindex in a meta robots tag or X-Robots-Tag header.
  • Canonicalisation: Google picked another URL as the main version.
  • Duplicate or thin content that adds little compared to pages already indexed.
  • Crawled, currently not indexed: Google fetched it but decided not to keep it, often a quality signal.
  • Discovered, currently not indexed: Google knows the URL exists but has not crawled it yet, often a crawl capacity or internal linking issue.
  • Soft 404s, server errors or redirects.

Why it matters beyond classic search. Google's AI Overviews and AI Mode draw on the same search index. A page that is not indexed cannot be shown as a source there either. Other AI assistants that search the web rely on their own indexes or on a search partner, so being properly indexed in more than one search engine improves your chances of being cited.

How to check. The Page indexing report in Google Search Console lists indexed and non-indexed pages with the reason for each. The URL Inspection tool shows the status of one URL, the canonical Google selected and whether the page can be indexed right now. A site: search gives only a rough sample and should not be used as an exact count.

Practical tips:

  • Do not panic when not every URL is indexed. Tag pages, filtered views and paginated archives often should stay out.
  • Focus on the pages that matter: are your key categories, products and articles indexed?
  • For "crawled, currently not indexed" on important pages, improve the content and internal links rather than repeatedly clicking Request indexing.
  • Keep sitemaps clean so they list only indexable, canonical URLs that return a 200 status.

For publishers, indexing is a basic quality check. A sponsored article that is not indexed passes little value to the advertiser's link, so it is worth confirming that new articles get indexed within a reasonable time.

Questions about Indexing

How long does it take for Google to index a new page?

It ranges from a few hours to several weeks. Pages on sites that Google crawls often, with good internal links, usually get indexed fastest.

Why is my page crawled but not indexed?

Google fetched it but decided it was not worth storing, often because the content is thin, duplicates other pages or the site has broader quality issues. Improving the content and linking to it from strong pages usually helps.

Can I force Google to index a page?

No. You can request indexing in the URL Inspection tool, but Google still decides whether to include the page.

Related terms