Noindex
Noindex is a robots directive, set in a meta robots tag or an X-Robots-Tag HTTP header, that tells search engines not to show a page in their results. The page must remain crawlable, otherwise the directive is never seen.
A noindex directive tells search engines: you may visit this page, but do not include it in your search results. It is the standard way to keep pages out of Google that need to exist for users but have no value in search.
Two ways to set it:
- A meta robots tag in the HTML head, with the content value noindex. You can target all crawlers with name="robots" or one specific crawler, such as name="googlebot".
- An X-Robots-Tag HTTP header with the value noindex. This works for non-HTML files such as PDFs and images, and can be set for whole folders in the server configuration.
Typical pages to noindex:
- Internal search result pages.
- Thank-you pages, login areas, shopping carts and account pages.
- Thin tag or archive pages that duplicate content found elsewhere.
- Filtered or sorted product views that you do not want indexed but cannot block entirely.
- Staging and test environments, although password protection is a safer choice there.
The robots.txt trap. Google has to crawl a page to read the noindex. If you block the same URL in robots.txt, Googlebot never sees the directive, and the URL can stay indexed (usually without a description) when other pages link to it. To remove a page from Google, allow crawling and add noindex. Once it has dropped out, you can decide whether to block crawling as well. Since 2019 Google no longer supports a noindex rule inside robots.txt itself.
Noindex and links. On a noindexed page, Google can still follow links at first. Google has said that pages left on noindex for a long time are eventually crawled less and their links may stop being followed. So do not rely on noindexed pages as an important part of your internal linking structure.
Common mistakes:
- A site launched with the "discourage search engines" setting still on from the development phase, keeping the whole site out of Google. Check this after every launch and migration.
- Noindex combined with a canonical to another URL. These are contradictory signals; pick one.
- Noindexed pages listed in the XML sitemap.
- JavaScript that removes a noindex after the page loads. If the raw HTML says noindex, Google may skip rendering and keep the page out.
- Noindexing paginated category pages, which can stop products or articles deeper in the series from being discovered.
How to check. The URL Inspection tool in Google Search Console shows whether indexing is allowed for a URL and why. The Page indexing report lists every page "Excluded by noindex tag". Scan that list regularly: important pages ending up there by accident is one of the most common technical SEO errors.
For publishers, a sponsored article must never carry noindex. A noindexed article will not appear in search and passes very little value, which defeats the purpose of the placement.
Questions about Noindex
What is the difference between noindex and robots.txt Disallow?
Disallow stops crawling but does not guarantee removal from the index. Noindex lets Google crawl the page but keeps it out of search results. For removal, use noindex and leave the page crawlable.
How long does it take for a noindexed page to disappear from Google?
Once Google recrawls the page and sees the directive, it is dropped. That can take days to weeks, depending on how often the page is crawled. Requesting indexing in URL Inspection can speed up the recrawl.
Does Google follow links on a noindex page?
Initially yes, but pages left on noindex for a long time tend to be crawled less, and Google may stop following their links.