
Quick answer
The robots meta tag is an HTML tag in a page’s head that tells search engines how to index and display that page. Its most common directives are noindex, which keeps a page out of search results, and nofollow, which asks crawlers not to follow the page’s links. Others control snippets and previews. For non-HTML files, the same directives are sent in an X-Robots-Tag HTTP header. The page must not be blocked by robots.txt, or crawlers never see the tag.

Some pages should not appear in search results. Thank-you pages, internal search results, staging copies, and thin tag archives are common examples. The robots meta tag is the standard way to tell search engines to leave them out.
This guide explains what the robots meta tag does, every directive Google supports, how it differs from robots.txt, how to use the X-Robots-Tag header for files, and the mistakes that most often remove the wrong pages from search.
In this guide
- What Is the Robots Meta Tag?
- Robots Meta Tag Directives Google Supports
- Robots Meta Tag vs Robots.txt
- When to Use Noindex
- When to Use Nofollow at Page Level
- Snippet and Preview Controls
- Using the X-Robots-Tag Header
- How to Add a Robots Meta Tag in WordPress
- Removing Pages From Search Quickly
- How to Check Robots Meta Tags
- Frequently asked questions
What Is the Robots Meta Tag?
The robots meta tag is a meta element in the head section of an HTML page. It has a name attribute, usually robots, and a content attribute listing one or more directives separated by commas. A tag with the content noindex, for example, tells search engines not to show the page in their results.
The name can also target a specific crawler. A name of googlebot applies the rules only to Google’s crawler, while robots applies them to all search engines that support the tag. If both are present, Google combines the restrictive rules from each.
By default, with no robots meta tag at all, Google treats a page as having no restrictions. Google’s documentation calls this default all. There is no need to add index or follow, because they are already assumed.
Robots Meta Tag Directives Google Supports
| Directive | What it does |
|---|---|
| noindex | Keeps the page out of search results |
| nofollow | Asks Google not to follow the links on the page |
| none | Equivalent to noindex and nofollow together |
| nosnippet | Prevents text snippets and video previews in results |
| indexifembedded | Allows content embedded through iframes to be indexed despite noindex |
| max-snippet | Limits snippet length to a set number of characters |
| max-image-preview | Sets the largest image preview: none, standard, or large |
| max-video-preview | Limits video previews to a set number of seconds |
| notranslate | Stops Google offering a translation of the page in results |
| noimageindex | Prevents images on the page from being indexed |
| unavailable_after | Removes the page from results after a specified date and time |
Google’s documentation notes that some older directives are no longer used. Noarchive and nocache related to cached page links, which Google no longer offers, and nositelinkssearchbox related to the sitelinks search box, which Google retired. Google ignores all three.
When rules conflict, Google applies the more restrictive one. A page with both max-snippet set to 50 and nosnippet will show no snippet.
Robots Meta Tag vs Robots.txt
These two are often confused, but they do different jobs.
| Tool | Controls | Where it lives |
|---|---|---|
| robots.txt | Whether crawlers may fetch a URL | A file at the root of the site |
| Robots meta tag | Whether a crawled page is indexed or shown, and how | The head of each HTML page |
| X-Robots-Tag | The same as the meta tag, for any file type | The HTTP response header |
The critical point is that they interact. If robots.txt blocks a URL, Google never fetches the page, so it never sees a noindex tag on it. The URL can still appear in results if other pages link to it, typically with no description.
The Rule
To remove a page from search, allow it to be crawled and add noindex. Do not block it in robots.txt at the same time, or Google will never see the noindex.
When to Use Noindex
Noindex suits pages that serve users but add nothing to search results.
- Thank-you and order confirmation pages.
- Internal site search results pages.
- Login, account, and cart pages.
- Thin tag or date archives that duplicate category content.
- Staging and test versions of a site, ideally also protected by a password.
- Campaign landing pages meant only for paid traffic.
Be cautious with category and product pages on shops. Filtered and sorted variations are often better handled with canonical tags or crawl controls, while the main category page should stay indexable, because it is frequently one of the strongest pages on the site.
Do not noindex pages you want to rank, and avoid using noindex to choose between duplicate versions of the same page. Google’s documentation recommends canonical tags for that, because noindex removes the page entirely.
When to Use Nofollow at Page Level
The page-level nofollow directive asks Google not to follow any link on the page. It is rarely needed. Most situations calling for nofollow involve individual links, such as paid or user-generated ones, which should use rel attributes on those links instead.
Page-level nofollow can make sense on pages full of untrusted links, such as unmoderated user submissions, but in most cases it is better to qualify the specific links.
Snippet and Preview Controls
The snippet directives control how a page appears rather than whether it appears. Nosnippet removes the text description. Max-snippet limits it. Max-image-preview sets how large an image thumbnail may be, and large previews can help pages appear in Discover. Max-video-preview limits video previews.
These controls also apply to featured snippets and to Google’s AI features in Search, which treat content as eligible according to the same snippet rules. Restricting snippets can therefore reduce visibility in several places at once.
Using the X-Robots-Tag Header
The X-Robots-Tag is an HTTP header that carries the same directives as the meta tag. It is the only way to apply them to files without an HTML head, such as PDFs, images, and video files. It can also be applied to many URLs at once through server configuration.
Example
A company publishes price lists as PDFs that change monthly. To keep old versions out of search, it configures the server to send an X-Robots-Tag header with noindex for every PDF in its archive folder, while leaving current price list pages indexable.
How to Add a Robots Meta Tag in WordPress
WordPress has a site-wide setting under Settings, Reading, called Discourage search engines from indexing this site. When ticked, it adds a noindex robots tag to every page. It is useful on staging sites and dangerous on live ones.
For individual pages, most SEO plugins add a robots setting to the post editor. Many also let you noindex whole groups, such as tag archives or author pages, from their settings.
Common Mistake
Launching a site with the Discourage search engines setting still ticked from development. The whole site is noindexed, and traffic falls to zero until it is fixed. Check this setting after every launch or migration.
Removing Pages From Search Quickly
Noindex works on Google’s schedule, because the page has to be recrawled before the tag takes effect. When a page must disappear quickly, for example because it was published by mistake or contains private information, use the Removals tool in Google Search Console. It hides the URL from results for about six months while a permanent fix, such as noindex, a password, or deleting the page, takes effect.
For content that should never be public, noindex is not enough on its own. Protect it with a password or remove it from the server. A noindexed page is still accessible to anyone who has the link.
How to Check Robots Meta Tags
Check the HTTP headers as well as the HTML. A noindex can be sent in an X-Robots-Tag header by a server rule, a plugin, or a CDN without appearing anywhere in the page source, which makes it easy to miss.
The URL Inspection tool in Google Search Console shows whether Google found a noindex directive on a page and whether the page is indexed. The Page indexing report lists every URL excluded by a noindex tag, which makes it easy to spot pages excluded by mistake. Crawlers such as Screaming Frog report robots directives for every page and header across a site.
Key takeaways
- The robots meta tag tells search engines how to index and display a page.
- Noindex keeps a page out of results; nofollow asks crawlers not to follow its links.
- Google ignores noarchive, nocache, and nositelinkssearchbox, which relate to retired features.
- Never block a noindexed page in robots.txt, or Google will not see the tag.
- Use the X-Robots-Tag header for PDFs and other files without an HTML head.
Frequently Asked Questions
What does the robots meta tag do?
It tells search engines whether to index a page, whether to follow its links, and how to show its snippet.
What is the difference between noindex and robots.txt?
Robots.txt controls crawling. Noindex controls indexing. A page blocked in robots.txt cannot have its noindex seen.
Do I need to add index, follow to my pages?
No. With no robots meta tag, Google already treats the page as indexable with followable links.
How do I noindex a PDF?
Send an X-Robots-Tag HTTP header with noindex from your server for that file.
How long does noindex take to work?
Google needs to recrawl the page. Once it sees the tag, it drops the page from results, often within days or weeks.
Related guides
Continue learning
Sources
Researched from Google documentation, published studies and hands-on experience, then reviewed for accuracy before publishing. Learn how we work in our editorial policy.

