Noindex, meta robots and X-Robots-Tag: how to check what a page tells Google
A page can ask search engines to keep it out of results in three places: a robots meta tag, a googlebot meta tag, and an X-Robots-Tag HTTP header. Google reads all of them together, and when they disagree the more restrictive rule wins. So "index" in the HTML means nothing if the server sends X-Robots-Tag: noindex. To check a page, look at the page source and the response headers, or run the snippet below.
Where the instructions live
Most people check the meta tag and stop there. The header is invisible on the page, and it's often set by a server or CDN rule that nobody on the SEO side has seen.
<meta name="robots" content="index, follow">not setX-Robots-Tag: noindexThe HTML says "index", the header says "noindex", so the page is noindexed. A googlebot tag doesn't override robots either: Google adds up the restrictive rules from both.
What each directive does
index, follow Google ignores them. A page with no robots rules can already be indexed and its links followed. If a page says both index and noindex, Google applies the more restrictive one: noindex.noindex Don't show this page in search results. Google still has to crawl the page to see it.nofollow Don't follow the links on this page. Pages that are only linked from here may be found more slowly.none Short for noindex, nofollow. Easy to miss in an audit because it doesn't contain the word "noindex".nosnippet, max-snippet Control the text shown under the result. They don't affect whether the page is indexed.The header can also name a crawler, such as X-Robots-Tag: googlebot: noindex, which applies only to Google. On files that have no HTML, such as PDFs, the header is the only way to say noindex.
Noindex vs robots.txt
These do different jobs, and using both on one page is a classic mistake:
| Google crawls it? | Can it appear in results? | Use it for | |
|---|---|---|---|
noindex | Yes | No | Pages that must stay out of results |
robots.txt Disallow | No | Yes, as a bare URL if other pages link to it | Keeping crawlers off pages that don't matter, such as internal search |
| Both together | No | Yes: Google never sees the noindex | Never |
If a page is blocked in robots.txt, Google can't read the page, so it can't find the noindex on it. To get a page out of results, allow crawling and use noindex.
Noindex and JavaScript
Google warns that when it finds noindex in the HTML your server sends, it may skip running JavaScript on the page altogether. So if the server sends noindex and a script removes it after loading, Google may never see the script's version. The page stays out.
The other direction is safer: Google itself suggests adding noindex with JavaScript to error pages in single-page apps, though crawlers that don't run JavaScript won't see it. The safe rule is to make the server HTML say what you mean, and check it separately from what the browser renders.
How to check by hand
1Check all three places at once 1 minute
Open DevTools (F12), go to the Console and paste:
// What this page tells search engines: server HTML, rendered page and HTTP header
const res = await fetch(location.href, { cache: 'no-store' });
const server = new DOMParser().parseFromString(await res.text(), 'text/html');
const metas = doc => [...doc.querySelectorAll('meta[name="robots" i], meta[name="googlebot" i]')]
.map(m => m.getAttribute('name') + ': ' + m.getAttribute('content'));
const header = res.headers.get('x-robots-tag') || '';
// Directives as tokens; a 'bingbot:' or similar prefix in the header aims the rule at another crawler
const tokens = [...metas(server), ...metas(document), header].join(',').toLowerCase()
.split(',').map(t => t.trim().replace(/^(robots|googlebot):\s*/, ''))
.filter(t => !/^[\w-]+:\s/.test(t)).flatMap(t => t.split(/\s+/));
const has = d => tokens.includes(d) || tokens.includes('none');
console.table({
'Server HTML': metas(server).join(' | ') || '(none)',
'Rendered page': metas(document).join(' | ') || '(none)',
'X-Robots-Tag header': header || '(none)',
'Google applies': (has('noindex') ? 'noindex' : 'index') + ', ' + (has('nofollow') ? 'nofollow' : 'follow'),
});You get one row for the server HTML, one for the rendered page, one for the header, and what Google ends up applying. If the server HTML and rendered rows differ, JavaScript is changing the robots tag. If they differ, see "Noindex and JavaScript" above: Google may only act on the server HTML.
It re-requests the page, so it sees what your browser is sent. Some sites send Googlebot something different. A header rule aimed at another crawler (bingbot: noindex) is ignored, but only for the first directive after the crawler's name.
2Read the response header yourself 1 minute
- Open DevTools, go to the Network tab and reload the page.
- Click the first request (the page itself) and open Headers.
- Under Response Headers, look for
x-robots-tag. No line means no header rule.
Then press Ctrl+U for the page source and search for robots and googlebot to see the meta tags as the server sent them.
3Ask Google what it saw 2 minutes
In Google Search Console, paste the URL into URL Inspection and open Page indexing. "Crawl allowed?" tells you about robots.txt, and "Indexing allowed?" says whether Google found a noindex and whether it came from the meta tag or the X-Robots-Tag header. Use Test live URL to check a fix before Google recrawls.
How to fix it
Most accidental noindexes come from a handful of places:
X-Robots-Tag added in nginx, Apache, a CDN rule or an edge function, often for a whole folder.- Decide what each page type should do, then make every place agree. For pages that should be indexed, remove noindex from the meta tags, the header and the template.
- Set it in one place. Use the meta tag for HTML pages and the header for files like PDFs. Fewer places means fewer disagreements.
- Don't block noindexed pages in robots.txt, or Google won't see the noindex.
- Check again with method 1, then use Request indexing in URL Inspection for the pages that matter most.
If there's no noindex and the page still isn't indexed, see Crawled and Discovered, currently not indexed.
On Shopify, WordPress and Next.js
- WordPress: Check Settings > Reading first. Discourage search engines from indexing this site is a common reason a live site carries a noindex after launch.
- Next.js: Set
robots: { index: false }in the page's metadata. Metadata from a layout passes down, so check the parent layouts as well.
How crawlers report this
If a crawl sent you here, these are the labels to look for. Our guide to Screaming Frog and Sitebulb issues covers other common ones.
| Screaming Frog | Sitebulb | What it means |
|---|---|---|
| Directives: Noindex | No direct match | The page asks not to be indexed. It's fine for checkout and account pages, and serious on a page that should rank. |
| Directives: Nofollow | No direct match | The page asks crawlers not to follow its links. |
| No direct match | Mismatched noindex directives in HTML and header | The HTML and the response header give different instructions. |
Doing this on every page?
SEOSnip checks every robots instruction for you
Open the extension on any page and it reads the robots and googlebot meta tags in the server HTML and the rendered page, the X-Robots-Tag header, and the site's robots.txt rules for Googlebot. You get:
- A flag for noindex or nofollow in the meta tags or header, saying where it's set
- The meta tag and the header side by side when they disagree
- A flag when the server HTML and the rendered page say different things
- A ticket with that evidence, and a "Check if fixed" button that re-runs the check
Robots meta tag and X-Robots-Tag header disagree
| Robots meta tag | X-Robots-Tag header | |
|---|---|---|
| Value Differs | index, follow | noindex |
| noindex Differs | no | yes |
| nofollow Same | no | no |