X-Robots-Tag
X-Robots-Tag is the HTTP-header equivalent of the robots meta tag. It works for any file type Google crawls — PDFs, images, videos, JSON APIs — not just HTML, making it the only way to deindex non-HTML resources.
What is X-Robots-Tag?
X-Robots-Tag is an HTTP response header that delivers the same indexing directives as the <meta name="robots"> tag, but at the response level rather than in HTML. Because it travels in headers, it can be applied to any file type: PDFs, images, videos, JSON APIs, plain text — not just HTML.
It was introduced by Google in 2007 to solve a specific problem: how do you tell search engines not to index a PDF or image, when those files have no place to embed a meta tag? The answer is to add the directive to the HTTP response itself.
Syntax
X-Robots-Tag: noindex, nofollowYou can also target specific user-agents:
X-Robots-Tag: googlebot: noindex
X-Robots-Tag: bingbot: noindex, nofollow
X-Robots-Tag: noindex, noarchiveWhen multiple X-Robots-Tag headers are sent, all directives combine. The most restrictive interpretation wins.
Supported Directives
| Directive | Effect | Use Case |
|---|---|---|
noindex | Page will not appear in search results | Thin tag archives, internal search results |
nofollow | Links on the page do not pass equity | User-generated content pages |
none | Shorthand for noindex, nofollow | Pages to fully exclude from SEO |
noarchive | No cached copy shown in SERPs | Time-sensitive pricing pages |
nosnippet | No description shown in SERPs | Premium content previews |
max-snippet:N | Cap description length at N chars | Brand-controlled previews |
noimageindex | Images on page excluded from Image Search | Stock photo-heavy pages |
unavailable_after: [date] | Drop from index after date | Event landing pages, expiring promos |
How to Implement
Nginx
location ~* \.(pdf|docx|xlsx)$ {
add_header X-Robots-Tag "noindex, nofollow";
}Apache (.htaccess)
<FilesMatch "\.(pdf|docx|xlsx)$">
Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>Node.js / Express
app.get('/private-pdf', (req, res) => {
res.set('X-Robots-Tag', 'noindex, nofollow');
res.sendFile('file.pdf');
});X-Robots-Tag vs Meta Robots vs Robots.txt
| Method | Works For | Prevents Crawl? | Prevents Index? |
|---|---|---|---|
| Robots.txt Disallow | Any URL | Yes | No (can still appear) |
<meta robots> | HTML only | No | Yes |
| X-Robots-Tag | Any file type | No | Yes |
Frequently Asked Questions
Can I use X-Robots-Tag for HTML pages?
Yes. For HTML, X-Robots-Tag and meta robots are functionally identical. Most teams pick one convention per project. X-Robots-Tag is preferred when you want to control directives at the CDN or reverse proxy layer without touching templates.
Does Google honor X-Robots-Tag from a CDN?
Yes. As long as the directive is in the response Google receives, it does not matter whether the header originated at the origin server, CDN edge, or middleware layer.
What happens if X-Robots-Tag and meta robots conflict?
Google combines all directives from both sources and applies the most restrictive interpretation. So if meta says 'index' and X-Robots-Tag says 'noindex', the page will not be indexed.
Is X-Robots-Tag case-sensitive?
The header name is case-insensitive per HTTP spec. The directive values should be lowercase as a convention, but Google parses them case-insensitively as well.
Can I use X-Robots-Tag to remove a URL from Google's index?
Yes, but only after Google recrawls the URL and sees the header. For faster removal, also use the Removals tool in Search Console. X-Robots-Tag is the permanent fix; the Removals tool is a 6-month emergency override.
Related Terms & Resources
- Robots.txt Best Practices — why robots.txt cannot deindex
- Robots.txt Tester — verify your crawl rules
- Meta Tag Generator — build robots meta tags with preview
- Crawl Budget glossary — how indexing rules affect crawl
Part of the PositiveBacklink SEO Glossary. Updated May 2026.