Robots Meta Tags and Scraping: What Your Crawler Must Read

A robots meta tag does not stop your scraper from fetching a page. It is a signal from the site owner about indexing and link-following, and your crawler can only read it after it has already downloaded the page. How you handle that signal is a policy decision, and it should be an explicit one: parse it, store it next to the data, and decide in code what each directive means for your pipeline.

This guide covers where the tags live, what the common directives mean, how to resolve conflicting rules, how they differ from robots.txt, and what to record.

TL;DR

• A robots meta tag is a hint in the page <head>. Google describes it as an instruction to cooperative search engine bots, not an access control.
• Your scraper must fetch the page to see the tag, so the tag can never prevent the request itself.
• Read both the <meta name="robots"> element and the X-Robots-Tag HTTP header. When rules conflict, apply the most restrictive one.
• robots.txt governs crawling. Meta tags govern indexing and presentation. Check robots.txt first.
• Store the directives you saw with every record, so each row of data stays traceable.

Table of Contents

• What a robots meta tag is and where to find it
• What the common directives mean
• Conflicting and combined rules
• Robots meta tags vs. robots.txt
• Does a robots meta tag stop a scraper?
• What to record and decide
• Our perspective
• Run it with predictable costs
• FAQ
• Sources

What a robots meta tag is and where to find it

The tag is a <meta> element in the <head> of an HTML document, with name="robots" and a content attribute holding one or more comma-separated directives:

<meta name="robots" content="noindex, nofollow">

Per MDN, the robots value defines crawl and indexing behavior, and a crawler must access the resource to read it. That last clause is the whole story for scrapers: the instruction travels with the content it describes.

There are two other places to look:

1. Bot-specific tags. A tag named after a specific crawler (for example, a search engine's user agent token) applies to that crawler only. Your scraper is not that crawler, so the generic robots tag is the one that speaks to you, but log the specific ones too.
2. The X-Robots-Tag HTTP header. It works like the meta tag but sits in the response headers. MDN notes that it lets site owners set rules for non-HTML resources such as PDFs and images, where there is no <head> to put a tag in.

If you only parse HTML, you will miss every directive attached to a PDF. Read headers on every response, whatever the content type.

What the common directives mean

The directives are written for search engines. Here is what each says, and what it plausibly implies for a data pipeline.

These definitions come from Google Search Central. The none equivalence is stated there explicitly, and so is the note that without nofollow, search engines may use a page's links for discovery.

Two parsing details save you bugs. Both the tag names and content values are case-insensitive, so normalize before comparing. And treat directives as a set of tokens, not a string match: content="NoIndex, nofollow" is valid.

Conflicting and combined rules

Real pages carry several rules at once: a meta tag in the HTML, an X-Robots-Tag header from the CDN, sometimes a second meta tag injected by a plugin. Google's documentation says rules can be combined and that the more restrictive rule applies when they conflict.

A sound merge algorithm:

1. Collect directives from the response headers and every robots-type meta element in the document.
2. Lowercase and tokenize on commas.
3. Union the restrictive tokens. If any source says noindex, the merged result says noindex.
4. Keep the raw values per source in your log, so you can tell a header-level rule from a page-level one later.

One caveat: the directives are not a formal standard. MDN describes indexing directives in headers and meta tags as a de-facto standard for search-related crawlers. Expect odd values, and design your parser to ignore unknown tokens rather than crash on them.

Robots meta tags vs. robots.txt

They answer different questions. robots.txt says which paths a crawler may request. Meta tags and headers say what should happen with a page after it has been fetched. MDN puts it this way: robots.txt controls crawling, while robots meta tags influence indexing and other behavior.

The interaction matters. If a URL is disallowed in robots.txt and a compliant crawler never fetches it, the tag is never seen. Google states that in that case the rules will not be found and will be ignored. So a site that blocks a path in robots.txt and also puts noindex on it has written a rule that cannot fire.

Order of operations for a polite scraper:

1. Fetch and cache robots.txt. Respect its rules for your user agent. Our write-up on robots.txt compliance for developers covers what to do when that file returns a server error.
2. Fetch the page if allowed.
3. Read headers and meta tags.
4. Apply your policy to the result.

If you want to see how a site's crawl rules parse before you point a job at it, a robots.txt validator and checker helps separate crawl blocking from the page-level directives discussed here.

One more honest point: robots.txt is not enforcement either. The OWASP Web Security Testing Guide notes that crawlers can intentionally ignore its Disallow directives. Compliance is something your code chooses to do. That is exactly why it should be written down.

Does a robots meta tag stop a scraper?

No. MDN is plain on this: the crawler still has to access the resource to read the tag. It also points out that robots.txt is more bandwidth-efficient, because it prevents the fetch entirely while a meta tag requires the page download first.

So the tag is not a technical barrier. It is a statement of the owner's intent, and it tells you something about the content: a noindex page is often a staging page, a thin utility page, a login-adjacent page or a duplicate. Sometimes that is exactly the signal your data quality filter wants.

What to record and decide

Don't bury this in a boolean. For each fetched URL, store:

• The URL and fetch timestamp.
• The raw X-Robots-Tag header values.
• The raw content of each robots meta element, including bot-specific ones.
• The merged, normalized directive set.
• The robots.txt verdict that allowed the fetch.
• The action your pipeline took: kept, dropped, kept without links, kept without republishing.

Then write the policy as code, not tribal knowledge. A starting point, to adapt with your own counsel for your use case:

• nofollow or none: don't enqueue links from that page.
• noindex or none: tag the record, and exclude it from anything resembling a public search index or republished corpus.
• nosnippet: don't surface excerpts of that page in user-facing output.
• Anything unparseable: log it, keep the page, and flag it for review.

This matters most for compliance and financial-research workflows, where an analyst months later needs to know why a page was in the dataset and what the site said about it. If you traverse sites, pair this with sensible limits: see our notes on polite crawling defaults and crawl depth limits. If your agents browse on their own, the AI agent web browsing checklist puts these checks in one place.

Our perspective

We build a web data API, so we see the same mistake repeatedly: teams treat crawl policy as a pass/fail gate and discard the evidence. A fetch either "worked" or "failed", and the reason the site gave is thrown away.

We think that is backwards. A directive is data. If a page says noindex, that is a fact about the page, as real as its title. Our approach to the API is that failure modes and content signals should be surfaced in typed responses, not hidden, so that an agent or a pipeline can reason about them instead of guessing. The same logic applies to robots directives: capture them, keep them with the record, and make the decision visible in your own code.

We would rather you see a signal and choose to ignore it with a documented reason than never know it existed. Your bill shouldn't surprise you, and neither should your audit.

— Glen

Run it with predictable costs

If you are building this pipeline, Gyrence gives you Fetch, Extract, Traverse and Map as separate primitives behind one API, plus a hosted MCP endpoint for agents, with spending caps so a runaway crawl can't run up your bill. Paid plans start with Standard at $75 per month; Growth is $299 per month and Scale is $549 per month. There is also a Free plan, and see the pricing page for Founders and Pay-As-You-Go details.

If you need to know when a page's content or signals change over time, WebDoppler monitors pages and sends webhook alerts.

FAQ

Do I have to obey a robots meta tag when scraping?

The tag is a de-facto convention aimed at search engines, not a technical barrier, and your crawler can read the page regardless. Whether to honor it is a policy and legal question for your use case. Decide it deliberately, write it down, and ask a lawyer if the data is sensitive or the use is commercial.

What is the difference between noindex and nofollow?

noindex asks that the page not be indexed. nofollow asks crawlers not to follow the links on it. none means both, per Google's documentation.

Can a PDF have robots directives?

Yes, through the X-Robots-Tag HTTP header. Files without an HTML <head> can't carry a meta tag, so the header is the only place the rule can live. Read response headers for every content type.

What if the meta tag and the header disagree?

Apply the more restrictive rule. Google's guidance is that the most restrictive directive wins when rules conflict. Keep both raw values in your log.

Does robots.txt override a noindex tag?

Not exactly. If a path is blocked in robots.txt, a compliant crawler never fetches the page and never sees the tag. The two mechanisms don't cooperate; they just apply at different stages.

Sources

• Robots meta tag specifications, Google Search Central
• Robots meta tag specifications (updated March 2026), Google Search Central
• meta name="robots" attribute value, MDN
• X-Robots-Tag header, MDN
• Review webserver metafiles for information leakage, OWASP WSTG

Recommended

• Engineers: 5 Audit Ready Controls for Compliance Friendly Scraping
• Cut Blocking Risk: 20s Backoff + Spending Caps on Scraping Rate Limits
• Robots.txt Compliance for Developers
• Polite Crawling: Developer Defaults for 429s, Crawl Caps, and Audits