dotvitals

A noindex directive is unreachable behind a robots.txt block

HighConfirmedQuick winseo.indexability.noindex-blocked-by-robots

What this check looks for

This page carries a noindex directive and is also blocked in robots.txt. Because crawlers never fetch the page, they never see the directive — so the page can still appear in results.

Why it matters

This combination produces the opposite of what it looks like. The site owner believes the page is excluded; search engines see a URL they are told not to fetch, and may list it anyway from the links pointing at it, with no description.

When the check passes, your report says: “The page's crawler directives are reachable by crawlers”.

What it costs your score

When this check fails it removes 20 points from your SEO score, before the status, confidence and repeat multipliers are applied. SEO carries a weight of 5 in the overall score.

It shares the seo.indexability family ceiling of 45 points: however many findings that family produces, together they cannot remove more than that from SEO. One underlying problem showing up in several places is still one problem.

Severity
high
Default confidence
confirmed
Status when triggered
fail
Deduction
20 points
Family cap
seo.indexability · 45
Category
SEO
Module
Seo audit
Fix owned by
user
In the ruleset since
2026.09

How the whole score is calculated

How to fix it

Pick one mechanism. To exclude the page from results, remove the robots.txt block and keep noindex.

A crawler that is told not to fetch the page never reads the directive telling it not to index the page.

  1. Decide the intent: keep the page out of the index, or keep the crawler off a section.

  2. To exclude from the index: remove the matching rule in robots.txt and leave noindex in place.

  3. To save crawl budget: keep the robots.txt rule and remove the noindex, which is doing nothing while the block stands.

  4. For a page already listed, use the removals tool in Search Console for an immediate effect while the recrawl catches up.

How to confirm it worked

  • curl -sS ‹robots url› — the matching rule should be gone

  • curl -sS ‹url› | grep -i 'name="robots"' — noindex should still be present

A named slot like ‹domain› — and the braces left in the configuration below — is filled in with your own values when this rule appears on a report.

Technical detail

‹rule› in ‹robots url› blocks this path, and the page carries noindex in ‹source›.

Google's documentation states this directly: a noindex rule is ineffective if the URL is blocked by robots.txt, because the crawler must be able to fetch the page to see it. The two mechanisms answer different questions — robots.txt controls *fetching*, noindex controls *indexing* — and using both on one URL disables the second.

The fix depends on which you meant.

  • **To keep the page out of results:** remove the robots.txt block and keep noindex. Let the crawler in so it can read the instruction.
  • **To save crawl budget on a large blocked section:** keep the block and accept that URLs may be listed without content. Remove the noindex, since it is doing nothing, and use the URL removal tool in Search Console for anything already listed.
  • **To do both properly:** serve the page with noindex until it has been recrawled and dropped from the index, then add the robots.txt block.

Standards and references

Test this on your domain

Run the check that produces this finding, on its own, against any domain.

Open the seo audit checker

Other seo audit checks