dotvitals

robots.txt blocks this page from being crawled

HighConfirmedQuick winseo.indexability.blocked-by-robots-txt

What this check looks for

Your robots.txt tells crawlers not to fetch this page. They will not read it, so its content cannot be used in search results.

Why it matters

A blocked page is not necessarily absent from search — Google can still list a URL it has never fetched, using only the anchor text of links to it — but it will have no description and no content behind it. In practice that is a result nobody clicks.

When the check passes, your report says: “robots.txt allows crawlers to fetch this page”.

What it costs your score

When this check fails it removes 30 points from your SEO score, before the status, confidence and repeat multipliers are applied. SEO carries a weight of 5 in the overall score.

It shares the seo.indexability family ceiling of 45 points: however many findings that family produces, together they cannot remove more than that from SEO. One underlying problem showing up in several places is still one problem.

Severity
high
Default confidence
confirmed
Status when triggered
fail
Deduction
30 points
Family cap
seo.indexability · 45
Category
SEO
Module
Seo audit
Fix owned by
user
In the ruleset since
2026.09

How the whole score is calculated

How to fix it

Remove or narrow the robots.txt rule that matches this page.

A crawler that cannot fetch the page cannot use anything on it, so the page competes on its inbound links alone.

  1. Open ‹robots url› and find the ‹rule› line. Check which User-agent group it is in.

  2. Narrow it to the paths you actually meant to block, rather than removing the file.

  3. Remember that Allow overrides Disallow when it is the longer match, so a targeted Allow can carve out one page without rewriting the group.

  4. If the goal was to keep the page out of results rather than out of the crawl, remove the block and serve noindex instead.

How to confirm it worked

  • curl -sS ‹robots url› — confirm the rule is gone or narrowed

  • Use the robots.txt report in Google Search Console to confirm the URL is now allowed

A named slot like ‹domain› — and the braces left in the configuration below — is filled in with your own values when this rule appears on a report.

Remediation by platform

Site root files
User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /

Sitemap: https://example.com/sitemap.xml

  • Block the paths you mean, not a prefix that happens to contain them. Disallow: /c blocks /cart/, /careers/ and /contact/ alike, which is how a page ends up blocked by accident.

Technical detail

The rule ‹rule› in ‹robots url› matches this page's path for the ‹user agent› token, and it is the longest matching rule, so by RFC 9309 §2.2.2 it decides.

The group we evaluated is the one that applies to ‹user agent› specifically, which is frequently not the User-agent: * group. A site that blocks * and allows Googlebot is correctly configured for Google and blocked for everything else — including the AI crawlers that increasingly matter.

**Blocking and de-indexing are different operations and are often confused.** Disallow stops the fetch; it does not remove a URL from the index. To keep a page out of results, allow the crawl and serve noindex — the crawler has to fetch the page to see the directive. A page that is both blocked and marked noindex gets the worst of both: the directive is never read.

Standards and references

Test this on your domain

Run the check that produces this finding, on its own, against any domain.

Open the seo audit checker

Other seo audit checks