The robots.txt at the site root was not readable as directives

LowConfirmedQuick winseo.robots-txt.unreadable

What this check looks for

/robots.txt did not return a file a crawler can read: an HTML error page served with a success status, a refusal, a redirect this scan does not follow, or a server error. Whatever the file was meant to say, that is not what was served.

Why it matters

Every directive anyone adds to that file is ignored, including the sitemap line and any path you meant to keep crawlers out of. The file looks present, so nobody checks it again.

What it costs your score

When this check fails it removes 5 points from your SEO score, before the status, confidence and repeat multipliers are applied. SEO carries a weight of 5 in the overall score.

It shares the seo.crawl family ceiling of 15 points: however many findings that family produces, together they cannot remove more than that from SEO. One underlying problem showing up in several places is still one problem.

Severity: low
Default confidence: confirmed
Status when triggered: warn
Deduction: 5 points
Family cap: seo.crawl · 15
Category: SEO
Module: Seo audit
Fix owned by: user
In the ruleset since: 2026.09

How the whole score is calculated

How to fix it

Serve a real robots.txt at the site root with a 200, or let the path return 404.

Directives nobody can read are doing nothing — and while the file answers with a server error, RFC 9309 has crawlers assume they may crawl nothing at all.

Place a plain-text robots.txt in the document root, served as text/plain with a 200.
On a single-page application or a framework with a catch-all route, exclude /robots.txt from the rewrite so the static file wins.
Do not redirect /robots.txt. It is per host and per scheme, so the right file is the one that answers on this host directly; crawlers follow a redirect but not every consumer of the file does.
Do not put it behind authentication, an IP allow-list, a WAF rule or a rate limiter — a 401, 403 or 429 at this path means no crawler ever reads your directives.
If you have no directives to publish, a 404 is a correct and well-defined answer. Do not serve an HTML page instead.
Add a Sitemap: line while you are there — it is the file's most useful content for most sites.

How to confirm it worked

curl -sS -o /dev/null -w '%{http_code} %{content_type}\n' ‹robots url› — expect 200 text/plain, or 404
curl -sS ‹robots url› | head -5 — should show directives, not markup

A named slot like ‹domain› — and the braces left in the configuration below — is filled in with your own values when this rule appears on a report.

Remediation by platform

Site root files

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Next.js

// Put robots.txt in `public/` so the static file is served before the catch-all route,
// or generate it from `app/robots.ts` which Next.js serves at /robots.txt.

Plain text, served as text/plain, at the root of the host. robots.txt is per host and per scheme — the one at https://example.com/ does not apply to https://shop.example.com/.
The failure this finding reports is almost always a catch-all route answering first. Check the served content type, not just the status.

Technical detail

‹robots url› returned ‹status›‹content type›. ‹detail›

What was served begins:

‹excerpt›

The commonest cause of the HTML case is a catch-all route or a single-page-application rewrite that serves index.html for any path the server does not recognise, with a 200 status. Crawlers parse what they get line by line and discard anything that is not a directive, so the practical effect is usually 'allow everything' — reached by accident, and quietly wrong the moment a rule is added.

RFC 9309 §2.3.1 assigns each status class a meaning, and they are not the same meaning: a 404 or 410 (§2.3.1.3) means no file exists and everything may be crawled; any other 4xx is in the same 'unavailable' class, so crawlers also crawl everything; a 3xx (§2.3.1.2) must be followed for up to five hops and the file at the end applies; a 5xx (§2.3.1.4) means the file is undefined and a crawler must assume **complete disallow** while it lasts. That last one is why a robots.txt behind a flaky server is worse than no robots.txt at all.

This scan does not follow the redirect, because a redirect target is a second destination that has to be validated in its own right. So when the status is a 3xx this finding records what was seen and withholds the verdict rather than reporting the file as empty.

The fix is to serve a real file, or to let the path 404. A genuine 404 is a *better* outcome than any of the above: RFC 9309 gives it a defined meaning, and nothing about it is misleading.