No robots.txt is published
Updated
What this check looks for
There is no robots.txt at the site root. This is not a fault — RFC 9309 says a crawler that gets a 404 may access everything — and it is worth knowing because the file is also where sitemaps are declared.
Why it matters
Nothing breaks without a robots.txt. The only thing you give up is a convenient place to point crawlers at your sitemap, which is how several crawlers discover it without being told.
What it costs your score
This check never deducts. It reports a fact — or reports that something could not be verified — and is shown on the report without moving the score.
- Severity
- info
- Default confidence
- confirmed
- Status when triggered
- info
- Deduction
- 0 points
- Family cap
- none
- Category
- SEO
- Module
- Seo audit
- Fix owned by
- user
- In the ruleset since
- 2026.09
Technical detail
‹robots url› returned ‹status›.
RFC 9309 §2.3.1.3 states that a 404 or 410 means the crawler may access any resource. So an absent robots.txt is equivalent to User-agent: * with nothing disallowed, which is the right configuration for most sites. This finding deducts nothing and is reported so the absence is a known fact rather than an assumption.
The reason to add one anyway is the Sitemap: directive: it is the discovery route that does not require submitting anything to anyone, and it works for crawlers that have no webmaster console.
A robots.txt is not an access control. Anything listed in it is a public statement of a path you would rather people did not visit, which is the opposite of hiding it. Use authentication for anything that must not be reached.
Standards and references
Test this on your domain
Run the check that produces this finding, on its own, against any domain.