Blocked by robots.txt

Blocked by robots.txt means a rule in your robots.txt file told Googlebot not to fetch the URL, so Google never read the page. What to do depends on one thing: did you mean to block it?

If you did not, find the rule and remove it. If you did, because you want the page out of Google, robots.txt is the wrong tool. Google’s own documentation says it plainly: robots.txt “is not a mechanism for keeping a web page out of Google”. A blocked URL can still be indexed from links elsewhere, shown without a description, and then reported as Indexed, though blocked by robots.txt.

If you want the page out of Google

Remove the robots.txt block and add noindex to the page instead. That is Google’s advice in its Page indexing help, and the order matters: Google has to fetch the page to see the noindex, and a block stops it fetching. Leave both in place and Google sees neither the page nor the instruction.

For pages that should not be seen by anyone, such as a staging site or a client area, a password does what neither file can.

Some blocks are fine as they are. The app behind this site, check.indexaction.com, disallows everything: there is nothing in it for a search engine to read, and Search Console lists its signup URLs under this status. Nobody links to those pages from the open web, so there is nothing to keep out. That is the report describing a decision, not a fault.

If you want the page indexed

Find the rule that blocks it

Paste your robots.txt and the URL into our robots.txt tester to see which line decides it, or open Settings → robots.txt in Search Console to see the version Google last fetched. The rules that catch people out:

• The most specific rule wins, judged by the length of the path. When an Allow and a Disallow are equally specific, Google uses the less restrictive one.
• Paths are case-sensitive. Disallow: /Blog does not block /blog.
• Each host has its own file. www.example.com, example.com and shop.example.com read separate robots.txt files, and so do http and https.

The usual accidental blocks

Rule	What happened
Disallow: /	A staging site’s file went to production with the launch
Disallow: /*?	Meant for tracking parameters, it also blocks every real page whose URL has a query string
Disallow: /blog	Without a trailing slash it also blocks /blog-tips and /blogging
A CMS “discourage search engines” setting	The CMS writes the block itself, so it is not in any file you edited

Remove it, then wait the right way

Google generally caches robots.txt for up to 24 hours, so a fix is not seen instantly. After that, run Test live URL in URL Inspection to confirm the page can be fetched, and Request indexing once.

When robots.txt itself is broken

How the file fails matters more than people expect:

• A 404 for robots.txt is read as “no rules”. Google crawls everything.
• A 5xx or a timeout is not. Google stops crawling the site for up to 12 hours while it retries, then falls back to the last copy it fetched, for up to 30 days.

So a server that errors on /robots.txt can stall crawling of a whole site while every page still loads in a browser. If this report grows suddenly with no rule change, check that the file itself returns 200.

When the answer is a different status

If the page is fetched and carries noindex while your sitemap lists it, see submitted URL marked ‘noindex’. If Google knows the URL but has not fetched it, and nothing blocks it, see discovered – currently not indexed. The full list of causes is in why is my page not indexed.