Indexed, though blocked by robots.txt: what it means and how to fix it

Google Search Console reports "Indexed, though blocked by robots.txt" and the status is marked as a warning. Two things about it confuse almost everyone: the page is in the index despite being blocked, and the fix is the opposite of what people try first.

Blocking a page harder makes this worse, not better. Here is why, and what to do instead.

"Indexed, though blocked by robots.txt" means Google found links to a URL and added it to the index without fetching it, because robots.txt disallowed crawling. The result is a search listing with no description. To remove it, allow Google to crawl the URL and serve a noindex directive, or return a 404 or 410. Adding a noindex tag while the URL is still blocked has no effect, because Google cannot read the tag.

Why a blocked page ends up indexed

Crawling and indexing are separate steps, and robots.txt only controls the first one.

When another page links to your blocked URL, Google learns the URL exists. It respects robots.txt and does not fetch the content — but it still has the URL and the anchor text pointing at it, and that can be enough to justify a listing. You end up with a search result showing the bare URL and a note that no information is available.

That is why the status is a warning and not an error. Nothing is broken. Google is doing exactly what you told it: not crawling. You simply told it the wrong thing for the outcome you wanted.

The mistake that keeps the page stuck

The instinctive response is to add a noindex meta tag while leaving the robots.txt block in place. This does not work, and it is worth being precise about why:

The noindex directive lives inside the page. Google must fetch the page to see it. robots.txt forbids fetching the page. Therefore Google never sees the directive.

The block that was meant to hide the page is the thing preventing it from being removed. Pages can sit in this state for months.

Decide what you actually want

Three different goals, three different fixes. Picking the wrong one is what causes this issue in the first place.

Goal	Correct method	Do not use
Keep the URL out of search results	noindex, with crawling allowed	robots.txt Disallow
Stop crawlers wasting budget on endless filter URLs	robots.txt Disallow	noindex — the page still gets crawled
Merge duplicate URLs into one	Canonical tag or a 301 redirect	robots.txt — it hides the signal
Remove a deleted page permanently	Return 410 Gone (or 404)	robots.txt — the old status is frozen in place
Prevent anyone accessing it	Authentication	robots.txt — the file is public and readable by anyone

How to fix it, step by step

1. Confirm the URL is genuinely blocked. Test the exact path against your rules — our robots.txt generator and tester applies the same longest-match precedence the search engines use, so an Allow rule overriding a broader Disallow shows up correctly.
2. Decide: should this page exist at all? If it should not, return 410 and skip to step 5. That is faster and cleaner than a noindex.
3. Remove the Disallow rule for that path. This feels wrong and is the necessary step.
4. Serve a noindex directive. Either <meta name="robots" content="noindex"> in the head, or an X-Robots-Tag: noindex HTTP header — the header is the only option for PDFs and images.
5. Wait for a recrawl. Anywhere from days to several weeks depending on how often that URL is visited. Use the URL Inspection tool and request indexing to prompt it.
6. Only after it has dropped out, restore the robots.txt block if you still want to save crawl budget.

For anything urgent — a leaked page, personal data — use the Removals tool in Search Console for a temporary hide while the permanent fix propagates.

Which URLs usually trigger this

• Internal search result pages — /search?q=, blocked in robots.txt and linked from somewhere.
• Faceted navigation on ecommerce — filter and sort parameters generating effectively unlimited URLs.
• Staging subdomains that someone linked to publicly.
• Thank-you and checkout pages blocked for tidiness, then linked in an email or a public post.
• Admin login paths — listing them in robots.txt tells everyone exactly where they are, which is a second problem on top of this one.
• PDFs and downloads blocked by directory but linked from articles.

Two things worth checking while you are in there

Do not block CSS and JavaScript. Google renders pages to evaluate them. If the assets needed to render are disallowed, it sees a broken layout, which affects mobile usability assessment. Leave public assets crawlable.

Make sure robots.txt returns 200. A 5xx response on the file itself can cause Google to treat the whole site as disallowed for a period — a far worse outcome than having no file at all, since a 404 is read as permission to crawl everything.

While you are auditing indexing signals, it is worth confirming that your titles, descriptions and canonical tags agree with each other. Contradictory signals — a canonical pointing one way while internal links point another — get resolved by Google rather than by you. Our meta tag generator previews all of them together.

Frequently asked questions

How long does it take to disappear after I fix it?

Typically days to a few weeks, depending on how frequently Google recrawls that URL. Requesting indexing through URL Inspection usually speeds it up. Low-value URLs are recrawled rarely, so those take longest.

Is this warning hurting my rankings?

Not directly, and a handful of these is normal on any large site. It becomes a real problem when hundreds of thin or duplicate URLs are listed, because that is a signal about site quality rather than about any single page.

Can I just delete the robots.txt file?

You can, and everything becomes crawlable — which may be worse. A missing file is interpreted as permission to crawl everything, including the filter URLs you were trying to control. Remove the specific rule instead.

What is the difference between this and "Blocked by robots.txt"?

"Blocked by robots.txt" means the URL is excluded and not indexed — that is the block working as intended. "Indexed, though blocked" means it got in anyway via links, which is the case that needs action.

Should I use noindex and Disallow together?

Never at the same time — that combination is exactly what creates this problem. Use them in sequence: allow crawling with noindex first, wait until the page drops out, then block it if you still want to.

Does robots.txt keep a page private?

No. The file is publicly readable, so listing sensitive paths advertises them, and non-compliant crawlers ignore it entirely. Anything that must stay private needs authentication.

The short version

robots.txt controls crawling. noindex controls indexing. To remove an indexed page you must let Google in so it can read the instruction to leave — then block it again once it has gone.