Crawl control: what robots.txt does and does not do
robots.txt is a plain-text file at the root of a host that tells compliant crawlers which URL paths they may request. It controls crawling — not indexing, and not access.
Almost every serious robots.txt mistake comes from one misunderstanding: people expect it to hide pages. It does not. A disallowed URL can still appear in search results if other pages link to it — Google will simply list it without a description, because it was never allowed to fetch the page and read one.
This page covers the matching rules crawlers actually apply, the precedence logic this generator’s tester implements, and the specific configurations that cause real traffic loss. If you only take one thing away: robots.txt manages crawl budget and server load, while noindex manages what appears in search results.
The matching rules, precisely
These are the rules the built-in path tester applies, and they follow the behaviour of the major search crawlers.
- Longest match wins
- When several Allow and Disallow rules match a URL, the one with the longest path pattern is applied. Disallow: /blog/ and Allow: /blog/public/ means /blog/public/post is crawlable, because the Allow rule is more specific.
- The * wildcard
- Matches any sequence of characters. Disallow: /*?sort= blocks every URL containing that query parameter, wherever it appears in the path.
- The $ anchor
- Anchors the pattern to the end of the URL. Disallow: /*.pdf$ blocks PDF files but leaves /guide.pdf?download=1 crawlable, because the query string means the URL no longer ends in .pdf.
- Group selection
- A crawler obeys the single most specific user-agent group that names it, and ignores every other group — including the * group. Rules are not inherited or merged, which surprises people constantly.
- Case sensitivity
- Paths are case-sensitive. Disallow: /Admin/ does not block /admin/. Directive names are not case-sensitive.
- Scope
- A robots.txt file applies to exactly one scheme, host and port. https://example.com and https://shop.example.com need their own files, and so does the http:// version if it serves content.
Rules that behave differently than people expect
Example 1: A specific Allow inside a broad Disallow
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php/wp-admin/options.php → blocked
/wp-admin/admin-ajax.php → allowedThe Allow is longer and therefore more specific, so it wins for that one file. This exact pattern is why the standard WordPress robots.txt looks the way it does.
Example 2: A named group silently disables the wildcard group
User-agent: *
Disallow: /private/
User-agent: Googlebot
Disallow: /tmp/Crawls /private/ freely.
Only /tmp/ is blocked for it.Googlebot uses its own group and ignores the * group entirely. If a rule must apply to everyone, repeat it in every group you define.
Example 3: Blocking a page you also want de-indexed
Disallow: /old-landing-page
<meta name="robots" content="noindex">The page stays in the index.The crawler is forbidden from fetching the page, so it never sees the noindex tag. To remove a URL from search, allow crawling and serve noindex — then block it later, once it has dropped out.
Configurations that cause real traffic loss
Shipping the staging robots.txt to production
Disallow: / on a live site removes it from search within days. This is the single most expensive robots.txt mistake, and it happens on nearly every large migration. Check the live file first thing after any deployment.
Blocking CSS and JavaScript
Google renders pages to evaluate them. Blocking the assets needed to render means it sees a broken layout, which affects mobile-friendliness assessment and can affect ranking. Leave public assets crawlable.
Expecting robots.txt to protect private data
The file is public and readable by anyone. Listing /admin/ or /internal-reports/ advertises exactly where to look, and non-compliant crawlers ignore it entirely. Private means authentication, not a directive.
A 5xx response on robots.txt
If the file returns a server error, Google may treat the whole site as disallowed for a period. An unreachable robots.txt is more dangerous than a missing one — a 404 is interpreted as "crawl everything".
Crawl-delay does nothing for Google
Google ignores the Crawl-delay directive; Bing and some others honour it. Use Search Console’s crawl-rate settings for Googlebot rather than assuming the directive applies.
Placing the file anywhere but the root
Only /robots.txt is read. A file at /blog/robots.txt or /public/robots.txt is ignored completely. The file must also be plain text, and Google reads up to 500 KiB of it.
Before you upload the file
- Test at least five real URLs in the built-in tester, including one that must stay crawlable.
- Include a Sitemap: line with the absolute URL — it works regardless of which user-agent group is being read.
- Repeat critical rules in every named user-agent group, since groups do not inherit.
- Confirm no rule blocks CSS, JavaScript or image directories the pages need in order to render.
- Re-check the live file after every deployment; a staging file overwriting production is the classic failure.
- Pair crawl rules with matching page-level directives from the meta tag generator so the two never contradict each other.
robots.txt vs. noindex vs. canonical vs. authentication
Direct answer: each of these solves a different problem, and using the wrong one is why pages that should be gone are still in search results.
| Goal | Correct mechanism | Why not robots.txt |
|---|---|---|
| Keep a URL out of search results | A noindex meta tag or header | A blocked page can still be indexed as a URL-only result. |
| Stop crawlers wasting budget on infinite filter URLs | robots.txt | This is exactly what it is for. |
| Consolidate duplicate URLs into one | A canonical tag | Blocking a duplicate hides the canonical signal instead of passing it. |
| Prevent anyone reading a page | Authentication | robots.txt is a public request, not access control. |
| Remove a page after it is gone | A 410 status, or a 301 to the replacement | A blocked URL keeps its old status indefinitely because nobody can re-crawl it. Build the redirect with the .htaccess redirect generator. |
How this page is maintained
The longest-match precedence, wildcard and $ anchor behaviour described here match the logic implemented in this tool’s path tester. Crawler-specific behaviour — group selection, Crawl-delay support and the 500 KiB parsing limit — reflects the search engines’ own published documentation at the time of review.
Written and reviewed by the ToolsRacks editorial team. Last reviewed: .