A robots.txt mistake is any rule that blocks crawlers from pages you want ranked or cited, or fails to block what you intended. The common ones are a leftover Disallow: /, blocked CSS and JavaScript, noindexed pages crawlers cannot reach, prefix and case errors, ignored Crawl-delay lines, and blanket rules that shut out AI crawlers.
What makes these mistakes expensive is that they are silent. The file is valid, the site loads, and the crawler does exactly what it was told. Rankings decay over weeks, AI answers stop citing you, and nobody connects it to a two-line text file edited during a migration. This guide covers how Google actually parses robots.txt, the ten misconfigurations that cause most of the damage, and a testing routine that catches them before they ship. To run the checks as you read, open the robots.txt Analyzer alongside, and see how to use the robots.txt Analyzer for a walkthrough of each panel.
How does Google actually read robots.txt?
Most robots.txt mistakes come from reading the file top to bottom like a script. Crawlers do not. Since September 2022 the format has been an IETF standard, RFC 9309, and Google's robots.txt specification documents how Googlebot applies it. Five rules explain almost every surprise:
- One file per host. The file at
https://www.example.com/robots.txtgoverns only that protocol, host, and port. Subdomains and the non-www host each need their own. - One group per crawler. A crawler obeys the most specific User-agent group that names it and ignores the rest. If a
User-agent: Googlebotgroup exists, Googlebot skips theUser-agent: *group entirely. Rules are not inherited. - Longest match wins. Within that group, the rule whose path matches the most characters of the URL decides. Order in the file is irrelevant. If an Allow and a Disallow tie on length, the Allow wins.
- Paths are prefixes and case sensitive.
Disallow: /blogmatches any URL that starts with those exact characters. The only special characters are*(any sequence) and$(end of URL). - Unknown lines are ignored. Misspelled directives, unsupported ones like noindex, and anything beyond the first 500 KiB of the file are skipped without warning. Google also caches the file, generally for up to 24 hours.
Here is the precedence logic in a file you can paste straight into the analyzer:
User-agent: *
Disallow: /shop/
Allow: /shop/sale/
Disallow: /shop/sale/archive
User-agent: Googlebot
Disallow: /checkout/
# /shop/sale/boots -> Bingbot: allowed (Allow: /shop/sale/ is longer)
# /shop/sale/archive-2024 -> Bingbot: blocked (longest match is the Disallow)
# /shop/shoes -> Googlebot: ALLOWED (Googlebot ignores the * group)That last line is the one that catches teams out. Adding a Googlebot group to fine-tune one folder silently cancels every wildcard rule for Google.
What are the most common robots.txt mistakes?
These are ordered roughly by how much damage they do when they slip through.
1. Disallow: / left over from staging
Staging environments are often protected with User-agent: * and Disallow: /. When the file is committed to the repo, baked into a theme, or set by a CMS “discourage search engines” checkbox, it follows the site to production. Fix it twice: protect staging with authentication instead of robots.txt, and add a post-deploy check that fails if the live file contains a bare site-wide disallow.
2. Blocking CSS, JavaScript, and images
Old advice said to keep crawlers out of asset folders to save crawl budget. Google renders pages much like a browser, so a blocked stylesheet or script can leave it looking at a broken layout or missing content. Disallows on /assets/, /js/, or a framework build folder should go. WordPress sites that block /wp-includes/ deserve a second look for the same reason.
3. Blocking pages you also noindex
This one feels like belt and braces and works like neither. A noindex tag lives inside the page, so a crawler blocked from fetching the page never sees it. If the URL has links pointing at it, Google can still index the bare address, and Search Console reports it as “Indexed, though blocked by robots.txt”. Google also stopped supporting Noindex: lines inside robots.txt in 2019. To deindex a page, allow crawling and serve a noindex meta tag or X-Robots-Tag header. Paste the page head into the Meta Tag Analyzer to confirm the tag is there; the guide on how to use the Meta Tag Analyzer covers the rest of the head.
4. Wildcard and precedence errors
Wildcards are powerful and easy to overreach with. Disallow: /*? blocks every URL with a query string, which includes archives paginated with ?page= and any landing page you share with tracking parameters. File-type rules cut both ways. Without an anchor, Disallow: /*.pdf matches “.pdf” anywhere, so /files/report.pdf-preview/ is blocked too. With it, Disallow: /*.pdf$ misses /report.pdf?download=1, because the query string comes after the extension. Pick the one you mean and test both. The precedence trap from the previous section belongs here too: a crawler-specific group replaces the wildcard group, so copy every rule you still need into it.
5. Case sensitivity in paths
Disallow: /Private/ does nothing for /private/. This bites hardest on sites migrating from Windows hosting or older CMSes where mixed-case URLs were common, and on rules copied from documentation. Match the casing your server actually serves, and if both casings resolve, fix the duplication with redirects rather than two robots.txt rules.
6. Trailing slashes and accidental prefixes
Disallow: /blog blocks /blog, /blog/, and everything below it, but also /blogging-tips and /blog-archive. Disallow: /blog/ blocks the folder contents but leaves the bare /blog URL open. Decide which you mean and test both forms. Related: an empty Disallow: with no path blocks nothing, which surprises people expecting the opposite.
7. Blocking AI crawlers by accident
The deliberate question of whether to block AI training crawlers is covered in should you block AI crawlers. The mistake here is blocking them without deciding to. The usual culprit is an allowlist pattern: Disallow: / for *, then Allow groups for Googlebot and Bingbot only. It reads as “let the search engines in”, but it also shuts out OAI-SearchBot, PerplexityBot, Claude-SearchBot, and every other retrieval crawler that would have cited you in an AI answer.
The second culprit is blocking the wrong token. OpenAI's crawler documentation separates GPTBot (training) from OAI-SearchBot (search) so you can opt out of one and keep the other. On Google's side, AI Overviews draw on the regular Googlebot crawl, so blocking Googlebot to avoid them removes you from Search; Google-Extended is the separate token for Gemini training and does not affect Search inclusion. Google lists each of its tokens in its crawler documentation. For how each bot behaves on your server, see how AI crawlers work.
8. Relying on Crawl-delay for Google
Google ignores Crawl-delay. Bing and some other crawlers honor it, so a large value can throttle Bing (and the AI products that draw on Bing's index) while doing nothing about the Googlebot traffic that prompted it. If a crawler is genuinely overloading your server, returning 503 or 429 temporarily slows Google down; caching and capacity fix it properly.
9. A broken or missing Sitemap line
The Sitemap directive needs a full absolute URL, including protocol. Relative paths, an http address on an https site, a staging hostname, or a sitemap that no longer exists all waste the line. The quieter failure is a sitemap that lists URLs your robots.txt blocks, which sends crawlers two contradictory instructions. Run the sitemap through the XML Sitemap Validator and spot-test a few of its URLs against your rules.
10. robots.txt returning a 5xx, a redirect chain, or an HTML page
The response matters as much as the contents. Per Google's documentation, a 5xx or 429 makes Google stop crawling the site for the first 12 hours while it retries, then fall back to the last cached copy for up to 30 days. A 404 or other 4xx means “no restrictions”, so a deleted file silently opens everything you had blocked. Google follows at least five redirect hops before giving up and treating the file as missing. And a CDN or single-page app that answers /robots.txt with an HTML page produces a file full of lines crawlers cannot parse. Check the status with the HTTP Header Checker and any hops with the Redirect Chain Checker.
Which robots.txt mistakes do the most damage?
A quick reference for triage, with what each mistake looks like and what crawlers actually do with it:
| Mistake | What crawlers do | Impact | Fix |
|---|---|---|---|
| Staging Disallow: / in production | Stop fetching every page | Severe, site-wide | Auth on staging, post-deploy check |
| 5xx on robots.txt | Google pauses crawling, then uses a stale copy | Severe while it lasts | Serve a static 200 text file |
| Allowlist blocks retrieval bots | AI search crawlers skip the site | High for AI visibility | Name the bots you allow explicitly |
| Blocked CSS and JS | Render a broken page | Medium to high | Remove asset disallows |
| Blocked plus noindex | Never see the noindex | Unwanted URLs stay indexed | Allow crawl, keep noindex |
| Specific group drops * rules | Follow only the named group | Medium, often unnoticed | Repeat needed rules in each group |
| Prefix, slash, or case errors | Match more or fewer URLs than meant | Varies by path | Test real URLs before deploy |
| Crawl-delay for Google | Google ignores it, Bing obeys it | Low, but can slow Bing | Remove unless Bing load is real |
How do I test robots.txt before and after a deploy?
A useful test checks three things: the response, the rules, and real URLs against those rules. This routine takes five minutes and belongs in every launch checklist.
Before deploy: test the draft
Paste the new file into the robots.txt Analyzer. Clear every error, then use its URL Path Tester on a short list you keep for the site: homepage, a money page, a blog post, a paginated URL, a URL with tracking parameters, and each path you intend to block. Test as Googlebot, Bingbot, GPTBot, and the any-agent option. If you are building the file from scratch, start in the Robots.txt Generator and verify its output the same way.
After deploy: test the live response
# Status code, content type, and any redirect
curl -sI https://www.example.com/robots.txt
# The exact bytes crawlers receive
curl -s https://www.example.com/robots.txt
# Flag any group with a bare site-wide disallow
curl -s https://www.example.com/robots.txt | grep -qx "Disallow: /" && echo "BLOCKED SITE"The last line is a blunt check, but blunt is right for the mistake that costs the most. It flags any group with a bare Disallow: /, so review a hit rather than assuming disaster, since a deliberate GPTBot block will trip it too.
Within a week: confirm with Google and your logs
In Search Console, the robots.txt report shows which file Google fetched, when, and whether it hit errors, and URL Inspection confirms whether Googlebot may crawl a specific page. Google's introduction to robots.txt links to both. For crawlers Google does not report on, server logs are the source of truth: if OAI-SearchBot or PerplexityBot hits stop after a deploy, the file is the first suspect. If unblocked pages are still slow to show up, work through how to get pages indexed faster.
How do robots.txt mistakes affect AI visibility?
AI assistants can only cite what they, or the search index behind them, can fetch. A retrieval crawler blocked by a blanket rule cannot read the page, so it cannot quote it, and the answer goes to a competitor whose file let the bot through. Every other piece of generative engine optimization depends on that access. If your content is crawlable and still not showing up, the causes are usually structural, covered in why your content is not cited by ChatGPT and internal linking for AI search.
Robots.txt says who may fetch. The companion llms.txt file, explained in the llms.txt and AI crawler guide, suggests what is worth reading, but it cannot override a Disallow. Once the file is clean, run the AI Visibility Score to see whether assistants are actually surfacing your brand, and re-run it a few weeks after any fix.
Frequently Asked Questions
What is the most common robots.txt mistake?
A site-wide Disallow: / that ships to production from a staging or pre-launch environment. It blocks every compliant crawler from every page, nothing in analytics flags it, and traffic simply decays over the following weeks. Check the live file after every launch, migration, or hosting change.
Can a page blocked by robots.txt still appear in Google?
Yes. Robots.txt stops crawling, not indexing. If other pages link to a blocked URL, Google can index the address without reading the content, which shows up in Search Console as Indexed, though blocked by robots.txt. To remove a page from results, allow crawling and use a noindex meta tag or X-Robots-Tag header.
Which rule wins when Allow and Disallow conflict?
Under RFC 9309, which Google follows, the rule with the longest matching path wins, regardless of the order rules appear in the file. If an Allow and a Disallow match with equal length, the Allow rule wins. The rules only compete within the single group that applies to the crawler.
Are robots.txt rules case sensitive?
The paths are. Disallow: /Admin/ does not block /admin/. Directive names such as Disallow and the user-agent value are matched without regard to case, so User-agent: googlebot and User-agent: Googlebot behave the same. Match the exact casing of the URLs your site serves.
Does Google respect Crawl-delay?
No. Google ignores the Crawl-delay directive. Bing and some other crawlers honor it. If Googlebot is overloading your server, the documented short-term lever is to return 503 or 429 responses, which slow crawling, and the longer-term fix is server capacity or caching.
What happens if robots.txt returns a server error?
Google treats a 5xx or 429 response as a temporary error and stops crawling the site for the first 12 hours while it retries, then falls back to its last cached copy for up to 30 days. A 404 or other 4xx is treated as no robots.txt at all, which means full access. Either way, the file you wrote is not the one being applied.