How to Use the robots.txt Analyzer: Score, Test, and Fix

A robots.txt analyzer parses the rules in your robots.txt file, groups them by crawler, flags syntax errors and risky directives, and tells you whether a specific URL is allowed or blocked for a specific bot. GrowthGPT's free version runs entirely in your browser, scores the file out of 100, and includes a URL path tester.

Robots.txt is the shortest file on most websites and one of the few that can remove a whole site from search in a single line. Nothing in your analytics warns you when it goes wrong; traffic just fades while the crawler politely does what it was told. This guide walks through every part of the robots.txt Analyzer: what it checks, how to read the score, how to test real URLs, and where its job ends. For the catalogue of misconfigurations it helps you catch, pair it with robots.txt mistakes that quietly kill rankings and AI visibility.

What does the robots.txt Analyzer check?

The analyzer reads the five directives that matter for crawl control: User-agent, Allow, Disallow, Sitemap, and Crawl-delay. Comments and blank lines are skipped. Consecutive User-agent lines are grouped into one block, the way crawlers read them, and every rule keeps its original line number so you can find it again in your source file. On top of that parse, it runs these checks:

• Syntax errors. Lines with no colon, Allow or Disallow rules that appear before any User-agent line, and Crawl-delay values that are not numbers. Each is reported with its line number.
• Site-wide block. A wildcard group (User-agent: *) containing Disallow: / with no Allow rules is flagged as an error, because it shuts every compliant crawler out of the whole site.
• Blocked rendering resources. Disallow rules on common asset folders (/css/, /js/, /images/, /img/, /assets/, /fonts/, /media/) raise a warning, since search engines need those files to render the page.
• Missing or duplicate sitemaps. No Sitemap line is a warning; the same sitemap URL listed twice is another.
• Rule hygiene. Empty user-agent blocks, duplicate rules inside a block, and the same path carrying both an Allow and a Disallow rule are all warnings. A block whose only rule is Allow: / gets an info note, because allowing everything is already the default.

If any block sets a Crawl-delay, a separate card lists each value per crawler with a reminder that Google does not honor the directive. Bing and some other crawlers do.

How do I analyze my robots.txt file?

The whole pass takes about two minutes. Here is the sequence that catches the most problems.

Step 1: Copy the live file, not the one in your repo

Open https://yourdomain.com/robots.txt in a browser and copy the response. The analyzer does not fetch URLs itself, and that is useful discipline: the file a crawler reads is whatever your server, CDN, or CMS plugin actually returns, which is not always what sits in version control. Check each subdomain separately, because blog.yourdomain.com has its own robots.txt and inherits nothing from the main domain.

Step 2: Paste or upload, then analyze

Paste the contents into the editor, or use Upload File to load a .txt file from disk. The editor highlights directives as you type: User-agent and Allow lines in green, Disallow in red, Sitemap in blue, Crawl-delay in amber, and comments greyed out. A Disallow that shows up red where you expected green is often the first clue. If you are drafting from scratch, the Basic, E-commerce, WordPress, and Strict templates give you a working starting point. Click Analyze robots.txt to run the checks.

Step 3: Read the score and the summary line

The score card shows a 0 to 100 rating with a verdict (Looking good, Room for improvement, or Needs attention) and four counts: user-agent blocks, total rules, sitemaps, and issues. Sanity-check the counts against what you think the file contains. If you wrote six groups and the analyzer found four, two of your User-agent lines are being merged into a group you did not intend.

Step 4: Clear errors first, then warnings

The Issues and Warnings list is colour-coded by severity. Errors are the ones that can stop crawling outright, such as a site-wide block or a rule with no User-agent above it. Fix those before anything else. Warnings are usually rendering or discoverability problems, and info notes are simplifications you can take or leave.

Step 5: Audit each user-agent block line by line

Each block renders as a card listing its crawlers and a table of every Allow and Disallow rule with its path and source line. Read it with one question in mind: is anything here blocking a page I want ranked or cited? Also look for what is missing. The parser silently skips directives it does not recognise, so a misspelled Dissallow will not appear in the table at all. Crawlers ignore it the same way, which means the path you meant to block is wide open.

Step 6: Test the URLs that pay the bills

Finish with the URL Path Tester, covered in detail below. Test your homepage, a top product or service page, a blog post, and anything you deliberately blocked. Four tests take thirty seconds and catch the mistakes that a clean score hides.

How is the robots.txt SEO score calculated?

The score starts at 100 and deducts points for what the checks find. Every error costs 15 points and every warning costs 5. A file with no user-agent blocks at all loses another 20, a file with no Sitemap line loses 10 on top of its warning, and a file that has blocks but no wildcard group loses 5. Info notes cost nothing. The result is clamped between 0 and 100: 80 and above reads as Looking good, 50 to 79 as Room for improvement, and below 50 as Needs attention.

Treat the score as a hygiene check, not a verdict on strategy. It measures whether the file is well formed and avoids the known traps. It cannot know that /pricing/ is your highest-converting page, so a perfectly formatted rule that blocks it still scores 100. That gap is what the tester is for.

How do I test whether a URL is blocked by robots.txt?

Enter a path such as /blog/my-post (a missing leading slash is added for you), pick a crawler, and click Test. The dropdown covers Googlebot, Bingbot, GPTBot, CCBot, Slurp, DuckDuckBot, and a * option for any agent. The tester collects every group that names that crawler (merging them, as Google does), falls back to the wildcard groups if none does, and then picks the matching rule with the longest path, with Allow winning a tie. That mirrors how Google resolves conflicts under RFC 9309. The result says Allowed or Blocked, names the matched agent, and shows the exact rule that decided it. If no rule matches, the path is allowed by default. Wildcard patterns are supported too, including ones that end in a $ anchor, such as /*.pdf$.

Here is a short file worth running through it:

User-agent: *
Disallow: /search
Disallow: /cart/
Allow: /blog/
Disallow: /blog/drafts/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

It scores 100: no errors, no warnings, a sitemap, and a wildcard group. Now test three paths:

• /blog/drafts/q4-plan as Googlebot. Blocked by Disallow: /blog/drafts/. Both the Allow and the Disallow match, and the longer rule wins. That is the intended result.
• /blog/robots-guide as GPTBot. Blocked by Disallow: / in the GPTBot group. GPTBot follows its own group and ignores the wildcard one, so the blog Allow does not apply to it. Intended if you meant to opt out of OpenAI training.
• /searchlight-case-study as Googlebot. Blocked by Disallow: /search. Rules are prefix matches, so a rule meant for internal search results also takes out a case study. Replacing it with Disallow: /search? and Disallow: /search/ blocks the result URLs without catching the case study. Re-test to confirm.

For crawlers not in the dropdown, such as ClaudeBot, OAI-SearchBot, or PerplexityBot, choose the * option. If your file has no group that names the bot, the wildcard group is exactly what it will follow. If you are unsure which of those bots fetch for training and which fetch to answer live questions, read how AI crawlers work before you write rules for them, and should you block AI crawlers for the business decision itself.

robots.txt Analyzer vs Google Search Console: which should I use?

Both, at different moments. Google retired its standalone robots.txt Tester and now shows the robots.txt files it has fetched, along with fetch status and parsing problems, in the robots.txt report inside Search Console. The URL Inspection tool tells you whether Googlebot is allowed to crawl a specific live URL. Those are the final word for Google. The analyzer is the faster loop for everything before and around that.

The practical workflow: draft and test in the analyzer, deploy, then confirm in Search Console. Google documents the full matching behaviour in its robots.txt specification, which builds on the IETF standard RFC 9309.

What can the robots.txt Analyzer not tell me?

Knowing the edges of a tool is part of using it well. Four things sit outside this one:

• The HTTP status of the file. A robots.txt that returns a server error is treated very differently from one that returns a 404. Run curl -I https://yourdomain.com/robots.txt and paste the output into the HTTP Header Checker to confirm it returns 200 with a text content type.
• Whether pages are indexable. Robots.txt controls crawling, not indexing. A noindex lives in the page itself, so paste the page head into the Meta Tag Analyzer to check the robots meta tag.
• Directives beyond the core five. Host, Clean-param, and other non-standard lines are skipped rather than validated. Google ignores them too, so this rarely matters for search.
• What your policy should be. The analyzer tells you what a file does, not what it should do. Whether to block training crawlers, retrieval crawlers, or neither is a business call.

One habit avoids a subtle class of errors: keep each crawler in a single group. Google merges multiple groups that name the same user agent, but humans reading the file rarely do, and a second Googlebot group added months later is easy to miss in review.

What should I do after the analysis?

If the file needs more than a couple of edits, regenerate it rather than patching. The Robots.txt Generator builds a clean file from presets and per-crawler toggles, and how to use the Robots.txt Generator walks through deployment. Then paste the generated output back into the analyzer to verify it before it goes live. Google generally caches robots.txt for up to 24 hours, so a fix may take a day to be picked up.

Next, check the sitemap your Sitemap line points to. If it lists URLs your robots.txt blocks, you are sending crawlers contradictory instructions. Paste it into the XML Sitemap Validator, or rebuild it with the steps in how to use the XML Sitemap Generator. If unblocked pages are still slow to appear, the playbook in how to get pages indexed faster picks up where robots.txt leaves off.

Finally, zoom out. Robots.txt is one line item in technical health. The Website Grader gives the broad view, the llms.txt and AI crawler guide covers the companion file some AI tools read, and the AI Visibility Score shows whether assistants are actually surfacing your brand once crawlers can reach you. Google's introduction to robots.txt and OpenAI's crawler documentation are the two primary sources worth bookmarking.

Run the robots.txt Analyzer after every deploy that touches routing, CMS plugins, or hosting. It is the cheapest insurance in technical SEO.

Frequently Asked Questions

Is the robots.txt Analyzer free, and does it store my file?

Yes, it is free with no sign-up. The parsing, scoring, and URL testing all run in your browser, so the robots.txt content you paste or upload is not sent to a server or stored. That makes it safe to check a staging file or a draft you have not deployed yet.

Can the analyzer fetch robots.txt directly from my domain?

No. You paste the file contents into the editor or upload a .txt file. Open yourdomain.com/robots.txt in a browser and copy what the server actually returns, because that live response is what crawlers read, not the copy in your repository or CMS settings.

What does a score of 100 mean?

It means the analyzer found no syntax errors, no warnings, a sitemap reference, and a wildcard group. It does not mean the rules match your intent. A file that accidentally blocks your blog for every crawler can still score 100, which is why you should always test your most important URL paths with the URL Path Tester.

Why does the URL Path Tester say a path is blocked when I never blocked it?

Robots.txt rules are prefix matches. A rule like Disallow: /search also blocks /searchlight and /search-tips, because both paths start with /search. The tester shows the exact rule that decided the result, so you can tighten it, for example by adding a trailing slash so it only matches the folder.

Does the analyzer check AI crawlers like GPTBot and ClaudeBot?

The URL Path Tester includes GPTBot and CCBot alongside Googlebot, Bingbot, Slurp, and DuckDuckBot. For crawlers not in the list, such as ClaudeBot, OAI-SearchBot, or PerplexityBot, choose the any agent option: if your file has no group naming that bot, the wildcard group is exactly what it will follow.