The Robots.txt Generator is a free GrowthGPT tool that builds a valid robots.txt file from toggles instead of hand-written syntax. You set access for 17 named crawlers across search, AI, and social categories, pick a wildcard default, add disallowed paths, set an optional crawl delay and sitemap URL, then copy or download the result.
Robots.txt is four lines of plain text that quietly decides whether anything else you do with SEO or AI visibility has a chance to work. A stray Disallow: / left over from a staging deploy can remove a site from search and from every AI assistant at once. This guide covers what the Robots.txt Generator produces, which preset to start from, how the AI crawler toggles actually behave, and the mistakes worth checking for before you ship the file.
Image: A toggle panel of search, AI, and social crawler names on the left, with a live robots.txt preview updating line by line on the right
What does the Robots.txt Generator do?
It turns crawler policy into a set of switches and writes the syntax for you. Everything runs client-side, so there is no rate limit, no account, and no waiting: the preview pane updates as you toggle. The controls are:
- Presets. Standard, Block AI Bots, Strict, and Allow All. Each preset rewrites every toggle at once, so apply the preset first and adjust afterwards.
- Search crawlers. Googlebot, Bingbot, Yandexbot, DuckDuckBot, Baiduspider, and Slurp.
- AI crawlers. GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot, Bytespider, and Amazonbot.
- Social crawlers. Applebot, Twitterbot, facebot, and LinkedInBot, which power link previews rather than search.
- Wildcard access. The default rule for every agent you did not name. This is the single most consequential setting in the file.
- Disallowed paths, crawl delay, and sitemap URL. Path blocks apply to the wildcard group, the delay is optional, and the sitemap line points crawlers at your index.
Which preset should I start from?
| Preset | What it sets | Use it when |
|---|---|---|
| Standard | All crawlers allowed, blocks /admin/, /private/, /tmp/ | Almost every marketing site and blog |
| Block AI Bots | Search engines allowed, all AI crawlers disallowed | Paid or licensed content you do not want in training data |
| Strict | Wildcard disallow, only Googlebot, Bingbot, and Slurp allowed | Private tools, internal apps, staging hosts |
| Allow All | No restrictions at all | Documentation and reference sites that want maximum reach |
For a business that wants to be found, Standard is the answer roughly nine times out of ten. Strict is the preset to handle carefully: a wildcard disallow means any crawler you have not explicitly named, including every AI assistant, is shut out. That is a deliberate choice for a staging host and an expensive accident on a production site.
How do the AI crawler toggles actually behave?
This is where most robots.txt files go wrong, because the AI companies use different agents for different jobs and the names do not announce which is which. Training crawlers collect content for model training. Search crawlers fetch pages so an assistant can retrieve and cite them live. Blocking the first costs you nothing in visibility. Blocking the second removes you from AI answers entirely.
OpenAI documents the distinction in its bots documentation, and Google separates Google-Extended from Googlebot for the same reason, which its AI features documentation spells out: opting out of Google-Extended does not affect crawling, indexing, or ranking in Search. The practical rule is to block training agents if licensing matters to you, and never block retrieval agents unless you have decided you do not want AI citations. The generator covers the training-side agents by name; if you also want to control an agent that is not in the list, add the extra User-agent block to the downloaded file before deploying it.
The strategic question of whether to block anything at all is worth a separate think. Should you block AI crawlers works through the trade-off by business model, and how AI crawlers work explains what each agent does once it arrives.
How do I generate and deploy the file?
Step 1: Apply a preset, then adjust
Presets overwrite individual toggles, so choosing one after you have customised the list will discard your work. Preset first, tweak second.
Step 2: Add the paths that genuinely need blocking
Good candidates are admin areas, internal search results, faceted filter URLs that generate infinite combinations, cart and checkout steps, and thank-you pages. Bad candidates are CSS and JavaScript directories: blocking those stops rendering and can wreck how a page is understood. Remember that a disallowed path is public information, so never use robots.txt as a hiding place for anything sensitive.
Step 3: Add the sitemap URL
The sitemap directive is the cheapest discovery win in the file, and it is host-independent, so it works for every crawler that reads robots.txt. Point it at your live index. If you do not have one yet, build it with the XML Sitemap Generator and check it with the Sitemap Validator first.
Step 4: Read the preview before you copy it
The preview is the actual file. Read it top to bottom and confirm two things: that no group you care about carries Disallow: /, and that the wildcard group says what you intended. Most catastrophic robots.txt incidents are visible in ten seconds of reading and invisible for months afterwards.
Step 5: Deploy at the root and verify live
The file must be served from the root of each host as plain text. Subdomains need their own file, and so does each protocol. On Next.js, prefer generating it from app/robots.ts so the rules ship with the code and get reviewed like code. After deploying, load the URL yourself, then run it through the Robots.txt Analyzer to confirm the rules parse the way you read them. Google’s Search Essentials cover the crawlability baseline this file protects.
What are the most common robots.txt mistakes?
- Shipping the staging file. A site-wide disallow that survives a deploy is the single most expensive SEO bug there is, and nothing in your analytics announces it. Check the live file after every launch.
- Expecting it to deindex pages. Robots.txt governs crawling. A blocked URL can still surface in results from external links, and the crawler can never see the noindex tag you added, because it is not allowed to fetch the page.
- Blocking assets. Disallowing CSS, JS, or image directories degrades rendering and hurts how both search and AI systems interpret the page.
- Blocking retrieval agents by accident. Reaching for a broad AI block to protect training data usually removes you from AI answers as well. Decide the two questions separately.
- Forgetting subdomains.A blog on a subdomain has its own robots.txt, and inheriting nothing from the apex is the normal cause of “why is our blog not indexed?”
Once the file is correct, the rest of technical visibility is worth a pass too. Run the Website Grader for the broad view, the GEO Audit for AI readiness, and the AI Visibility Score to confirm assistants can actually see and cite you afterwards.
Frequently Asked Questions
What does the Robots.txt Generator do?
The Robots.txt Generator is a free GrowthGPT tool that builds a valid robots.txt file from toggles instead of hand-written syntax. You set access for 17 named crawlers across search, AI, and social categories, choose a wildcard default, add disallowed paths, set an optional crawl delay and sitemap URL, and copy or download the finished file. It runs entirely in the browser with a live preview.
Which preset should I choose?
Standard suits almost every site: all crawlers allowed, with admin, private, and temp paths blocked. Block AI Bots keeps search engines allowed while disallowing every AI crawler. Strict disallows everything by default and allows only Googlebot, Bingbot, and Slurp. Allow All removes every restriction. Start from Standard unless you have a specific reason not to.
Does blocking AI crawlers hurt my Google rankings?
No. Google-Extended is a separate token from Googlebot, so disallowing it stops your content being used for Gemini model training without affecting crawling, indexing, or ranking in Google Search. The same separation applies to OpenAI, which uses different user agents for training and for search retrieval. Blocking the wrong one costs you AI citations rather than rankings.
Where does the robots.txt file go?
It must sit at the root of each host and protocol, served at https://yourdomain.com/robots.txt as plain text. A file at a subfolder path is ignored, and subdomains need their own file. On Next.js, generate it from app/robots.ts so it ships with your deploy rather than living as a forgotten static file.
Does robots.txt stop a page appearing in search results?
No. Robots.txt controls crawling, not indexing. A disallowed URL can still be indexed without its content if other pages link to it. To keep a page out of results, allow crawling and use a noindex meta robots tag, or protect it with authentication. Blocking in robots.txt actually prevents the crawler from ever seeing the noindex tag.
Should I set a crawl delay?
Rarely. Crawl-delay is not supported by Googlebot and is honoured inconsistently elsewhere. Use it only when server logs show an aggressive crawler causing real load, and prefer fixing the underlying capacity or blocking that specific agent. Setting a broad delay slows discovery of new content for no benefit.