Blocking AI crawlers means adding disallow rules for agents like GPTBot, ClaudeBot, Google-Extended, and CCBot so they stop fetching your pages. For most businesses that sell something other than the content itself, blocking costs more visibility than it protects, because AI answers are now where buyers research.
The debate usually gets framed as a moral one. It is a commercial one, and the answer differs by business model. This guide separates the two kinds of crawler that get lumped together, walks the decision by business type, covers what robots.txt can and cannot actually enforce, and ends with a default policy most teams can ship today. For the mechanics of writing the file, pair this with how to use the Robots.txt Generator.
Image: A fork in a path, one branch labelled training crawlers leading to a licensing gate, the other labelled retrieval crawlers leading to a cited answer with a brand mention
What is the difference between training and retrieval crawlers?
Almost every bad decision in this area comes from treating “AI bots” as one category. They do two different jobs, and the commercial consequences of blocking them are opposite.
Training crawlers gather text that may be used to train future models. Nothing you publish today shows up in an answer because of them for months, and blocking them has no effect on whether an assistant can cite you tomorrow. Retrieval crawlers fetch pages so an assistant can quote and link them in an answer being composed right now. Block those and you are simply absent from the answer. OpenAI publishes the distinction between its agents in the OpenAI bots documentation, and Google keeps Google-Extended separate from Googlebot for exactly the same reason, as described in its AI features documentation.
| Crawler purpose | What blocking prevents | What blocking costs | Sensible default |
|---|---|---|---|
| Model training | Future models learning from your text | Nothing measurable in traffic or rankings | Business decision, safe either way |
| Live retrieval and citation | Assistants quoting and linking you | Citations, referral clicks, brand presence | Allow |
| Search indexing | Classic crawling and indexing | Organic search entirely | Allow |
| Link previews | Social cards rendering | Click-through on every shared link | Allow |
Should my business block AI crawlers?
One question settles most of it: is the content the product, or is the content the thing that sells the product?
Content is marketing: do not block
SaaS companies, agencies, ecommerce brands, consultants, and local businesses publish content to be found. Being quoted by an assistant in front of someone actively researching a purchase is the outcome you were paying for, and it arrives with attribution attached. Blocking here is self-harm dressed as caution. The bigger risk for these businesses is not being visible at all, which is what the AI Visibility Score exists to measure.
Content is the product: block training, allow retrieval
Subscription journalism, paid research, course libraries, and data products have a genuine conflict: free reuse of the full text substitutes for the sale. The workable shape is to gate the substance behind authentication, publish generous summaries and methodology pages openly, block training agents, and let retrieval agents cite the open pages. You keep the discovery and sell the depth.
Regulated or confidential material: authenticate, do not disallow
If a page must not be read by machines, robots.txt is the wrong control. It is a public list of the paths you consider sensitive, and it depends on the crawler choosing to obey. Put real access control in front of the content instead.
Can robots.txt actually stop AI companies using my content?
Partly, and it is worth being precise about the limits before you build a policy on it:
- It is voluntary. Major documented crawlers honour it. Scrapers that do not identify themselves simply ignore it.
- It is not retroactive. Content already collected in previous crawls stays collected. Blocking today changes tomorrow, not yesterday.
- It does not stop republication. An assistant can describe your material from a third-party summary that quoted you, without ever fetching your site.
- It does not deindex. Robots.txt governs crawling. Keeping a page out of results needs noindex or authentication.
- It is public. Anyone can read which paths you consider worth hiding.
Check what your file currently says before assuming it matches your intent. The Robots.txt Analyzer parses the live file the way a crawler does, which regularly surprises teams who inherited their rules from a template three years ago.
What does blocking actually cost?
The visible cost is referral traffic from assistants, which is small today for most sites and growing. The larger cost is absence from the answer itself. When a buyer asks an assistant which vendor suits their situation, the shortlist is composed from sources it could retrieve. A blocked site is not evaluated and rejected. It is never considered, and nothing in your analytics reports the loss.
There is an upside worth naming too: content that is structured to be cited tends to be structured well for people. The Princeton GEO study found that adding quotations, statistics, and citations lifted visibility in generative answers by up to 40 percent. Those are the same edits that make a page more credible to a human reader.
What is the recommended default policy?
For a business whose content exists to win customers:
- Allow all search crawlers and all retrieval crawlers without exception.
- Decide training crawlers deliberately. Blocking Google-Extended and similar training tokens is safe for rankings if you want the licensing stance.
- Disallow only operational paths: admin, cart and checkout, internal search, and infinite filter combinations.
- Put anything that must stay private behind authentication, never behind a disallow rule.
- Generate the file with the Robots.txt Generator, verify it live with the analyzer, and re-check it after every major deploy.
Then spend the energy on the part that actually moves the number. Google’s Search Essentials still describe the crawlability floor, and being reachable is only step one. Run a GEO Audit and the AEO Ready Checker on your key pages, and if you are reachable but still absent from answers, why your content is not cited by ChatGPT and query fan-out cover the two usual causes.
Frequently Asked Questions
Should I block AI crawlers?
For most businesses that sell something other than the content itself, no. Blocking removes you from AI answers where buyers now research, and it does not stop content already in older training sets. Blocking makes sense when the content is the product, such as subscription journalism, licensed research, or paid courses, where free reuse cannibalises revenue.
What is the difference between a training crawler and a retrieval crawler?
Training crawlers collect content that may be used to train future models. Retrieval crawlers fetch pages live so an assistant can quote and cite them in an answer right now. Blocking a training crawler costs you nothing in visibility. Blocking a retrieval crawler removes you from AI answers entirely, which is the mistake most sites make when they apply a broad AI block.
Does blocking Google-Extended affect my Google rankings?
No. Google documents Google-Extended as a separate control from Googlebot: opting out affects whether your content helps train and ground Gemini models, not whether Google Search crawls, indexes, or ranks your pages. It is the one block you can apply with confidence that classic search performance is unaffected.
Can robots.txt actually stop AI companies using my content?
Only for crawlers that choose to honour it. Robots.txt is a voluntary protocol with no enforcement, so it reliably controls the major documented agents and does nothing about scrapers that ignore it or content already collected in past crawls. If you need enforcement rather than a request, you need authentication, rate limiting at the edge, or licensing terms.
What happens to my traffic if I block AI crawlers?
You lose the citations and the referral clicks that come with them, and competitors fill the space in answers your buyers read. You do not lose Google rankings if you blocked only AI-specific tokens. The honest trade is fewer AI-driven visits and less brand presence in assistant answers, in exchange for less unattributed reuse of your material.
What is the recommended default policy?
Allow retrieval crawlers everywhere, decide training crawlers by business model, and block nothing at path level except admin, checkout, and internal search. Publish the policy in robots.txt, put your best material where assistants can reach it, and gate the parts that genuinely carry revenue behind authentication rather than a robots directive.