Blocking AI Crawlers Doesn't Make You Invisible

You are about to edit your robots.txt, or you edited it months ago and something has just told you that was a mistake. A scanner flagged it red. A post said that if you block AI crawlers, you disappear from AI.

Most of that advice is wrong, and it is wrong in a way that costs real visibility. It collapses two unrelated decisions into one switch: whether your content trains a model, and whether an assistant can cite you. One of those is free. The other is not. What follows is what the 2026 data actually shows about each, engine by engine, using the operators' own documentation and the peer-reviewed research we could find on how crawlers behave.

Does blocking AI crawlers make you invisible to AI?

No. Blocking a training crawler costs you nothing in Search or AI Overviews. Blocking an answer-time crawler - the one an AI assistant sends to fetch your page while it is writing an answer - sharply reduces how often that particular assistant cites you. Those are two different decisions, and most advice in this space treats them as one.

The confusion is understandable, because one file controls both. Your robots.txt holds a single kind of instruction: this user agent may or may not fetch these paths. But the crawlers listed in it are doing two unrelated jobs.

Training access decides whether your content can be used to train a model. That is an intellectual property question. Answer access decides whether the crawlers that let an assistant cite you can reach your pages. That is a visibility question.

Blocking training is a legitimate, common choice. It is not a misconfiguration, and any audit that flags it as one is measuring the wrong thing.

What the evidence actually shows

cloro.dev, an AI-search research site, compared robots.txt rules against citation rates for 1,058 domains in July 2026. It counted citations per Google organic ranking, so a site ranking for a thousand keywords is not automatically credited with more citations than one ranking for ten. A score of 1.167 means roughly one AI citation for every keyword the site ranks for in Google. A score of 0 means the assistant never cited the site at all. Higher is better.

Domains blocking PerplexityBot scored a median of 0 in Perplexity, against 1.167 for domains allowing it. Domains blocking OpenAI's OAI-SearchBot scored 0 in ChatGPT, against 0.394.

Goodie, an AI-search analytics vendor, analysed 31 million citations across 11 AI surfaces and 105 news publishers. The New York Times blocks every major AI crawler. It took zero citations from ChatGPT, zero from Gemini and one from Claude.

In the same dataset, over the same period, the same newspaper took 37,642 citations from Grok, 8,007 from Google's AI Overviews, 4,303 from Perplexity and 971 from Copilot.

That is the whole story in one row. Against the AI companies that honour robots.txt, the block very nearly held: three compliant assistants produced one citation between them. The tens of thousands came almost entirely from other assistants that did not act on those rules.

So the honest unit of measurement is the assistant, never "AI" in aggregate. There is no single number for your AI visibility, and a tool that reports one is averaging away the only thing that matters. If you want a number, you want one per engine.

Three limits on both studies above. They are observational industry datasets rather than controlled experiments, so they show association rather than cause. Sites that block one crawler may well block several, and no published study we found separates those rules from each other. And across a sweep of arXiv, the ACM Digital Library, SSRN and the major web conferences, we found no peer-reviewed causal study of citation loss after blocking.

One more thing this article cannot tell you: what a citation is worth. We found no reliable published link between AI citations and traffic or revenue. The datasets above show presence and absence, not value. If you are trying to decide whether losing your Perplexity citations matters commercially, that number does not exist yet.

Does blocking GPTBot remove you from ChatGPT?

No. GPTBot is a training crawler, and OpenAI says so in its own crawler documentation: "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models."

Two other user agents decide whether ChatGPT can cite you.

OAI-SearchBot is the one that matters most. OpenAI: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."

ChatGPT-User handles fetches a person triggers by pasting a link. It carries a caveat that most blocklists ignore: "Because these actions are initiated by a user, robots.txt rules may not apply." Blocking it is a request, not a guarantee.

The number that appears to contradict this

The cloro.dev study cited earlier also found that domains blocking GPTBot scored a median of 0.003 in ChatGPT, against 0.417 for domains allowing it. Read quickly, that says blocking GPTBot removes you from ChatGPT, which contradicts OpenAI's own documentation.

The number cannot separate the two rules. The study read robots.txt files and counted citations; it did not isolate sites that had blocked GPTBot and nothing else. Blanket "block all AI bots" lists take OAI-SearchBot out at the same time, so a file that blocks GPTBot frequently blocks the citation crawler in the same breath, and the score reflects whatever the file did in total. We found no published measurement that isolates one rule from the other.

Which is the practical point rather than a footnote. If you block GPTBot alone, OpenAI's documentation says what you get: out of training, still eligible for citation. If you paste someone else's blocklist, you are buying an outcome nobody has measured.

OpenAI does note one complication: "we may use the results from just one crawl for both use cases to avoid duplicative crawling." That describes shared fetching, not shared permissions. The opt-out signal still attaches to the token you disallowed.

What does blocking Google-Extended actually control?

Less than almost anyone writing the rule expects, but not nothing. Google-Extended is not a crawler at all. Google's documentation is explicit that it "doesn't have a separate HTTP request user agent string" and that "the robots.txt user-agent token is used in a control capacity."

Nothing arrives when you allow it. Nothing stops arriving when you block it. It is a policy label applied to pages Googlebot already fetched.

What it controls is use of your content for training and for grounding in Gemini Apps and Vertex AI. That second half is a real cost and it is routinely left out: if you care about being surfaced inside Gemini specifically, blocking Google-Extended is not free.

What it does not control is Google Search. Google states that blocking it "does not impact a site's inclusion in Google Search."

That includes AI Overviews. Google is direct about which token governs its generative results: "robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search." AI Overviews and AI Mode run off the same index as blue links. Block Google-Extended and you remain fully eligible for both.

Apple ships a similar design. Applebot-Extended "does not crawl webpages" and "is only used to determine how to use the data crawled by the Applebot user agent."

Apple also documents a trap that breaks a common assumption. "If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions." A crawler you never named is not automatically unrestricted. It may be following rules you wrote for somebody else.

What each token actually controls

The pattern holds across every operator. GPTBot, ClaudeBot and the -Extended tokens are training controls with no documented cost to search or AI Overviews. Everything else is answer access.

Which robots.txt rules do nothing at all?

Three common ones, and the reason is always the same: the file's layout decides which crawler gets which rule, and it decides it without reference to what you meant.

First, the vocabulary, because the specification's term does the damage. A group is a User-agent: line plus the rules underneath it. RFC 9309, the official specification that defines how robots.txt files are read, works out which crawler belongs to which group purely from the file's structure.

Which crawler lands in which group is the whole game, and the file decides it by line position alone. Not by what you meant.

The file that names twenty crawlers and changes nothing

Consecutive User-agent: lines form a single group sharing the rules that follow. So a file listing twenty AI crawlers, ending with User-agent: *, then one set of rules, gives every named crawler exactly the same treatment as everybody else.

The naming accomplishes nothing at all. You can check this one yourself: as of August 2026, Squarespace's robots.txt opens with more than two dozen AI crawler tokens on consecutive lines - GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider and the rest - ending with User-agent: *, and only then reaches its first rule, Disallow: /config. Every one of those named crawlers gets exactly the same treatment as every other bot on the internet.

This is a platform default rather than anyone's decision. The Squarespace sites in our own sample carry the same file, byte-for-byte, differing only in the sitemap URL.

Which is why it matters more than a curiosity. An owner who opens that file sees twenty-something AI crawler names and reasonably concludes something is being managed on their behalf. Nothing is.

The rule that does the opposite of what it says

This one fires on people who were paying attention.

A named group replaces the wildcard group. It does not add to it. The specification is clear that the * group applies only "if no matching group exists" for that crawler. So this file does not do what it looks like:

User-agent: *
Allow: /
Disallow: /private/

User-agent: GPTBot
Allow: /

Every crawler on the internet is kept out of /private/. GPTBot is not, because GPTBot now has its own group and never inherits the disallow. The author wrote a restriction and produced a permission, for precisely the crawler they were being careful about.

We found this pattern in a live file belonging to a site we monitor, where the owner had written his intent in a comment above it. He knew what he wanted. The file did the reverse, and nothing would have told him.

The rule meant for one crawler that lands on another

Only Allow and Disallow are rules under RFC 9309. Crawl-delay is not one, so it cannot end a group, and neither can a blank line or a comment. The specification is unusually direct about this, in section 2.2.4:

"Parsing of other records MUST NOT interfere with the parsing of explicitly defined records in Section 2. For example, a 'Sitemaps' record MUST NOT terminate a group."

So this file does not do what it plainly says:

User-agent: SlowBotA
Crawl-delay: 10

# Blocking this one, it hammers the server
User-agent: BadBotB
Disallow: /

The author wrote a rate limit for one crawler and a ban for another. Both crawlers get the ban. SlowBotA inherits a Disallow: / that nobody wrote for it, two lines and a section comment later.

All three share one sentence: group membership is decided by the file's structure, not by the author's intent. In one case nothing changes, in one a crawler escapes a rule everyone else has, and in one a crawler inherits a rule meant for its neighbour.

Two of the three only fire on people who were paying attention. A one-line robots.txt cannot hit any of them. Writing a more detailed file is not the same as writing a safer one.

How common are they? Nobody knows, and that includes us. We checked the HTTP Archive Web Almanac, Google's own published robots.txt analyses, Cloudflare Radar and the academic web-measurement literature: all of them count directives and user-agent names, none of them reconstruct which crawler ends up in which group. The shapes above are demonstrable. Their prevalence is unmeasured.

Rules aimed at a token its own vendor does not document

anthropic-ai appears widely in copied blocklists. Anthropic does not document this token. Its support documentation describes three crawlers: ClaudeBot for training, Claude-SearchBot for search indexing, and Claude-User for fetches triggered by a person.

Squarespace's file, quoted earlier, lists anthropic-ai fourth.

Anthropic is unusually clear about consequences, and worth quoting because so few operators are this direct. Blocking Claude-User "prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility."

The lesson is not that anthropic-ai is harmful. It is that blocklists get copied from other blocklists rather than from operator documentation, and they accumulate tokens their own vendors never published.

Blocks that only partly block

Perplexity states that PerplexityBot "will not index the full or partial text content of any site that disallows it via robots.txt." Then it adds the part people miss: "if a page is blocked, we may still index the domain, headline, and a brief factual summary."

A block there is not an absence.

What can't your robots.txt tell you?

Five things, plus one the courts have now confirmed: whether the crawler obeys, what the crawlers you cannot see are doing, whether fetched content may be used, what your CDN is enforcing, and who chose the rules in the first place.

Whether the crawler obeys

The best measurement available is Kim et al., ACM Internet Measurement Conference 2025. Read it for what it is: a controlled experiment on a single website at one US university, four robots.txt regimes over eight weeks, with a baseline measured earlier in the year rather than alongside. One careful site's experience, not the web's.

Three things in it matter to you. Scores run 0 to 1, where 1.000 means the crawler complied every time it was tested against that directive.

The major labs honoured the directive you would actually use. GPTBot, ClaudeBot and ChatGPT-User all scored 1.000 on disallow.

"Does this crawler obey robots.txt" has no answer. Only "does it obey this directive" does.

ClaudeBot is perfect on page access and weakest of the four on crawl-delay, on the same site in the same weeks. PerplexityBot scores low on disallow, though the study found only its endpoint-restriction change statistically significant, so read that row as unsettled.

AI crawlers behaved better than ordinary search crawlers, not worse. AI search crawlers scored 0.631 and AI data scrapers 0.559, against 0.447 for conventional search engine crawlers. The authors call the result surprising themselves.

And one finding that changes what you do: fewer than 40% of AI bots re-fetched robots.txt within a week, the lowest rate of any category. Changing your rules and having your rules take effect are different events, and nothing tells you when the second one happens.

The crawlers nobody can measure

That study logged 19,250 unique user agents across 231,859 IP addresses. Just 405 were known bots. The least compliant category it measured was headless browsers, at 0.155, which announce themselves as nothing at all.

Every compliance figure in this article, and every one you will read elsewhere, describes the self-declaring minority.

Access is not the same as use

Robots.txt governs whether a crawler may reach a page. A second family of directives governs whether content that was fetched perfectly legally may be used.

Google's robots meta tag documentation states that nosnippet "will also prevent the content from being used as a direct input for AI Overviews and AI Mode." Microsoft documents that NOARCHIVE "prevents content from being used in Copilot responses and grounding results." Apple documents nosnippet for Apple Intelligence.

These are per-vendor behaviours. No single directive means "no AI" across all of them. But a site can sit in the search index and be absent from the assistant, and nothing in its robots.txt would show it.

The surface no file exposes

A site can block Googlebot, bingbot and Applebot at its CDN while its robots.txt says nothing about any of them.

Those three build the indexes that AI Overviews, Copilot and Apple Intelligence draw on. So it is possible to read a robots.txt file correctly, conclude that answer access is untouched, and be wrong, because enforcement was happening at a layer that publishes nothing.

No file-based check can close that gap. Confirming whether a site lets Googlebot through means fetching as Googlebot, which means impersonating it. Worth knowing before trusting any robots.txt report, including reports that disagree with each other.

Rules that nobody chose

From 15 September 2026, Cloudflare applies new defaults: "For all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads." Search crawlers remain allowed. Owners can opt out in their security settings before that date.

Cloudflare also offers a managed robots.txt that writes training-crawler blocks on a site's behalf. We have seen a production file where every AI rule sat inside a # BEGIN Cloudflare Managed content marker. Nobody typed those rules.

After September, the presence of a training block stops being proof that a human decided anything.

And the courts read it the same way

In Ziff Davis v. OpenAI, the Southern District of New York rejected the argument that robots.txt is a technological measure that effectively controls access under the DMCA. A legal analysis of the ruling summarises the reasoning: robots.txt is "more akin to a sign than a barrier," and following it is "effectively voluntary" because a bot has to be built to look for the file at all.

Your robots.txt is a published request. Most large operators honour it. It was never a lock.

The proposed standards that would change this are not here yet. The IETF's AI preferences drafts remain pre-standard. Cloudflare's Content Signals policy is published, and no major AI lab documents honouring it. The noai and noimageai tags sit in no official specification, and Originality.ai, whose adoption dashboard tracks them, notes that major AI crawlers have not formally committed to honouring them.

So should you worry, and what should you do?

Three situations, three answers.

You have never edited your robots.txt. You are almost certainly fine on the file itself. Check your CDN anyway, below.

You pasted a blocklist from somewhere. This is the case worth ten minutes today. Open your file and look for OAI-SearchBot, Claude-SearchBot, Claude-User, ChatGPT-User and PerplexityBot. If any are disallowed, you opted out of citation in that assistant, probably without intending to.

You deliberately blocked training crawlers only. You are fine, and you are in good company. No tool should be telling you otherwise.

That last group is larger than the discourse suggests. Across 66 days we checked 71 domains. Sixty-one published a robots.txt whose contents we retained. Five of those sixty-one block an AI crawler at the site root - and all five block training crawlers only. Not one blocks an answer-time crawler.

That counts only root-level blocks, so five is a floor rather than a total: a site restricting an AI crawler from a single section does not appear in it. Small, mixed sample, checked a median of once each. Directional, not representative.

If you want to do the same, here is a file that separates the two decisions without falling into any of the traps above:

# Training crawlers blocked. No documented cost to Search or AI Overviews.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Everyone else, including answer-time crawlers, allowed.
User-agent: *
Allow: /
Disallow: /private/

The important detail is what is absent. OAI-SearchBot, Claude-SearchBot and PerplexityBot are not named anywhere in that file, so they fall under the * group and keep their access.

Do not be tempted to add an Allow line for them. The moment a crawler gets its own named section, it stops reading the * section - so your Disallow: /private/ would no longer apply to it.

If you do give a crawler its own group, repeat every rule it needs. A named group starts empty.

Do not swap tokens on advice, including this article's. Crawler tokens move, and a rename that looks like a cleanup can retire a rule that was working. If you are unsure, add the new token and keep the old line.

Then check the layer your file cannot see. In Cloudflare, that is Security, then Bots, then the AI bot policy; other CDNs put it under bot management or WAF rules. You are looking for whether the Training, Agent or Search categories are blocked at the edge, because those settings outrank anything in your file and nothing in your file will mention them.

Then re-check on a schedule, one reading per assistant. Operators changed their documented behaviour twice while this article was being written, and nothing notifies you. CompetLab tracks AI visibility per engine across ChatGPT, Claude and Gemini. If your crawlers are getting through and your content still is not being read, the JavaScript rendering problem is the next thing to check.