Free AI Crawler Checker

Almost every tool in this category sorts crawlers into two piles: bots that collect training data, and bots that fetch pages to cite in answers. It is a useful rule of thumb and it is how most advice is written. Two cases break it — and both break it in the direction that flatters you, which is why they are easy to miss and expensive to get wrong.

• bingbot is Microsoft's search crawler, and nobody writing a rule for it in 2019 was thinking about AI. But Microsoft Copilot has no crawler of its own. Microsoft describes the same crawl as serving both jobs — it collects pages once, and both Bing search and Copilot draw on what it collected. So a rule you wrote years ago for search decides an AI assistant today, and you cannot change that by naming an AI bot. Read as “just search” and skipped over, it quietly decides an assistant.
• Google-Extended never visits your site. It sends no requests at all — it is a permission setting Google applies to pages Googlebot had already collected. Google documents it as covering both model training andwhether Gemini Apps may draw on your pages when answering. Filed under “training” it looks free to block; in practice that one line protects your content and removes you from Gemini Apps at the same time. The trap is not the cost. It is that you think you are making one decision.

This is why the tool reports what each crawler is for, which assistants it actually decides, and the line that decided them — instead of sorting your file into two piles and handing you a number.

Plenty of publishers act on this split, and a good share of them end up somewhere they did not intend. In a 2026 study of 100 top US and UK news sites, BuzzStream found that 79% block at least one AI crawler of some kind — most often a training crawler, which costs them nothing in whether assistants can cite them — while 71% also block at least one of the crawlers that fetch pages at answer time. That second group is the one paying a price it may not have chosen. (A hundred large news publishers is not the web; read it as a pattern among big publishers, not a general rate.)

The distinction itself is real and worth knowing. Training crawlers collect content to build and refine models: once your pages are in a training set there is no un-train button, and blocking one only prevents future collection. Crawlers that fetch pages at answer time are the opposite — block one and your content stops appearing in that assistant's answers straight away, because nothing can be cited that cannot be read.

Just don't treat it as a filing system. As the two cases above show, a search crawler can decide an assistant, and something that never visits you at all can still cost you one. The version of this advice that survives contact with a real file is simple: read your own result rather than working from someone's list.

No. And this is not a matter of studies or averages — it is simply how the plumbing works. Google Search is crawled by Googlebot. A rule aimed at GPTBot, ClaudeBot or any other operator's crawler is not a rule aimed at Googlebot, so it cannot change what Google crawls, indexes or ranks. Google states the same thing directly about its own opt-out setting: Google-Extended “does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.” Google's crawler documentation.

What that same page does not say is that Google-Extended only affects training. It covers training andwhether Gemini Apps may pull up your pages while answering someone. Your search rankings are safe either way; your presence in Gemini Apps is not. It is the one place where “blocking training costs nothing” turns out to be false.

AI Overviews — the AI answers that appear inside Google Search itself — are a separate question again. They draw on the ordinary Search index, so they are decided by Googlebot, not by Google-Extended. You do not need to allow Google-Extended to show up there.

The real calculation is not “will blocking hurt my SEO”. It is “which assistants am I closing, and did I mean to” — which you cannot answer from a list of bot names, only from your own file.

Your robots.txt is a declaration, not a fence. It is a plain text file at your domain root stating which crawlers may visit which paths, and for AI crawlers it is currently the main control anyone has. But following it is voluntary: nothing physically stops a crawler that decides to ignore it. Four things follow from that, and together they mark out what reading the file tells you and what it does not.

• Access is not visibility. Being crawlable, being indexed, being retrieved and being cited are four different things. A rule that permits an assistant to fetch your pages does not mean it will quote you, and this tool reports permission — never citation.
• Compliance is not universal. Major operators publish that they honour robots.txt, and some publish exemptions for pages a user asks for by name. At least one crawler has been measured continuing to fetch pages it had been disallowed. Where an operator's compliance is disputed or measured as absent, the results say so rather than picking a side.
• Blocking a crawler is not removing your content. Content already collected stays in models that were trained on it. Your material can also reach an assistant through republication, through open datasets, or through someone simply pasting your page into a chat.
• Asking is not blocking. Your robots.txt asks crawlers to stay out; it cannot force them. Actually turning a crawler away happens at your server or your CDN, and that is a different kind of product. Cloudflare's AI crawler tooling, for example, watches the traffic actually arriving at your site, blocks the crawlers you choose, and can spot ones ignoring your file. That is the right tool for enforcement, and worth knowing about — this one answers the question you have first, which is what your file is currently telling each assistant.

All of which leaves one question to answer first, and it is the one this tool exists for: what is the file you already publish actually saying to each assistant right now, and which line is saying it? Most site owners have never seen that answer for their own domain.

1. Start with the assistants, not the bots. The headline counts how many of the six named assistants can get at your content. An assistant that reaches most of your site still counts as reaching it — a site with one folder shut off is not the same thing as a site nobody can read. You will see “part of your site” when a rule closes off something that looks like content people came to read: a blog, docs, product or pricing section. Rules covering carts, logins, search results and the other plumbing every site has are left alone, because flagging those would mean raising an alarm about a site that is set up correctly.
2. Read the crawlers beside each verdict. Each assistant names the crawlers that decided it. This is where the two awkward cases show up in your own file: Copilot decided by a search crawler, Gemini Apps decided by something that never visits your site at all.
3. Check whether a rule named the crawler or swept it up. Each row says either that a group names that crawler, or that your User-agent: * catch-all caught it. This matters more than it sounds: the moment you write a rule naming a crawler, that crawler stops reading your catch-all rules entirely and follows only the ones addressed to it. So anything still being caught by the catch-all is usually governed by a rule written before that crawler existed — which makes those the lines worth looking at first.
4. Treat training access as a separate decision. Who may train on your content is reported apart from who may reach it, because blocking training crawlers is a legitimate content choice that usually costs no visibility. The exception is flagged where it applies.
5. Look past robots.txt. The results also read your homepage for AI meta directives and the X-Robots-Tag header, and check whether you publish llms.txt. If any of those could not be read, the result says so instead of quietly treating it as missing.
6. Compare against the notable sites, not against a score. The ten-site table above is read live from those sites' own files, so it shows what they do today. Your own result is deliberately not marked against the published industry figure shown beside it, because that figure counts something different — sites blocking any AI crawler at all, including the training ones that cost nothing. Where that figure is an estimate rather than a measurement for your industry, it says so.

People often reach for these four in the same week, so it helps to know which one answers which question. Cloudflare sits in front of your site and actually turns crawlers away. Google Search Console is the definitive record of what Google itself did, on sites you have verified. Screaming Frog crawls all your pages rather than reading one file. This tool answers the question none of them do: which named AI assistants your rules currently reach, and the line responsible for each.

Short version: robots.txt is policy, not behaviour. If you need to know which bots actually hit your server, or to stop the ones that ignore your file, use edge controls or log analysis — this reads the file, not your access logs. What it does that they do not is map your file onto the assistants people actually use, and show you the line that decided each one.