AI Crawler Checker

REST endpoints for AI Crawler Checker. Base URL https://api.competlab.com; every request needs a CL-API-Key header (see Authentication). Responses wrap in { item } or { items }; errors in { error }.

Check AI crawler accessibility for a domain

POST/v1/tools/ai-crawler-checker

Check which AI ASSISTANTS can fetch a public domain’s pages, read from its robots.txt. assistantAccess is the answer: one verdict per assistant — ChatGPT, Claude, Perplexity, Microsoft Copilot, Google AI Overviews, Gemini Apps — with the crawlers that decided each named beside it. Count that array for totals; no count is stored and there is deliberately no overall score, because a weighted index has no unit and shifts when our crawler catalog moves rather than when the site does. Every tracked crawler is also returned individually with what it is FOR, the line of robots.txt that decided it, and whether a rule NAMED it or a User-agent: * catch-all swept it up — the second is usually accidental and is the more actionable finding. modelTrainingAccess is reported separately and NEUTRALLY: blocking training crawlers is a legitimate content decision and is never a gap, with one mechanical exception the response documents. This reads permission, not outcomes — it never claims a site is or is not cited. When robots.txt cannot be read, the read outcome is returned and the entire verdict layer is ABSENT rather than assumed. Optionally tailor the industry context. Stateless live check — no project required, no data is persisted.

Returns · { item }

Request body — application/json

Field	Type	Description
`domain`	`string`	Target domain to check. Accepts a bare hostname or a full URL — normalized (lowercased, scheme and path stripped) before checking.
`industry`	`string`	Industry context used to tailor crawler recommendations and benchmarks. Defaults to “other” when omitted. Values: `news-media`, `arts-entertainment`, `law-government`, `finance-healthcare`, `saas-tech`, `ecommerce`, `other`.

Request


curl -X POST "https://api.competlab.com/v1/tools/ai-crawler-checker" \
  -H "CL-API-Key: YOUR_COMPETLAB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"stripe.com","industry":"saas-tech"}'

Response 200 OK

Example


{
  "item": {
    "domain": "competlab.com",
    "fetchedAt": "2026-05-16T14:30:00.000Z",
    "industry": "saas-tech",
    "scopeOfAnswer": "This reads your robots.txt — the access rules you publish. It does not measure whether AI assistants actually cite you. Rules written into the page itself, such as meta robots and X-Robots-Tag, don't change these verdicts; we report those separately in the homepage results, along with whether we could read that page at all.",
    "crawlers": [
      {
        "userAgent": "GPTBot",
        "displayName": "GPTBot",
        "operator": "OpenAI",
        "crawlerPurpose": "collects_training_data",
        "honoursRobotsTxt": "yes",
        "honoursRobotsTxtEvidence": "operator_documented",
        "operatorDocumentationUrl": "https://platform.openai.com/docs/bots",
        "operatorDocsLastCheckedOn": "2026-08-13",
        "status": "blocked",
        "matchingDirective": "Disallow: /",
        "matchingDirectiveLine": 14,
        "ruleAudience": "wildcard_catch_all"
      }
    ],
    "robotsTxt": {
      "found": true,
      "read": {
        "state": "read",
        "reason": "access_http_error_status",
        "httpStatusCode": 503
      },
      "content": "User-agent: GPTBot\nDisallow: /\n\nUser-agent: *\nAllow: /\n",
      "statusCode": 200
    },
    "homepage": {
      "read": {
        "state": "read",
        "reason": "access_http_error_status",
        "httpStatusCode": 503
      },
      "metaTags": [
        {
          "name": "robots",
          "content": "noai, noimageai"
        }
      ],
      "xRobotsTag": "noai, noimageai"
    },
    "llmsTxt": {
      "read": {
        "state": "read",
        "reason": "access_http_error_status",
        "httpStatusCode": 503
      },
      "sectionCount": 12,
      "linkCount": 47,
      "approxTokens": 1850
    },
    "llmsFullTxt": {
      "read": {
        "state": "read",
        "reason": "access_http_error_status",
        "httpStatusCode": 503
      },
      "sectionCount": 12,
      "linkCount": 47,
      "approxTokens": 1850
    },
    "assistantAccess": [
      {
        "assistantName": "ChatGPT",
        "crawlerAccessStatus": "can_reach_site",
        "appliesTo": "whole_site",
        "decidedByCrawlers": [
          {
            "userAgentToken": "OAI-SearchBot",
            "crawlerPurpose": "fetches_pages_to_cite_in_answers",
            "honoursRobotsTxt": "yes",
            "honoursRobotsTxtEvidence": "operator_documented",
            "decidedByRuleFor": "wildcard_catch_all",
            "decidedByDirective": "Disallow: /",
            "decidedByLineNumber": 14
          }
        ],
        "explanations": [
          {
            "code": "training_blocked",
            "text": "Blocking a training crawler asks these operators not to use your content for model training. It does not affect whether AI assistants can cite you."
          }
        ]
      }
    ],
    "modelTrainingAccess": [
      {
        "modelOperatorName": "OpenAI",
        "trainingAccessStatus": "may_use_content_for_training",
        "decidedByCrawlers": [
          {
            "userAgentToken": "OAI-SearchBot",
            "crawlerPurpose": "fetches_pages_to_cite_in_answers",
            "honoursRobotsTxt": "yes",
            "honoursRobotsTxtEvidence": "operator_documented",
            "decidedByRuleFor": "wildcard_catch_all",
            "decidedByDirective": "Disallow: /",
            "decidedByLineNumber": 14
          }
        ],
        "explanations": [
          {
            "code": "training_blocked",
            "text": "Blocking a training crawler asks these operators not to use your content for model training. It does not affect whether AI assistants can cite you."
          }
        ]
      }
    ],
    "industryPosition": {
      "industryLabel": "SaaS & Tech",
      "industryAnyAiCrawlerBlockRate": 25,
      "figureIsMeasured": false,
      "source": "arxiv:2510.09031",
      "sourceDate": "2025-10",
      "insight": "~25% of saas & tech sites block at least one AI crawler of some kind — most often a training crawler, which does not affect whether assistants can cite them. (arxiv:2510.09031, 2025-10.)"
    },
    "recommendations": [
      {
        "id": "answer-time-crawler-swept-up-by-wildcard",
        "title": "A catch-all rule is blocking crawlers it was never written for",
        "description": "OAI-SearchBot and PerplexityBot are blocked by your `User-agent: *` rule, not by any rule naming them. A named group replaces the wildcard group rather than adding to it, so naming a crawler and allowing it lifts it out of that rule and changes nothing else you wrote.",
        "priority": "high"
      }
    ]
  }
}

Fields

item — object

Field	Type	Description
`domain`	`string`	Normalized domain, as actually analyzed (after normalization)
`fetchedAt`	`string`	ISO-8601 timestamp of when the scan completed
`industry`	`string`	Industry echoed back — defaults to ‘other’ if caller didn’t supply one. Values: `news-media`, `arts-entertainment`, `law-government`, `finance-healthcare`, `saas-tech`, `ecommerce`, `other`.
`scopeOfAnswer`	`string`	What this answer covers and what it does not, in one sentence, meant to be shown beside the result rather than buried. Present whenever a verdict was reached.
`crawlers`	`array<object>`	Every catalog crawler with its derived status — evidence you can check line by line, not a scoreboard. Nothing counts this array; the headline counts assistants. Absent when robots.txt could not be read, because we evaluated no crawler and an empty array would read as ‘we looked and found none’. See `item.crawlers[]`.
`robotsTxt`	`object`	robots.txt fetch result. See `item.robotsTxt`.
`homepage`	`object`	The homepage read, carrying the AI meta robots directives and X-Robots-Tag it produced. Both live under ONE read outcome because they come from ONE request — and that request is the one most likely to fail, since robots.txt is a small text file while the homepage is HTML behind whatever bot protection the site runs. Check `homepage.read` before concluding anything from what is or is not here: an unread page used to publish as an empty tag list plus a missing header, which reads as two findings agreeing that the site has no page-level directives when it is one failure counted twice. See `item.homepage`.
`llmsTxt`	`object`	/llms.txt — the read outcome, and stats only where we read one. See `item.llmsTxt`.
`llmsFullTxt`	`object`	/llms-full.txt — the read outcome, and stats only where we read one. See `item.llmsFullTxt`.
`assistantAccess`	`array<object>`	THE HEADLINE: whether each AI assistant can fetch this site’s pages, with the crawlers that decided each verdict named beside it. Count this array to say ‘4 of 6’ — no count is stored anywhere, so a number and the names behind it cannot disagree. There is deliberately no aggregate score: a weighted 0-100 has no unit, no defensible denominator, and moves when our crawler catalog changes rather than when the site does. Absent when robots.txt could not be read. See `item.assistantAccess[]`.
`modelTrainingAccess`	`array<object>`	Whether each model operator may use this site’s content for training — reported as a NEUTRAL fact, never as a problem. Blocking a training crawler usually protects content at no cost to whether assistants can cite the site, because the two are governed by different crawlers, and many well-advised sites block training deliberately. ⚠️ THE EXCEPTION IS MECHANICAL AND MUST BE CHECKED, NOT ASSUMED: a few tokens govern training AND an assistant at once — Google documents `Google-Extended` as controlling both model training and grounding in Gemini Apps. Where a token under `decidedByCrawlers` here also appears under `assistantAccess[].decidedByCrawlers`, that block DOES cost reach. Match on `userAgentToken` before repeating the general rule. Absent when robots.txt could not be read. See `item.modelTrainingAccess[]`.
`industryPosition`	`object`	A published industry figure for context, with its source. Read the field descriptions before comparing it to anything — it counts a different quantity from assistantAccess. Absent when robots.txt could not be read. See `item.industryPosition`.
`recommendations`	`array<object>`	What to do, ordered by consequence. Absent when robots.txt could not be read — advice built on a file we never saw could break a working configuration. See `item.recommendations[]`.

item.crawlers[] — each entry

Field	Type	Description
`userAgent`	`string`	Robots.txt user-agent token for this crawler, exactly as the operator publishes it. Matching is case-insensitive per RFC 9309.
`displayName`	`string`	Name to show. Identical to the user-agent token — a prettified variant would be a second name for the same thing, and the point of showing it is that you can search your own file for it.
`operator`	`string`	The organization that operates this crawler. Where the operator does not document the token at all, the name says so rather than implying they claim it.
`crawlerPurpose`	`string`	What this crawler is FOR — the field that says what blocking it actually costs. ‘fetches_pages_to_cite_in_answers’: blocking costs VISIBILITY in AI answers. ‘collects_training_data’: blocking protects your content and costs no visibility; it is a legitimate choice, not a problem. ‘builds_search_index_that_grounds_assistants’: an ordinary search crawler that an assistant grounds on, so a rule written for search reaches the assistant too. ‘policy_token_does_not_crawl’: fetches nothing; it carries a policy that applies to other crawlers’ fetches. ‘seo_or_backlink_analysis’, ‘renders_link_previews’, ‘checks_ad_landing_pages’: no AI consequence. ‘purpose_unknown’: we see the token in the wild and could not source what it does — our gap, not a defect in your file. Values: `fetches_pages_to_cite_in_answers`, `collects_training_data`, `builds_search_index_that_grounds_assistants`, `policy_token_does_not_crawl`, `seo_or_backlink_analysis`, `renders_link_previews`, `checks_ad_landing_pages`, `purpose_unknown`.
`honoursRobotsTxt`	`string`	Whether this crawler’s operator says it obeys robots.txt. ‘not_for_user_initiated_requests’ matters most: several operators publish that their rules may not apply when a person pastes a URL and asks about it directly, so a rule against those reads as ‘you have asked them not to crawl you’, not ‘they cannot reach you’. Values: `yes`, `no`, `not_for_user_initiated_requests`, `not_for_security_or_integrity_checks`, `unknown`.
`honoursRobotsTxtEvidence`	`string`	How well we know that, which governs how firmly the claim is worded. ‘operator_documented’: their own words. ‘independently_measured’: a study, cited with its limits. ‘disputed’: a credible allegation the operator denies, reported as a dispute and never as a verdict. ‘undocumented’: nobody has published anything — which is not the same as ‘ignores robots.txt’. Values: `operator_documented`, `independently_measured`, `disputed`, `undocumented`.
`operatorDocumentationUrl`	`string`	The operator’s own documentation page. Absent where we could not find one.
`operatorDocsLastCheckedOn`	`string`	When a person last read that operator page. Operators move tokens and rewrite policies without notice, so this says what was published ON A DATE rather than implying it is current. Claims soften as it ages.
`status`	`string`	Whether this crawler can reach the site. ‘conditional’ means the homepage is reachable but a rule closes a recognised content area to it — ordinary operational paths such as /wp-admin/ do not count. Values: `allowed`, `blocked`, `conditional`.
`matchingDirective`	`string \| null`	The robots.txt line that decided the status, verbatim, or null when no directive matched. Nullable.
`matchingDirectiveLine`	`number`	1-based line number of that directive, so you can go straight to it in your own file.
`ruleAudience`	`string`	WHO THE RULE WAS WRITTEN FOR, which decides what you would edit. ‘names_this_crawler’: a group naming this crawler decided it — a policy somebody chose. ‘wildcard_catch_all’: no group names it and your `User-agent: *` rule decided it — frequently a rule written before the crawler existed, applying to it by accident. A named group REPLACES the wildcard group rather than adding to it (RFC 9309 §2.2.1), so naming a crawler lifts it out of the catch-all. Absent when nothing restricted this crawler. Values: `names_this_crawler`, `wildcard_catch_all`.

item.robotsTxt — object

Field	Type	Description
`found`	`boolean`	Whether a robots.txt file was read. False covers both ‘the site publishes none’ and ‘we could not read it’ — see ‘read’ for which.
`read`	`object`	Whether we actually read the file, and if not, why. Check this before interpreting anything else in the response. See `item.robotsTxt.read`.
`content`	`string`	Raw text content of the robots.txt file when fetched successfully
`statusCode`	`number`	HTTP status code of the robots.txt fetch

item.homepage — object

Field	Type	Description
`read`	`object`	Whether we read the homepage, and if not, why. Read this before interpreting `metaTags` or `xRobotsTag` — their absence means nothing until you know the page was fetched. See `item.homepage.read`.
`metaTags`	`array<object>`	AI-related meta robots directives found on the homepage. ABSENT when the page was not read — never an empty array, which would claim we looked and the page carries none. See `item.homepage.metaTags[]`.
`xRobotsTag`	`string`	Value of the X-Robots-Tag response header, where the homepage carried one. Present even on an error status, because a header on an error response is still a real observation.

item.llmsTxt — object

Field	Type	Description
`read`	`object`	Whether we read the file, and if not, why. There is deliberately no `found` boolean: `read` means the file exists and was read, `measured_absence` means the server returned 404 and it genuinely is not published, and `could_not_read` means we do not know — a refused request, an error status, or markup served with a 200 by a bot-protection page. A single boolean collapsed all of those into ‘this site has no llms.txt’, which is a claim about the site invented from a failed request. See `item.llmsTxt.read`.
`sectionCount`	`number`	Count of markdown section headers (`# foo` lines)
`linkCount`	`number`	Count of markdown inline links `[text](url)`
`approxTokens`	`number`	Rough token estimate (chars / 4, ceil)

item.llmsFullTxt — object

Field	Type	Description
`read`	`object`	Whether we read the file, and if not, why. There is deliberately no `found` boolean: `read` means the file exists and was read, `measured_absence` means the server returned 404 and it genuinely is not published, and `could_not_read` means we do not know — a refused request, an error status, or markup served with a 200 by a bot-protection page. A single boolean collapsed all of those into ‘this site has no llms.txt’, which is a claim about the site invented from a failed request. See `item.llmsFullTxt.read`.
`sectionCount`	`number`	Count of markdown section headers (`# foo` lines)
`linkCount`	`number`	Count of markdown inline links `[text](url)`
`approxTokens`	`number`	Rough token estimate (chars / 4, ceil)

item.assistantAccess[] — each entry

Field	Type	Description
`assistantName`	`string`	The assistant, as its users know it.
`crawlerAccessStatus`	`string`	Whether this assistant can obtain this site’s content: `can_reach_site`, `cannot_reach_site`, `can_reach_part_of_site`, or `blocked_but_may_not_be_honoured` — the last meaning a rule is written against crawlers whose operators publish an exemption, so protection is not guaranteed. Count the assistants in this array to get totals; no count is stored. IT IS NOT A CITATION MEASUREMENT: it says an assistant is permitted to obtain the content, never that it mentions or cites this site. Note also that a `cannot_reach_site` may be decided by a crawler whose `crawlerPurpose` is `policy_token_does_not_crawl` — that is consistent, not contradictory: see that field.
`appliesTo`	`string`	`whole_site` or `homepage_only`. robots.txt is site-wide; `nosnippet` and NOARCHIVE are per page and we read the homepage, so a verdict leaning on them describes that page alone.
`decidedByCrawlers`	`array<object>`	The crawlers that decided this verdict — the names behind the status. Never publish the verdict without them: ‘blocked’ is a claim, ‘blocked because these two tokens are disallowed’ is a finding. See `item.assistantAccess[].decidedByCrawlers[]`.
`explanations`	`array<object>`	Sentences to render verbatim about THIS assistant. ABSENT when there is nothing to say — never an empty array, which would claim we considered what needed saying and found nothing. See `item.assistantAccess[].explanations[]`.

item.modelTrainingAccess[] — each entry

Field	Type	Description
`modelOperatorName`	`string`	The operator whose training use is governed. Usually the company whose models are affected; `Common Crawl` is the exception — it trains nothing, but publishes the open corpus much training is built from, so a site owner blocking it is making this same decision.
`trainingAccessStatus`	`string`	`may_use_content_for_training`, `may_not_use_content_for_training`, `may_use_part_of_content_for_training`, or `blocked_but_may_not_be_honoured` — the last meaning a rule is written but the deciding crawler is not reliably bound by robots.txt, either because its operator publishes an exemption or because it is measured not to obey the file at all. Read `decidedByCrawlers[].honoursRobotsTxt` and `honoursRobotsTxtEvidence` to tell those two apart; they are different facts and only the second means the content should be treated as taken. AS A RULE, blocking training crawlers costs no AI visibility and is a legitimate content decision — do not report it as a problem, a gap, or something to undo. THE RULE HAS ONE EXCEPTION AND YOU MUST CHECK FOR IT: a few tokens govern training AND an assistant at once, `Google-Extended` being the documented case. Where the same crawler in `decidedByCrawlers` also appears under an entry in `assistantAccess`, that block DOES carry a visibility consequence and the exception applies — report it. The check is mechanical: match on `userAgentToken`. Never suppress a finding on the strength of the general rule without running it.
`decidedByCrawlers`	`array<object>`	The crawlers that decided this verdict. Match their `userAgentToken` against the tokens under `assistantAccess[].decidedByCrawlers` to find the dual-purpose case the status description warns about. See `item.modelTrainingAccess[].decidedByCrawlers[]`.
`explanations`	`array<object>`	Sentences to render verbatim about THIS operator. ABSENT when there is nothing to say — never an empty array, which would claim we considered what needed saying and found nothing. See `item.modelTrainingAccess[].explanations[]`.

item.industryPosition — object

Field	Type	Description
`industryLabel`	`string`	Human-readable industry label
`industryAnyAiCrawlerBlockRate`	`number`	Published share of sites in this vertical blocking at least one AI crawler OF ANY KIND, training crawlers included. ⚠️ This is NOT the same quantity as assistantAccess: it is dominated in practice by training crawlers, which do not affect whether assistants can cite a site. Do not read it as ‘this share of the industry is unreachable to AI assistants’.
`figureIsMeasured`	`boolean`	False where the published figure is interpolated between studies rather than directly measured for this vertical. The insight sentence marks interpolated figures with ’~’.
`source`	`string`	The study the figure comes from.
`sourceDate`	`string`	When the source was published or last read. Absent where we do not know — absent means unknown, never current.
`insight`	`string`	The figure in a sentence, stating what it counts. It deliberately does not place this site on a scale against it, because the two measure different things.

item.recommendations[] — each entry

Field	Type	Description
`id`	`string`	Stable machine-readable id
`title`	`string`	Short title
`description`	`string`	The detail. Recommendations EXPLAIN what a configuration costs; they suggest undoing one only where the evidence says it was not chosen deliberately — a rule naming a crawler is a decision, a wildcard rule that swept one up is usually not. Blocking a training crawler is never presented as a problem to fix.
`priority`	`string`	Priority, ordered by consequence rather than by how many crawlers are involved. Values: `high`, `medium`, `low`.

item.robotsTxt.read — object

Field	Type	Description
`state`	`string`	Whether we read THIS resource, and if not, why. The same block appears on every resource the check reads — robots.txt, the homepage, llms.txt and llms-full.txt — so read it against the field it sits on rather than assuming robots.txt. ‘read’ means we retrieved and interpreted it. ‘measured_absence’ means the server answered 404 and the resource genuinely is not published — a real result, and for robots.txt specifically that is the state RFC 9309 treats as allowing every crawler. ‘could_not_read’ means we could not reach or interpret it, so nothing beside it may be read as a finding about the site; where that happens to robots.txt, the entire verdict layer is absent rather than assumed. A failed read is not an open site. Values: `read`, `measured_absence`, `could_not_read`.
`reason`	`string`	Why the read did not complete. Present only when state is ‘could_not_read’. Each value names what WE observed, never a claim about the site. ‘access_transport_error’ — no answer came back at all. ‘access_http_error_status’ — the server answered with a status that settles nothing, such as 5xx, 403 or 429. ‘access_body_not_robots’ — a success status carrying markup rather than a robots.txt, usually a bot-protection page. ‘access_body_not_expected_format’ — the same thing on llms.txt: a success status carrying markup where a plain-text file was expected, which is a page we could not read and never evidence that the file is absent. Values: `access_transport_error`, `access_http_error_status`, `access_body_not_robots`, `access_body_not_expected_format`.
`httpStatusCode`	`number`	HTTP status observed for this file, where one was observed

item.homepage.read — object

Field	Type	Description
`state`	`string`	Whether we read THIS resource, and if not, why. The same block appears on every resource the check reads — robots.txt, the homepage, llms.txt and llms-full.txt — so read it against the field it sits on rather than assuming robots.txt. ‘read’ means we retrieved and interpreted it. ‘measured_absence’ means the server answered 404 and the resource genuinely is not published — a real result, and for robots.txt specifically that is the state RFC 9309 treats as allowing every crawler. ‘could_not_read’ means we could not reach or interpret it, so nothing beside it may be read as a finding about the site; where that happens to robots.txt, the entire verdict layer is absent rather than assumed. A failed read is not an open site. Values: `read`, `measured_absence`, `could_not_read`.
`reason`	`string`	Why the read did not complete. Present only when state is ‘could_not_read’. Each value names what WE observed, never a claim about the site. ‘access_transport_error’ — no answer came back at all. ‘access_http_error_status’ — the server answered with a status that settles nothing, such as 5xx, 403 or 429. ‘access_body_not_robots’ — a success status carrying markup rather than a robots.txt, usually a bot-protection page. ‘access_body_not_expected_format’ — the same thing on llms.txt: a success status carrying markup where a plain-text file was expected, which is a page we could not read and never evidence that the file is absent. Values: `access_transport_error`, `access_http_error_status`, `access_body_not_robots`, `access_body_not_expected_format`.
`httpStatusCode`	`number`	HTTP status observed for this file, where one was observed

item.homepage.metaTags[] — each entry

Field	Type	Description
`name`	`string`	Meta tag name (currently always ‘robots’)
`content`	`string`	Directive value, e.g. ‘noai’, ‘noimageai’, ‘noml’, ‘noimageml’

item.llmsTxt.read — object

Field	Type	Description
`state`	`string`	Whether we read THIS resource, and if not, why. The same block appears on every resource the check reads — robots.txt, the homepage, llms.txt and llms-full.txt — so read it against the field it sits on rather than assuming robots.txt. ‘read’ means we retrieved and interpreted it. ‘measured_absence’ means the server answered 404 and the resource genuinely is not published — a real result, and for robots.txt specifically that is the state RFC 9309 treats as allowing every crawler. ‘could_not_read’ means we could not reach or interpret it, so nothing beside it may be read as a finding about the site; where that happens to robots.txt, the entire verdict layer is absent rather than assumed. A failed read is not an open site. Values: `read`, `measured_absence`, `could_not_read`.
`reason`	`string`	Why the read did not complete. Present only when state is ‘could_not_read’. Each value names what WE observed, never a claim about the site. ‘access_transport_error’ — no answer came back at all. ‘access_http_error_status’ — the server answered with a status that settles nothing, such as 5xx, 403 or 429. ‘access_body_not_robots’ — a success status carrying markup rather than a robots.txt, usually a bot-protection page. ‘access_body_not_expected_format’ — the same thing on llms.txt: a success status carrying markup where a plain-text file was expected, which is a page we could not read and never evidence that the file is absent. Values: `access_transport_error`, `access_http_error_status`, `access_body_not_robots`, `access_body_not_expected_format`.
`httpStatusCode`	`number`	HTTP status observed for this file, where one was observed

item.llmsFullTxt.read — object

Field	Type	Description
`state`	`string`	Whether we read THIS resource, and if not, why. The same block appears on every resource the check reads — robots.txt, the homepage, llms.txt and llms-full.txt — so read it against the field it sits on rather than assuming robots.txt. ‘read’ means we retrieved and interpreted it. ‘measured_absence’ means the server answered 404 and the resource genuinely is not published — a real result, and for robots.txt specifically that is the state RFC 9309 treats as allowing every crawler. ‘could_not_read’ means we could not reach or interpret it, so nothing beside it may be read as a finding about the site; where that happens to robots.txt, the entire verdict layer is absent rather than assumed. A failed read is not an open site. Values: `read`, `measured_absence`, `could_not_read`.
`reason`	`string`	Why the read did not complete. Present only when state is ‘could_not_read’. Each value names what WE observed, never a claim about the site. ‘access_transport_error’ — no answer came back at all. ‘access_http_error_status’ — the server answered with a status that settles nothing, such as 5xx, 403 or 429. ‘access_body_not_robots’ — a success status carrying markup rather than a robots.txt, usually a bot-protection page. ‘access_body_not_expected_format’ — the same thing on llms.txt: a success status carrying markup where a plain-text file was expected, which is a page we could not read and never evidence that the file is absent. Values: `access_transport_error`, `access_http_error_status`, `access_body_not_robots`, `access_body_not_expected_format`.
`httpStatusCode`	`number`	HTTP status observed for this file, where one was observed

item.assistantAccess[].decidedByCrawlers[] — each entry

Field	Type	Description
`userAgentToken`	`string`	The crawler’s user-agent token exactly as its operator publishes it.
`crawlerPurpose`	`string`	What blocking this crawler costs. `fetches_pages_to_cite_in_answers` costs visibility; `collects_training_data` costs none and is a content decision; `builds_search_index_that_grounds_assistants` costs visibility indirectly, through the assistant built on that index; `seo_or_backlink_analysis`, `renders_link_previews` and `checks_ad_landing_pages` have no AI consequence. `policy_token_does_not_crawl` names a token that issues no HTTP requests of its own — it is a CONTROL the operator applies to content fetched by a different crawler, and blocking it still changes what an assistant may do, which is why such a token can decide a `cannot_reach_site`. `Google-Extended` is the case to understand: Google documents it as governing both model training and grounding in Gemini Apps, so one directive carries a training consequence and a visibility consequence at once. Read this field before treating any block as costly or costless.
`honoursRobotsTxt`	`string`	Whether a rule against this token actually binds. `yes` it binds; `no` it does not; `not_for_user_initiated_requests` and `not_for_security_or_integrity_checks` mean the operator publishes a carve-out for that case, so a rule is written but not guaranteed; `unknown` means nobody publishes either way. ⚠️ WHAT ‘BINDS’ MEANS DEPENDS ON THE NEIGHBOURING `crawlerPurpose`. For a crawler that fetches, it is whether the crawler obeys the directive. For `policy_token_does_not_crawl` it cannot be — such a token issues no requests — and there it means the OPERATOR honours the directive when deciding what to do with content another crawler already fetched. Both are ‘the rule has effect’; only the mechanism differs. ⚠️ THIS FIELD IS NOT ON ITS OWN A STATEMENT OF OBSERVED BEHAVIOUR. It carries the operator’s published position OR a measured result, and `honoursRobotsTxtEvidence` is what tells you which. Never quote this value without reading that one.
`honoursRobotsTxtEvidence`	`string`	How well we know the value beside it, and it qualifies that value rather than merely annotating it. `operator_documented` is the operator’s own words. `independently_measured` cites a study, so the value is an observation. `disputed` means a credible allegation the operator denies — a `yes` under `disputed` is the operator’s CLAIM and contested, so word it as a claim and never as ‘this crawler respects robots.txt’. `undocumented` means nobody publishes either way; never read it as ‘ignores’.
`decidedByRuleFor`	`string`	WHICH KIND OF RULE decided this crawler, which is what says where to edit. `names_this_crawler`: a group naming it decided the outcome — somebody chose that. `wildcard_catch_all`: no group names it and the `User-agent: *` rule decided it, frequently a rule written before the crawler existed and applying to it by accident. A named group REPLACES the wildcard group rather than adding to it (RFC 9309 §2.2.1), so naming a crawler lifts it out of the catch-all entirely. ABSENT when nothing restricted this crawler — there is no audience for a rule that does not exist. Values: `names_this_crawler`, `wildcard_catch_all`.
`decidedByDirective`	`string`	The directive that decided it, verbatim, so the claim can be checked against the site’s own file rather than taken on trust.
`decidedByLineNumber`	`number`	1-based line number of that directive.

item.assistantAccess[].explanations[] — each entry

Field	Type	Description
`code`	`string`	Stable identifier, so a consumer can branch without parsing prose.
`text`	`string`	The sentence. Render it verbatim; do not paraphrase or substitute your own.

item.modelTrainingAccess[].decidedByCrawlers[] — each entry

Field	Type	Description
`userAgentToken`	`string`	The crawler’s user-agent token exactly as its operator publishes it.
`crawlerPurpose`	`string`	What blocking this crawler costs. `fetches_pages_to_cite_in_answers` costs visibility; `collects_training_data` costs none and is a content decision; `builds_search_index_that_grounds_assistants` costs visibility indirectly, through the assistant built on that index; `seo_or_backlink_analysis`, `renders_link_previews` and `checks_ad_landing_pages` have no AI consequence. `policy_token_does_not_crawl` names a token that issues no HTTP requests of its own — it is a CONTROL the operator applies to content fetched by a different crawler, and blocking it still changes what an assistant may do, which is why such a token can decide a `cannot_reach_site`. `Google-Extended` is the case to understand: Google documents it as governing both model training and grounding in Gemini Apps, so one directive carries a training consequence and a visibility consequence at once. Read this field before treating any block as costly or costless.
`honoursRobotsTxt`	`string`	Whether a rule against this token actually binds. `yes` it binds; `no` it does not; `not_for_user_initiated_requests` and `not_for_security_or_integrity_checks` mean the operator publishes a carve-out for that case, so a rule is written but not guaranteed; `unknown` means nobody publishes either way. ⚠️ WHAT ‘BINDS’ MEANS DEPENDS ON THE NEIGHBOURING `crawlerPurpose`. For a crawler that fetches, it is whether the crawler obeys the directive. For `policy_token_does_not_crawl` it cannot be — such a token issues no requests — and there it means the OPERATOR honours the directive when deciding what to do with content another crawler already fetched. Both are ‘the rule has effect’; only the mechanism differs. ⚠️ THIS FIELD IS NOT ON ITS OWN A STATEMENT OF OBSERVED BEHAVIOUR. It carries the operator’s published position OR a measured result, and `honoursRobotsTxtEvidence` is what tells you which. Never quote this value without reading that one.
`honoursRobotsTxtEvidence`	`string`	How well we know the value beside it, and it qualifies that value rather than merely annotating it. `operator_documented` is the operator’s own words. `independently_measured` cites a study, so the value is an observation. `disputed` means a credible allegation the operator denies — a `yes` under `disputed` is the operator’s CLAIM and contested, so word it as a claim and never as ‘this crawler respects robots.txt’. `undocumented` means nobody publishes either way; never read it as ‘ignores’.
`decidedByRuleFor`	`string`	WHICH KIND OF RULE decided this crawler, which is what says where to edit. `names_this_crawler`: a group naming it decided the outcome — somebody chose that. `wildcard_catch_all`: no group names it and the `User-agent: *` rule decided it, frequently a rule written before the crawler existed and applying to it by accident. A named group REPLACES the wildcard group rather than adding to it (RFC 9309 §2.2.1), so naming a crawler lifts it out of the catch-all entirely. ABSENT when nothing restricted this crawler — there is no audience for a rule that does not exist. Values: `names_this_crawler`, `wildcard_catch_all`.
`decidedByDirective`	`string`	The directive that decided it, verbatim, so the claim can be checked against the site’s own file rather than taken on trust.
`decidedByLineNumber`	`number`	1-based line number of that directive.

item.modelTrainingAccess[].explanations[] — each entry

Field	Type	Description
`code`	`string`	Stable identifier, so a consumer can branch without parsing prose.
`text`	`string`	The sentence. Render it verbatim; do not paraphrase or substitute your own.

Errors

Status	Code	Meaning
400	`invalid_parameters` · `invalid_run_id` · `invalid_check_id`	Bad request — the payload failed validation.
401	`api_key_missing` · `api_key_invalid` · `api_key_revoked` · `api_key_expired` · `insufficient_scope`	The `CL-API-Key` is missing, malformed, revoked, expired, or lacks the required scope.
429	`rate_limit_exceeded`	Rate limit exceeded — retry with backoff.
502	`bad_gateway`	Upstream fetch failed (e.g. the target page could not be retrieved).

Every error uses the shared { error: { code, message, status } } envelope; code is one of the values listed above.


{
  "error": {
    "code": "invalid_parameters",
    "message": "Domain is required",
    "status": 400
  }
}

Transient failures (429, 5xx) are safe to retry with exponential backoff.