AI Crawler Checker

REST endpoints for AI Crawler Checker. Base URL https://api.competlab.com; every request needs a CL-API-Key header (see Authentication). Responses wrap in { item } or { items }; errors in { error }.

Check AI crawler accessibility for a domain

POST/v1/tools/ai-crawler-checker

Check which AI crawlers are allowed, disallowed, or unconfigured on any public domain. Inspects robots.txt for major AI bots (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and more), checks meta tags and the presence of /llms.txt, and surfaces an accessibility verdict, opt-in/opt-out strategy detection, and how the domain compares to industry peers. Optionally tailor the analysis to a specific industry. Stateless live check — no project required, no data is persisted. Suitable for AI readiness audits, content-strategy reviews, and pre-launch compliance checks.

Returns · { item }

Request body — application/json

Field	Type	Description
`domain`	`string`	Target domain to check for AI-crawler accessibility. Accepts a bare hostname or a full URL — normalized (lowercased, scheme and path stripped) before checking.
`industry`	`string`	Industry context used to tailor crawler recommendations and benchmarks. Defaults to “other” when omitted. Values: `news-media`, `arts-entertainment`, `law-government`, `finance-healthcare`, `saas-tech`, `ecommerce`, `other`.

Request


curl -X POST "https://api.competlab.com/v1/tools/ai-crawler-checker" \
  -H "CL-API-Key: YOUR_COMPETLAB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"stripe.com","industry":"saas-tech"}'

Response 200 OK

Example


{
  "item": {
    "domain": "competlab.com",
    "fetchedAt": "2026-05-16T14:30:00.000Z",
    "industry": "saas-tech",
    "crawlers": [
      {
        "userAgent": "GPTBot",
        "displayName": "GPTBot",
        "operator": "OpenAI",
        "category": "training",
        "status": "blocked",
        "matchingDirective": "Disallow: /"
      }
    ],
    "robotsTxt": {
      "found": true,
      "content": "User-agent: GPTBot\nDisallow: /\n\nUser-agent: *\nAllow: /\n",
      "statusCode": 200
    },
    "metaTags": [
      {
        "name": "robots",
        "content": "noai, noimageai"
      }
    ],
    "headers": {
      "xRobotsTag": "noai, noimageai"
    },
    "llmsTxt": {
      "found": true,
      "sectionCount": 12,
      "linkCount": 47,
      "approxTokens": 1850
    },
    "llmsFullTxt": {
      "found": true,
      "sectionCount": 12,
      "linkCount": 47,
      "approxTokens": 1850
    },
    "accessibility": {
      "score": 65,
      "label": "Mixed Access",
      "interpretation": "Selective access — training bots blocked, retrieval bots allowed. Common pattern for content-led brands protecting IP while staying answer-engine visible.",
      "color": "orange"
    },
    "strategy": {
      "type": "block-training-allow-retrieval",
      "label": "Block training, allow retrieval",
      "description": "You're blocking AI model training crawlers (GPTBot, ClaudeBot) while allowing retrieval bots (ChatGPT-User, PerplexityBot). Protects IP, maintains answer-engine visibility."
    },
    "industryPosition": {
      "industryLabel": "SaaS / Tech",
      "industryAvgBlockRate": 42,
      "typicalScoreRange": [
        55,
        80
      ],
      "position": "average",
      "insight": "Your access score of 65 sits in the typical SaaS / Tech range (55–80). 42% of peers block at least one AI bot."
    },
    "recommendations": [
      {
        "id": "ip-protection",
        "title": "Add llms.txt for retrieval-bot context",
        "description": "ChatGPT-User and PerplexityBot can fetch your homepage but lack a curated context file. Add /llms.txt with key pages and definitions to improve citation quality.",
        "priority": "medium"
      }
    ],
    "robotsSnippet": "User-agent: GPTBot\nDisallow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n"
  }
}

Fields

item — object

Field	Type	Description
`domain`	`string`	Normalized domain, as actually analyzed (after normalization)
`fetchedAt`	`string`	ISO-8601 timestamp of when the scan completed
`industry`	`string`	Industry echoed back — defaults to ‘other’ if caller didn’t supply one. Values: `news-media`, `arts-entertainment`, `law-government`, `finance-healthcare`, `saas-tech`, `ecommerce`, `other`.
`crawlers`	`array<object>`	All 21 catalog entries with derived per-crawler status. See `item.crawlers[]`.
`robotsTxt`	`object`	robots.txt fetch result. See `item.robotsTxt`.
`metaTags`	`array<object>`	AI-related meta robots directives captured from the homepage. See `item.metaTags[]`.
`headers`	`object`	Values captured from homepage response headers. See `item.headers`.
`llmsTxt`	`object`	/llms.txt stats (AI search file spec). See `item.llmsTxt`.
`llmsFullTxt`	`object`	/llms-full.txt stats (AI search file spec). See `item.llmsFullTxt`.
`accessibility`	`object`	Accessibility verdict — score + label + interpretation. See `item.accessibility`.
`strategy`	`object`	Detected strategy — opt-in/opt-out pattern across crawler categories. See `item.strategy`.
`industryPosition`	`object`	Industry-context positioning — benchmark percentile and insight. See `item.industryPosition`.
`recommendations`	`array<object>`	Prioritized recommendations for closing gaps. See `item.recommendations[]`.
`robotsSnippet`	`string`	Ready-to-paste robots.txt patch for the detected blocked crawlers

item.crawlers[] — each entry

Field	Type	Description
`userAgent`	`string`	Robots.txt user-agent token for this crawler
`displayName`	`string`	Human-readable display name
`operator`	`string`	Operating organization
`category`	`string`	Crawler purpose category. Values: `training`, `retrieval`, `hybrid`.
`status`	`string`	Per-crawler access status derived from robots.txt rules. Values: `allowed`, `blocked`, `conditional`.
`matchingDirective`	`string \| null`	Raw robots.txt line that triggered the status, or null if no directive matched. Nullable.

item.robotsTxt — object

Field	Type	Description
`found`	`boolean`	Whether the robots.txt file was successfully fetched
`content`	`string`	Raw text content of the robots.txt file when fetched successfully
`statusCode`	`number`	HTTP status code of the robots.txt fetch

item.metaTags[] — each entry

Field	Type	Description
`name`	`string`	Meta tag name (currently always ‘robots’)
`content`	`string`	Directive value, e.g. ‘noai’, ‘noimageai’, ‘noml’, ‘noimageml’

item.headers — object

Field	Type	Description
`xRobotsTag`	`string`	Value of the X-Robots-Tag header from the homepage response

item.llmsTxt — object

Field	Type	Description
`found`	`boolean`	Whether the file was found at /llms.txt or /llms-full.txt
`sectionCount`	`number`	Count of markdown section headers (`# foo` lines)
`linkCount`	`number`	Count of markdown inline links `[text](url)`
`approxTokens`	`number`	Rough token estimate (chars / 4, ceil)

item.llmsFullTxt — object

Field	Type	Description
`found`	`boolean`	Whether the file was found at /llms.txt or /llms-full.txt
`sectionCount`	`number`	Count of markdown section headers (`# foo` lines)
`linkCount`	`number`	Count of markdown inline links `[text](url)`
`approxTokens`	`number`	Rough token estimate (chars / 4, ceil)

item.accessibility — object

Field	Type	Description
`score`	`number`	0-100 weighted composite accessibility score
`label`	`string`	Coarse-grained accessibility label. Values: `Fully Open`, `Mostly Open`, `Mixed Access`, `Mostly Blocked`.
`interpretation`	`string`	Human-readable interpretation of the score
`color`	`string`	UI hint color for the score. Values: `teal`, `blue`, `orange`, `red`.

item.strategy — object

Field	Type	Description
`type`	`string`	Stable machine-readable strategy type. Values: `block-nothing`, `block-training-allow-retrieval`, `block-everything`, `block-retrieval-allow-training`, `mixed`.
`label`	`string`	Human-readable strategy label
`description`	`string`	Description of the detected strategy

item.industryPosition — object

Field	Type	Description
`industryLabel`	`string`	Human-readable industry label
`industryAvgBlockRate`	`number`	Average percentage of sites in this industry blocking at least one AI bot
`typicalScoreRange`	`array<number>`	Typical accessibility score range for this industry [min, max]
`position`	`string`	Where this domain sits relative to the typical score range. Values: `below-average`, `average`, `above-average`.
`insight`	`string`	Human-readable sentence explaining the position in industry context

item.recommendations[] — each entry

Field	Type	Description
`id`	`string`	Stable machine-readable id (e.g. ‘ip-protection’, ‘visibility-risk’)
`title`	`string`	Short recommendation title
`description`	`string`	Recommendation detail
`priority`	`string`	Priority for the recommendation. Values: `high`, `medium`, `low`.

Errors

Status	Code	Meaning
400	`invalid_parameters`	Bad request — the payload failed validation.
401	`api_key_missing` · `api_key_invalid` · `api_key_revoked` · `api_key_expired` · `insufficient_scope`	The `CL-API-Key` is missing, malformed, revoked, expired, or lacks the required scope.
429	`rate_limit_exceeded`	Rate limit exceeded — retry with backoff.
502	`bad_gateway`	Upstream fetch failed (e.g. the target page could not be retrieved).

Every error uses the shared { error: { code, message, status } } envelope; code is one of the values listed above.


{
  "error": {
    "code": "invalid_parameters",
    "message": "Domain is required",
    "status": 400
  }
}

Transient failures (429, 5xx) are safe to retry with exponential backoff.