Fetch URL

REST endpoints for Fetch URL. Base URL https://api.competlab.com; every request needs a CL-API-Key header (see Authentication). Responses wrap in { item } or { items }; errors in { error }.

Fetch URL body and headers with JS-rendering + bot-protection handling

POST/v1/tools/fetch-url

Fetches the body and headers of any URL with automatic JavaScript rendering and common bot-protection handling. Tune via optional parameters (bodyNeeded, headersNeeded, cleanHtml, maxTimeoutMs, bodyMaxBytes). Response includes body, headers, cleanStats. The cleanHtml flag strips HTML noise (scripts, styles, comments) while preserving text content — significant token-cost reduction for LLM consumption; per-request reduction reported in cleanStats. Returns headersAvailable: false when advanced behavioral fingerprinting prevents header access. Intended for AI agents and customer integrations that need direct URL content access.

Returns · { item }

Request body — application/json

Field	Type	Description
`url`	`string`	Target URL to fetch. Must use http:// or https:// and resolve to a public host.
`bodyNeeded`	`boolean`	Include `body` and `contentType` in the response. Defaults to service-controlled value when omitted.
`headersNeeded`	`boolean`	Include `headers` and `headersAvailable` in the response. Defaults to service-controlled value when omitted. When the target site uses advanced behavioral fingerprinting, `headersAvailable` will be `false` and `headers` will be absent.
`cleanHtml`	`boolean`	When `true` and the response content-type is `text/html`, strip HTML noise (scripts, styles, comments) while preserving text content. Significant token-cost reduction for LLM consumption — per-request reduction reported in `cleanStats`. Requires `bodyNeeded`.
`maxTimeoutMs`	`number`	Caller-side timeout budget in milliseconds. Accepted range 1000–120000. Defaults to service-controlled value when omitted. Range `1000`–`120000`.
`bodyMaxBytes`	`number`	Per-request response body cap in bytes. Accepted range 1024–104857600 (1 KiB – 100 MiB). Oversize responses are rejected pre-buffer. Defaults to service-controlled value when omitted. Range `1024`–`104857600`.

Request


curl -X POST "https://api.competlab.com/v1/tools/fetch-url" \
  -H "CL-API-Key: YOUR_COMPETLAB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://stripe.com","bodyNeeded":true,"headersNeeded":true,"cleanHtml":false,"maxTimeoutMs":60000,"bodyMaxBytes":5242880}'

Response 200 OK

Example


{
  "item": {
    "statusCode": 200,
    "body": "<!DOCTYPE html><html>…</html>",
    "headers": {
      "content-type": "text/html; charset=utf-8",
      "strict-transport-security": "max-age=63072000"
    },
    "headersAvailable": true,
    "contentType": "text/html",
    "cleanStats": {
      "originalSize": 248532,
      "cleanedSize": 41218,
      "reductionPercent": 83.4,
      "originalSizeKB": 242.7,
      "cleanedSizeKB": 40.3,
      "savedKB": 202.4
    },
    "finalUrl": "https://stripe.com/",
    "durationMs": 1842
  }
}

Field	Type	Description
`statusCode`	`number`	HTTP status code returned by the target after the cascade resolved
`body`	`string`	Response body. Present when the request had `bodyNeeded: true`.
`headers`	`object`	Response headers, lower-cased keys. Present when the request had `headersNeeded: true` AND `headersAvailable` is `true`.
`headersAvailable`	`boolean`	Honest-degradation discriminator. `true` when headers were observed end-to-end; `false` when the target site uses advanced behavioral fingerprinting that prevents header access — `headers` will be absent in that case.
`contentType`	`string`	Content-type of the resolved response. Present when the request had `bodyNeeded: true`. Values: `text/html`, `text/plain`, `application/xml`, `application/json`.
`cleanStats`	`object`	HTML cleanup reduction stats. Present when the request had `cleanHtml: true` AND the resolved content-type is `text/html`. See `item.cleanStats`.
`finalUrl`	`string`	Resolved URL after any redirects
`durationMs`	`number`	End-to-end fetch duration in milliseconds

Field	Type	Description
`originalSize`	`number`	Original HTML body size in bytes (pre-cleanup)
`cleanedSize`	`number`	Cleaned HTML body size in bytes (post-cleanup)
`reductionPercent`	`number`	Reduction percentage (0–100, one decimal place)
`originalSizeKB`	`number`	Original size in kilobytes (convenience)
`cleanedSizeKB`	`number`	Cleaned size in kilobytes (convenience)
`savedKB`	`number`	Bytes saved expressed as kilobytes (convenience)

Errors

Status	Code	Meaning
400	`invalid_parameters`	Bad request — the payload failed validation.
401	`api_key_missing` · `api_key_invalid` · `api_key_revoked` · `api_key_expired` · `insufficient_scope`	The `CL-API-Key` is missing, malformed, revoked, expired, or lacks the required scope.
429	`rate_limit_exceeded`	Rate limit exceeded — retry with backoff.
502	`bad_gateway` · `bot_protection_blocked` · `fetch_failed`	Upstream fetch failed (e.g. the target page could not be retrieved).
503	`target_unreachable`	Service temporarily unavailable — retry shortly.

Every error uses the shared { error: { code, message, status } } envelope; code is one of the values listed above.


{
  "error": {
    "code": "invalid_parameters",
    "message": "Domain is required",
    "status": 400
  }
}

Transient failures (429, 5xx) are safe to retry with exponential backoff.

Fetch URL

Fetch URL body and headers with JS-rendering + bot-protection handling

Example

Fields