Sitemap Visualizer

REST endpoints for Sitemap Visualizer. Base URL https://api.competlab.com; every request needs a CL-API-Key header (see Authentication). Responses wrap in { item } or { items }; errors in { error }.

Analyze sitemap structure of a domain

POST/v1/tools/sitemap-visualizer

Discover and analyze the sitemap structure of any public domain. Reads the conventional /sitemap.xml AND every Sitemap: directive robots.txt declares — a site publishing several sitemaps returns all of them, deduplicated (or pass an explicit URL to read just that one). Categorizes URLs by content type, validates structure, and surfaces freshness and depth-distribution insights with sample URLs per category. Optionally include the full URL list (off by default to keep payload bounded). Where a declared sitemap could not be retrieved, truncated is true and unreadSitemaps names it, so a total is never reported as complete over a corpus only partly read. Stateless live check — no project required, no data is persisted. Suitable for content-strategy audits, SEO discovery, and content-volume benchmarking.

Returns · { item }

Request body — application/json

Field	Type	Description
`domain`	`string`	Target domain whose sitemap to analyze. Accepts a bare hostname or a full URL — normalized (lowercased, scheme and path stripped) before analysis.
`sitemapUrl`	`string`	Explicit sitemap URL to fetch directly, short-circuiting /sitemap.xml then robots.txt discovery. Must be a full http:// or https:// URL with a public hostname.
`includeUrls`	`boolean`	When true, include the full raw `urls[]` list in the response. Defaults to false, which returns a lean insights-only payload without the URL list.

Request


curl -X POST "https://api.competlab.com/v1/tools/sitemap-visualizer" \
  -H "CL-API-Key: YOUR_COMPETLAB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"stripe.com","sitemapUrl":"https://stripe.com/sitemap.xml","includeUrls":false}'

Response 200 OK

Example


{
  "item": {
    "domain": "competlab.com",
    "fetchedAt": "2026-05-16T14:30:00.000Z",
    "status": "ok",
    "source": "sitemap-xml",
    "sitemaps": [
      {
        "url": "https://competlab.com/sitemap.xml",
        "type": "urlset",
        "urlCount": 247,
        "actualUrlCount": 247
      }
    ],
    "truncated": false,
    "unreadSitemaps": [
      "https://example.com/sitemap-archive.xml"
    ],
    "unreadSitemapCount": 2,
    "accessDenied": false,
    "metrics": {
      "totalUrls": 247,
      "totalActualUrls": 247,
      "totalSitemaps": 3,
      "categoriesFound": 6,
      "maxDepth": 4,
      "avgDepth": 2.6
    },
    "urls": [
      {
        "url": "https://competlab.com/blog/competitive-intelligence-guide",
        "category": "blog",
        "depth": 2,
        "lastmod": "2026-04-12"
      }
    ],
    "categories": {
      "blog": {
        "count": 47,
        "percentage": 18.4
      },
      "docs": {
        "count": 47,
        "percentage": 18.4
      },
      "tools": {
        "count": 47,
        "percentage": 18.4
      },
      "landing": {
        "count": 47,
        "percentage": 18.4
      },
      "legal": {
        "count": 47,
        "percentage": 18.4
      },
      "case-studies": {
        "count": 47,
        "percentage": 18.4
      },
      "comparison": {
        "count": 47,
        "percentage": 18.4
      },
      "integrations": {
        "count": 47,
        "percentage": 18.4
      },
      "changelog": {
        "count": 47,
        "percentage": 18.4
      },
      "webinars": {
        "count": 47,
        "percentage": 18.4
      },
      "careers": {
        "count": 47,
        "percentage": 18.4
      },
      "programmatic": {
        "count": 47,
        "percentage": 18.4
      },
      "other": {
        "count": 47,
        "percentage": 18.4
      }
    },
    "validation": {
      "sitemapFound": {
        "passed": true,
        "message": "Found valid sitemap at /sitemap.xml"
      },
      "validXml": {
        "passed": true,
        "message": "Found valid sitemap at /sitemap.xml"
      },
      "nonEmpty": {
        "passed": true,
        "message": "Found valid sitemap at /sitemap.xml"
      },
      "validIndex": {
        "passed": true,
        "message": "Found valid sitemap at /sitemap.xml"
      }
    },
    "insights": {
      "freshness": {
        "fresh": 32,
        "recent": 18,
        "aging": 41,
        "stale": 122,
        "noDate": 34
      },
      "depthDistribution": {
        "0": 1,
        "1": 24,
        "2": 87,
        "3": 92,
        "4+": 43
      },
      "stalePageCount": 163,
      "sampleUrlsByCategory": {
        "blog": [
          "https://competlab.com/blog/competitive-intelligence-guide",
          "https://competlab.com/blog/ai-visibility-tracking"
        ],
        "docs": [
          "https://competlab.com/docs/getting-started"
        ],
        "case-studies": [
          "https://competlab.com/case-studies/acme"
        ]
      },
      "summary": "Content leans towards landing (48% of URLs). Active comparison-page SEO (74 pages). Strong social proof (113 case studies)."
    }
  }
}

Fields

item — object

Field	Type	Description
`domain`	`string`	Normalized domain, as actually analyzed (after normalization)
`fetchedAt`	`string`	ISO-8601 timestamp of scan completion
`status`	`string`	Overall outcome of the scan — ‘ok’/‘partial’ for success, ‘not-found’/‘access-denied’/‘invalid’ as first-class envelope states. ‘partial’ means the scan did not cover the whole corpus: either a limit stopped it early, or a sitemap the site declares could not be read — check `truncated` and `unreadSitemaps` for which. ‘invalid’ means a document was retrieved and could not be parsed as XML, never that nothing arrived. Values: `ok`, `partial`, `not-found`, `access-denied`, `invalid`.
`source`	`string`	Where discovery started, not the only place it looked. Every scan reads both the conventional /sitemap.xml and every Sitemap: directive robots.txt declares, so a site publishing several sitemaps returns all of them; `sitemaps` is the authoritative list of what was actually read. ‘sitemap-xml’ means the conventional path returned a sitemap; ‘robots-txt’ that it did not and robots.txt declared at least one instead — which may itself have failed to load, so check `sitemaps`; ‘provided’ that you supplied the URL; ‘none’ that neither source yielded anything. Values: `sitemap-xml`, `robots-txt`, `provided`, `none`.
`sitemaps`	`array<object>`	Every sitemap resource actually read, in the order the scan read them. When `urls` is returned it is the concatenation of each ‘urlset’ row’s URLs in this same order. A sitemap that could not be read is not here: if the site declared it in robots.txt it is named in `unreadSitemaps`, and if it was a child listed inside an index, `validation.validIndex` reports that the index names children the scan could not retrieve. See `item.sitemaps[]`.
`truncated`	`boolean`	True if the scan stopped short of the whole corpus: more than 10,000 URLs, its ceiling of 20 sitemaps under any one index or 20 declared in robots.txt, sitemap indexes nested deeper than it walks, or a declared sitemap it could not read. The counts below describe what was read, not the whole site.
`unreadSitemaps`	`array<string>`	Sitemaps declared in this site’s robots.txt that the scan could not read: the URL is not fetchable as written (relative rather than absolute, or a host we will not follow), a redirect led off the site’s own domain, the server refused it, or nothing came back. Each one is a promise robots.txt makes and does not keep — search engines and AI crawlers read the same directives, so it is worth fixing on your own site. Absent when every declared sitemap was read. Two cases are deliberately not listed: one that answered 404, which is a confirmed absence rather than an unread document; and one the scan skipped after using up its own budget, which `truncated` alone covers. Capped at 20 entries.
`unreadSitemapCount`	`number`	How many declared sitemaps went unread. Absent when none did. This is the number to report — it is not `unreadSitemaps.length`, which is capped at 20 and omits any value carrying a scheme the scan will not repeat back. A scan can report unread sitemaps with an empty list, and that still means the site’s robots.txt is at fault rather than the scan’s limits.
`accessDenied`	`boolean`	True if the conventional /sitemap.xml returned 401/403
`metrics`	`object`	Quantitative metrics — URL counts, sitemap count, category count, depth stats. See `item.metrics`.
`urls`	`array<object>`	Raw categorized URLs. Present only when request had includeUrls: true. Capped at 10,000; check `truncated` flag. See `item.urls[]`.
`categories`	`object`	Per-category breakdown. Every category key is populated (count may be 0). See `item.categories`.
`validation`	`object`	Quality checks for sitemap discovery and parsing. See `item.validation`.
`insights`	`object`	Interpreted findings — freshness, depth distribution, stale-page count, category samples, summary. See `item.insights`.

item.sitemaps[] — each entry

Field	Type	Description
`url`	`string`	Absolute URL of the fetched sitemap resource
`type`	`string`	Sitemap resource shape. Values: `index`, `urlset`.
`urlCount`	`number`	Number of URLs from this sitemap included in the analysis, which covers at most 10,000 URLs in total. A row of type ‘index’ carries the total for the sitemaps it lists, which are also listed individually — so summing every row double-counts. For the analysis-wide total use `metrics.totalUrls`.
`actualUrlCount`	`number`	Number of URLs this sitemap declares, before the 10,000-URL analysis limit. On a row of type ‘index’, the total its own children declare.

item.metrics — object

Field	Type	Description
`totalUrls`	`number`	Count of URLs analyzed (post 10k-cap)
`totalActualUrls`	`number`	True upstream URL count before the 10k cap
`totalSitemaps`	`number`	Number of sitemap resources actually read, counting both ‘index’ and ‘urlset’ rows. Never what the site declares — anything the scan could not read is in `unreadSitemaps` instead.
`categoriesFound`	`number`	Number of distinct content categories with at least one URL
`maxDepth`	`number`	Maximum path depth observed
`avgDepth`	`number \| null`	Average pathname depth across all URLs. Null when urls[] is empty. Nullable.

item.urls[] — each entry

Field	Type	Description
`url`	`string`	Absolute URL of the page
`category`	`string`	Content-category classification (kebab-case). One of 13 values (e.g. `blog`, `docs`, `tools`, …).
`depth`	`number`	Path depth — segment count of URL.pathname after filtering empty parts
`lastmod`	`string \| null`	Value of <lastmod> from the source XML if present, else null. Nullable.

item.categories — object

Field	Type	Description
`blog`	`object`	Blog posts and editorial content. See `item.categories.blog`.
`docs`	`object`	Product documentation. See `item.categories.docs`.
`tools`	`object`	Free tools and utilities. See `item.categories.tools`.
`landing`	`object`	Landing / marketing pages. See `item.categories.landing`.
`legal`	`object`	Legal pages — privacy, terms, DPA, etc. See `item.categories.legal`.
`case-studies`	`object`	Customer case studies. See `item.categories.case-studies`.
`comparison`	`object`	Comparison pages (X vs Y). See `item.categories.comparison`.
`integrations`	`object`	Integration pages. See `item.categories.integrations`.
`changelog`	`object`	Changelog / release notes. See `item.categories.changelog`.
`webinars`	`object`	Webinars and events. See `item.categories.webinars`.
`careers`	`object`	Careers / hiring pages. See `item.categories.careers`.
`programmatic`	`object`	Templated pages generated from a database or pattern — per-item catalog entries, reference tables. Assigned to a group of 25 or more sibling URLs under one parent path whose slugs are machine-generated, never to a single URL. Describes how the pages are generated, not their purpose or quality. Example URLs ship in `insights.sampleUrlsByCategory.programmatic`. See `item.categories.programmatic`.
`other`	`object`	Uncategorized pages. See `item.categories.other`.

item.validation — object

Field	Type	Description
`sitemapFound`	`object`	Whether a sitemap was discovered (via /sitemap.xml, robots.txt, or provided URL). See `item.validation.sitemapFound`.
`validXml`	`object`	Whether the discovered XML parsed successfully. See `item.validation.validXml`.
`nonEmpty`	`object`	Whether the sitemap contained at least one URL. See `item.validation.nonEmpty`.
`validIndex`	`object`	Whether sitemap index referenced valid child sitemaps (if applicable). See `item.validation.validIndex`.

item.insights — object

Field	Type	Description
`freshness`	`object`	URL counts grouped by how recently each page was last modified. See `item.insights.freshness`.
`depthDistribution`	`object`	URL count grouped by path depth (0 = home, ‘4+’ = 4+ segments). See `item.insights.depthDistribution`.
`stalePageCount`	`number \| null`	Count of URLs older than ~6 months (180 days). Null when no <lastmod> data is present at all. Nullable.
`sampleUrlsByCategory`	`object`	Up to N sample URLs per populated category, largest categories first. Keys are content categories; absent keys mean the category had zero URLs.
`summary`	`string`	Plain-language read of what this sitemap shows, in 1-3 sentences. The same sitemap always produces the same text.

item.categories.blog — object

Field	Type	Description
`count`	`number`	Number of URLs in this category
`percentage`	`number`	Percentage of total URLs (0-100, one decimal place)

item.categories.docs — object

Field	Type	Description
`count`	`number`	Number of URLs in this category
`percentage`	`number`	Percentage of total URLs (0-100, one decimal place)

item.categories.tools — object

Field	Type	Description
`count`	`number`	Number of URLs in this category
`percentage`	`number`	Percentage of total URLs (0-100, one decimal place)

item.categories.landing — object

Field	Type	Description
`count`	`number`	Number of URLs in this category
`percentage`	`number`	Percentage of total URLs (0-100, one decimal place)

item.categories.legal — object

Field	Type	Description
`count`	`number`	Number of URLs in this category
`percentage`	`number`	Percentage of total URLs (0-100, one decimal place)

item.categories.case-studies — object

Field	Type	Description
`count`	`number`	Number of URLs in this category
`percentage`	`number`	Percentage of total URLs (0-100, one decimal place)

item.categories.comparison — object

Field	Type	Description
`count`	`number`	Number of URLs in this category
`percentage`	`number`	Percentage of total URLs (0-100, one decimal place)

item.categories.integrations — object

Field	Type	Description
`count`	`number`	Number of URLs in this category
`percentage`	`number`	Percentage of total URLs (0-100, one decimal place)

item.categories.changelog — object

Field	Type	Description
`count`	`number`	Number of URLs in this category
`percentage`	`number`	Percentage of total URLs (0-100, one decimal place)

item.categories.webinars — object

Field	Type	Description
`count`	`number`	Number of URLs in this category
`percentage`	`number`	Percentage of total URLs (0-100, one decimal place)

item.categories.careers — object

Field	Type	Description
`count`	`number`	Number of URLs in this category
`percentage`	`number`	Percentage of total URLs (0-100, one decimal place)

item.categories.programmatic — object

Field	Type	Description
`count`	`number`	Number of URLs in this category
`percentage`	`number`	Percentage of total URLs (0-100, one decimal place)

item.categories.other — object

Field	Type	Description
`count`	`number`	Number of URLs in this category
`percentage`	`number`	Percentage of total URLs (0-100, one decimal place)

item.validation.sitemapFound — object

Field	Type	Description
`passed`	`boolean`	Whether the validation check passed
`message`	`string`	Human-readable reason

item.validation.validXml — object

Field	Type	Description
`passed`	`boolean`	Whether the validation check passed
`message`	`string`	Human-readable reason

item.validation.nonEmpty — object

Field	Type	Description
`passed`	`boolean`	Whether the validation check passed
`message`	`string`	Human-readable reason

item.validation.validIndex — object

Field	Type	Description
`passed`	`boolean`	Whether the validation check passed
`message`	`string`	Human-readable reason

item.insights.freshness — object

Field	Type	Description
`fresh`	`number`	URLs last modified within the last 3 months (0-90 days)
`recent`	`number`	URLs last modified 3-6 months ago (90-180 days)
`aging`	`number`	URLs last modified 6-12 months ago (180-360 days)
`stale`	`number`	URLs last modified more than a year ago (>360 days)
`noDate`	`number`	URLs missing the <lastmod> tag

item.insights.depthDistribution — object

Field	Type	Description
`0`	`number`	URLs at path depth 0 (home pages)
`1`	`number`	URLs at path depth 1
`2`	`number`	URLs at path depth 2
`3`	`number`	URLs at path depth 3
`4+`	`number`	URLs at path depth 4 or deeper

Errors

Status	Code	Meaning
400	`invalid_parameters` · `invalid_run_id` · `invalid_check_id`	Bad request — the payload failed validation.
401	`api_key_missing` · `api_key_invalid` · `api_key_revoked` · `api_key_expired` · `insufficient_scope`	The `CL-API-Key` is missing, malformed, revoked, expired, or lacks the required scope.
429	`rate_limit_exceeded`	Rate limit exceeded — retry with backoff.
502	`bad_gateway`	Upstream fetch failed (e.g. the target page could not be retrieved).

Every error uses the shared { error: { code, message, status } } envelope; code is one of the values listed above.


{
  "error": {
    "code": "invalid_parameters",
    "message": "Domain is required",
    "status": 400
  }
}

Transient failures (429, 5xx) are safe to retry with exponential backoff.