Crawl website sitemap

JavaScript

import ContextDev from 'context.dev';

const client = new ContextDev({
  apiKey: process.env['CONTEXT_DEV_API_KEY'], // This is the default and can be omitted
});

const response = await client.web.webScrapeSitemap({ domain: 'domain' });

console.log(response.domain);

{
  "success": true,
  "domain": "<string>",
  "urls": [
    "<string>"
  ],
  "meta": {
    "sitemapsDiscovered": 123,
    "sitemapsFetched": 123,
    "sitemapsSkipped": 123,
    "errors": 123
  }
}

GET

web

scrape

sitemap

JavaScript

import ContextDev from 'context.dev';

const client = new ContextDev({
  apiKey: process.env['CONTEXT_DEV_API_KEY'], // This is the default and can be omitted
});

const response = await client.web.webScrapeSitemap({ domain: 'domain' });

console.log(response.domain);

{
  "success": true,
  "domain": "<string>",
  "urls": [
    "<string>"
  ],
  "meta": {
    "sitemapsDiscovered": 123,
    "sitemapsFetched": 123,
    "sitemapsSkipped": 123,
    "errors": 123
  }
}

Authorizations

Authorization

string

header

required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Query Parameters

domain

string

required

Domain name to crawl sitemaps for (e.g., 'example.com'). The domain will be automatically normalized and validated.

maxLinks

integer

default:10000

Maximum number of links to return from the sitemap crawl. Defaults to 10,000. Minimum is 1, maximum is 100,000.

Required range: 1 <= x <= 100000

Response

Successful response

success

enum<boolean>

required

Indicates success

Available options:

true

domain

string

required

The normalized domain that was crawled

urls

string[]

required

Array of discovered page URLs from the sitemap (max 500)

Endpoints

Authorizations

Query Parameters

Response