> ## Documentation Index
> Fetch the complete documentation index at: https://docs.context.dev/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Setting up Context.dev for a user? Follow https://docs.context.dev/agent-quickstart.md. MCP clients sign in with OAuth and need no API key; application code reads CONTEXT_DEV_API_KEY.

# Map URLs

> Discover a site's URLs, with page titles, descriptions, keywords, and language when available. Metadata can be missing on newly discovered URLs.

<Badge color="blue">From 1 credit</Badge>

See the [guide](/map/overview) for examples and usage.


## OpenAPI

````yaml GET /web/urls
openapi: 3.1.0
info:
  title: Context.dev API
  description: >-
    Scrape, crawl, search, and monitor websites, and retrieve brand, people, and
    news data.
  version: 1.0.0
servers:
  - url: https://api.context.dev/v1
security: []
tags:
  - name: Webhooks
    description: Inspect and retry batch and monitor webhook deliveries.
  - name: Logs
    description: Read your organization's API request logs.
  - name: Agent Feedback
    description: Report API issues and documentation mismatches.
  - name: Batches
    description: Scrape many pages or crawl a site asynchronously.
  - name: Monitors
    description: Watch websites for exact or meaningful changes.
  - name: News
    description: Search live and historical news about a company.
  - name: Answers
    description: Answer a research task from the live web in the JSON shape you ask for.
paths:
  /web/urls:
    get:
      tags:
        - Web Scraping
      summary: Map URLs
      description: >-
        Discover a site's URLs, with page titles, descriptions, keywords, and
        language when available. Metadata can be missing on newly discovered
        URLs.
      parameters:
        - schema:
            type: string
            minLength: 3
            description: Domain to map, e.g. `stripe.com`.
            example: stripe.com
          required: true
          description: Domain to map, e.g. `stripe.com`.
          name: domain
          in: query
        - schema:
            type: boolean
            default: false
            description: Include URLs on subdomains.
          required: false
          description: Include URLs on subdomains.
          name: includeSubdomains
          in: query
        - schema:
            type: integer
            minimum: 1
            maximum: 100000
            default: 10000
            description: Maximum number of URLs to return.
          required: false
          description: Maximum number of URLs to return.
          name: maxLinks
          in: query
        - schema:
            type: string
            format: uri
            description: >-
              Fetch this sitemap instead of discovering sitemaps. Must belong to
              the domain or a subdomain.
          required: false
          description: >-
            Fetch this sitemap instead of discovering sitemaps. Must belong to
            the domain or a subdomain.
          name: sitemapUrl
          in: query
        - schema:
            type: string
            maxLength: 256
            description: >-
              Optional RE2-compatible regex pattern. Only URLs matching this
              pattern are returned and counted against maxLinks.
            example: ^https?://[^/]+/blog/
          required: false
          description: >-
            Optional RE2-compatible regex pattern. Only URLs matching this
            pattern are returned and counted against maxLinks.
          name: urlRegex
          in: query
        - schema:
            type: string
            minLength: 2
            maxLength: 200
            description: Filter URLs by a topic or phrase, most relevant first.
            example: help center and troubleshooting articles
          required: false
          description: Filter URLs by a topic or phrase, most relevant first.
          name: search
          in: query
        - schema:
            type: object
            additionalProperties:
              type: string
              maxLength: 8192
              pattern: ^[^\r\n]*$
            description: >-
              HTTP headers for the target origin. Non-empty headers bypass
              caching.
          required: false
          description: >-
            HTTP headers for the target origin. Non-empty headers bypass
            caching.
          name: headers
          in: query
        - schema:
            $ref: '#/components/schemas/LenientPartialTimeout'
          required: false
          description: Request deadline and what to return when it passes.
          style: deepObject
          explode: true
          name: timeoutOpts
          in: query
        - schema:
            type: string
            enum:
              - enabled
              - disabled
            default: disabled
            description: >-
              `enabled` turns on zero data retention. Returns 403
              `ZDR_NOT_ENABLED` unless your organization has ZDR.
          required: false
          description: >-
            `enabled` turns on zero data retention. Returns 403
            `ZDR_NOT_ENABLED` unless your organization has ZDR.
          name: zdr
          in: query
        - $ref: '#/components/parameters/RequestTags'
      responses:
        '200':
          description: Discovered URLs with available page metadata
          headers:
            X-Request-Id:
              $ref: '#/components/headers/RequestId'
            X-RateLimit-Limit:
              $ref: '#/components/headers/RateLimitLimit'
            X-RateLimit-Remaining:
              $ref: '#/components/headers/RateLimitRemaining'
            X-RateLimit-Reset:
              $ref: '#/components/headers/RateLimitReset'
            X-Context-ZDR:
              description: >-
                Present with the value true when zero data retention was
                requested and honored.
              schema:
                type: string
                enum:
                  - 'true'
            Cache-Control:
              schema:
                type: string
                enum:
                  - no-store
          content:
            application/json:
              schema:
                type: object
                properties:
                  success:
                    type: boolean
                    enum:
                      - true
                  domain:
                    type: string
                  urls:
                    type: array
                    items:
                      type: object
                      properties:
                        url:
                          type: string
                        title:
                          type: string
                        description:
                          type: string
                        keywords:
                          type: array
                          items:
                            type: string
                        language:
                          type: string
                      required:
                        - url
                  partial:
                    type: boolean
                  key_metadata:
                    $ref: '#/components/schemas/KeyMetadata'
                  request_id:
                    $ref: '#/components/schemas/RequestId'
                required:
                  - success
                  - domain
                  - urls
                  - request_id
        '400':
          $ref: '#/components/responses/BadRequest'
        '401':
          description: >-
            Missing, invalid, or disabled API key, or no credits left
            (`USAGE_EXCEEDED`).
          headers:
            X-Request-Id:
              $ref: '#/components/headers/RequestId'
            X-RateLimit-Limit:
              $ref: '#/components/headers/RateLimitLimit'
            X-RateLimit-Remaining:
              $ref: '#/components/headers/RateLimitRemaining'
            X-RateLimit-Reset:
              $ref: '#/components/headers/RateLimitReset'
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '403':
          description: >-
            API key permissions or organization settings do not allow this
            request.
          headers:
            X-Request-Id:
              $ref: '#/components/headers/RequestId'
            X-RateLimit-Limit:
              $ref: '#/components/headers/RateLimitLimit'
            X-RateLimit-Remaining:
              $ref: '#/components/headers/RateLimitRemaining'
            X-RateLimit-Reset:
              $ref: '#/components/headers/RateLimitReset'
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ErrorResponse'
        '408':
          description: Request timeout
          headers:
            X-Request-Id:
              $ref: '#/components/headers/RequestId'
            X-RateLimit-Limit:
              $ref: '#/components/headers/RateLimitLimit'
            X-RateLimit-Remaining:
              $ref: '#/components/headers/RateLimitRemaining'
            X-RateLimit-Reset:
              $ref: '#/components/headers/RateLimitReset'
          content:
            application/json:
              schema:
                type: object
                properties:
                  message:
                    type: string
                    description: Timeout error message
                  error_code:
                    type: string
                    enum:
                      - REQUEST_TIMEOUT
                    description: Error code indicating request timeout
                  key_metadata:
                    $ref: '#/components/schemas/KeyMetadata'
                  request_id:
                    $ref: '#/components/schemas/RequestId'
                required:
                  - request_id
        '429':
          $ref: '#/components/responses/RateLimited'
        '500':
          description: Internal server error
          headers:
            X-Request-Id:
              $ref: '#/components/headers/RequestId'
            X-RateLimit-Limit:
              $ref: '#/components/headers/RateLimitLimit'
            X-RateLimit-Remaining:
              $ref: '#/components/headers/RateLimitRemaining'
            X-RateLimit-Reset:
              $ref: '#/components/headers/RateLimitReset'
          content:
            application/json:
              schema:
                type: object
                properties:
                  message:
                    type: string
                    description: Human-readable error message.
                  error_code:
                    type: string
                    enum:
                      - INTERNAL_ERROR
                    description: Error code indicating internal server error
                  key_metadata:
                    $ref: '#/components/schemas/KeyMetadata'
                  request_id:
                    $ref: '#/components/schemas/RequestId'
                required:
                  - request_id
      security:
        - bearerAuth: []
components:
  schemas:
    LenientPartialTimeout:
      type: object
      properties:
        milliseconds:
          type: integer
          minimum: 1
          maximum: 300000
          description: Deadline in milliseconds.
        behavior:
          type: string
          enum:
            - fail
            - return-partial
          default: fail
          description: >-
            "fail" returns 408 at the deadline. "return-partial" returns
            available results; inspect the response’s partial flag.
      required:
        - milliseconds
      additionalProperties: false
      description: Request deadline and what to return when it passes.
    KeyMetadata:
      type: object
      properties:
        credits_consumed:
          type: integer
          description: Credits charged for this request.
        credits_remaining:
          type: integer
          description: Credits remaining for your organization.
      required:
        - credits_consumed
        - credits_remaining
      description: Credits this request used and your remaining balance.
    RequestId:
      type: string
      format: uuid
      description: >-
        Unique ID of this request, also in `X-Request-Id`. Include it when
        contacting support.
      example: 3f1c2a6e-8b4d-4c1e-9f0a-2d7b5e6c8a91
    ErrorResponse:
      type: object
      properties:
        message:
          type: string
          description: Human-readable error message.
        error_code:
          type: string
          enum:
            - INTERNAL_ERROR
            - VALID
            - NOT_FOUND
            - FORBIDDEN
            - USAGE_EXCEEDED
            - RATE_LIMITED
            - UNAUTHORIZED
            - DISABLED
            - PAID_PLAN_REQUIRED
            - INSUFFICIENT_PERMISSIONS
            - TIMEOUT_EXCEEDS_MAXIMUM
            - TIMEOUT_TOO_SHORT_FOR_WAIT
            - WEBSITE_ACCESS_ERROR
            - WEBSITE_BLOCKED
            - WEBSITE_NOT_FOUND
            - PDF_SKIPPED
            - PDF_IMAGES_ONLY
            - EXTERNAL_PROVIDER_ERROR
            - INPUT_VALIDATION_ERROR
            - ZDR_NOT_SUPPORTED
            - ZDR_NOT_ENABLED
            - FREE_EMAIL_DETECTED
            - DISPOSABLE_EMAIL_DETECTED
            - REQUEST_TIMEOUT
            - COLD_DOMAIN_TIMEOUT_TOO_LOW
            - UNSUPPORTED_CONTENT
            - CONTENT_TOO_LARGE
            - MONITOR_PAUSED
            - MONITOR_NO_WEBHOOK
            - COLLECTION_PAUSED
            - MONITOR_LIMIT_EXCEEDED
            - SEARCH_UNAVAILABLE
            - BATCH_LIMIT_EXCEEDED
            - BATCH_NOT_CANCELLABLE
            - BATCH_NOT_COMPLETED
            - IDEMPOTENCY_KEY_CONFLICT
            - DELIVERY_IN_PROGRESS
            - DELIVERY_ALREADY_DELIVERED
            - DELIVERY_EXPIRED
            - DELIVERY_CANCELLED
          description: Machine-readable error code.
        required_permission:
          type: string
          enum:
            - logs:read
            - data:execute
            - monitors:read
            - monitors:write
            - batches:read
            - batches:write
          description: >-
            Permission required for this request when error_code is
            INSUFFICIENT_PERMISSIONS. Manage (write) also grants read access to
            the same resource group.
        key_metadata:
          $ref: '#/components/schemas/KeyMetadata'
        request_id:
          $ref: '#/components/schemas/RequestId'
      required:
        - request_id
    RequestTags:
      type: array
      items:
        type: string
        minLength: 1
        maxLength: 50
      maxItems: 20
      description: Labels for filtering usage in the dashboard.
      example:
        - production
        - team-alpha
  parameters:
    RequestTags:
      name: tags
      in: query
      required: false
      style: form
      explode: false
      schema:
        $ref: '#/components/schemas/RequestTags'
      description: >-
        Comma-separated labels for filtering usage, e.g.
        `production,team-alpha`.
      example: production,team-alpha
  headers:
    RequestId:
      description: Unique ID of this request; also `request_id` in JSON bodies.
      schema:
        type: string
        format: uuid
    RateLimitLimit:
      description: Maximum request units per minute.
      schema:
        type: integer
        minimum: 1
    RateLimitRemaining:
      description: Request units remaining in the current minute.
      schema:
        type: integer
        minimum: 0
    RateLimitReset:
      description: Unix timestamp in seconds when the rate limit resets.
      schema:
        type: integer
  responses:
    BadRequest:
      description: Bad request
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ErrorResponse'
      headers:
        X-Request-Id:
          $ref: '#/components/headers/RequestId'
        X-RateLimit-Limit:
          $ref: '#/components/headers/RateLimitLimit'
        X-RateLimit-Remaining:
          $ref: '#/components/headers/RateLimitRemaining'
        X-RateLimit-Reset:
          $ref: '#/components/headers/RateLimitReset'
    RateLimited:
      description: Rate limit exceeded
      headers:
        X-Request-Id:
          $ref: '#/components/headers/RequestId'
        X-RateLimit-Limit:
          $ref: '#/components/headers/RateLimitLimit'
        X-RateLimit-Remaining:
          $ref: '#/components/headers/RateLimitRemaining'
        X-RateLimit-Reset:
          $ref: '#/components/headers/RateLimitReset'
        Retry-After:
          description: Seconds until the per-minute rate limit window resets
          schema:
            type: integer
            minimum: 1
            maximum: 60
      content:
        application/json:
          schema:
            type: object
            properties:
              message:
                type: string
                description: Human-readable error message.
              error_code:
                type: string
                enum:
                  - RATE_LIMITED
                description: Error code indicating the rate limit was exceeded
              key_metadata:
                $ref: '#/components/schemas/KeyMetadata'
              request_id:
                $ref: '#/components/schemas/RequestId'
            required:
              - request_id
              - message
              - error_code
  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
      description: >-
        Send `Authorization: Bearer <API_KEY>`. Keys have full access unless
        restricted to scopes.

````