> ## Documentation Index
> Fetch the complete documentation index at: https://docs.context.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl Async

> Crawl up to 25,000 pages in a background batch, track progress, and retrieve Markdown or HTML when the job finishes.

<Info>
  Use async crawling for large sites or jobs that should run in the background, up to 25,000 pages. For fewer than 500 pages when you need results in one response, use [Crawl Sync](/guides/crawl-website).
</Info>

Submit a starting URL to `POST /batch/submit`. The API discovers linked pages and processes them as a background job. Save the batch ID, poll for completion, then read or download the results.

## Submit a crawl

Choose cURL or [install an SDK](/sdks) for your language. Each example creates its own client.

Export an API key from the [dashboard](https://context.dev/dashboard):

```bash theme={null}
export CONTEXT_DEV_API_KEY="ctxt_secret_..."
```

This request follows links within the documentation site, up to three hops from the starting page. It caps the job at 100 pages so you can check coverage before increasing the limit.

<CodeGroup>
  ```bash cURL theme={null}
  curl -X POST https://api.context.dev/v1/batch/submit \
    -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
    -H "Content-Type: application/json" \
    -H "Idempotency-Key: docs-crawl-v1" \
    -d '{
      "input": {
        "mode": "crawl",
        "data": {
          "format": "markdown",
          "source": {
            "type": "start_url",
            "url": "https://docs.context.dev/introduction",
            "controls": {
              "maxUrls": 100,
              "maxDepth": 3,
              "followSubdomains": false,
              "regex": "^https://docs\\.context\\.dev/"
            }
          },
          "options": {
            "useMainContentOnly": true
          }
        }
      },
      "tags": ["docs-crawl"]
    }'
  ```

  ```typescript TypeScript theme={null}
  import ContextDev from "context.dev";

  const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

  const response = await client.batch.submit({
    input: {
      mode: "crawl",
      data: {
        format: "markdown",
        source: {
          type: "start_url",
          url: "https://docs.context.dev/introduction",
          controls: {
            maxUrls: 100,
            maxDepth: 3,
            followSubdomains: false,
            regex: "^https://docs\\.context\\.dev/",
          },
        },
        options: {
          useMainContentOnly: true,
        },
      },
    },
    tags: ["docs-crawl"],
    "Idempotency-Key": "docs-crawl-v1",
  });
  console.log(response);
  ```

  ```python Python theme={null}
  import os
  from context.dev import ContextDev

  client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

  response = client.batch.submit(
      input={
          "mode": "crawl",
          "data": {
              "format": "markdown",
              "source": {
                  "type": "start_url",
                  "url": "https://docs.context.dev/introduction",
                  "controls": {
                      "max_urls": 100,
                      "max_depth": 3,
                      "follow_subdomains": False,
                      "regex": "^https://docs\\.context\\.dev/",
                  },
              },
              "options": {
                  "use_main_content_only": True,
              },
          },
      },
      tags=["docs-crawl"],
      idempotency_key="docs-crawl-v1",
  )
  print(response)
  ```

  ```ruby Ruby theme={null}
  require "cgi/core"
  require "context_dev"

  client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

  response = client.batch.submit(
    input: {
      mode: "crawl",
      data: {
        format: "markdown",
        source: {
          type: "start_url",
          url: "https://docs.context.dev/introduction",
          controls: {
            max_urls: 100,
            max_depth: 3,
            follow_subdomains: false,
            regex: "^https://docs\\.context\\.dev/",
          },
        },
        options: {
          use_main_content_only: true,
        },
      },
    },
    tags: ["docs-crawl"],
    idempotency_key: "docs-crawl-v1",
  )
  puts response.inspect
  ```

  ```go Go theme={null}
  package main

  import (
  	"context"
  	"fmt"
  	"os"

  	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
  	"github.com/context-dot-dev/context-go-sdk/v2/option"
  )

  func main() {
  	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))

  	response, err := client.Batch.Submit(context.Background(), contextdev.BatchSubmitParams{
  		Input: contextdev.BatchSubmitParamsInputUnion{
  			OfCrawl: &contextdev.BatchSubmitParamsInputCrawl{
  				Mode: "crawl",
  				Data: contextdev.BatchSubmitParamsInputCrawlDataUnion{
  					OfMarkdown: &contextdev.BatchSubmitParamsInputCrawlDataMarkdown{
  						Format: "markdown",
  						Source: contextdev.BatchSubmitParamsInputCrawlDataMarkdownSourceUnion{
  							OfStartURL: &contextdev.BatchSubmitParamsInputCrawlDataMarkdownSourceStartURL{
  								Type: "start_url",
  								URL:  "https://docs.context.dev/introduction",
  								Controls: contextdev.BatchSubmitParamsInputCrawlDataMarkdownSourceStartURLControls{
  									MaxURLs:          contextdev.Int(100),
  									MaxDepth:         contextdev.Int(3),
  									FollowSubdomains: contextdev.Bool(false),
  									Regex:            contextdev.String("^https://docs\\.context\\.dev/"),
  								},
  							},
  						},
  						Options: contextdev.BatchSubmitParamsInputCrawlDataMarkdownOptions{
  							UseMainContentOnly: contextdev.Bool(true),
  						},
  					},
  				},
  			},
  		},
  		Tags:           []string{"docs-crawl"},
  		IdempotencyKey: contextdev.String("docs-crawl-v1"),
  	})
  	if err != nil {
  		panic(err)
  	}
  	fmt.Println(response)
  }
  ```

  ```php PHP theme={null}
  <?php
  require __DIR__.'/vendor/autoload.php';

  use ContextDev\Client;

  $client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

  $response = $client->batch->submit(
      input: [
          "mode" => "crawl",
          "data" => [
              "format" => "markdown",
              "source" => [
                  "type" => "start_url",
                  "url" => "https://docs.context.dev/introduction",
                  "controls" => [
                      "maxURLs" => 100,
                      "maxDepth" => 3,
                      "followSubdomains" => false,
                      "regex" => "^https://docs\\.context\\.dev/",
                  ],
              ],
              "options" => [
                  "useMainContentOnly" => true,
              ],
          ],
      ],
      tags: ["docs-crawl"],
      idempotencyKey: "docs-crawl-v1",
  );
  var_dump($response);
  ```
</CodeGroup>

An accepted request returns `202` with `status: "queued"` and an `id`. Save that ID before starting another job. Submission reserves credits immediately; see [costs and limits](#costs-and-limits) before raising `maxUrls`.

Choose a new `Idempotency-Key` for each new crawl. Reuse the same key and body when retrying a submission to recover the original batch instead of creating a duplicate.

## Control page discovery

Set these fields under `input.data.source.controls`:

| Setting            | Behavior                                                                                                             |
| ------------------ | -------------------------------------------------------------------------------------------------------------------- |
| `maxUrls`          | Maximum pages to fetch, from 1 to 25,000. Defaults to 100; the crawl can finish below this cap.                      |
| `maxDepth`         | Maximum link depth, from 0 to 50. The starting page is depth 0. Omit it for no depth limit.                          |
| `regex`            | RE2 pattern for eligible URLs, up to 256 characters. The starting URL is always included, even if it does not match. |
| `followSubdomains` | Follow links to subdomains. Defaults to `false`.                                                                     |

Use `input.data.format: "html"` for HTML instead of Markdown. Per-page controls belong in `input.data.options`, including content selectors, rendering waits, and PDF parsing. `maxAgeMs` defaults to one day; set it to `0` to fetch pages fresh.

<Accordion title="Start from a sitemap">
  To scrape the URLs listed in a sitemap, submit a `sitemap` source:

  <CodeGroup>
    ```bash cURL theme={null}
    curl https://api.context.dev/v1/batch/submit \
      --request POST \
      --header "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
      --header "Content-Type: application/json" \
      --data '{
      "input": {
        "mode": "crawl",
        "data": {
          "format": "markdown",
          "source": {
            "type": "sitemap",
            "domain": "example.com",
            "controls": {
              "maxUrls": 100,
              "regex": "^https://example\\.com/docs/"
            }
          }
        }
      }
    }'
    ```

    ```typescript TypeScript theme={null}
    import ContextDev from "context.dev";

    const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

    const response = await client.batch.submit({
      input: {
        mode: "crawl",
        data: {
          format: "markdown",
          source: {
            type: "sitemap",
            domain: "example.com",
            controls: {
              maxUrls: 100,
              regex: "^https://example\\.com/docs/",
            },
          },
        },
      },
    });
    console.log(response);
    ```

    ```python Python theme={null}
    import os
    from context.dev import ContextDev

    client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

    response = client.batch.submit(
        input={
            "mode": "crawl",
            "data": {
                "format": "markdown",
                "source": {
                    "type": "sitemap",
                    "domain": "example.com",
                    "controls": {
                        "max_urls": 100,
                        "regex": "^https://example\\.com/docs/",
                    },
                },
            },
        },
    )
    print(response)
    ```

    ```ruby Ruby theme={null}
    require "cgi/core"
    require "context_dev"

    client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

    response = client.batch.submit(
      input: {
        mode: "crawl",
        data: {
          format: "markdown",
          source: {
            type: "sitemap",
            domain: "example.com",
            controls: {
              max_urls: 100,
              regex: "^https://example\\.com/docs/",
            },
          },
        },
      },
    )
    puts response.inspect
    ```

    ```go Go theme={null}
    package main

    import (
    	"context"
    	"fmt"
    	"os"

    	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
    	"github.com/context-dot-dev/context-go-sdk/v2/option"
    )

    func main() {
    	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))

    	response, err := client.Batch.Submit(context.Background(), contextdev.BatchSubmitParams{
    		Input: contextdev.BatchSubmitParamsInputUnion{
    			OfCrawl: &contextdev.BatchSubmitParamsInputCrawl{
    				Mode: "crawl",
    				Data: contextdev.BatchSubmitParamsInputCrawlDataUnion{
    					OfMarkdown: &contextdev.BatchSubmitParamsInputCrawlDataMarkdown{
    						Format: "markdown",
    						Source: contextdev.BatchSubmitParamsInputCrawlDataMarkdownSourceUnion{
    							OfSitemap: &contextdev.BatchSubmitParamsInputCrawlDataMarkdownSourceSitemap{
    								Type:   "sitemap",
    								Domain: "example.com",
    								Controls: contextdev.BatchSubmitParamsInputCrawlDataMarkdownSourceSitemapControls{
    									MaxURLs: contextdev.Int(100),
    									Regex:   contextdev.String("^https://example\\.com/docs/"),
    								},
    							},
    						},
    					},
    				},
    			},
    		},
    	})
    	if err != nil {
    		panic(err)
    	}
    	fmt.Println(response)
    }
    ```

    ```php PHP theme={null}
    <?php
    require __DIR__.'/vendor/autoload.php';

    use ContextDev\Client;

    $client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

    $response = $client->batch->submit(
        input: [
            "mode" => "crawl",
            "data" => [
                "format" => "markdown",
                "source" => [
                    "type" => "sitemap",
                    "domain" => "example.com",
                    "controls" => [
                        "maxURLs" => 100,
                        "regex" => "^https://example\\.com/docs/",
                    ],
                ],
            ],
        ],
    );
    var_dump($response);
    ```
  </CodeGroup>

  Sitemap batches scrape matching listed URLs without following links from those pages. Only `maxUrls` and `regex` are supported in their controls; omit `maxDepth` and `followSubdomains`. A full URL in `domain` is reduced to its domain.
</Accordion>

## Track progress

Replace `batch_9f2c8a` with the ID returned by submission. For cURL, save it in `BATCH_ID`:

```bash theme={null}
export BATCH_ID="batch_9f2c8a"
```

<CodeGroup>
  ```bash cURL theme={null}
  curl "https://api.context.dev/v1/batch/$BATCH_ID" \
    -H "Authorization: Bearer $CONTEXT_DEV_API_KEY"
  ```

  ```typescript TypeScript theme={null}
  import ContextDev from "context.dev";

  const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

  const response = await client.batch.retrieve("batch_9f2c8a");
  console.log(response);
  ```

  ```python Python theme={null}
  import os
  from context.dev import ContextDev

  client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

  response = client.batch.retrieve(
      "batch_9f2c8a",
  )
  print(response)
  ```

  ```ruby Ruby theme={null}
  require "cgi/core"
  require "context_dev"

  client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

  response = client.batch.retrieve(
    "batch_9f2c8a",
  )
  puts response.inspect
  ```

  ```go Go theme={null}
  package main

  import (
  	"context"
  	"fmt"
  	"os"

  	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
  	"github.com/context-dot-dev/context-go-sdk/v2/option"
  )

  func main() {
  	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))

  	response, err := client.Batch.Get(context.Background(), "batch_9f2c8a")
  	if err != nil {
  		panic(err)
  	}
  	fmt.Println(response)
  }
  ```

  ```php PHP theme={null}
  <?php
  require __DIR__.'/vendor/autoload.php';

  use ContextDev\Client;

  $client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

  $response = $client->batch->retrieve(
      "batch_9f2c8a",
  );
  var_dump($response);
  ```
</CodeGroup>

Poll every 10 to 30 seconds while `status` is `queued`, `running`, or `cancelling`. Stop when it reaches `completed`, `cancelled`, or `failed`. Use the status, not a percentage of `maxUrls`, to decide whether the job has finished.

`progress.succeeded` and `progress.failed` count page outcomes. `page_errors` groups page failures by code; `failure` describes a batch-level failure. A completed batch can still contain failed pages.

You can also add `webhookUrl` to the submit body. [Completion webhooks](/guides/scrape-websites-in-batches#receive-a-webhook) are attempted once, so keep polling as a fallback and save the signing secret returned at submission.

## Read the results

After the batch reaches a final status, request its results as JSON:

<CodeGroup>
  ```bash cURL theme={null}
  curl -G "https://api.context.dev/v1/batch/$BATCH_ID/results" \
    -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
    --data-urlencode "limit=100"
  ```

  ```typescript TypeScript theme={null}
  import ContextDev from "context.dev";

  const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

  const response = await client.batch.getResults("batch_9f2c8a", {
    limit: 100,
  });
  console.log(response);
  ```

  ```python Python theme={null}
  import os
  from context.dev import ContextDev

  client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

  response = client.batch.get_results(
      "batch_9f2c8a",
      limit=100,
  )
  print(response)
  ```

  ```ruby Ruby theme={null}
  require "cgi/core"
  require "context_dev"

  client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

  response = client.batch.get_results(
    "batch_9f2c8a",
    limit: 100,
  )
  puts response.inspect
  ```

  ```go Go theme={null}
  package main

  import (
  	"context"
  	"fmt"
  	"os"

  	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
  	"github.com/context-dot-dev/context-go-sdk/v2/option"
  )

  func main() {
  	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))

  	response, err := client.Batch.GetResults(context.Background(), "batch_9f2c8a", contextdev.BatchGetResultsParams{
  		Limit: contextdev.Int(100),
  	})
  	if err != nil {
  		panic(err)
  	}
  	fmt.Println(response)
  }
  ```

  ```php PHP theme={null}
  <?php
  require __DIR__.'/vendor/autoload.php';

  use ContextDev\Client;

  $client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

  $response = $client->batch->getResults(
      "batch_9f2c8a",
      limit: 100,
  );
  var_dump($response);
  ```
</CodeGroup>

Each record describes one page. Check `status` before reading its content:

```json sample response expandable theme={null}
{
  "data": [
    {
      "url": "https://docs.context.dev/introduction",
      "status": "ok",
      "http_status": 200,
      "final_url": "https://docs.context.dev/introduction",
      "markdown": "# Introduction\n\n...",
      "metadata": {
        "sourceUrl": "https://docs.context.dev/introduction",
        "finalUrl": "https://docs.context.dev/introduction",
        "title": "Introduction"
      },
      "cache_metadata": { "status": "miss", "age_ms": 0 }
    },
    {
      "url": "https://docs.context.dev/missing-page",
      "status": "error",
      "error_code": "NOT_FOUND",
      "message": "Page returned 404"
    }
  ],
  "has_more": true,
  "next_cursor": "..."
}
```

While `has_more` is `true`, pass the returned `next_cursor` as `cursor`. Replace `CURSOR_FROM_PREVIOUS_RESPONSE` in the SDK examples, or set `NEXT_CURSOR` for cURL:

<CodeGroup>
  ```bash cURL theme={null}
  curl -G "https://api.context.dev/v1/batch/$BATCH_ID/results" \
    -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
    --data-urlencode "limit=100" \
    --data-urlencode "cursor=$NEXT_CURSOR"
  ```

  ```typescript TypeScript theme={null}
  import ContextDev from "context.dev";

  const client = new ContextDev({ apiKey: process.env.CONTEXT_DEV_API_KEY });

  const response = await client.batch.getResults("batch_9f2c8a", {
    limit: 100,
    cursor: "CURSOR_FROM_PREVIOUS_RESPONSE",
  });
  console.log(response);
  ```

  ```python Python theme={null}
  import os
  from context.dev import ContextDev

  client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

  response = client.batch.get_results(
      "batch_9f2c8a",
      limit=100,
      cursor="CURSOR_FROM_PREVIOUS_RESPONSE",
  )
  print(response)
  ```

  ```ruby Ruby theme={null}
  require "cgi/core"
  require "context_dev"

  client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

  response = client.batch.get_results(
    "batch_9f2c8a",
    limit: 100,
    cursor: "CURSOR_FROM_PREVIOUS_RESPONSE",
  )
  puts response.inspect
  ```

  ```go Go theme={null}
  package main

  import (
  	"context"
  	"fmt"
  	"os"

  	contextdev "github.com/context-dot-dev/context-go-sdk/v2"
  	"github.com/context-dot-dev/context-go-sdk/v2/option"
  )

  func main() {
  	client := contextdev.NewClient(option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")))

  	response, err := client.Batch.GetResults(context.Background(), "batch_9f2c8a", contextdev.BatchGetResultsParams{
  		Limit:  contextdev.Int(100),
  		Cursor: contextdev.String("CURSOR_FROM_PREVIOUS_RESPONSE"),
  	})
  	if err != nil {
  		panic(err)
  	}
  	fmt.Println(response)
  }
  ```

  ```php PHP theme={null}
  <?php
  require __DIR__.'/vendor/autoload.php';

  use ContextDev\Client;

  $client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

  $response = $client->batch->getResults(
      "batch_9f2c8a",
      limit: 100,
      cursor: "CURSOR_FROM_PREVIOUS_RESPONSE",
  );
  var_dump($response);
  ```
</CodeGroup>

`limit` accepts 1 to 100 records and defaults to 25. A page can close early to stay under approximately 8 MB, so follow the cursor even when fewer than 100 records arrive.

For large imports, use the signed URLs in the retrieved batch's `results.files` to download gzipped NDJSON. Set `RESULT_URL` to a file's `url`, then stream its records:

```bash theme={null}
curl --fail --location "$RESULT_URL" | gzip --decompress
```

Download each file. Links expire at `results.expires_at`; retrieve the batch again for fresh links. File order is not guaranteed.

## Costs and limits

Each successfully scraped page costs 1 credit. A start-URL crawl reserves its page budget upfront: `input.reserved_is_ceiling` is `true` because the reachable page count is not yet known. A sitemap batch reserves for the exact accepted URL count instead.

Credits for pages that do not succeed, including unused crawl capacity, are refunded when the batch settles. If PDF OCR is enabled, each recovered PDF page costs 1 additional credit. OCR is off by default. Read `credits.net` after settlement for the final cost: `reserved - refunded + ocr_charged`.

Batch management uses a separate [rate-limit bucket](/optimization/rate-limits#batch-api-has-its-own-bucket). Submission consumes 50 of its 1,000 units per minute; each poll, results request, or cancellation consumes 1. These management calls cost zero API credits, separate from the page credits reserved at submission.

## Handle failures

Retry only the failed pages when practical, using a [fixed URL batch](/guides/scrape-websites-in-batches#submit-a-fixed-url-list). Keep successful content and inspect each failed record's `error_code` and `message` before retrying.

| Response                       | What to do                                                         |
| ------------------------------ | ------------------------------------------------------------------ |
| `400`                          | Check the request fields, page limits, and source URL.             |
| `403 BATCH_LIMIT_EXCEEDED`     | Wait for an active batch to finish before submitting another.      |
| `409 IDEMPOTENCY_KEY_CONFLICT` | Reuse the original request body or choose a new key for a new job. |
| `409 BATCH_NOT_COMPLETED`      | Keep polling before requesting results.                            |
| `429 RATE_LIMITED`             | Honor `Retry-After` and retry with backoff.                        |
| `500` at submission            | Retry with the same idempotency key and body.                      |

To stop a crawl, call `POST /batch/{batch_id}/cancel`. This prevents new pages from starting; pages already in progress finish. Continue polling until the job settles and unused credits are refunded.

## Next steps

<CardGroup cols={2}>
  <Card title="Crawl Sync" icon="bolt" href="/guides/crawl-website">
    Collect a smaller site in one synchronous response.
  </Card>

  <Card title="Submit a batch" icon="code" href="/api-reference/batches/submit">
    Review the full request schema and per-page options.
  </Card>

  <Card title="Build a RAG pipeline" icon="database" href="/use-cases/build-rag-from-websites">
    Turn crawled Markdown into a searchable knowledge base.
  </Card>
</CardGroup>