Scrape

Scrape visits a URL and returns the outputs you enable in formats. Combine formats in one request and check each output’s success before using its data.

Make a request

Set up your API key and SDK in the Quickstart. This request returns the page’s main content as Markdown.

import ContextDev from "context.dev";

const client = new ContextDev({
  apiKey: process.env.CONTEXT_DEV_API_KEY,
});

const page = await client.web.scrape({
  url: "https://example.com",
  formats: { markdown: true },
  sharedParams: { mainContentOnly: true },
});

console.log(page.metadata.title);
console.log(page.markdown.data);

import os
from context.dev import ContextDev

client = ContextDev(api_key=os.environ["CONTEXT_DEV_API_KEY"])

page = client.web.scrape(
    url="https://example.com",
    formats={"markdown": True},
    shared_params={"main_content_only": True},
)

print(page.metadata.title)
print(page.markdown.data)

require "cgi/core"
require "context_dev"

client = ContextDev::Client.new(api_key: ENV.fetch("CONTEXT_DEV_API_KEY"))

page = client.web.scrape(
  url: "https://example.com",
  formats: {markdown: true},
  shared_params: {main_content_only: true},
)

puts page.metadata.title
puts page.markdown.data

package main

import (
    "context"
    "fmt"
    "os"

    contextdev "github.com/context-dot-dev/context-go-sdk/v2"
    "github.com/context-dot-dev/context-go-sdk/v2/option"
)

func main() {
    client := contextdev.NewClient(
        option.WithAPIKey(os.Getenv("CONTEXT_DEV_API_KEY")),
    )

    page, err := client.Web.Scrape(context.Background(), contextdev.WebScrapeParams{
        URL:          "https://example.com",
        Formats:      contextdev.WebScrapeParamsFormats{Markdown: contextdev.Bool(true)},
        SharedParams: contextdev.WebScrapeParamsSharedParams{MainContentOnly: contextdev.Bool(true)},
    })
    if err != nil {
        panic(err)
    }

    fmt.Println(page.Metadata.Title)
    fmt.Println(page.Markdown.Data)
}

<?php

require __DIR__.'/vendor/autoload.php';

use ContextDev\Client;

$client = new Client(apiKey: getenv('CONTEXT_DEV_API_KEY'));

$page = $client->web->scrape(
    formats: ['markdown' => true],
    url: 'https://example.com',
    sharedParams: ['mainContentOnly' => true],
);

echo $page->metadata->title, PHP_EOL;
echo $page->markdown->data, PHP_EOL;

curl https://api.context.dev/v1/web/scrape \
  -H "Authorization: Bearer $CONTEXT_DEV_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": { "markdown": true },
    "sharedParams": { "mainContentOnly": true }
  }'

Read the response

Every output has requested, success, and data. A successful output has success: true. Failed outputs have success: false and data: null; outputs you did not request have success: null. One failed output can coexist with successful ones in an HTTP 200 response. Abridged response:

{
  "url": "https://example.com/",
  "markdown": {"requested": true, "success": true, "data": "# Example Domain"},
  "metadata": {"title": "Example Domain"},
  "cache_metadata": {"status": "miss", "age_ms": 0},
  "request_id": "d7aa1c3a-38d2-4fb5-92c8-0f0cac3efa37"
}

isPartial flags incomplete results. metadata describes the page; key_metadata reports request usage. Save request_id when investigating failures.

Formats

Format	Output	Guide
`html`	Rendered HTML	HTML
`markdown`	Text with Markdown formatting	Markdown
`screenshot`	Inline image data URL	Screenshot
`images`	Image URLs and optional enrichment	Images
`bytes`	Original response bytes as base64	Bytes
`parse`	CSS-selected fields in `parsed.data`	Parse fields
`highlights`	Passages relevant to a question	Highlights
`json`	An object matching your JSON Schema	JSON
`product`	Product details and availability	Product

Shape the page visit

Use sharedParams for browser actions, waiting and rendering, content filtering, location and headers, or PDF parsing. Use maxAgeMs for freshness, timeoutOpts for deadlines, zdr for zero data retention, and tags for usage tracking. The reference lists the complete request and response.

Choose another API

Use Map to find a site’s URLs, Crawl to read several pages, or Batches to process a URL list asynchronously. Search finds pages from a query; Answers researches a question. Parse accepts uploaded files.

Move from the per-format endpoints

The older routes remain available. New integrations can request several formats through POST /web/scrape:

Earlier route	Request on Scrape	Read
`GET /web/scrape/html`	`formats.html: true`	`html.data`
`GET /web/scrape/markdown`	`formats.markdown: true`	`markdown.data`
`GET /web/scrape/screenshot`	`formats.screenshot: true`	`screenshot.data`
`GET /web/scrape/images`	`formats.images: true`	`images.data`
`GET /web/scrape/bytes`	`formats.bytes: true`	`bytes.data`
`GET /web/scrape/sitemap`	Use Map URLs	`urls`

The option names and response shapes differ. Use the format guides when migrating.

Get started

Web data

Company and people data

Automation

Account and billing

Production

Make a request

Read the response

Formats

Shape the page visit

Choose another API

Move from the per-format endpoints

​Make a request

​Read the response

​Formats

​Shape the page visit

​Choose another API

​Move from the per-format endpoints

Make a request

Read the response

Formats

Shape the page visit

Choose another API

Move from the per-format endpoints