Web Scrape

Web Scrape stage showing content extraction from URLs

The Web Scrape stage fetches content from web URLs using the Engine’s Playwright service. It handles JavaScript rendering and main-content extraction to add web content to your retrieval pipeline.

Stage Category: APPLY (Enriches documents with scraped content)Transformation: N documents → N documents (with web content added)

When to Use

Use Case	Description
URL enrichment	Extract content from URLs in documents
Reference expansion	Scrape linked references for context
Content aggregation	Pull in external content sources
Real-time content	Access current webpage content

When NOT to Use

Scenario	Recommended Alternative
Searching the web	`external_web_search` (Exa)
Static content already indexed	Use indexed content
High-volume scraping	Pre-index content in Mixpeek

Parameters

Parameter	Type	Default	Description
`url_field`	string	`metadata.url`	Document field containing the URL to scrape
`output_field`	string	`web_content`	Field that receives the extracted content
`strategy`	string	`auto`	`static`, `javascript`, or `auto`
`timeout`	integer	`10`	Request timeout in seconds (1 to 60)
`include_html`	boolean	`false`	Include the raw HTML in the output
`min_content_length`	integer	`500`	With `auto`, fall back to JavaScript when static parsing returns fewer characters
`when`	object	none	Only scrape documents that match this condition
`on_error`	string	`skip`	`skip`, `remove`, or `raise`
`batch_size`	integer	`5`	Concurrent fetch requests (1 to 20)

Configuration Examples

{
  "stage_name": "web_scrape",
  "stage_type": "apply",
  "config": {
    "stage_id": "web_scrape",
    "parameters": {
      "url_field": "metadata.source_url",
      "output_field": "page_content"
    }
  }
}

{
  "stage_name": "web_scrape",
  "stage_type": "apply",
  "config": {
    "stage_id": "web_scrape",
    "parameters": {
      "url_field": "metadata.url",
      "output_field": "content",
      "strategy": "javascript",
      "timeout": 30
    }
  }
}

{
  "stage_name": "web_scrape",
  "stage_type": "apply",
  "config": {
    "stage_id": "web_scrape",
    "parameters": {
      "url_field": "metadata.reference_url",
      "output_field": "reference",
      "when": {
        "AND": [
          {"field": "score", "operator": "gte", "value": 0.8}
        ]
      }
    }
  }
}

{
  "stage_name": "web_scrape",
  "stage_type": "apply",
  "config": {
    "stage_id": "web_scrape",
    "parameters": {
      "url_field": "url",
      "output_field": "html_content",
      "include_html": true,
      "timeout": 15
    }
  }
}

Output Schema

Default Output

The extracted content is stored as an object at output_field.

{
  "document_id": "doc_123",
  "metadata": {
    "source_url": "https://example.com/article"
  },
  "page_content": {
    "url": "https://example.com/article",
    "final_url": "https://example.com/article",
    "title": "Article Title",
    "text": "Article content here...",
    "content_length": 1842,
    "metadata": {
      "author": "Jane Doe",
      "date": "2025-01-15",
      "description": "Meta description",
      "sitename": "Example",
      "language": "en"
    },
    "strategy_used": "static"
  }
}

With Raw HTML

Set include_html to true to add the page HTML to the same object.

{
  "document_id": "doc_123",
  "html_content": {
    "url": "https://example.com/article",
    "final_url": "https://example.com/article",
    "title": "Article Title",
    "text": "Article content here...",
    "content_length": 1842,
    "metadata": {},
    "strategy_used": "static",
    "html": "<html>...</html>"
  }
}

Failed Fetch

With the default on_error: skip, a document whose fetch fails passes through unchanged, without the output field.

Scraping Strategies

Strategy	Description
`static`	Fast HTML parsing without JavaScript. Best for articles and blogs
`javascript`	Full browser rendering. Slower, and required for single-page apps
`auto`	Tries `static` first and falls back to `javascript` when the extracted text is shorter than `min_content_length`

Use strategy: "javascript" when scraping JavaScript-heavy sites, and raise timeout to 20-30 seconds.

Performance

Metric	Value
Latency	1-10s (depends on page complexity)
Concurrent requests	`batch_size`, 5 by default and up to 20
Timeout default	10 seconds

Web scraping adds significant latency. Use sparingly and consider pre-indexing frequently accessed content.

Common Pipeline Patterns

Enrich Documents with Referenced Content

[
  {
    "stage_name": "semantic_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [
          { "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 10 }
        ],
        "final_top_k": 10
      }
    }
  },
  {
    "stage_name": "structured_filter",
    "stage_type": "filter",
    "config": {
      "stage_id": "attribute_filter",
      "parameters": {
        "field": "metadata.has_url",
        "operator": "eq",
        "value": true
      }
    }
  },
  {
    "stage_name": "web_scrape",
    "stage_type": "apply",
    "config": {
      "stage_id": "web_scrape",
      "parameters": {
        "url_field": "metadata.source_url",
        "output_field": "source_content"
      }
    }
  }
]

Scrape and Summarize

[
  {
    "stage_name": "web_scrape",
    "stage_type": "apply",
    "config": {
      "stage_id": "web_scrape",
      "parameters": {
        "url_field": "metadata.url",
        "output_field": "page_content"
      }
    }
  },
  {
    "stage_name": "json_transform",
    "stage_type": "apply",
    "config": {
      "stage_id": "json_transform",
      "parameters": {
        "template": {
          "content": "{{ DOC.page_content.text }}",
          "source": "{{ DOC.metadata.url }}"
        }
      }
    }
  },
  {
    "stage_name": "summarize",
    "stage_type": "reduce",
    "config": {
      "stage_id": "summarize",
      "parameters": {
        "provider": "google",
        "model_name": "gemini-2.5-flash-lite",
        "prompt": "Summarize the key points from these webpages:\n\n{{DOCUMENTS}}"
      }
    }
  }
]

Error Handling

Error	Behavior
Missing or empty URL field	Follows `on_error`
Invalid URL, timeout, or HTTP error such as 404 or 403	Follows `on_error`

on_error accepts three values. skip (default) passes the document through unchanged. remove drops the document from the results. raise fails the stage.

Rate Limits and Best Practices

Batch wisely: Limit to 5-10 URLs per pipeline run
Cache results: Consider storing scraped content
Use timeouts: Set an appropriate timeout (in seconds) for your use case

External Web Search - Search the web (Exa)
API Call - General HTTP enrichment
Document Enrich - Collection joins

Get started

Connect your data

Extract features

Build retrievers

Enrich & organize

Integrate & operate

Resources

When to Use

When NOT to Use

Parameters

Configuration Examples

Output Schema

Default Output

With Raw HTML

Failed Fetch

Scraping Strategies

Performance

Common Pipeline Patterns

Enrich Documents with Referenced Content

Scrape and Summarize

Error Handling

Rate Limits and Best Practices

​When to Use

​When NOT to Use

​Parameters

​Configuration Examples

​Output Schema

​Default Output

​With Raw HTML

​Failed Fetch

​Scraping Strategies

​Performance

​Common Pipeline Patterns

​Enrich Documents with Referenced Content

​Scrape and Summarize

​Error Handling

​Rate Limits and Best Practices

​Related

When to Use

When NOT to Use

Parameters

Configuration Examples

Output Schema

Default Output

With Raw HTML

Failed Fetch

Scraping Strategies

Performance

Common Pipeline Patterns

Enrich Documents with Referenced Content

Scrape and Summarize

Error Handling

Rate Limits and Best Practices

Related