Content
# advanced-fetch-mcp
English | [English](README.en.md)
Provide easy-to-use, powerful, and token-saving web scraping capabilities for Agents.
More powerful than vanilla fetch and more concise than Playwright.
## Features
- **Content Extraction**: Based on trafilatura's powerful content extraction capabilities, configurable extraction strategies and scope, maximum noise reduction to save tokens.
- **Support Dynamic Websites**: Based on Playwright's dynamic website crawling capabilities, intelligent identification of page stable state.
- **Support Authentication**: Request automatically carries cookie authentication information. Agent can open a visible browser for users to complete login. After logging in once, subsequent requests can continue to reuse login information.
- **Citation and Segmentation Processing**: Support searching and segment reading within pages. Subsequent processing of the same page can use refid to reference.
- **Anti-Scraping**: Request limit minimum interval for the same hostname to prevent triggering rate limiting. Includes Playwright-Stealth to mimic real requests and prevent detection as robots.
- **Proxy Support**: Supports `HTTP_PROXY` / `HTTPS_PROXY` / `NO_PROXY`
- **LLM Sampling (Experimental)**: Through `sampling.prompt`, refine webpage content and return concise results. Supported clients include VS Code GitHub Copilot, goose, Amp, etc.
- **Network Image Reading and Resource Download**: Through `read_image`, multi-modal reading of network images; through `download`, download network files to local paths.
## MCP Client Configuration
```json
{
"mcpServers": {
"advanced-fetch": {
"command": "uvx",
"args": [
"--from",
"git+https://github.com/john-walks-slow/advanced-fetch-mcp",
"advanced-fetch-mcp"
],
"env": {
"SCHEMA_LANGUAGE": "en"
}
}
}
}
```
## Schema
### I. Top-Level Parameters
| Parameter Name | Type | Default Value | Description |
| --- | --- | --- | --- |
| `url` | `string \| Array<string>` | Required | The complete URL of the target webpage, the reference ID of the previous result (reuse crawling result), or URL list. When passing a list, process multiple pages simultaneously and return results array. |
| `operation` | `"view" \| "find" \| "sampling" \| "eval" \| "elicit"` | `"view"` | Operation type: view, search within page, LLM extraction, execute JS, or request user manual operation (only when blocked by captcha/login wall). |
| `fetch` | `object` | See below | Page acquisition method and waiting strategy configuration. |
| `view` | `object` | See below | View operation configuration |
| `find` | `object \| null` | `null` | Find operation configuration |
| `sampling` | `object \| null` | `null` | Sampling operation configuration |
| `eval` | `object \| null` | `null` | Eval operation configuration |
| `elicit` | `object \| null` | `null` | Elicit operation configuration |
| `cursor` | `integer \| null` | `null` | Offset for continuing to read. Effective for view and find operations. |
| `max_length` | `integer` | `20000` | Maximum result length. |
| `output_to_file` | `string \| null` | `null` | If specified, write the result to this file path in JSON format instead of returning directly, ignoring max_length. |
### II. `fetch` Object
| Path | Type | Default Value | Description |
| --- | --- | --- | --- |
| `fetch.mode` | `"dynamic" \| "static"` | `"static"` | Crawling method: dynamic= browser, static=request. Automatically reuse authentication information. |
| `fetch.min_stable_seconds` | `number` | `3.0` | Minimum stable time (seconds) for dynamic crawling to wait for content stability. |
| `fetch.timeout` | `number` | `12.0` | Crawling timeout in seconds. Return current obtained content after timeout. |
### III. `view` Object
| Path | Type | Default Value | Description |
| --- | --- | --- | --- |
| `view.output_format` | `"markdown" \| "html"` | `"markdown"` | Content output format. |
| `view.markdown_engine` | `"article" \| "full"` | `"article"` | Markdown extraction engine. article uses trafilatura to extract article content; full uses markdownify to extract complete page. |
| `view.max_length` | `integer` | `20000` | Maximum result length. |
| `view.links` | `boolean` | `true` | Whether to extract links in the page. |
| `view.with_screenshot` | `boolean` | `false` | Whether to take a screenshot. Automatically use dynamic mode to obtain the page and capture the first screen, returning base64-encoded PNG. |
### IV. `find` Object
| Path | Type | Default Value | Description |
| --- | --- | --- | --- |
| `find.query` | `string` | Required | Text or regular expression to search for. |
| `find.regex` | `boolean` | `false` | Whether to treat query as a regular expression. |
### V. `sampling` Object
| Path | Type | Default Value | Description |
| --- | --- | --- | --- |
| `sampling.prompt` | `string` | Required | Prompt word to guide LLM to extract information from page content. |
| `sampling.model` | `string \| null` | `null` | Preferred model name. |
### VI. `eval` Object
| Path | Type | Default Value | Description |
| --- | --- | --- | --- |
| `eval.script` | `string` | Required | JavaScript code executed in the page context. |
### VII. Usage Constraints
| Rule | Description |
| --- | --- |
| Operation-specific configuration | Only when `operation` is the corresponding value, can `find`, `sampling`, or `eval` objects be provided, and the three are mutually exclusive. |
| `eval` mode restriction | When `operation="eval"`, `fetch.mode` must be `"dynamic"`. |
| `elicit` mode restriction | When `operation="elicit"`, `fetch.mode` must be `"dynamic"`. |
| `max_length` scope | Effective for `view`, `find`, `sampling`, and `eval`, limiting the final return result. |
| `cursor` scope | Effective for `view` and `find`. Used to continue reading from the last returned `next_cursor` position. |
| Continued reading consistency | When continuing to read with `cursor`, fill in the `refid` of the last result as `url` to reuse the cache and ensure referencing the same page snapshot. |
## Return Value Format
### General Return Structure
```json
{
"success": true,
"final_url": "https://example.com/final",
"result": "...",
"refid": "a1b2c3d4e5f6",
"timed_out": true,
"timeout_stage": "network_idle",
"intervention_ended_by": "timeout",
"truncated": true,
"next_cursor": 8000,
"warnings": ["..."]
}
```
### General Field Description
| Field | Type | Necessity | Description |
| --- | --- | --- | --- |
| `success` | `boolean` | Yes | Always `true` when successful. |
| `final_url` | `string` | Yes | Final page URL, which may differ from the input `url`. |
| `result` | `string` | Yes | Main return content. `view`/`sampling`/`eval` returns text results; `find` currently fixed to empty string. |
| `refid` | `string` | No | Reference ID of this crawling result. Filling this value into the `url` parameter of subsequent requests can directly reuse the cache. |
| `timed_out` | `boolean` | No | Appears when crawling stage timeout occurs. |
| `timeout_stage` | `string` | No | Stage where timeout occurred. |
| `intervention_ended_by` | `string` | No | Reason for manual intervention ending, such as `timeout` or `page_closed`. |
| `truncated` | `boolean` | No | Appears when return result is truncated by `max_length`. |
| `next_cursor` | `integer` | No | Offset for continuing to read or search when returned. |
| `warnings` | `string[]` | No | List of warning messages. |
### `view` Return
```json
{
"success": true,
"final_url": "https://example.com/final",
"result": "Page content fragment",
"truncated": true,
"next_cursor": 8000
}
```
Description:
- `result` is the text content of the current window.
- When the content is not fully read, `next_cursor` will be returned.
### `find` Return
```json
{
"success": true,
"final_url": "https://example.com/final",
"result": "",
"found": true,
"matches": [
{
"snippet": "... nearby text fragment ...",
"cursor": 1234
}
],
"matches_total": 3,
"matches_truncated": false,
"next_cursor": 1234
}
```
### `find` Specific Fields
| Field | Type | Description |
| --- | --- | --- |
| `found` | `boolean` | Whether a match is found. |
| `matches` | `object[]` | List of match summaries. |
| `matches_total` | `integer` | Total number of matches. |
| `matches_truncated` | `boolean` | Whether match summaries are truncated due to excessive number. |
### `find.matches` Item Structure
| Field | Type | Description |
| --- | --- | --- |
| `snippet` | `string` | Nearby text summary of the match. |
| `cursor` | `integer` | Offset for continuing to read or search. |
### `sampling` Return (Experimental)
> Supported clients include VS Code GitHub Copilot, goose, Amp, Glama, Joey, fast-agent, mcp-use, Postman, etc.
```json
{
"success": true,
"final_url": "https://example.com/final",
"result": "Refined result text",
"truncated": true
}
```
Description:
- `result` is the text result refined by LLM.
- If `sampling` fails, it will fall back to the original page content and explain in `warnings`.
### `eval` Return
```json
{
"success": true,
"final_url": "https://example.com/final",
"result": "{\n \"title\": \"Example\"\n}",
"truncated": false
}
```
Description:
- `result` is the stringified content of the script execution result.
- If the return value is an object, array, boolean, or number, it will be serialized to a JSON string and returned.
## Examples
Extract content:
```yaml
url: https://example.com
operation: view
```
Get complete page markdown (without extracting content, preserving all):
```yaml
url: https://example.com
operation: view
view:
markdown_engine: full
```
Extract links in the page:
```yaml
url: https://example.com
operation: view
view:
links: true
```
Output in HTML format:
```yaml
url: https://example.com
operation: view
view:
output_format: html
```
Set timeout:
```yaml
url: https://example.com
operation: view
fetch:
timeout: 60
```
Search for keywords:
```yaml
url: https://example.com
operation: find
find:
query: price
```
Continue reading from a specified position (with `find` returned `matches[].cursor` or `next_cursor`):
```yaml
url: <refid>
operation: view
cursor: 300
```
Intelligent Sampling (let the model organize):
```yaml
url: https://example.com
operation: sampling
sampling:
prompt: Extract product name and price
```
Execute JS within the page:
```yaml
url: https://example.com
operation: eval
fetch:
mode: dynamic
eval:
script: |
() => ({
title: document.title,
href: location.href,
itemCount: document.querySelectorAll('.item').length
})
```
Websites requiring login:
```yaml
url: https://private-site.com
operation: elicit
fetch:
mode: dynamic
```
## `read_image` — Get Images
Independent tool, not dependent on `advanced_fetch`.
```yaml
# Single image
read_image:
url: https://example.com/photo.png
# Multiple images
read_image:
url:
- https://example.com/photo1.png
- https://example.com/photo2.jpg
```
**Parameters**:
| Parameter | Type | Default Value | Description |
| --- | --- | --- | --- |
| `url` | `string \| string[]` | Required | Image URL, can pass a single URL or URL list. |
| `timeout` | `number` | `30` | Timeout in seconds (default 30). |
**Output**: Returns `ImageContent` list (MCP native image), returns `TextContent` to explain error reason on failure. Single URL failure does not affect other URLs.
## `download` — Download File
Standalone tool, independent of `advanced_fetch`.
```yaml
download:
url: https://example.com/document.pdf
file_path: /path/to/save/document.pdf
```
**Parameters**:
| Parameter | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `url` | `string` | Required | Download source URL. |
| `file_path` | `string` | Required | Local save path. Automatically creates parent directory, automatically resolves to absolute path. |
| `overwrite` | `boolean` | `false` | If `false` and file exists, throws an error; set to `true` to overwrite existing file. |
| `timeout` | `number` | `120` | Timeout in seconds (default 120). |
**Output**: On success, returns a JSON containing `file_path`, `size`, and `content_type`; on failure, returns `{success: false, error: ...}`. Download interruptions automatically clean up residual files.
## Session Mode
When crawling internal network pages, use `operation="elicit"` to open a visible browser. After logging in once, cookies are automatically saved through `storage_state.json`. Subsequent silent requests (without `operation="elicit"`) automatically carry the login state.
## Cache
Each crawl result generates a `refid` that is returned. To reuse the cache, directly fill the `refid` into the `url` parameter of subsequent requests without needing to crawl again.
- `refid` is a 12-character hex string with a cache validity period of 24 hours.
- Using `refid` as a URL does not generate a new `refid` (the same `refid` always points to the same content).
- Directly passing a regular URL always results in a new crawl, without implicitly using the cache.
- If `refid` has expired or does not exist, an error is returned, and the original URL must be used for a new crawl.
## Environment Variables
### General
- `FETCH_TIMEOUT`: Total crawl timeout in seconds. Default `12`.
- `PER_SITE_RATE_LIMIT_SECONDS`: Minimum crawl interval in seconds for the same hostname. Default `1.0`. Set to `0` to disable. Serial crawls include a small random jitter to avoid overly fixed request rhythms.
### Auto-Wait
- `AUTO_WAIT_POLL_INTERVAL`: Dynamic crawl stability detection polling interval (seconds). Default `0.25`.
- `AUTO_WAIT_MIN_STABLE_SECONDS`: Minimum stable content wait time during dynamic crawls (seconds). Default `3.0`.
- `AUTO_WAIT_MIN_CONTENT_LENGTH`: Minimum content length for dynamic crawls. Content is considered stable and ready only if it reaches this threshold. Default `150`.
- `AUTO_WAIT_SAMPLE_EDGE_CHARS`: Number of characters from the start and end used for comparison during stability detection. Default `200`.
### Extraction / LLM
- `DEFAULT_MAX_LENGTH`: Default maximum returned length. Default `20000`.
- `ENABLE_PROMPT_EXTRACTION`: Whether to enable `sampling`. Default `false`. Experimental feature. Supported clients include VS Code GitHub Copilot, goose, Amp, Glama, Joey, fast-agent, mcp-use, Postman, etc.
- `PROMPT_INPUT_MAX_CHARS`: Maximum input characters sent to LLM. Default `64000`.
- `MAX_FIND_MATCHES`: Maximum number of matches returned for page search. Default `12`.
- `FIND_SNIPPET_MAX_CHARS`: Maximum length of each search hit snippet. Default `240`.
- `MAX_LINKS_COUNT`: Maximum number of links returned by the `links` operation. Default `30`.
- `SCHEMA_LANGUAGE`: Schema description language. Default `zh`. Supports `zh` / `en`.
### Browser / Session
- `BROWSER_CHANNEL`: Browser channel passed to Playwright. Default `chrome`. Options include `chrome`, `chrome-beta`, `chrome-dev`, `msedge`, `msedge-beta`, `msedge-dev`, `chromium`.
- `BROWSER_AUTH_STORAGE_STATE`: Path to `storage_state.json` in `auth` mode. Default `~/.advanced-fetch-auth/storage_state.json`.
- `BROWSER_LOCALE`: Browser locale. Default empty string. Leave empty to use the system default.
- `BROWSER_TIMEZONE_ID`: Browser timezone. Default empty string. Leave empty to use the system default.
- `BROWSER_COLOR_SCHEME`: Color scheme. Default `light`.
- `BROWSER_VIEWPORT_WIDTH`: Viewport width. Default `1440`.
- `BROWSER_VIEWPORT_HEIGHT`: Viewport height. Default `900`.
- `ENABLE_AUTH_STEALTH`: Whether to enable stealth in `auth` mode. Default `true`.
- `INTERVENTION_TIMEOUT_SECONDS`: Timeout in seconds for user manual operation wait. Default `600`.
### Proxy
- `ENABLE_STATIC_PROXY`: Whether to enable a proxy in static mode. Default `true`.
- `ENABLE_DYNAMIC_PROXY`: Whether to enable a proxy in dynamic mode (browser). Default `false`.
- `HTTP_PROXY`: HTTP proxy address. Default empty string.
- `HTTPS_PROXY`: HTTPS proxy address. Default empty string.
- `NO_PROXY`: Proxy bypass list. Default empty string.
### Env Loading
- `ADVANCED_FETCH_ENV_FILE`: Explicitly specify the dotenv file path. Default empty string.
### Others
- `IGNORE_SSL_ERRORS`: Whether to ignore HTTPS / SSL certificate errors. Default `false`.
## Local Installation
```bash
uv sync
```
## Testing
```bash
uv run python -m unittest discover -s tests
```
