fetch-mcp

<div align="center"> <img src="https://raw.githubusercontent.com/jae-jae/fetcher-mcp/refs/heads/main/icon.svg" width="100" height="100" alt="Fetcher MCP Icon" /> </div> [中文](https://www.readme-i18n.com/jae-jae/fetcher-mcp?lang=zh) | [Deutsch](https://www.readme-i18n.com/jae-jae/fetcher-mcp?lang=de) | [Español](https://www.readme-i18n.com/jae-jae/fetcher-mcp?lang=es) | [français](https://www.readme-i18n.com/jae-jae/fetcher-mcp?lang=fr) | [日本語](https://www.readme-i18n.com/jae-jae/fetcher-mcp?lang=ja) | [한국어](https://www.readme-i18n.com/jae-jae/fetcher-mcp?lang=ko) | [Português](https://www.readme-i18n.com/jae-jae/fetcher-mcp?lang=pt) | [Русский](https://www.readme-i18n.com/jae-jae/fetcher-mcp?lang=ru) # Fetcher MCP MCP server for fetch web page content using Playwright headless browser. > 🌟 **Recommended**: [OllaMan](https://ollaman.com/) - Powerful Ollama AI Model Manager. ## Advantages - **JavaScript Support**: Unlike traditional web scrapers, Fetcher MCP uses Playwright to execute JavaScript, making it capable of handling dynamic web content and modern web applications. - **Intelligent Content Extraction**: Built-in Readability algorithm automatically extracts the main content from web pages, removing ads, navigation, and other non-essential elements. - **Flexible Output Format**: Supports both HTML and Markdown output formats, making it easy to integrate with various downstream applications. - **Parallel Processing**: The `fetch_urls` tool enables concurrent fetching of multiple URLs, significantly improving efficiency for batch operations. - **Resource Optimization**: Automatically blocks unnecessary resources (images, stylesheets, fonts, media) to reduce bandwidth usage and improve performance. - **Robust Error Handling**: Comprehensive error handling and logging ensure reliable operation even when dealing with problematic web pages. - **Configurable Parameters**: Fine-grained control over timeouts, content extraction, and output formatting to suit different use cases. ## Quick Start Run directly with npx: ```bash npx -y fetcher-mcp ``` First time setup - install the required browser by running the following command in your terminal: ```bash npx playwright install chromium ``` ### HTTP and SSE Transport Use the `--transport=http` parameter to start both Streamable HTTP endpoint and SSE endpoint services simultaneously: ```bash npx -y fetcher-mcp --log --transport=http --host=0.0.0.0 --port=3000 ``` After startup, the server provides the following endpoints: - `/mcp` - Streamable HTTP endpoint (modern MCP protocol) - `/sse` - SSE endpoint (legacy MCP protocol) Clients can choose which method to connect based on their needs. ### Debug Mode Run with the `--debug` option to show the browser window for debugging: ```bash npx -y fetcher-mcp --debug ``` ## Configuration MCP Configure this MCP server in Claude Desktop: On MacOS: `~/Library/Application Support/Claude/claude_desktop_config.json` On Windows: `%APPDATA%/Claude/claude_desktop_config.json` ```json { "mcpServers": { "fetcher": { "command": "npx", "args": ["-y", "fetcher-mcp"] } } } ``` ## Docker Deployment ### Running with Docker ```bash docker run -p 3000:3000 ghcr.io/jae-jae/fetcher-mcp:latest ``` ### Deploying with Docker Compose Create a `docker-compose.yml` file: ```yaml version: "3.8" services: fetcher-mcp: image: ghcr.io/jae-jae/fetcher-mcp:latest container_name: fetcher-mcp restart: unless-stopped ports: - "3000:3000" environment: - NODE_ENV=production # Using host network mode on Linux hosts can improve browser access efficiency # network_mode: "host" volumes: # For Playwright, may need to share certain system paths - /tmp:/tmp # Health check healthcheck: test: ["CMD", "wget", "--spider", "-q", "http://localhost:3000"] interval: 30s timeout: 10s retries: 3 ``` Then run: ```bash docker-compose up -d ``` ## Features - `fetch_url` - Retrieve web page content from a specified URL - Uses Playwright headless browser to parse JavaScript - Supports intelligent extraction of main content and conversion to Markdown - Supports the following parameters: - `url`: The URL of the web page to fetch (required parameter) - `timeout`: Page loading timeout in milliseconds, default is 30000 (30 seconds) - `waitUntil`: Specifies when navigation is considered complete, options: 'load', 'domcontentloaded', 'networkidle', 'commit', default is 'load' - `extractContent`: Whether to intelligently extract the main content, default is true - `maxLength`: Maximum length of returned content (in characters), default is no limit - `returnHtml`: Whether to return HTML content instead of Markdown, default is false - `waitForNavigation`: Whether to wait for additional navigation after initial page load (useful for sites with anti-bot verification), default is false - `navigationTimeout`: Maximum time to wait for additional navigation in milliseconds, default is 10000 (10 seconds) - `disableMedia`: Whether to disable media resources (images, stylesheets, fonts, media), default is true - `debug`: Whether to enable debug mode (showing browser window), overrides the --debug command line flag if specified - `fetch_urls` - Batch retrieve web page content from multiple URLs in parallel - Uses multi-tab parallel fetching for improved performance - Returns combined results with clear separation between webpages - Supports the following parameters: - `urls`: Array of URLs to fetch (required parameter) - Other parameters are the same as `fetch_url` - `browser_install` - Install Playwright Chromium browser binary automatically - Installs required Chromium browser binary when not available - Automatically suggested when browser installation errors occur - Supports the following parameters: - `withDeps`: Install system dependencies required by Chromium browser, default is false - `force`: Force installation even if Chromium is already installed, default is false ## Tips ### Handling Special Website Scenarios #### Dealing with Anti-Crawler Mechanisms - **Wait for Complete Loading**: For websites using CAPTCHA, redirects, or other verification mechanisms, include in your prompt: ``` Please wait for the page to fully load ``` This will use the `waitForNavigation: true` parameter. - **Increase Timeout Duration**: For websites that load slowly: ``` Please set the page loading timeout to 60 seconds ``` This adjusts both `timeout` and `navigationTimeout` parameters accordingly. #### Content Retrieval Adjustments - **Preserve Original HTML Structure**: When content extraction might fail: ``` Please preserve the original HTML content ``` Sets `extractContent: false` and `returnHtml: true`. - **Fetch Complete Page Content**: When extracted content is too limited: ``` Please fetch the complete webpage content instead of just the main content ``` Sets `extractContent: false`. - **Return Content as HTML**: When HTML format is needed instead of default Markdown: ``` Please return the content in HTML format ``` Sets `returnHtml: true`. ### Debugging and Authentication #### Enabling Debug Mode - **Dynamic Debug Activation**: To display the browser window during a specific fetch operation: ``` Please enable debug mode for this fetch operation ``` This sets `debug: true` even if the server was started without the `--debug` flag. #### Using Custom Cookies for Authentication - **Manual Login**: To login using your own credentials: ``` Please run in debug mode so I can manually log in to the website ``` Sets `debug: true` or uses the `--debug` flag, keeping the browser window open for manual login. - **Interacting with Debug Browser**: When debug mode is enabled: 1. The browser window remains open 2. You can manually log into the website using your credentials 3. After login is complete, content will be fetched with your authenticated session - **Enable Debug for Specific Requests**: Even if the server is already running, you can enable debug mode for a specific request: ``` Please enable debug mode for this authentication step ``` Sets `debug: true` for this specific request only, opening the browser window for manual login. ## Development ### Install Dependencies ```bash npm install ``` ### Install Playwright Browser Install the browsers needed for Playwright: ```bash npm run install-browser ``` ### Build the Server ```bash npm run build ``` ## Debugging Use MCP Inspector for debugging: ```bash npm run inspector ``` You can also enable visible browser mode for debugging: ```bash node build/index.js --debug ``` ## Related Projects - [g-search-mcp](https://github.com/jae-jae/g-search-mcp): A powerful MCP server for Google search that enables parallel searching with multiple keywords simultaneously. Perfect for batch search operations and data collection. ## License Licensed under the [MIT License](https://choosealicense.com/licenses/mit/) [](https://dartnode.com "Powered by DartNode - Free VPS for Open Source")