webcrawl-mcp

# webcrawl-mcp [](https://pypi.org/project/webcrawl-mcp/) [](https://pypi.org/project/webcrawl-mcp/) [](LICENSE) A lightweight MCP server that gives Claude Code (or any MCP client) the ability to scrape, search, map, and crawl the web — using free, open-source libraries. Firecrawl is supported as an **optional** fallback for JS-heavy sites when you have a key. ## Why Most scraping doesn't actually need a headless browser. `trafilatura` handles the ~80% case (articles, docs, blogs) locally, which is faster and keeps external API usage to a minimum. This server routes the easy stuff through local extraction and only falls back to Firecrawl when content quality is genuinely poor. ## Tools | Tool | Purpose | |------|---------| | `webcrawl_scrape` | Fetch a single URL → `{content, source}` (markdown + provenance) | | `webcrawl_search` | DuckDuckGo search (optionally scrape results, each with provenance) | | `webcrawl_map` | Discover same-domain URLs from a starting page | | `webcrawl_crawl` | BFS crawl multiple pages (each result includes provenance) | The `source` field on scraped content is one of `static_http`, `static_http_retry`, `firecrawl_transport_fallback`, or `firecrawl_quality_fallback` — see [Fallback behavior](#fallback-behavior). ## Install ```bash pip install webcrawl-mcp ``` Requires Python 3.12+. Quick smoke test (should print `Webcrawl MCP server running` then exit cleanly with Ctrl-C): ```bash webcrawl-mcp ``` ### Install from source (for development) ```bash git clone https://github.com/andyliszewski/webcrawl-mcp.git cd webcrawl-mcp python -m venv venv source venv/bin/activate # Windows: venv\Scripts\activate pip install -e . ``` ## Configure your MCP client ### Claude Code Create `.mcp.json` in your project root (or merge into `~/.claude/settings.json`): ```json { "mcpServers": { "webcrawl": { "command": "uvx", "args": ["webcrawl-mcp"] } } } ``` This uses [`uvx`](https://docs.astral.sh/uv/guides/tools/) to run the package in a temporary environment — no manual install needed. If `uvx` is unavailable, install via `pip install webcrawl-mcp` and use `"command": "webcrawl-mcp"` with no args instead. For a source checkout (development), point `command` at your venv's Python and use `"args": ["-m", "webcrawl_mcp"]`. Then in a Claude Code session run `/mcp` — you should see `webcrawl` listed with four tools. ### Claude Desktop Same JSON shape, placed in `claude_desktop_config.json` (see [Anthropic's docs](https://modelcontextprotocol.io/quickstart/user) for the OS-specific path). Restart Claude Desktop after editing. ### Other MCP clients (Cursor, Cline, Continue, Zed, etc.) The `command` / `args` / `env` shape is standardized. Consult your client's MCP docs for where to put it. ## Verify it's working Ask your agent something like: > Use the `webcrawl_scrape` tool to fetch `https://docs.python.org/3/library/asyncio.html` and summarize the first section. You should see the tool invocation in the client UI, followed by a summary grounded in the live page content. If nothing happens, see [Troubleshooting](#troubleshooting). ## Environment variables All optional: | Variable | Default | Purpose | |----------|---------|---------| | `USER_AGENT` | `Mozilla/5.0 (compatible; WebcrawlMCP/1.0; …)` | HTTP User-Agent | | `REQUEST_TIMEOUT` | `30` | Seconds before request timeout | | `FIRECRAWL_API_KEY` | *(unset)* | If set, enables Firecrawl fallback for low-quality extractions. **Leave unset for a fully free setup.** | | `FIRECRAWL_API_URL` | `https://api.firecrawl.dev/v1` | Firecrawl endpoint | | `FALLBACK_ON_TRANSPORT_ERROR` | `false` | If `true` **and** `FIRECRAWL_API_KEY` is set, route bot-blocked statuses (`403`, `429`, `503`) to Firecrawl instead of raising. Opt-in. | | `POLITE_MODE` | `true` | On a `429` with a parseable `Retry-After`, retry the original request once after the indicated wait (capped at `REQUEST_TIMEOUT`) before falling through. | Set these inside the `env` block of your MCP config, not in your shell — MCP servers run under the client, not your terminal. ## Fallback behavior The scraper distinguishes **extraction-quality** failure from **transport** failure and reports which path produced the content via the `source` field. **Extraction-quality path (default):** 1. `trafilatura` extracts main content from HTML → `source: static_http`. 2. If that fails or returns <200 chars, `markdownify` converts the raw HTML. 3. If the result is still low-quality **and** `FIRECRAWL_API_KEY` is set, Firecrawl is used as a last resort → `source: firecrawl_quality_fallback`. **Transport path (opt-in):** If a request returns `403`, `429`, or `503` (typical bot-blocking responses): - With `POLITE_MODE=true` (default), a `429` carrying a `Retry-After` header gets one bounded retry of the original request → `source: static_http_retry` on success. - With `FALLBACK_ON_TRANSPORT_ERROR=true` and `FIRECRAWL_API_KEY` set, the request routes to Firecrawl instead of raising → `source: firecrawl_transport_fallback`. - Otherwise, the transport error is raised to the caller (current behavior). Without a Firecrawl key, the tool is fully self-contained and free; `FALLBACK_ON_TRANSPORT_ERROR` is a no-op without a key. ## Troubleshooting **Client doesn't list `webcrawl` under MCP servers.** The `command` path is almost always the problem. It must be an absolute path to the Python binary *inside* your venv, not just `python`. Test it in a terminal: `/path/you/configured -m webcrawl_mcp` should print `Webcrawl MCP server running`. **`ModuleNotFoundError: webcrawl_mcp`.** Either `pip install -e .` didn't run in the same venv as `command`, or `PYTHONPATH` is missing/wrong. Double-check both point at the same checkout. **Python version mismatch.** Requires 3.12+. `python --version` inside your venv should report ≥ 3.12. If not, recreate the venv with a newer Python. **Scrapes return very little text.** Some sites render with JavaScript and can't be extracted statically. Either set `FIRECRAWL_API_KEY` to enable the fallback path, or accept that this tool isn't the right fit for that specific site. **Search is slow or rate-limited.** DuckDuckGo throttles bursty querying. Space out searches, or reduce `num_results`. ## Responsible use This tool is for fetching public web content for research, coding assistance, and similar legitimate uses. You are responsible for: - Respecting each target site's Terms of Service and `robots.txt`. - Not overloading servers — the built-in per-domain rate limiter helps, but don't circumvent it. - Complying with applicable laws around automated access and data use in your jurisdiction. ## License MIT — see [LICENSE](LICENSE). ## Acknowledgements Built on top of [`trafilatura`](https://github.com/adbar/trafilatura), [`httpx`](https://www.python-httpx.org/), [`beautifulsoup4`](https://www.crummy.com/software/BeautifulSoup/), [`markdownify`](https://github.com/matthewwithanm/python-markdownify), [`ddgs`](https://github.com/deedy5/ddgs), and [`fastmcp`](https://github.com/jlowin/fastmcp). Firecrawl integration uses the public [Firecrawl API](https://firecrawl.dev) (not affiliated).