Content
# mcp-web-tools
MCP server providing web search and content extraction tools for AI assistants. **Works out of the box with zero config** — no API keys, no Docker. Search and extract backends are decoupled and individually swappable between [SearXNG](https://searxng.org) (public or self-hosted), [Tavily](https://tavily.com) (hosted), and [Firecrawl](https://firecrawl.dev) (keyless or self-hosted).
## Install
No API keys, no Docker, no config files. Just add the server — it works out of the box using a public SearXNG instance for search and Firecrawl keyless for extraction.
**Claude Code (CLI):**
```bash
claude mcp add mcp-web-tools -- npx -y mcp-web-tools
```
**Claude Desktop** — add to `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows):
```json
{
"mcpServers": {
"mcp-web-tools": {
"command": "npx",
"args": ["-y", "mcp-web-tools"]
}
}
}
```
**OpenCode** — add to `~/.config/opencode/opencode.json` (global) or `.opencode.json` in your project root:
```jsonc
"mcp-web-tools": {
"type": "local",
"command": ["npx", "-y", "mcp-web-tools"]
}
```
That's it — search and extraction both work with zero configuration. The sections below cover customizing the backends.
## How it works
The server has two independent backends, each overridable via env vars in your client config:
- **Search** (`WEBTOOLS_SEARCH_PROVIDER`): defaults to `searxng` — uses a [public SearXNG instance](https://search.mdosch.de). Override with `tavily`, or point `SEARXNG_URL` at your own instance.
- **Extract** (`WEBTOOLS_EXTRACT_PROVIDER`): defaults to `firecrawl` keyless — no API key needed. Override with `tavily`, or set to `none` to disable `web_read`.
> ⚠️ The default public SearXNG instance is volunteer-operated. If it's down or rate-limited, set `SEARXNG_URL` to another instance or [run your own](#self-hosting-searxng-optional).
## Customizing backends
### Use a different SearXNG instance
Point `SEARXNG_URL` at any SearXNG instance — public, self-hosted, or team-shared:
```bash
claude mcp add mcp-web-tools -e SEARXNG_URL=https://your-instance.example -- npx -y mcp-web-tools
```
Find public instances at [searx.space](https://searx.space/).
### Use Tavily instead (single key, both tools)
```bash
claude mcp add mcp-web-tools -e WEBTOOLS_SEARCH_PROVIDER=tavily -e TAVILY_API_KEY=your-key -- npx -y mcp-web-tools
```
Get a key at [tavily.com](https://tavily.com). Extract automatically uses Tavily too (key reuse).
### Firecrawl options
The default Firecrawl keyless tier gives ~1000 extractions/month. For higher limits:
```bash
claude mcp add mcp-web-tools -e FIRECRAWL_API_KEY=fc-your-key -- npx -y mcp-web-tools
```
### Provider combinations
| Search | Extract | Env vars |
|---|---|---|
| SearXNG (default) | Firecrawl keyless (default) | *none* |
| SearXNG | Firecrawl keyed | `FIRECRAWL_API_KEY` |
| SearXNG | Tavily | `WEBTOOLS_EXTRACT_PROVIDER=tavily` + `TAVILY_API_KEY` |
| SearXNG | none (`web_read` disabled) | `WEBTOOLS_EXTRACT_PROVIDER=none` |
| Tavily | Tavily | `WEBTOOLS_SEARCH_PROVIDER=tavily` + `TAVILY_API_KEY` |
More `claude mcp add` one-liners:
```bash
# Point at your own SearXNG
claude mcp add mcp-web-tools -e SEARXNG_URL=http://localhost:8080 -- npx -y mcp-web-tools
# SearXNG search + Tavily extract
claude mcp add mcp-web-tools -e WEBTOOLS_EXTRACT_PROVIDER=tavily -e TAVILY_API_KEY=your-key -- npx -y mcp-web-tools
# Search only, web_read disabled
claude mcp add mcp-web-tools -e WEBTOOLS_EXTRACT_PROVIDER=none -- npx -y mcp-web-tools
# Firecrawl keyed (higher limits)
claude mcp add mcp-web-tools -e FIRECRAWL_API_KEY=fc-your-key -- npx -y mcp-web-tools
```
## Self-hosting SearXNG (optional)
For full control over availability and rate limits, run your own SearXNG instance and point `SEARXNG_URL` at it.
The key requirement: SearXNG must have **JSON output enabled**. The default Docker image only serves HTML, so you need a `settings.yml` with `json` in `search.formats`. See the official docs:
→ [SearXNG — Installation & Docker](https://docs.searxng.org/admin/installation-docker.html)
→ [SearXNG — settings.yml reference](https://docs.searxng.org/admin/settings/settings.html)
Minimal `settings.yml` (mount this into the container):
```yaml
use_default_settings: true
server:
secret_key: "your-random-secret"
search:
formats: [html, json]
```
Then point the server at it:
```bash
claude mcp add mcp-web-tools -e SEARXNG_URL=http://localhost:8080 -- npx -y mcp-web-tools
```
## Full self-host (optional)
Run everything with no external network calls: SearXNG for search, Firecrawl for extract, both on your machine.
**1. SearXNG** — see [Self-hosting SearXNG](#self-hosting-searxng-optional) above.
**2. Firecrawl** — use Firecrawl's official self-host guide. Their stack needs Postgres, RabbitMQ, Redis, and Playwright:
→ [firecrawl/firecrawl — SELF_HOST.md](https://github.com/firecrawl/firecrawl/blob/main/SELF_HOST.md)
**3. Point mcp-web-tools at both:**
```bash
claude mcp add mcp-web-tools -e SEARXNG_URL=http://localhost:8080 -e FIRECRAWL_URL=http://localhost:3002 -- npx -y mcp-web-tools
```
## Tools
### `web_search`
Search the web. Returns results with titles, URLs, and snippets. Backend is configured server-side (Tavily or SearXNG).
**Parameters:**
| Parameter | Type | Default | Description |
|---|---|---|---|
| `query` | string | *required* | The search query |
| `maxResults` | number | 5 | Maximum number of results (1–20) |
| `searchDepth` | string | `"basic"` | `"basic"`, `"fast"`, `"ultra-fast"` (1 credit) or `"advanced"` (2 credits) |
| `topic` | string | `"general"` | `"general"`, `"news"`, or `"finance"` |
| `timeRange` | string | — | `"day"`, `"week"`, `"month"`, or `"year"` |
| `startDate` | string | — | Start date filter (`YYYY-MM-DD`) |
| `endDate` | string | — | End date filter (`YYYY-MM-DD`) |
| `includeDomains` | string[] | — | Only include results from these domains |
| `excludeDomains` | string[] | — | Exclude results from these domains |
| `includeAnswer` | boolean | `true` | Include an AI-generated answer summary |
### `web_read`
Extract clean content from web pages. Returns page text stripped of navigation, ads, and scripts. Supports up to 20 URLs per request. Backend is configured server-side (Tavily or Firecrawl). Not registered when `WEBTOOLS_EXTRACT_PROVIDER=none`.
**Parameters:**
| Parameter | Type | Default | Description |
|---|---|---|---|
| `urls` | string[] | *required* | URLs to extract content from (1–20). Must be `http://` or `https://` |
| `extractDepth` | string | — | `"basic"` (1 credit per 5 URLs, raw page HTML as markdown including navigation/boilerplate) or `"advanced"` (2 credits per 5 URLs, clean article content with nav/ads/sidebar/footer stripped) |
| `includeImages` | boolean | `false` | Include extracted image URLs |
### `credit_balance`
Check your provider credit balance and usage. Returns "not supported" when the active provider has no credit system (e.g. SearXNG).
## GitHub Tools
Three tools for reading public GitHub repository content directly via the GitHub REST API. These do not go through the search provider — they live in their own namespace and work anonymously.
### `github_get_repo_structure`
List the directory structure of a GitHub repository path (single level, non-recursive). Useful for understanding project layout.
**Parameters:**
| Parameter | Type | Default | Description |
|---|---|---|---|
| `repo` | string | *required* | Repository in the form `owner/repo` (e.g. `vitejs/vite`) |
| `path` | string | `/` | Directory path inside the repo |
| `ref` | string | default branch | Git ref (branch, tag, or commit SHA) |
### `github_read_file`
Read the full content of a file in a GitHub repository. Files larger than 512 KB or detected as binary are rejected.
**Parameters:**
| Parameter | Type | Default | Description |
|---|---|---|---|
| `repo` | string | *required* | Repository in the form `owner/repo` |
| `path` | string | *required* | Path to the file inside the repo |
| `ref` | string | default branch | Git ref (branch, tag, or commit SHA) |
### `github_search_repo`
Search a GitHub repository across code, issues, and pull requests in one call. When `kind=all` (the default), all three searches run in parallel and results are grouped by type. Code search uses GitHub's `/search/code` endpoint, which **requires a `GITHUB_TOKEN`** (anonymous code search is not supported by GitHub — the call returns 401 without a token). Issues and PRs use `/search/issues` and work anonymously.
**Parameters:**
| Parameter | Type | Default | Description |
|---|---|---|---|
| `repo` | string | *required* | Repository in the form `owner/repo` |
| `query` | string | *required* | Search query. GitHub search syntax is supported (e.g. `"auth token"`, `"bug in login"`, `"label:bug"`) |
| `kind` | string | `"all"` | `"code"`, `"issues"`, `"prs"`, or `"all"` (fans out across all three in parallel) |
| `state` | string | `"all"` | Filter for issues/PRs only (`"open"`, `"closed"`, or `"all"`). Ignored for code |
| `maxResults` | number | `5` | Max results per kind (1–10). When `kind=all`, the tool returns up to 3× this number of results |
## Environment Variables
| Variable | Required | Default | Description |
|---|---|---|---|
| `WEBTOOLS_SEARCH_PROVIDER` | No | `searxng` | Search backend: `tavily` or `searxng` |
| `SEARXNG_URL` | No | `https://search.mdosch.de` | Base URL of a SearXNG instance. Defaults to a public instance; set to your own for reliability |
| `TAVILY_API_KEY` | Conditional | — | Tavily API key. Required when search or extract provider is `tavily`. Get one at [tavily.com](https://tavily.com) |
| `WEBTOOLS_EXTRACT_PROVIDER` | No | derived\* | Extract backend: `tavily`, `firecrawl`, or `none`. When unset: `tavily` if search is tavily, otherwise `firecrawl` |
| `FIRECRAWL_URL` | No | `https://api.firecrawl.dev` | Base URL of a Firecrawl instance. Accepts any HTTP(S) URL (local, hosted, or self-hosted) |
| `FIRECRAWL_API_KEY` | No | — | Optional. Omit for keyless mode (~1000 credits/month, IP-bucketed). Set for higher hosted limits. Ignored on self-hosted bypass mode |
| `GITHUB_TOKEN` | No | — | Optional GitHub token for the `github_*` tools. Lifts rate limits from 60 to 5000 requests/hour. Create one at [github.com/settings/tokens](https://github.com/settings/tokens) (no scopes needed for public repos) |
| `WEBTOOLS_MAX_RESULTS` | No | `5` | Default number of results (must be 1–20) |
| `WEBTOOLS_SEARCH_DEPTH` | No | `basic` | Default search depth |
| `WEBTOOLS_CACHE_TTL` | No | `3600` | Cache TTL in seconds (0 disables) |
| `WEBTOOLS_CACHE_MAX_ENTRIES` | No | `100` | Maximum cache entries before LRU eviction (0 disables caching) |
| `WEBTOOLS_CACHE_SWEEP_INTERVAL_MS` | No | `300000` | Interval in ms to sweep expired entries (0 disables periodic sweep) |
| `WEBTOOLS_DEBUG` | No | — | Set to any value to enable debug logging to stderr |
\* `WEBTOOLS_EXTRACT_PROVIDER` defaults to `tavily` when `WEBTOOLS_SEARCH_PROVIDER=tavily`, and to `firecrawl` otherwise.
## Provider limitations
Each backend supports a different subset of features. Unsupported params are silently ignored.
| Feature | Tavily | SearXNG | Firecrawl |
|---|---|---|---|
| `web_search` | ✅ | ✅ | — (extract only) |
| `web_read` / extract | ✅ | — (search only) | ✅ |
| AI answer (`includeAnswer`) | ✅ | ❌ ignored | — |
| Native domain filter | ✅ | post-filtered (may return fewer than `maxResults`) | — |
| `topic=finance` | ✅ | maps to `general` | — |
| `searchDepth` | ✅ | ❌ ignored | — |
| `extractDepth=advanced` | ✅ | — | ✅ (maps to `onlyMainContent`) |
| `includeImages` | ✅ | — | ❌ ignored |
| Credit/usage (`credit_balance`) | ✅ | ❌ "not supported" | ❌ "not supported" |
**Common pitfalls:**
- **SearXNG 403** — JSON output is disabled on the instance. Edit `settings.yml` (`search.formats` must include `json`) and restart. The `docker run` snippet in [Backends](#searxng-search-default) bakes this in via the mounted settings file.
- **SearXNG 429** — rate limiter is on. Disable `server.limiter` in `settings.yml`, or use a different instance.
- **Firecrawl 429** — keyless free tier exhausted (~1000 credits/month, 10/min). Set `FIRECRAWL_API_KEY` for higher limits, or self-host.
## Example Output
```
Node.js 22 introduces require() support for ES modules, a WebSocket client, and updates to the V8 JavaScript engine.
### 1. [Node.js — Node.js 22 is now available!](https://nodejs.org/blog/announcements/v22-release-announce)
> We're excited to announce the release of Node.js 22! Highlights include require()ing ES modules, a WebSocket client, updates of the V8 JavaScript engine, and more!
*Response time: 1.2s*
---
Sources:
- [Node.js — Node.js 22 is now available!](https://nodejs.org/blog/announcements/v22-release-announce)
```
## Security note
This server is designed for trusted AI clients (Claude Code, Claude Desktop, OpenCode). The `web_read` tool accepts any `http(s)://` URL supplied by the calling client and does not implement SSRF protections (e.g. blocking of private/internal IP ranges). Do not expose this server to untrusted input.
## License
MIT
MCP Config
Below is the configuration for this MCP Server. You can copy it directly to Cursor or other MCP clients.
mcp.json
Connection Info
You Might Also Like
No recommendations available