Content
# mcp-web-search
An [MCP](https://modelcontextprotocol.io) server that gives an AI agent two tools: **search the web**, and **read a page**.
It does this without a headless browser. Search hits DuckDuckGo's no-JavaScript HTML endpoint and parses the server-rendered result list. Page fetching uses [curl-impersonate](https://github.com/lwthiker/curl-impersonate) so requests carry a real browser's TLS/HTTP2 fingerprint — which is what gets past Cloudflare/DataDome-style bot walls that a plain HTTP client trips — then extracts the article text with Readability. No API keys, no accounts, no browser install.
## Tools
| Tool | What it does |
|------|--------------|
| `search` | Takes a query and result count (1–25). Returns rank, title, URL, and snippet for each organic result. |
| `fetch_page` | Takes a URL. Returns the main article text with title/author/site/published metadata, boilerplate stripped. Falls back to a raw text strip if Readability can't find an article. |
The intended loop is `search` to find a result, then `fetch_page` on its URL to read it in full.
## Requirements
The [.NET 10 SDK](https://dotnet.microsoft.com/download/dotnet/10.0). That's it — everything else, including the curl-impersonate native binaries, comes from NuGet on first build.
Native binaries ship for **win-x64** and **linux-x64**. Other platforms (notably Apple Silicon) won't have a native to load, so `fetch_page` will fail there; `search` still works.
## Run it
```bash
git clone https://github.com/MrJackSpade/mcp-web-search.git
cd mcp-web-search
dotnet run --project src/WebSearchMcp
```
It listens on `http://127.0.0.1:5079/` — localhost only, no authentication. Verify it's up:
```bash
curl -X POST http://127.0.0.1:5079/ \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"test","version":"1"}}}'
```
You should get a `serverInfo` response naming `WebSearchMcp`.
If port 5079 is taken, change `Urls` in `src/WebSearchMcp/appsettings.json` or pass `--urls http://127.0.0.1:PORT`.
## Connect a client
**Claude Code** — point it at the running server:
```bash
claude mcp add --transport http websearch http://127.0.0.1:5079/
```
**Claude Desktop** doesn't connect to plain local HTTP servers directly; bridge it with [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) in `claude_desktop_config.json`:
```json
{
"mcpServers": {
"websearch": {
"command": "npx",
"args": ["-y", "mcp-remote", "http://127.0.0.1:5079/"]
}
}
}
```
Any other MCP client works too — it's a standard streamable-HTTP MCP server.
## Configuration
All of `src/WebSearchMcp/appsettings.json` is optional; the defaults are what's shown here.
```json
{
"Search": {
"Provider": "duckduckgo",
"Region": "us-en",
"MaxResults": 25,
"TimeoutSeconds": 20
},
"Fetch": {
"BrowserProfile": "Chrome136",
"MaxContentChars": 40000,
"TimeoutSeconds": 30
}
}
```
- **`Search:Region`** — DuckDuckGo region code, e.g. `us-en`, `uk-en`, `de-de`.
- **`Search:MaxResults`** — hard ceiling on what `search` will return, whatever the caller asks for.
- **`Fetch:BrowserProfile`** — which browser to impersonate. Defaults to `Chrome136`: as of Loxifi.CurlImpersonate 1.3.0 the *newer* profiles (Chrome146/145/142, Firefox147) ship stale fingerprints that get 403'd, so Chrome136 is the newest one that actually passes. Worth revisiting when the library refreshes them.
- **`Fetch:MaxContentChars`** — truncation cap on returned body text, to keep responses inside a model's context.
Every value can also come from an environment variable instead: `WEBSEARCH_PROVIDER`, `WEBSEARCH_REGION`, `WEBSEARCH_USER_AGENT`, `WEBSEARCH_FETCH_PROFILE`.
## Optional: authentication for remote hosting
The default config has no auth because it binds to localhost. If you expose this beyond your own machine, turn auth on by adding an `Mcp` section:
```json
{
"Urls": "http://0.0.0.0:5079",
"Mcp": {
"ClientId": "web-search-mcp",
"ClientSecret": "<a long random string>",
"SigningKey": "<a different long random string, 32+ chars>",
"PublicUrl": "https://websearch.example.com"
}
}
```
Auth switches on automatically when `ClientId`, `ClientSecret`, and `SigningKey` are all present. You then get an OAuth 2.0 authorization server (`/authorize`, `/token`, PKCE, discovery at `/.well-known/oauth-authorization-server`) and every MCP request requires a bearer token. `PublicUrl` must be the externally reachable URL, since it becomes the token issuer/audience.
Generate the secrets yourself and keep them out of git — the values are yours, not defaults to copy.
## Layout
```
src/
├── WebSearchMcp/ the server
│ ├── Search/ ISearchProvider + DuckDuckGo implementation
│ ├── Fetch/ PageReader (curl-impersonate + Readability)
│ └── Tools/ the two MCP tool definitions
└── Mcp.Server.Harness/ MCP-over-HTTP hosting + optional OAuth
```
### Adding a search engine
`ISearchProvider` is the seam. Implement it, add one arm to the switch in `Program.cs:CreateProvider`, and select it with `Search:Provider`. Nothing else changes.
Google is deliberately absent: its no-JavaScript path was removed in January 2025 when Search began requiring JS, which rules out the parse-the-HTML approach this server is built on.
## License
MIT — see [LICENSE](LICENSE).
