Content
# websearch-gw
A self-hosted, zero-cost web-search gateway for coding agents (Pi, Claude Code) over MCP
streamable-http. SearXNG (search) + Crawl4AI (page reading) + Valkey (cache) behind one gateway
process that exposes exactly two tools: `web_search` and `web_read`. No API keys, no paid tiers,
no hosted fallback of any kind. See `PLAN.md` for the full design rationale and build history.
```
Claude Code ──┐ ┌── SearXNG :8080 search only
├──► websearch-gw :8090/mcp ──┼── Crawl4AI :11235 reading only
Pi agents ────┘ (streamable-http) └── Valkey :6379 cache only
```
## Prerequisites
- Docker (or OrbStack as the Docker backend — this project was built and tested against
OrbStack). If `docker` isn't on `PATH` in a fresh shell:
```bash
export PATH="/usr/local/bin:/opt/homebrew/bin:$HOME/.orbstack/bin:$PATH"
```
`docker version` should show `Context: orbstack` (or whichever backend you're using).
- Python 3.11+ (only needed for running the gateway's test suite locally, not for running
the stack itself — that's fully containerized).
## First-time setup
```bash
git clone <this-repo> websearch-gw && cd websearch-gw
./scripts/gen-secret.sh # generates searxng/settings.yml
cp .env.example .env
sed -i '' "s/^CRAWL4AI_API_TOKEN=$/CRAWL4AI_API_TOKEN=$(openssl rand -hex 32)/" .env
docker compose up -d --build
docker compose ps # all four should report healthy within ~30s
curl -s localhost:8090/health # {"status":"ok", ...}
```
**Why `.env` is required, not optional:** Crawl4AI's image has a deliberate security default —
with no `CRAWL4AI_API_TOKEN` set, its server binds to the *container's own* loopback and refuses
to expose itself at all, even to other containers on the same Docker network. Setting the token
both fixes the bind and switches Crawl4AI into Bearer-authenticated mode; the gateway holds this
token and agents never see it.
## Wiring up clients
**Pi** — install the MCP bridge extension and point it at the gateway:
```bash
pi install npm:pi-mcp-extension
```
`~/.pi/agent/mcp.json`:
```json
{
"settings": { "toolPrefix": "mcp" },
"mcpServers": {
"websearch-gw": {
"transport": "streamable-http",
"url": "http://127.0.0.1:8090/mcp",
"lifecycle": "eager"
}
}
}
```
**Claude Code**:
```bash
claude mcp add --transport http websearch-gw http://127.0.0.1:8090/mcp
```
## Tools
- **`web_search(query, depth, count, freshness)`** — `depth="links"` (default) returns raw
SearXNG results. `depth="snippets"`/`"full"` over-fetch a candidate pool and rerank it with
FlashRank (a free local cross-encoder) before trimming to `count`; `"full"` additionally
fetches page content for the top few reranked results.
- **`web_read(url, size, query, page)`** — reads a URL as clean markdown. `size` is `s`/`m`/`l`
(truncated) or `full` (paginated at 25K-char boundaries). `query`, if given, activates
query-relevance filtering instead of generic boilerplate removal. Results are cached in Valkey.
Fetches go through a tiered fallback (direct-API bypass → Crawl4AI → raw HTTP → Wayback
Machine) so one blocked or JS-heavy page doesn't just fail outright.
Both tools reject unsafe URLs (loopback/private/link-local IPs, a small hostname blocklist)
before any network call.
## Repo layout
| Path | What |
|---|---|
| `docker-compose.yml` | The 4-container stack — all ports bound to `127.0.0.1` only |
| `searxng/` | SearXNG config (`settings.yml.template` is committed; `settings.yml` is generated, gitignored) |
| `gateway/` | The MCP gateway itself — see `gateway/README.md` for its internals and test suite |
| `scripts/gen-secret.sh` | Generates `searxng/settings.yml` with a fresh secret key |
| `PLAN.md` | Design doc: architecture rationale, prior-art research, build order, open risks |
## Verify the stack
```bash
curl -s 'http://localhost:8080/search?q=test&format=json' | head -c 300 # SearXNG JSON API
curl -s http://localhost:11235/health # Crawl4AI liveness
docker exec websearch-valkey valkey-cli ping # -> PONG
curl -s localhost:8090/health # gateway + all 3 dependencies
```
## Run the gateway's test suite
```bash
cd gateway
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest -q -m "not live_stack" # deterministic subset, no running stack required
pytest -q # everything, incl. live smoke tests against the running stack
```
See `gateway/README.md` for what each test file covers.
## What's not built
Lockdown (making this the *only* path agents have to the web — blocking `bash curl`/`wget` in Pi,
denying `WebSearch`/`WebFetch` in Claude Code) and optional query-expansion are deliberately
unbuilt. See PLAN.md §7 for why lockdown is last in the build order.
