Content
# myscrape
[](https://github.com/Bonifatius94/myscrape/actions/workflows/ci.yml)
A self-contained, single-codebase **web-research MCP server** for local LLM agents.
It mimics the spirit of Claude's WebSearch / WebFetch and adds a cooked
`web_research` tool that runs the whole search → fetch → synthesize loop with a
local LLM inside the box.
📖 **[Full user guide → `docs/USER_GUIDE.md`](docs/USER_GUIDE.md)** — setup,
running (local/Docker), the tools, configuration, and troubleshooting.
## Tools
| Tool | Altitude | LLM inside? |
|----------------|----------|-------------|
| `web_search` | raw | no |
| `web_fetch` | raw | no |
| `web_research` | cooked | yes |
See [`specs/SPEC.md`](specs/SPEC.md) for the interface contract, and the design
docs: [`specs/IMPLEMENTATION_OPTIONS.md`](specs/IMPLEMENTATION_OPTIONS.md),
[`specs/TECH_STACK.md`](specs/TECH_STACK.md),
[`specs/BENCHMARK.md`](specs/BENCHMARK.md). Decisions and checkpoints are logged
in [`specs/DEVLOG.md`](specs/DEVLOG.md); empirical work in
[`specs/EXPERIMENTS.md`](specs/EXPERIMENTS.md); search-provider options in
[`specs/PROVIDERS.md`](specs/PROVIDERS.md).
## Tooling
Stable tools only — no type checker; we rely on tests + linting instead.
- **uv** — env, deps, locking, running
- **Ruff** — lint + format
- **pytest** — tests (TDD: red → green → refactor)
## Quickstart
```bash
uv sync # create venv, install deps
uv run pytest # run the test suite
uv run ruff check . && uv run ruff format --check . # lint + format gate
uv run bench # run the stability benchmark (any time)
```
## Running the MCP server
```bash
uv run myscrape # stdio transport — a local MCP client spawns this
```
`web_research` needs a local, OpenAI-compatible LLM endpoint (Ollama / llama.cpp /
LM Studio). Configure via `MYSCRAPE_*` env vars (see `src/myscrape/config.py`):
```bash
export MYSCRAPE_LLM_BASE_URL=http://localhost:11434/v1
export MYSCRAPE_LLM_MODEL=qwen2.5:14b # eval winner on a 12GB GPU (see specs/EXPERIMENTS.md)
```
`web_search` and `web_fetch` need no LLM. The proven stability operating point
(8s + 2s jitter) is the default; override with `MYSCRAPE_REQUEST_MIN_INTERVAL`.
### On a server (Docker)
Uses the **host's** native Ollama (GPU) — see [`specs/SERVING.md`](specs/SERVING.md).
```bash
ollama pull qwen2.5:14b # on the host (the eval winner)
docker compose up --build # myscrape on :8000, talks to host Ollama
```
The container is one process, includes Chromium for dynamic fetch, speaks
`streamable-http` at `http://localhost:8000/mcp`, and caps concurrent research
at 2 (the GPU serializes synthesis — see EXPERIMENTS E-007). For a self-contained
Ollama-in-container setup with GPU passthrough: `docker compose --profile ollama up`.
## Use it from a coding session (Claude Code)
A project [`.mcp.json`](.mcp.json) points at the running server:
```bash
claude mcp add --transport http myscrape http://localhost:8000/mcp # or use .mcp.json
```
Then `web_search`, `web_fetch`, and `web_research` are available as tools in the
session. (For local use without Docker, run `uv run myscrape` over stdio instead.)
## Tools
| Tool | Input (key fields) | Returns |
|------|--------------------|---------|
| `web_search` | `query`, `max_results` | ranked results (no fetch, no LLM) |
| `web_fetch` | `url`, `max_tokens` | clean markdown + metadata (no LLM) |
| `web_research` | `question`, `effort`, `return_mode` | cited answer + sources + coverage |
## Status
Implemented end-to-end: all three tools, the full search → fetch → rank →
synthesize loop, behind a stability gate that **passes** (`ratelimit_rate == 0`,
extraction 100% over fetchable pages). Static **and** dynamic (Playwright) fetch
work — `web_fetch` auto-escalates to a headless browser for JS-rendered pages
(live-validated on `quotes.toscrape.com/js`). See [`specs/DEVLOG.md`](specs/DEVLOG.md)
for the full build log and [`specs/EXPERIMENTS.md`](specs/EXPERIMENTS.md) for the
empirical work.
