webify-mcp-rust

# Webify (Rust) > Adaptive web research for AI coding agents 91% of Deep Research quality · 5% of the cost · Single static binary · Works as MCP server or CLI. [](LICENSE) [](https://rustup.rs/) Rust port of [webify-mcp](https://github.com/kunal12203/webify-mcp) --- ## Install ```bash cargo install --git https://github.com/bruj0/webify-mcp-rust webify-mcp ``` This installs both the `webify-mcp` binary (MCP stdio server) and the `webify` CLI. ### Use with an MCP client ```bash claude mcp add webify -- webify-mcp ``` Works in Claude Code, Cursor, VS Code, Windsurf, Zed, and any other MCP client. ### Use with Visual Studio Code VS Code reads MCP servers from `~/.config/Code/User/mcp.json` (Linux) / `%APPDATA%\Code\User\mcp.json` (Windows) / `~/Library/Application Support/Code/User/mcp.json` (macOS). Workspace-scoped servers also work via `.vscode/mcp.json` in any project. **Workspace scope** (recommended, per-project): create `.vscode/mcp.json` at the root of any project where you want webify available: ```json { "servers": { "webify-mcp": { "type": "stdio", "command": "webify-mcp", "env": { "WEBIFY_CACHE_DIR": "${HOME}/.cache/webify", "ANTHROPIC_API_KEY": "${env:ANTHROPIC_API_KEY}", "BRAVE_SEARCH_API_KEY": "${env:BRAVE_SEARCH_API_KEY}" } } } } ``` **User scope** (one config, every workspace): add the same `webify-mcp` block to `~/.config/Code/User/mcp.json` under the existing `"servers"` key. After saving the file, click the **Start** button that appears next to the server in the MCP panel, or reload the window (`Ctrl+Shift+P` → *Developer: Reload Window*). The four tools (`web_find`, `web_lookup`, `web_build`, `web_stats`) will then be available in Copilot Chat's **Agent** mode — Copilot will route research requests through `web_find` per the policy sent by the server itself. If your shell exports the API keys, you can also install via the CLI: ```bash code --add-mcp '{"name":"webify-mcp","type":"stdio","command":"webify-mcp"}' ``` --- ## What it does ```mermaid flowchart LR A[Query] --> B[Search\nBrave / DDG] B --> C1[Page 1] B --> C2[Page 2] C1 --> D[DOM Graph\n+ BM25] C2 --> D D --> E[Multi-aspect\nextraction] E --> F[Haiku\nsynthesis] F --> G["Answer\n(~800 tokens)"] ``` --- ## MCP tools | Tool | Args | Description | | -------------------- | ----------------------------- | -------------------------------------------------------------------------------------------- | | `web_find(query)` | query, num_sources?, max_per_source?, synthesize? | Multi-source research with LinUCB query reformulation and citation chasing | | `web_lookup(url, query)` | url, query | Structural sub-tree retrieval from a graph cached at `~/.cache/webify/<url_hash>.json` | | `web_build(url, force_refresh?)` | url | Force-rebuild the graph for `url` | | `web_stats(url)` | url | Graph statistics (nodes, edges, compression ratio, confidence) | Default `synthesize=true` requires `ANTHROPIC_API_KEY`. When unset or on API error, the server returns pre-synthesis fragments unchanged. ### Environment | Variable | Effect | | ----------------------- | ------------------------------------------------------------- | | `ANTHROPIC_API_KEY` | Enable Haiku synthesis in `web_find` | | `BRAVE_SEARCH_API_KEY` | Use Brave Search; falls back to DDG when unset | | `WEBIFY_CACHE_DIR` | Override cache directory (default `~/.cache/webify`) | | `RUST_LOG` | Tracing filter (default `warn`) | | `HTTP_PROXY` / `HTTPS_PROXY` | Forwarded to reqwest | --- ## CLI ```bash webify build <url> [--force] # Build graph webify lookup <url> <query> # Retrieve via graph webify stats <url> # Graph statistics (JSON) webify find <query> [--no-synthesize] webify search <query> [--max N] ``` --- ## Library ```rust use webify::{graph, retrieve, orchestrate, WebFindOptions}; // 1. Build a graph (cached at ~/.cache/webify/<hash>.json) let g = graph::build_graph("https://example.com/article", false).await?; // 2. Retrieve the structural sub-tree relevant to your query let result = retrieve::lookup("https://example.com/article", "memory safety", 10).await?; // 3. Or run a multi-source research synthesis let r = orchestrate::web_find("how does Rust ensure memory safety", WebFindOptions { synthesize: true, num_sources: None, max_results_per_source: None }).await; println!("{}", r.content); ``` --- ## Performance vs Python | Metric | Python (v0.7.x) | Rust (v0.1.0) | | ----------------- | -------------------- | -------------------------- | | Cold graph build | 1.8–3.2 s / page | ~0.6–1.1 s / page | | Hot graph lookup | 30–90 ms | <5 ms | | Memory (idle) | ~80 MB | ~6 MB | | Binary size | n/a (interpreted) | 11 MB stripped | | Startup | ~280 ms | <10 ms | | Tests | pytest, ~50 cases | cargo test, 156 cases | Roughly 3× faster on graph build, 10× faster on retrieval, ~10× lower idle memory. --- ## Architecture | Module | Responsibility | | ----------------------- | --------------------------------------------------------------------- | | `config` | Constants (paths, caps, bandit hyper-params) and env var lookups | | `error` | Typed errors (`HttpStatus`, `Fetch`, `CacheRead`, `NoSearchResults`) | | `entities` | Content hashing, token estimation, slugification | | `fetch::http` | `reqwest` GET with browser UA, gzip, timeout | | `fetch::fallback` | OpenAPI → raw source → Wayback → Google cache cascade | | `extract::markdown` | `_text_to_html`: headings, fenced code, lists | | `extract::embedded` | `__NEXT_DATA__`, JSON-LD (incl. FAQ Q&A), Nuxt data mining | | `extract` (root) | Readability scoring — DOM-based main-content extraction | | `graph::sections` | Markdown → Section tree + ContentBlock list | | `graph::nodes` | GraphNode / GraphEdge data types | | `graph::confidence` | Multi-signal heuristic (nav shells, thin content, SPA templates) | | `graph::meta` | `<meta>` extraction + citation URL mining | | `graph` (root) | Orchestrator: fetch → select → parse → cache | | `retrieve::bm25` | Per-arm IDF, BM25 score with type bonuses + nav-node gating | | `retrieve::subtree` | Adjacency BFS for structural sub-tree collection | | `retrieve` (root) | `retrieve`, `lookup`, `smart_lookup`, `retrieve_from_graph` | | `search::brave` | Brave Search API client (only used when key is set) | | `search::ddg` | DDG lite fallback (form POST → HTML parse) | | `search::primary` | Primary-source citation URL mining (DOI / PubMed / arXiv / .edu / …) | | `search` (root) | `search_web` (Brave → DDG cascade) | | `ml::bandit` | LinUCB (Sherman-Morrison rank-1 update, deterministic trigram proj.) | | `ml::domain` | Welford online mean/M2 per domain + UCB bonus | | `ml` (root) | On-disk `ml_state.json` (A⁻¹, b, n; domain stats; total_pulls) | | `synthesize::query` | Query complexity scoring + aspect decomposition | | `synthesize` (root) | Haiku synthesis with graceful fragment fallback | | `orchestrate` | `web_find` orchestration: search → parallel builds → merge → synth | | `main` | `webify-mcp` binary: MCP stdio server | | `bin/webify_cli` | `webify` binary: clap-based command-line front-end | All persistent state lives under `~/.cache/webify/`: - `<url_hash>.json` per URL graph (TTL 24 h) - `ml_state.json` LinUCB arm + per-domain statistics --- ## Build ```bash cargo build --release # Build gate (warnings as errors): cargo clippy --all-targets -- -W clippy::all -D warnings cargo test --lib ``` The default profile already uses LTO + strip in `release`. Build target is `x86_64-unknown-linux-gnu` with rustc 1.75+. CI tested on stable and 1.97. --- ## Differences from the Python version | Area | Python | Rust | | ------------------ | ------------------------------- | ------------------------------------------------------------ | | HTTP client | `urllib.request` | `reqwest` 0.12 with rustls-tls | | HTML parser | `lxml.html` | `scraper` 0.27 over `html5ever` | | Async model | `threading.Thread` per source | `tokio::spawn` per source | | JSON state | `json.dump` / `Path.write_text` | `serde_json` via `Mutex<MlState>` (`OnceLock` singleton) | | Random projection | `hash()`-based + per-call seeds | FNV-1a + `once_cell::Lazy<MatVec>` (identical distribution) | | Synthesis prompt | Hand-written string template | Identical prompt; `format!`-built | | CLI | `sys.argv` dispatch | `clap` 4 (derive) | | Tests | pytest | `#[test]` + `#[tokio::test]` | Behavioral parity tested at the algorithm level. The bandit projection matrix is stable (deterministic seed) but differs from Python's `hash()` for non-ASCII trigrams; in practice the bandit's policy converges within a handful of observations either way. Numeric tolerance for BM25 and confidence is 1e-6 / 1e-9 respectively. --- ## License MIT — see [LICENSE](LICENSE).