solocrawl

<p align="center"> <img src="assets/icon/solocrawl.png" alt="SoloCrawl logo" width="128"> </p> <h1 align="center">SoloCrawl</h1> <p align="center"> <strong>Web search, scraping, and live package-version lookup —<br> for your terminal, your code, and your local LLM.</strong><br> No accounts. No API keys. No cloud. </p> <p align="center"> <img src="https://img.shields.io/badge/python-3.14%2B-blue.svg" alt="Python 3.14+"> <img src="https://img.shields.io/badge/license-MIT-green.svg" alt="MIT License"> <img src="https://img.shields.io/badge/MCP-ready-8A2BE2.svg" alt="MCP ready"> <img src="https://img.shields.io/badge/API%20keys-none%20required-success.svg" alt="No API keys required"> </p> --- SoloCrawl is a small, self-hosted, fully async Python tool that does three things well: it **searches the web** across many sources at once, **scrapes pages into clean markdown** that's ready for an LLM, and **looks up the latest package version** from official registries. It runs on your machine, for free, and works the moment you install it — nothing to sign up for, no keys to paste. It's built for individual developers and people tinkering with local LLMs. Use it as an **MCP server** (LM Studio, Claude Desktop, OpenCode, …), straight from the **CLI**, or as a **Python library**. ## ✨ Highlights | | | |---|---| | 🔌 **Zero config** | Search, scrape, and package lookup all work out of the box — no accounts, no keys. | | 🔎 **Federated search** | Queries 11 possible sources, merges them with Reciprocal Rank Fusion and de-duplicates URLs into one clean ranking. | | 📄 **Smart scraping** | HTML → tidy markdown via `trafilatura`/`readability`, with a Playwright browser fallback for JS-heavy pages — used only when it's actually needed. | | 📦 **Live package versions** | 10 ecosystems (PyPI, npm, crates.io, Maven, Go, …) resolved live from official registries — never a stale local DB. | | 🤖 **MCP-native** | Drops straight into local LLM tooling as a stdio MCP server with five ready tools. | | ⚡ **Fully async, bounded** | One shared HTTP client, one recycled browser, global + per-domain concurrency limits. Fast without hammering anyone. | | 🧩 **Hackable** | Add a search or package provider as a single self-registering file — the core stays untouched. | | 🔒 **Safe by default** | Blocks localhost/private/cloud-metadata targets and honours `robots.txt`. | ## 🚀 Quick start The recommended way to install SoloCrawl is **pipx** — it drops the `solocrawl` and `solocrawl-mcp` commands onto your PATH in their own isolated environment, so you can run them from anywhere without juggling a virtualenv: ```bash git clone https://github.com/hlavacm/solocrawl.git pipx install ./solocrawl # or an absolute path: pipx install /path/to/solocrawl ``` > Already on PyPI? Then it's just `pipx install solocrawl` — no checkout needed. That's it — now run the three core commands from any directory: ```bash # Scrape a page to markdown solocrawl scrape https://example.com # Federated web search (Wikipedia + DuckDuckGo + StackExchange by default) solocrawl search "python asyncio semaphore" --limit 5 # Live package version lookup solocrawl package requests --ecosystem pypi ``` ### Updating ```bash # Installed from PyPI: pipx upgrade solocrawl # Installed from a local checkout — pull the latest changes, then reinstall: cd /path/to/solocrawl && git pull && pipx install --force . # alias: pipx reinstall solocrawl ``` After upgrading, restart your MCP client (LM Studio, Cursor, Claude Desktop, …) so it picks up the new `solocrawl-mcp` binary — your `mcp.json` needs no changes as long as it points at `solocrawl-mcp` on your PATH. ## What you can do ### 🔎 Search the web One query, many sources, a single merged ranking — no single provider deciding everything for you. ```bash solocrawl search "python asyncio semaphore" --limit 5 # Pick exactly which sources to hit, and get machine-readable output solocrawl search "django orm" --sources wikipedia,stackexchange --json ``` ### 📄 Scrape a page to clean markdown Turn any URL into LLM-ready markdown with page metadata (title, author, date, …) as front-matter. ```bash solocrawl scrape https://example.com # Save to a file, or force the browser for a JS-rendered page solocrawl scrape https://example.com --out page.md solocrawl scrape https://example.com --force-browser ``` ### 📦 Look up package versions The current version — and the one matching your constraint — straight from the official registry. ```bash solocrawl package react --ecosystem npm --constraint ">=18,<19" solocrawl package monolog/monolog --ecosystem packagist --json solocrawl package some-lib --ecosystem pypi --allow-prerelease ``` ### 🧪 Research in one shot The classic LLM workflow — search, scrape the top hits, and get back one aggregated, **cited** report. ```bash solocrawl research "python asyncio semaphore" --depth 3 ``` ### 🗂️ Batch-scrape many URLs Fetch a whole list at once under the same bounded concurrency; `--out-dir` writes one file per URL. ```bash solocrawl batch https://example.com https://www.python.org --out-dir /tmp/scrape solocrawl batch --from-file urls.txt --out-dir /tmp/scrape ``` ### …and see what's available ```bash # List every registered provider (search + package), default vs. opt-in solocrawl providers ``` ## 🤖 Use it with your local LLM (MCP) This is where SoloCrawl really shines — give your local model (LM Studio, OpenCode, Claude Desktop, …) the ability to search, scrape, and check versions. The `pipx install` from the [Quick start](#-quick-start) already put `solocrawl-mcp` on your PATH, so all that's left is pointing your MCP client at it. **LM Studio / Claude Desktop** — ready-to-use config at **[examples/mcp.json](examples/mcp.json)**. Drop it into your client's MCP settings (`mcp.json`): ```json { "mcpServers": { "solocrawl": { "command": "solocrawl-mcp", "args": [], "env": { "SOLOCRAWL_LOG_LEVEL": "INFO", "SOLOCRAWL_LOG_FILE": "~/.local/state/solocrawl/mcp.log" } } } } ``` **OpenCode** — uses a different config format. Copy **[examples/opencode.jsonc](examples/opencode.jsonc)** into `~/.config/opencode/opencode.jsonc` (global) or `opencode.jsonc` in your project root. OpenCode expects `type: "local"`, `command` as an array, and `environment` instead of `env`. If your MCP client doesn't inherit your shell `PATH`, replace `"solocrawl-mcp"` with the full path from `which solocrawl-mcp` (typically `~/.local/bin/solocrawl-mcp` after `pipx install`). Logs go to stderr (visible in LM Studio Developer Logs) and optionally to `SOLOCRAWL_LOG_FILE`. The server exposes **five tools**: - `web_search(query, limit=5, sources=None)` — federated search across enabled providers - `scrape(url)` — fetch and extract markdown (with page metadata) from a URL - `research(query, depth=3)` — search, scrape the top results, and return an aggregated cited report - `package_version(name, ecosystem, constraint=None, allow_prerelease=False)` — live registry lookup - `list_providers(provider_type="all")` — list registered search/package providers (default vs. opt-in) To check the active version and command path: ```bash pipx list | grep solocrawl which solocrawl-mcp ``` > **Working from a local clone?** A `pipx`-installed `solocrawl-mcp` is a *snapshot* — editing the > repo does **not** update the command on your `PATH`, so your MCP client keeps running the old > code. After changing the source, refresh it with `pipx install --force .` (or install once with > `pipx install --editable .` so future edits are picked up automatically). ## 🐍 Use it from Python ```python import asyncio from solocrawl.config import load_config from solocrawl.core.search import federated_search, select_providers from solocrawl.core.search.providers import duckduckgo, stackexchange, wikipedia # noqa: F401 async def main() -> None: providers = select_providers(load_config()) results = await federated_search(providers, "asyncio python", limit=3) for result in results: print(result.title, result.url) asyncio.run(main()) ``` See [examples/library_search.py](examples/library_search.py) for a runnable example. ## Search providers **Default (zero-config, always enabled):** | Provider | Source | |----------|--------| | `wikipedia` | MediaWiki API | | `duckduckgo` | `ddgs` package | | `stackexchange` | Stack Exchange API (Stack Overflow) | **Opt-in** (enable with `SOLOCRAWL_ENABLE_PROVIDERS`): | Provider | Source | |----------|--------| | `wikidata` | Wikidata entity search | | `hackernews` | Hacker News (Algolia) | | `arxiv` | arXiv Atom API | | `pubmed` | PubMed/NCBI E-utilities | | `github` | GitHub repository search | | `mdn` | MDN Web Docs search | | `reddit` | Reddit post search (`search.json`) | | `searxng` | Self-hosted SearXNG (set `SOLOCRAWL_SEARXNG_URL`) | ```bash SOLOCRAWL_ENABLE_PROVIDERS=arxiv,hackernews solocrawl search "transformer attention" --limit 6 SOLOCRAWL_ENABLE_PROVIDERS=github,mdn solocrawl search "fetch api" --limit 6 ``` ## Package registries Default ecosystems: **PyPI**, **npm**, **Packagist**, **crates.io**, **NuGet**, **Maven Central**, **RubyGems**, **Go modules**, **pub.dev**, **Swift**. Versions are always fetched live from official registries — SoloCrawl does not maintain its own version database. Swift packages have no central registry, so versions come from the repository's git tags (`owner/repo` on GitHub). ```bash solocrawl package serde --ecosystem crates solocrawl package Newtonsoft.Json --ecosystem nuget solocrawl package org.junit.jupiter:junit-jupiter --ecosystem maven solocrawl package github.com/gorilla/mux --ecosystem go solocrawl package apple/swift-argument-parser --ecosystem swift ``` ## Optional extras ```bash # Browser fallback for JS-heavy pages (Playwright) pip install -e ".[browser]" playwright install chromium solocrawl scrape https://example.com --force-browser # Install everything pip install -e ".[all]" ``` ## ⚙️ Configuration **All defaults work with no configuration.** Everything below is optional and uses the `SOLOCRAWL_` prefix. For local development, copy `.env.dist` to `.env` and uncomment what you need — SoloCrawl loads `.env` automatically via [python-dotenv](https://pypi.org/project/python-dotenv/), and existing shell environment variables take precedence. <details> <summary>Environment variables</summary> | Variable | Default | Purpose | |----------|---------|---------| | `SOLOCRAWL_ENABLE_PROVIDERS` | _(empty)_ | Comma-separated opt-in provider names | | `SOLOCRAWL_SEARXNG_URL` | _(empty)_ | Base URL of a self-hosted SearXNG instance (enables the `searxng` provider) | | `SOLOCRAWL_RESPECT_ROBOTS` | `true` | Honour `robots.txt` on scrape (fail-open); set `false` to skip | | `SOLOCRAWL_CACHE_TTL_SECONDS` | `0` | In-memory fetch cache TTL in seconds (`0` = disabled) | | `SOLOCRAWL_MAX_CONCURRENCY` | `10` | Global fetch concurrency limit | | `SOLOCRAWL_PER_DOMAIN_LIMIT` | `2` | Per-domain concurrency limit | | `SOLOCRAWL_TIMEOUT_SECONDS` | `30` | Per-request timeout in seconds | | `SOLOCRAWL_MAX_RETRIES` | `3` | Retries on network errors / rate limits | | `SOLOCRAWL_MAX_RESPONSE_BYTES` | `10485760` | Cap on fetched response body size (10 MiB); larger bodies are truncated | | `SOLOCRAWL_PROXY_ENABLED` | `false` | Enable optional proxy layer | | `SOLOCRAWL_PROXY_MODE` | `list` | Proxy mode: `list` (rotate a pool) or `endpoint` (single rotating endpoint) | | `SOLOCRAWL_PROXY_LIST` | _(empty)_ | Comma-separated proxy URLs | | `SOLOCRAWL_PROXY_ENDPOINT` | _(empty)_ | Single rotating proxy endpoint | | `SOLOCRAWL_PROXY_USERNAME` | _(empty)_ | Proxy auth username | | `SOLOCRAWL_PROXY_PASSWORD` | _(empty)_ | Proxy auth password | | `SOLOCRAWL_ALLOW_INTERNAL_URLS` | `false` | Allow scraping localhost/private IPs (dev only) | | `SOLOCRAWL_USER_AGENT` | _(SoloCrawl default)_ | Override HTTP User-Agent for API requests | | `SOLOCRAWL_BROWSER_ALLOWED` | `true` | Allow Playwright fallback when installed | | `SOLOCRAWL_LOG_LEVEL` | `WARNING` | Log level: `DEBUG`, `INFO`, `WARNING`, `ERROR` | | `SOLOCRAWL_LOG_FILE` | _(empty)_ | Optional log file path (also logs to stderr) | </details> ## 🔒 Security note on URL fetching By default SoloCrawl refuses to fetch localhost, link-local, private, reserved, and cloud-metadata addresses. It checks literal hosts, DNS-resolved A/AAAA records, HTTP redirect targets, and Playwright's final browser URL. SoloCrawl is still a single-user local tool, not a hostile-multi-tenant proxy — do not expose it to untrusted network callers. `SOLOCRAWL_ALLOW_INTERNAL_URLS=true` disables these internal-target checks entirely (intended for trusted local development only). ## 🧩 Extending it The whole point of the plugin layout is that adding a source is a single self-registering file — the core never changes. To add a search provider: 1. Create `src/solocrawl/core/search/providers/myprovider.py` implementing `SearchProvider`. 2. Register with `@register("myprovider", zero_config=True)` or as opt-in. 3. Import the module in `src/solocrawl/core/search/providers/__init__.py` so registration runs. 4. Add fixture-based tests in `tests/`. The same pattern applies to package providers under `src/solocrawl/core/packages/providers/`. ## Development Work from a checkout in a virtualenv with an editable install — this also drops the `solocrawl` and `solocrawl-mcp` scripts into `.venv/bin/`: ```bash python -m venv .venv && source .venv/bin/activate pip install -e ".[dev]" ``` Then run the quality gate: ```bash ruff check . && ruff format --check . pyright pytest # …or all in one line: ruff check . && pyright && pytest ``` ## Ethics and terms of use SoloCrawl is built for individual developers and local LLM tooling. It respects the robots.txt and terms of service of target sites — `scrape` consults `robots.txt` and refuses disallowed URLs by default (fail-open on errors; opt out with `SOLOCRAWL_RESPECT_ROBOTS=false`). The proxy and scraping features are **not** intended to bypass site rules, captchas, or anti-bot systems. Use responsibly and stay within legitimate access patterns. ## License MIT — see [LICENSE](LICENSE).