Content
# LionScraper MCP + CLI + HTTP API bridge
[简体中文](README_cn.md)
- **Website**: [lionscraper.com](https://www.lionscraper.com/)
- **npm**: [package `lionscraper`](https://www.npmjs.com/package/lionscraper)
- **PyPI**: [project `lionscraper`](https://pypi.org/project/lionscraper/)
## What is this?
**LionScraper** is a browser extension that can collect lists, articles, links, images, and more from web pages. This repository provides the companion **bridge** between your tools and that extension in three ways:
- **MCP** (`lionscraper-mcp`): connect an **AI app** (e.g. Cursor) so the model can call scraping tools over stdio.
- **CLI** (`lionscraper`): run **daemon**, **scrape**, **ping**, and more from a terminal on the same local HTTP/WebSocket port as the extension.
- **HTTP API**: when the daemon is running, call the same capabilities over **loopback JSON HTTP** (e.g. `/v1/...`) from scripts or any HTTP client—no MCP or CLI front-end required.
**The real scraping logic runs in the extension**; these packages connect and forward.
## Before you start
1. **Browser**: Chrome or Edge (follow what the extension supports).
2. **LionScraper extension**: install and enable from the store.
- **Chrome**: [Chrome Web Store — LionScraper](https://chromewebstore.google.com/detail/godiccfjpjdapemodajccjjjcdcccimf)
- **Microsoft Edge**: [Edge Add-ons — LionScraper](https://microsoftedge.microsoft.com/addons/detail/llfpnjbphhfkgbgljpngbjpjpnljkijk)
3. **Runtime** (pick one or both implementations):
- **Node.js** **18+** for the npm package — [Node.js](https://nodejs.org/)
- **Python** **3.10+** for the PyPI package — [Python](https://www.python.org/downloads/)
4. **For MCP**: an AI app that supports MCP (e.g. Cursor, Trae).
5. **For the HTTP API**: same browser, extension, and daemon as the CLI; see the package READMEs for paths and examples.
**HTTP fallback without Chrome/Edge**: If neither browser is detected under standard paths and the extension is not connected, MCP still starts; `ping` succeeds with **http_fetch** mode and `scrape*` use a minimal server-side HTTP GET (no JS execution). If a browser is installed but the extension is not connected, you still get the extension connection flow. The Node auto-spawn path fixes Unix installs where `lionscraper.js` was resolved without a leading `/` (e.g. Glama/Docker). The Python package uses **aiohttp** for outbound HTTP/WebSocket to the daemon.
## Two implementations
| | **Node.js (npm)** | **Python (pip)** |
|--|-------------------|------------------|
| **Registry** | `io.github.dowant/lionscraper-node` | `io.github.dowant/lionscraper-python` |
| **Docs (EN)** | [packages/node/README.md](packages/node/README.md) | [packages/python/README.md](packages/python/README.md) |
| **Docs (ZH)** | [packages/node/README_cn.md](packages/node/README_cn.md) | [packages/python/README_cn.md](packages/python/README_cn.md) |
Install one or both; they are separate packages with the same CLI command names.
### Install (npm)
Published as **[lionscraper](https://www.npmjs.com/package/lionscraper)** on npm.
```bash
npm install -g lionscraper
```
Without a global install, MCP can use **`npx`**; see the **npx** JSON examples under [Add MCP in your AI app](#add-mcp-in-your-ai-app).
### Install (pip)
Published as **[lionscraper](https://pypi.org/project/lionscraper/)** on PyPI.
```bash
pip install -U lionscraper
```
A **virtual environment** is recommended, or `pip install -U --user lionscraper` if you prefer not to install into the system interpreter.
### Commands (both packages)
| Command | Role |
|--------|------|
| **`lionscraper-mcp`** | Thin MCP server (stdio) for AI apps |
| **`lionscraper`** | CLI: `daemon`, `stop`, `scrape`, `ping`, … (also serves the **HTTP API** on the same port) |
After **`pip install -U lionscraper`**, if `lionscraper-mcp` is not on your `PATH`, use **`python -m lionscraper`** with **no extra arguments** for MCP stdio (see [packages/python/README.md](packages/python/README.md)).
**`PORT`** (default **13808**) must match the extension **bridge port** in all modes.
## CLI quick start
```bash
lionscraper daemon
lionscraper ping
lionscraper scrape -u https://www.example.com
```
Full flags, multiple URLs, pagination, and **HTTP API** details: [packages/node/README.md](packages/node/README.md) / [packages/python/README.md](packages/python/README.md).
## Add MCP in your AI app
Examples assume **`lionscraper-mcp`** is on your `PATH` (from npm or pip). In MCP JSON, every **`env` value is a string**.
**Minimal config** (`PORT` defaults to **13808**; must match the extension bridge port):
```json
{
"mcpServers": {
"lionscraper": {
"command": "lionscraper-mcp"
}
}
}
```
**Full `env` example** (omit keys you do not need):
```json
{
"mcpServers": {
"lionscraper": {
"command": "lionscraper-mcp",
"env": {
"PORT": "13808",
"TIMEOUT": "120000",
"LANG": "en-US",
"TOKEN": "",
"DAEMON": ""
}
}
}
}
```
**npx (no global install)** — requires Node.js; the first run may download the package. The npm **package name** is `lionscraper`; the executable is `lionscraper-mcp`. Use `command` **`npx`** and pass **`lionscraper`** then **`lionscraper-mcp`** in `args` (after `-y`).
**Minimal config (npx):**
```json
{
"mcpServers": {
"lionscraper": {
"command": "npx",
"args": ["-y", "lionscraper", "lionscraper-mcp"]
}
}
}
```
**Full `env` example (npx):**
```json
{
"mcpServers": {
"lionscraper": {
"command": "npx",
"args": ["-y", "lionscraper", "lionscraper-mcp"],
"env": {
"PORT": "13808",
"TIMEOUT": "120000",
"LANG": "en-US",
"TOKEN": "",
"DAEMON": ""
}
}
}
}
```
To pin a version, use e.g. `"lionscraper@1.0.1"` in place of `"lionscraper"` inside `args`.
- **`PORT`**: HTTP + WebSocket listen port; default **13808**; must match the extension **bridge port**.
- **`TIMEOUT`**: ms to wait for a previous instance to release the port; default **120000**; **`0`** forces takeover quickly.
- **`LANG`**: tool descriptions and stderr language (**`en-US`**, **`zh-CN`**, or POSIX forms).
- **`TOKEN`**: Bearer token shared with the daemon; empty means no auth.
- **`DAEMON`**: only **`0`** disables auto-starting `lionscraper daemon` from thin MCP.
Restart MCP or the host app after changing config.
### Python: MCP via `python -m`
```json
{
"mcpServers": {
"lionscraper": {
"command": "python",
"args": ["-m", "lionscraper"]
}
}
}
```
Use the same **`python`** you used to install the package (or `python3` on some systems).
## Match the port in the browser extension
1. Open LionScraper **settings / options**.
2. Set **bridge port** to the same value as **`PORT`** (e.g. `13808`).
3. If needed, use **Reconnect**, reload the extension, or restart the browser.
## Day-to-day use
1. Keep the extension **enabled** and target pages **open** as required.
2. Ask in natural language (e.g. check connection, scrape lists / article / emails / phones / links / images).
3. If you see “not connected” or timeouts, retry a connection check and confirm **PORT** matches.
## FAQ
**Extension not connected or scrape fails?**
- Is the extension enabled?
- Does **PORT** in the AI app match the extension **bridge port** exactly?
- One bridge per machine is usually enough; duplicate MCP configs can conflict.
**Seeing MCP tools in the client means everything works?**
Not necessarily. Tools only prove **AI → bridge**; the extension must also register on the same port.
## MCP Registry and directories
Official MCP Registry entries (both use `server.json`):
| Path | Registry name | Package |
|------|----------------|---------|
| [packages/node/server.json](packages/node/server.json) | `io.github.dowant/lionscraper-node` | [npm: lionscraper](https://www.npmjs.com/package/lionscraper) (`mcpName` in `package.json`) |
| [packages/python/server.json](packages/python/server.json) | `io.github.dowant/lionscraper-python` | [PyPI: lionscraper](https://pypi.org/project/lionscraper/) (`mcp-name` comment in English `README.md`) |
**Publish outline** (install the official CLI, see [Quickstart](https://modelcontextprotocol.io/registry/quickstart)):
1. Publish **npm / PyPI** at the version in each `server.json`.
2. In **`packages/node`**: `mcp-publisher login github`, then `mcp-publisher publish`.
3. In **`packages/python`**: `mcp-publisher publish` (login reused).
Third-party listings (e.g. [Glama](https://glama.ai/mcp/servers)) have their own rules; [Smithery](https://smithery.ai/docs/build/publish) targets public HTTPS/streaming setups rather than local stdio + npm/pip by default.
## Third-party directory (Glama)
This project is listed on Glama (e.g. [LionScraper on Glama](https://glama.ai/mcp/servers/dowant/lionscraper-mcp)). If the page shows **cannot be installed** or **license not found**, typical fixes are: add a root **`LICENSE`** (this repo includes [LICENSE](LICENSE)), add **`glama.json`** with maintainer **GitHub usernames** for org-owned repos ([glama.json](glama.json)—edit `maintainers` if claim fails), **claim** the server on Glama, and optionally complete Glama’s **Docker / release** flow if you need their install and security/quality checks—official install remains **`npm install -g lionscraper`** and **`pip install -U lionscraper`**. See also the [score / checklist page](https://glama.ai/mcp/servers/dowant/lionscraper-mcp/score).
## License
[MIT](LICENSE) (same as the npm and PyPI packages).
