Content
# Web Scout
> **Disclaimer**: This project is for learning, research, and technical exchange purposes only. Users should comply with the target website's `robots.txt` and terms of service, and bear all legal responsibilities. The project author does not encourage or participate in any usage that violates laws and regulations.
Helping AI discover web data sources with MCP Server - not a crawler, but a reconnaissance tool that lets AI know "where data is, and what it looks like".
[](LICENSE)
[](https://python.org)
> [English README](./README_EN.md)
## Positioning
**Web Scout is a discovery tool, not a crawler.**
| ✅ Done | ❌ Not done |
|---------|----------|
| Network panel → JSON API endpoints | XHR breakpoint tracing call chain |
| DOM → Duplicate structure + CSS selectors | JS encryption / wasm reverse engineering |
| Request parameters + Response structure extraction | WebSocket binary frame decoding |
| Full page text → Markdown for AI reading | E2EE decryption |
| Compressed field documentation → AI writes crawlers based on it | Automatically generate runnable crawler code |
Suitable for standard HTTP JSON API sites (e.g., Xiaohongshu, Bilibili, e-commerce sites). Not suitable for encrypted data streams, wasm obfuscation, and other reverse engineering scenarios.
## Principle
```
Website → Browser → Full text Markdown
↓ ↓
Network listening AI reads text → Select keywords
↓ ↓
API capture Search → Match APIs with keywords
↓ ↓
Field documentation ←──────────┘
Raw data packets saved to local
```
## Quick Start
```bash
git clone https://github.com/SanZiNEO/web-scout.git
cd web-scout
python -m venv .venv
.venv\Scripts\activate # Windows
pip install -e .
```
### MCP Configuration
Add to `kilo.json`:
```json
"web-scout": {
"type": "local",
"command": ["path\\to\\web-scout\\.venv\\Scripts\\web-scout.exe"],
"enabled": true
}
```
Optional environment variables:
| Variable | Default value | Description |
|------|--------|------|
| `HEADLESS` | `"false"` | Headless mode (`"true"` does not display browser window) |
| `BROWSER_PATH` | Auto | Browser path, `"edge"` uses Edge |
| `USER_DATA_DIR` | Temporary | Persistent user folder, preserves login state |
| `LOGIN_TIMEOUT` | `"300"` | Maximum login wait seconds |
| `MAX_TEXT_LENGTH` | `"3000"` | Maximum page text characters |
## Tools
| Tool | Description |
|------|------|
| `scout_open` | Open page → Extract full text Markdown → Start network listening |
| `scout_action` | Perform search, scrolling, and other operations |
| `scout_wait_login` | Wait for user manual login |
| `scout_list_apis` | List captured API endpoints |
| `scout_inspect_api` | View complete API request and response |
| `scout_list_elements` | List page elements for AI selection |
| `scout_click` | Click specified element |
| `scout_search` | Keyword search: first search API body, automatically fall back to DOM if no match |
| `scout_fetch_api` | Verification mode: open page → listen → match API by path → return details in one step |
| `scout_inspect_dom` | Verification mode: open page → scan DOM container by keyword → return in one step |
| `scout_export` | Export raw data packets + compressed field documentation |
| `scout_close` | Close current browser session, release resources |
## Example
```
AI: scout_open("https://xiaohongshu.com/explore")
→ "Page text: weight loss meal fitness plan OOTD …"
AI: scout_action("search", "weight loss meal")
→ "Captured 2 new APIs"
AI: scout_list_apis()
→ [1] POST /api/search/notes → 20 fields
AI: scout_inspect_api(1)
→ POST https://edith.xiaohongshu.com/api/search/notes
Request body: {"keyword": "weight loss meal", "page": 1, ...}
Response: code=0, data.items[]: this time=20, id=..., title=...
AI: scout_export(1)
→ Field documentation + saved: response/search_notes.json
```
## Architecture
```
src/web_scout/
├── server.py # FastMCP entry + 8 tools
├── browser.py # Chromium wrapper + text extraction + login detection
├── monitor.py # Network listening + API filtering + storage
├── dom.py # Element scanning + container merging
└── export.py # Compressed field documentation + raw data packet saving
```
## License
MIT © [ShanZhi](https://github.com/SanZiNEO)
---
> **Disclaimer**
>
> This project (Web Scout) is a general-purpose web data source discovery tool that does not initiate crawling requests, store, or transmit any website data. Users should:
>
> 1. Comply with the target website's `robots.txt` and terms of service
> 2. Control request frequency, not causing abnormal loads on the target website
> 3. Only crawl public data, not bypassing website authentication and authorization mechanisms
> 4. Bear all legal responsibilities for using this tool
>
> The project author (ShanZhi / SanZiNEO) does not encourage or participate in any usage that violates laws and regulations or website terms. This tool is for learning, research, and technical exchange purposes only.
---
**Statement:** This project is developed with AI assistance, aiming to help AI and developers quickly discover web data sources, without any cracking, bypassing, or malicious functions. Users should comply with the target website's `robots.txt` and relevant laws and regulations.
