cn-scraper-mcp

# 🔥 CN Scraper MCP **Empowering AI Agents to Directly Search the Chinese Internet - Taobao, JD, Xiaohongshu, Zhihu, Weibo, Zhixiaozhu... without being blocked by anti-scraping measures.** [](https://python.org) [](https://modelcontextprotocol.io) [](LICENSE) [](.) --- ## What is this? Every AI Agent (Codex, Claude Code, Cursor, Trae) can search the web. But Chinese platforms are not welcoming to bots: - **Taobao**: TLS fingerprint detection + MTOP HMAC-MD5 signature, curl directly pops up a slider - **JD**: Headless browser returns 0 results. Old selector `li.gl-item` is dead, `p.3.cn` DNS has disappeared - **Xiaohongshu**: Data center IP directly blocked (cookie not even checked), search results loaded by JS signature XHR - **Zhihu**: Visitor search is closed, all API requires login status - **Pinduoduo**: `anti_content` token mechanism, one browser session can only search once - **Zhixiaozhu**: Paid community, content hidden behind cookie-authenticated REST API **This project packages the pits I've stepped on for months into an MCP Server** - your Agent can search with one line of code: `taobao_search("children's learning desk")`. --- ## Platform Support ### E-commerce | Platform | Method | Requires Browser | Limitations | Stability | |------|------|-----------|------|--------| | **Taobao/Tmall** | `curl_cffi` + MTOP signature | ❌ | Loose¹ | ✅ Stable | | **JD/JD** | Chrome CDP headful | ✅ | Medium | ✅ Stable² | | **Pinduoduo/PDD** ❌ | — | — | — | ❌ Unavailable³ | > ¹ Taobao has no hard rate limiting, but the platform may tighten up at any time, and high-frequency batch crawling is not recommended. > ² JD relies on `div[data-sku]` selector, platform changes may cause adaptation to fail. > ³ Pinduoduo allows only one search per browser session, and subsequent searches return "system busy". Single search result has zero practical value, and the engine code is retained but not recommended for use. ### Content Community | Platform | Method | Requires Browser | Limitations | Stability | |------|------|-----------|------|--------| | **Xiaohongshu/XHS** | Local Chrome/Obscura CDP + cookie | ✅ | Medium⁴ | ✅ Stable | | **Zhihu/Zhihu** | REST API v4 | 🔑 Requires login | Normal | ✅ Stable | | **Zhixiaozhu/ZSXQ** | REST API v2 | ❌ | Normal | ✅ Stable | | **Weibo/Weibo** 🔑 | REST API | ❌ (hot search) / 🔑 (search) | Normal | ✅ Stable⁵ | | **Douyin/Douyin** ❌ | — | — | — | ❌ Not feasible⁶ | > ⁴ Xiaohongshu only allows residential IP - cloud browser/data center IP is blocked. Recommended to use Obscura (built-in anti-detection) or local Chrome. > ⁵ Weibo hot search does not require login, but must bring `Referer: https://weibo.com/` + `X-Requested-With: XMLHttpRequest` header (otherwise 403). Search API returns 432, requires `SUB` cookie (currently unavailable). > ⁶ Douyin requires encrypted signature request header (X-Gorgon/X-Khronos/X-Argus), and there is no available login-free entrance. ### API/Selector Life and Death Book | API / Selector | Status | Description | |-------------|------|------| | `mtop.taobao.wsearch.appsearch` → `itemsArray` | ✅ Alive | Correct field; `data.result` is always `[]` | | `h5api.m.taobao.com` h5search | ❌ Dead | Returns 502 | | `p.3.cn/prices/mgets` | ❌ Dead | DNS no longer resolves | | `club.jd.com/comment/productPageComments` | ❌ Blocked | Returns "system busy" (12 bytes) | | `li.gl-item` / `#J_goodsList` | ❌ Dead | JD changed layout | | `div[data-sku]` | ✅ Current | JD current product selector | | XHS `section.note-item` | ✅ | Search result DOM | | XHS `__INITIAL_STATE__.note.noteDetailMap` | ✅ | Note content + comments | | PDD `mobile.yangkeduo.com/search_result.html` | ⚠️ | Only valid for the first search | | ZSXQ `api.zsxq.com/v2/groups/{id}/topics` | ✅ | Cookie authentication, no browser required | | Zhihu `api/v4/search_v3` | 🔑 | Requires z_c0 + d_c0 | | Weibo `ajax/side/hotSearch` | ✅ | Visitor accessible (requires Referer + X-Requested-With header, otherwise 403) | | Weibo `m.weibo.cn/api/container/getIndex` | ❌ | Returns 432, search requires SUB cookie | | Douyin `aweme/v1/web/search/item/` | ❌ | Requires signature request header | --- ## Quick Start ### Installation > ⚠️ **Not yet released to PyPI**, please install from source: ```bash git clone https://github.com/goesByhc/cn-scraper-mcp.git cd cn-scraper-mcp pip install -e ".[dev]" ``` ### Cookie Configuration (one-time) Each platform requires a logged-in browser's Cookie. Stored in `~/.cn-scraper-cookies/`: ```bash mkdir -p ~/.cn-scraper-cookies ``` | Platform | Cookie File | Acquisition Method | |------|------------|---------| | Taobao | `taobao.json` | Log in to `m.taobao.com` → DevTools → Application → Cookies → Export JSON. Requires `_m_h5_tk`, `_tb_token_`, `cookie2`, `cna`, `unb`, and HttpOnly `sgcookie`/`tfstk`/`isg` (harvested using CDP) | | JD | `~/.jd_login_profile/` | Persistent Chrome Profile - log in to `jd.com` once, and the Profile will automatically remember | | Xiaohongshu | `xiaohongshu.json` | Export from `xiaohongshu.com` DevTools. Requires `web_session`, `a1`, `webId`, `gid` | | Zhihu | `zhihu.json` | Export from `zhihu.com` DevTools. Requires `z_c0`, `d_c0` | | Zhixiaozhu | `zsxq.json` | Export from `zsxq.com` DevTools. Requires `zsxq_access_token` | | Pinduoduo | `pdd.json` | Export from `yangkeduo.com` DevTools. Requires `PDDAccessToken`, `pdd_user_id`. ⚠️ Token expires in about 1 hour | > ⚠️ **Taobao HttpOnly Cookie**: `sgcookie`, `tfstk`, `isg`, `havana_lgc2_0` are HttpOnly - can't be obtained manually using DevTools. Use CDP `Network.getAllCookies` to harvest the complete set from a logged-in Chrome, or use the built-in `harvest_cookies` MCP tool. ### Start ```bash cn-scraper-mcp # or: python -m cn_scraper_mcp.server ``` ### Docker Pre-installed Chromium in the container, no local browser required: ```bash docker build -t cn-scraper-mcp . docker run -i --rm \ -v ~/.cn-scraper-cookies:/root/.cn-scraper-cookies \ -v ~/.jd_login_profile:/root/.jd_login_profile \ cn-scraper-mcp ``` Agent integration configuration: ```toml # Codex ~/.codex/config.toml [mcp_servers.cn-scraper] type = "stdio" command = "docker" args = ["run", "-i", "--rm", "-v", "~/.cn-scraper-cookies:/root/.cn-scraper-cookies", "-v", "~/.jd_login_profile:/root/.jd_login_profile", "cn-scraper-mcp"] ``` > Docker image built-in Chromium + `--no-sandbox`. JD headful mode requires Xvfb, set environment variable `XVFB_WRAPPER=1`. Xiaohongshu still requires residential IP - data center IP will be blocked. --- ## MCP Tool List (13 available / 16 registered) ### E-commerce Search | Tool | Description | |------|------| | `taobao_search` | Taobao/Tmall keyword search → price, sales, store | | `jd_search` | JD keyword search → SKU, price, product name | | `compare_prices` | Cross-platform price comparison → Taobao vs JD, lowest price/median price/price range | > ❌ `pdd_search` / `pdd_product_detail` registered but not available - Pinduoduo allows only one search per session, with no practical value. ### Content Community | Tool | Description | |------|------| | `xiaohongshu_search` | Xiaohongshu note search → title, author, likes | | `xiaohongshu_note` | Xiaohongshu note details → content, tags, comments | | `zhihu_search` | Zhihu search → questions, articles (requires login) | | `zhihu_hot_list` | Zhihu hot list (requires login) | | `weibo_hot_list` | Weibo hot search (no login required, requires Referer header) | | `zsxq_topics` | Zhixiaozhu paid community posts | > ❌ `weibo_search` not available - search API returns 432. `douyin_search` not available - requires encrypted signature. ### Diagnostics & Tools | Tool | Description | |------|------| | `check_cookies` | Check all platform Cookie status | | `diagnose` | Environment diagnosis - dependency version, browser, CDP port | | `harvest_cookies` | CDP automatic Cookie harvesting (including HttpOnly) | --- ## Python API ```python from cn_scraper_mcp.engines import ( TaobaoEngine, JDEngine, PDDEngine, XiaohongshuEngine, ZhihuEngine, ZsxqEngine, WeiboEngine, ) # Taobao - pure script, no browser required tb = TaobaoEngine() r = tb.search("Huawei mate70", limit=5) print(r["items"][0]["price"]) # "3099.00" # JD - requires Chrome headful jd = JDEngine() r = jd.search("JD Jingzao Muguang") # Pinduoduo - ⚠️ only one search pdd = PDDEngine() r = pdd.search("children's learning desk", limit=5) detail = pdd.product_detail("123456789") # unlimited times # Xiaohongshu - Obscura priority, Chrome fallback xhs = XiaohongshuEngine() notes = xhs.search("evaluation") detail = xhs.get_note(notes["items"][0]["noteId"]) # Zhihu - requires login Cookie zh = ZhihuEngine() r = zh.search("semiconductor") hot = zh.hot_list() # Zhixiaozhu - REST API zs = ZsxqEngine() topics = zs.get_topics("28888555451", count=5) # Weibo - hot search no login, search requires Cookie wb = WeiboEngine() hot = wb.hot_list() r = wb.search("hot search topic") # Cross-platform price comparison from cn_scraper_mcp.compare import compare_prices result = compare_prices("Huawei mate70", platforms=["taobao", "jd"]) print(result["best_deal"]) # lowest price product ``` --- ## Agent Integration ### Codex `~/.codex/config.toml`: ```toml [mcp_servers.cn-scraper] type = "stdio" command = "cn-scraper-mcp" args = [] autoApprove = ["*"] ``` ### Claude Code / Cursor / Trae ```json { "mcp": { "servers": { "cn-scraper": { "command": "cn-scraper-mcp", "args": [] } } } } ``` ### Reasonix ```toml [[plugins]] name = "cn-scraper" command = "cn-scraper-mcp" args = [] ``` --- ## Architecture ``` AI Agent (Codex / Claude / Cursor / Trae / Reasonix) │ MCP stdio ▼ ┌─────────────────────────────────────────────────────┐ │ server.py (FastMCP) │ │ 16 tools · parameter verification · unified error model · stderr log │ ├─────────────────────────────────────────────────────┤ │ Engine layer │ │ taobao.py ─→ curl_cffi + MTOP ──→ h5api.m.taobao │ │ jd.py ─→ Chrome CDP ──→ search.jd.com │ │ pdd.py ─→ Chrome CDP + iUA ──→ mobile.yangkeduo │ │ xiaohongshu.py ─→ Obscura/Chrome CDP ─→ xiaohongshu│ │ zhihu.py ─→ REST API v4 ──→ zhihu.com │ │ weibo.py ─→ REST API ──→ weibo.com │ │ zsxq.py ─→ REST API v2 ──→ api.zsxq.com │ │ douyin.py ─→ (skeleton, API not available) │ ├─────────────────────────────────────────────────────┤ │ Infrastructure │ │ auth.py — Cookie management and field verification │ │ http.py — timeout/retries/backoff/rate limiting │ │ cdp.py — Chrome/Obscura CDP driver + BrowserLock │ │ logging.py — stderr desensitized log + error recording │ │ errors.py — 8 types of exceptions + unified error_response() │ │ compare.py — cross-platform price comparison aggregation layer │ │ cookie_harvest.py — CDP automatic Cookie harvesting │ └─────────────────────────────────────────────────────┘ ``` --- ## Tests 454 unit tests, all mocked (no real network/Chrome/Cookie): ```bash pytest tests/ -v # 454 passed, ~30s ruff check src/ tests/ # zero warnings pytest tests/ --cov # coverage report ``` GitHub Actions CI: Python 3.11 / 3.12 / 3.13 matrix, Windows + Ubuntu, automatic Ruff check + Wheel build. --- ## FAQ **Q: Why not use Playwright / Selenium?** Heavier and slower, and many AI Agents can't run it. `curl_cffi` + native CDP WebSocket = minimal dependencies. **Q: Taobao returns `Session expired`.** `_m_h5_tk` cookie expired. Harvest again from the browser. **Q: JD returns 0 results.** Three possible reasons: (1) Chrome is set to headless → switch to headful. (2) Profile not logged in. (3) Cookie injection lacks real login status → use persistent Profile. **Q: Xiaohongshu searches for "IP exists risk".** You used a cloud browser/data center IP. Xiaohongshu blocks at the IP level - switch to **local Chrome** or residential IP. **Q: Pinduoduo can search once, but "system busy" the second time.** This is a platform limitation, not a bug. Each browser session only allows one search. Restart MCP Server to get a new session. **Q: Is it legal?** Only for **learning and research purposes**. Bulk crawling may violate platform service terms. Assume your own risk. Do not use for spam, DDoS, or large-scale commercial scraping. --- ## Roadmap - [x] Taobao/Tmall (curl_cffi + MTOP) - [x] JD (Chrome CDP headful) - [x] Pinduoduo (CDP + iPhone UA + single search limitation documented) - [x] Xiaohongshu (Obscura priority, local Chrome fallback) - [x] Zhihu (REST API, adapted to login requirements) - [x] Zhixiaozhu (REST API v2) - [x] Weibo (hot search + search) - [x] Cross-platform price comparison tool - [x] CDP automatic Cookie harvesting (including HttpOnly) - [x] Docker support - [x] GitHub Actions CI - [x] Unified error model + parameter verification - [x] Concurrent isolation (BrowserLock) - [x] Platform health check script - [ ] Release to PyPI - [ ] Cookie encrypted storage - [ ] Request metrics, caching, auditing --- ## License MIT — see [LICENSE](LICENSE). ## Acknowledgements - [curl_cffi](https://github.com/lexiforest/curl_cffi) — TLS fingerprint spoofing - [FastMCP](https://github.com/jlowin/fastmcp) — MCP Server framework - [websockets](https://github.com/python-websockets/websockets) — asynchronous WebSocket - [Obscura](https://github.com/h4ckf0r0day/obscura) — Rust lightweight anti-detection browser --- *Made with ☕ and months of frustration at Chinese platform anti-bot walls.* > 📄 English version: [README.en.md](README.en.md)