web-agent

# web-agent [](https://github.com/franciseliang99-dot/web-agent/actions/workflows/ci.yml) [](LICENSE) [](pyproject.toml) [](tests/) [](pyproject.toml) [](CHANGELOG.md) ## Featured Blogs (3-part series, all web-agent dogfooding publish) 1. **[50% Compliance, Not 0%: How a Logging Spike Almost Triggered the Wrong Architecture Rewrite](https://dev.to/francise_liang_e4544eadb9/50-compliance-not-0-how-a-logging-spike-almost-triggered-the-wrong-architecture-rewrite-1lna)** — dev.to · 8 min read Measurement layer story: W5-C.2 spike 7 versions closed-loop regex false negatives almost triggered wrong architecture rewrite (V0.16.27 dogfooding). 2. **[Why I Permanently NO-GO'd Patchright After a Spike (And the Anti-Detection Decision Tree)](https://dev.to/francise_liang_e4544eadb9/why-i-permanently-no-god-patchright-after-a-spike-and-the-anti-detection-decision-tree-3m11)** — dev.to · 7 min read Architecture layer story: V0.16.14 patchright spike NO-GO + V0.16.15 curl_cffi NO-GO + anti-detection decision tree 4-layer selection (V0.16.30 dogfooding). 3. **[Build Time vs Edit Time — My Web Agent Can Publish But Can't Edit (An Honest Capability-Boundary Spike)](https://dev.to/francise_liang_e4544eadb9/build-time-vs-edit-time-my-web-agent-can-publish-but-cant-edit-an-honest-capability-boundary-4lpl)** — dev.to · 6 min read Tool boundary story: V0.16.31 dogfooding 4/5 = 80% success rate, edit existing article triggers V0.5.0 anti-loop exposes actuator 5 actions boundary (V0.16.32 dogfooding 4th publish own blog). --- **MultiOn-style highly anthropomorphic Web Agent — Python + Playwright + VLM/SoM + stealth, BYO LLM (Anthropic/OpenAI/Kimi)**. Takes over your **logged-in Chrome** (no isolated Chromium launch, preserving cookies/extensions/profile), completing natural language tasks → real browser operations through Set-of-Mark visual annotation + Anthropic Claude Sonnet 4.6 vision tool calling. Supports 4 integration methods: **MCP server (stdio)** / **CLI** / **Python import** / **Claude Desktop** — see "Usage" below. Technical approach based on [`docs/高度模仿人操作网页的agent技术路径图.txt`](docs/高度模仿人操作网页的agent技术路径图.txt) (original user document). ## Project Features 🧠 **Decision-driven spike closed-loop**: Each anti-detection/planning scheme runs a spike to gather data before making decisions, not implementing something that seems useful. Representative closed-loops: - **patchright-python NO-GO** (V0.16.14) — sannysoft.com A=C 19/57 measured and refuted + ARCHITECTURE §1.3 permanently documented - **curl_cffi NO-GO** (V0.16.15) — Chrome already uses real BoringSSL, curl_cffi bypass has no effect on browser path - **W5-C.2 real plan-and-execute DEFER** (V0.16.16-22, 7-version closed-loop) — augmentation route 50% compliance data foundation + ARCHITECTURE §1.5 decision matrix 📊 **Observable**: Each step screenshot + thinking + action → SQLite persistence → single-file HTML replay panel; long-term memory across sessions (domain ↔ past goals/results, V0.13.0+). 🛡 **Three-layer release gate**: ruff 0 + mypy strict 0 + pytest 255 all green + GitHub Actions CI (V0.16.13). 🤝 **MCP server (V0.16.0-9)**: 3 tools + 2 resources + progress heartbeat + asyncio.Lock serialization + 9222 health check. Claude Desktop adds 2 lines of config for use. ## Current Status **V0.16.23** (2026-05-05) — 80+ commits, **255 tests passed + 2 smoke skips**, 3-layer release gate all green, GitHub Actions CI automated. **W milestone progress**: - W1 ✅ Wikipedia search term + extract first paragraph (skeleton + multi-LLM support) - W2 ✅ GitHub search repo + anthropomorphic actuator (3-stage Bezier + smootherstep + normal typing + typo+backspace) - W3 ✅ Gmail login state: read-only summary (W3-B) + compose write operation (W3-C, V0.16.17 real account E2E test passed) + `safety.py` authorization whitelist (W3-A) - W4 ✅ replay log panel (W4-1) + index page (W4-1.1) + Cloudflare/reCAPTCHA pause takeover UX (W4-2) + desktop notifications (W4-3) - W5 ✅ — self-reflection page-stuck hint (W5-A) + Shadow DOM penetration (W5-B) + cross-session long-term memory (W5-D) + memory inject planner (W5-D.2) + layered planning augmentation (W5-C) + W5-C.2 real plan-and-execute spike closed-loop maintenance DEFER (V0.16.16-22) **MCP server (V0.16.0-9)** ✅: stdio + 3 tools (web_agent_run / web_agent_get_replay / web_agent_query_memory) + 2 resources (`webagent://replay/<id>` / `webagent://memory/<domain>`) + progress wire + asyncio.Lock + health check + ARCHITECTURE §5 complete documentation. **Audit gap closure (6/6 complete)**: perceiver (V0.12.0) / trace (V0.12.4) / cli (V0.12.6) / loop main body (V0.12.8) / browser (V0.15.1) / anthropic (V0.15.1). **Architecture decisions**: see [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) — decision tree / module boundary / three-track same-track (reflect/memory/subgoal shared trace channel) / dual defense (safety+anti-loop+reflect+captcha signal orthogonal) / W5-C.2 spike 7-version closed-loop verdict. ## Stack - Python 3.12 + Playwright 1.59 async - Takes over user's local Chrome (`--remote-debugging-port=9222`) — no isolated Chromium launch, preserving login state/cookies/extensions. From V0.16.18, `start_chrome.sh` also supports Chromium-based forks: Brave / Edge / Vivaldi / Opera (CDP protocol zero-difference), or `CHROME_BIN=/path/to/...` env override. Firefox/Safari not supported (different protocol + launch mode loses login state). - **V0.16.19 auto-spawn Chrome**: 9222 unavailable cli/mcp_server automatically `subprocess.Popen scripts/start_chrome.sh` (start_new_session=True, exit does not kill), users do not need to start Chrome in advance — `WEB_AGENT_AUTO_SPAWN_CHROME=false` to disable. First login to Gmail still requires headed mode manual login. - Anthropic Claude Sonnet 4.6 (vision) + prompt caching; OpenAI / Kimi / OpenRouter same protocol access - Set-of-Mark (SoM) screenshot JS injection + Shadow DOM penetration (V0.12.0) + cookie/GDPR popup auto-close (V0.5.1) - Anthropomorphic actuator (3-stage Bezier + smootherstep + truncated normal typing + typo+backspace + mouse coherent landing) - ReAct loop + Action Trace (deque maxlen=20) + SQLite persistence + single-file HTML replay panel - Dual defense: `safety.py` whitelist blocks send/pay/delete + V0.5.0 anti-loop same action 3 times hard abort + V0.11.0 self-reflection page 3-step no-change soft hint ## Installation + Getting Started (3 steps, V0.16.19+ auto-spawn) ```bash # 1. Install package + browser cd web-agent uv sync uv run playwright install chromium # 2. Configure API key (choose one: Anthropic default / OpenAI / Kimi) cp .env.example .env # Edit .env: ANTHROPIC_API_KEY=sk-ant-xxx # 3. Run demo (Chrome 9222 unavailable auto-spawn, no need to start Chrome in advance) uv run python demos/wikipedia_search.py "quantum entanglement" ``` V0.16.19 auto-spawn Chrome: cli and mcp_server both `subprocess.Popen scripts/start_chrome.sh` (start_new_session=True, exit does not kill) before connecting, users **do not need to start Chrome in advance**. Env `WEB_AGENT_AUTO_SPAWN_CHROME=false` to disable. `scripts/start_chrome.sh` default `CHROME_MODE=auto`: installed `xvfb` → xvfb / has `$DISPLAY` → headed / none → `--headless=new`. SSH bare-metal recommended `apt install xvfb` to upgrade to xvfb and improve anti-scraping pass rate. From V0.16.18, `start_chrome.sh` automatically detects 11 Chromium-based binaries: Chrome / Chromium / **Brave / Edge / Vivaldi / Opera**, or `CHROME_BIN=/path/to/...` env override. Firefox/Safari not supported (different protocol + launch mode loses login state). ## 4 Integration Methods | Method | Applicable | Command | |---|---|---| | **MCP server (stdio)** | Claude Desktop / Cursor / Continue | `web-agent-mcp` (see config below) | | **CLI** | Any shell / Node / cross-language | `uv run web-agent "..."` | | **Python import** | Same venv Python agent | `from web_agent.cli import run_task` | | **demos** | Learn usage / reference implementation | `uv run python demos/<name>.py "..."` | ### CLI ```bash # Run task uv run web-agent "Search for quantum entanglement on Wikipedia and extract the first paragraph" --url https://zh.wikipedia.org/ # Replay panel (V0.8.0 W4-1 + V0.12.2 W4-1.1) uv run web-agent-replay # Render latest task → data/replays/<task_id>.html uv run web-agent-replay <task_id> # Render specific task uv run web-agent-replay --all # Render all tasks in DB + index.html index page (by started DESC) xdg-open data/replays/index.html # Open in browser ``` ```bash # Memory dump (V0.13.0 W5-D long-term memory) — cross-session same domain history query uv run web-agent-memory github.com # Default 5 entries uv run web-agent-memory github.com --limit 10 uv run web-agent-memory wikipedia.org --db data/memory.db # Output: [2026-05-03T12:34:56] OK Search trending repo[:60] -> repo: x/y, stars: 1k[:80] ``` ## Roadmap - W1 ✅ / W2 ✅ / W3 ✅ / W4 ✅ — See [`CHANGELOG.md`](CHANGELOG.md) - W5-A Self-reflection page-stuck soft hint ✅ (V0.11.0) - W5-B Shadow DOM piercing ✅ (V0.12.0) - W5-D Long-term memory cross-session episodic ✅ (V0.13.0 persistence + CLI dump) - W5-D.2 Memory injection into planner context ✅ (V0.14.0) - W5-C Hierarchical planning ✅ (V0.15.0, prompt augmentation roadmap; actual plan-and-execute = W5-C.2 **permanent DEFER**, V0.16.16 archived — SDK obstacles unresolved (OpenAI/Kimi vision cannot invoke zero screenshots) + ROI not quantified, trigger conditions 3 choose 1: ① user feedback augmentation failure cases ② OpenAI/Kimi support zero image vision ③ spike verifies plan-and-execute failure rate low >20%, see ARCHITECTURE §1.5) **MCP server** ✅ (V0.16.0 ~ V0.16.9 cumulative 10 commits, already shipped): - Exposes web-agent as MCP server (Claude Desktop / any MCP client can call `web_agent_run(goal, url)` via tool) - V0.16.0 ✅ 25 print statements → logger.info(stderr), zero business changes, 220 tests passed - V0.16.1 ✅ `mcp_server.py` uses official `mcp[cli]>=1.10` SDK to expose 3 tools + asyncio.Lock serialization + Chrome 9222 health check + 10 case tests - V0.16.4 ✅ progress_cb actually wires mcp ctx → cli → loop main loop + captcha poll heartbeat - V0.16.6 ✅ Resources (`webagent://replay/<id>` HTML + `webagent://memory/<domain>` JSON read-only view) - V0.16.7 ✅ V0.16.6 Resources `/simplify` pass - V0.16.8 ✅ ARCHITECTURE.md §5 MCP server 6 sections fully documented - V0.16.9 ✅ P1 decoupling: `Mark`/`Action` moved to `web_agent.types`, eliminating safety/llm.base reverse dependency **Future Optional** (not blocking main process): - Elicitation replaces `WEB_AGENT_AUTO_APPROVE` (human-in-the-loop approval) - HTTP transport (replaces stdio, facilitates remote MCP client) Run MCP server (add entry to Claude Desktop config): ```json { "mcpServers": { "web-agent": { "command": "uv", "args": ["--directory", "/home/myclaw/web-agent", "run", "web-agent-mcp"] } } } ``` After restarting Claude Desktop, three tools will appear: web_agent_run, web_agent_get_replay, and web_agent_query_memory. **Known Gaps** (not in main roadmap but need to be addressed): - ~~patchright-python decision~~ — V0.16.14 spike test closure: `connect_over_cdp` takeover mode patchright client patch bypass (A=C 19/32 identical), only effective in launch_persistent_context mode → conflicts with project CDP takeover core architecture, permanently NO-GO. See ARCHITECTURE §1.3 - ~~curl_cffi TLS fingerprinting~~ — V0.16.15 permanently NO-GO: web-agent takes over real Chrome, all browsing traffic goes through Chrome's own BoringSSL = real Chrome JA3/JA4, curl_cffi useless in browsing path (LLM API endpoints do not require anti-scraping). If "Python direct HTTP bypass" is introduced in W6+, reassess. See ARCHITECTURE §1.3 - **Residential proxy** (enables after Cloudflare/DataDome hit): compatible with connect_over_cdp — add `--proxy-server=` to `scripts/start_chrome.sh` ARGS, candidate IPRoyal $7/GB / Smartproxy $8.5/GB. Pit: Chrome `--proxy-server=` does not support user:pass inline credentials, requires IP whitelist mode - ~~Gmail real account end-to-end verification~~ — V0.16.17 already tested: user local 9222 Chrome (logged in state in user-data-dir) + `WEB_AGENT_TEST_RECIPIENT` + `WEB_AGENT_AUTO_APPROVE='*'` runs `demos/gmail_compose.py`, LLM completes compose process, safety auto_approve allows Send, email actually sent to inbox. See README "Run W3 demo" section - **Real LLM smoke + cassette** (V0.15.3 + V0.15.5 dual frameworks already landed, Anthropic + OpenAI(Kimi domestic version .cn)): ```bash cd /home/myclaw/web-agent # must be in project root, otherwise pytest-recording plugin not registered # Anthropic path (V0.15.3) ANTHROPIC_API_KEY=sk-ant-xxx uv run pytest tests/test_smoke_anthropic_real.py --record-mode=once # OpenAI/Kimi domestic version path (V0.15.5, platform.moonshot.cn real key, do not use sk-xxx placeholder) OPENAI_API_KEY=sk-real-key uv run pytest tests/test_smoke_openai_kimi_real.py --record-mode=once git add tests/cassettes/ # cassette header filtered, no key leakage ``` Afterwards, anyone/CI can run without keys (using cassette replay). Single recording cost: Anthropic ≈ $0.006 / Kimi ≈ ¥0.03 (~$0.004). Kimi international version .ai endpoint framework pending V0.15.6+ (modify `_KIMI_BASE_URL` and re-record or add dual framework). GPT framework V0.15.8 already landed, recording command: ```bash # Note: must explicitly set OPENAI_BASE_URL to prevent .env moonshot.cn hijacking OPENAI_BASE_URL=https://api.openai.com/v1 OPENAI_API_KEY=sk-real-OpenAI \ uv run pytest tests/test_smoke_openai_gpt_real.py --record-mode=once ``` OpenRouter / Azure / Bedrock path framework pending V0.16.0+, can be supplemented with helper template ## BYO LLM API key Supports multiple providers through `LLMClient` Protocol, selected by environment variables: ```bash # === Anthropic Claude (default) === ANTHROPIC_API_KEY=sk-ant-... WEB_AGENT_MODEL=claude-sonnet-4-6 # optional # === OpenAI GPT (install optional dependency) === # uv sync --extra openai OPENAI_API_KEY=sk-... WEB_AGENT_LLM_PROVIDER=openai # explicitly specify, or omit to auto-infer model prefix WEB_AGENT_MODEL=gpt-5.5 # or gpt-4o / o3-vision... # === Use proxy (OpenRouter / self-deployed LiteLLM / Azure OpenAI) === ANTHROPIC_BASE_URL=https://openrouter.ai/api ANTHROPIC_API_KEY=sk-or-v1-... # OpenRouter key WEB_AGENT_MODEL=anthropic/claude-sonnet-4.6 # or OpenAI-style proxy OPENAI_BASE_URL=https://openrouter.ai/api/v1 OPENAI_API_KEY=sk-or-v1-... WEB_AGENT_LLM_PROVIDER=openai WEB_AGENT_MODEL=openai/gpt-4o # === Kimi / Moonshot (OpenAI compatible) === # International version https://platform.kimi.ai OPENAI_API_KEY=sk-xxx OPENAI_BASE_URL=https://api.moonshot.ai/v1 WEB_AGENT_MODEL=kimi-k2.6 # auto-infers provider=openai # Domestic version https://platform.moonshot.cn use https://api.moonshot.cn/v1 ``` OpenAIClient automatically detects Kimi compatibility patches: - `max_completion_tokens` → `max_tokens` (Kimi does not recognize GPT-5.x new parameter name) - `tool_choice="required"` → `"auto"` (Kimi rejects required) - Single step cost (kimi-k2.6, ~3k input + 200 output): cache miss ≈ $0.004, cache hit ≈ $0.001 (automatic cache, 6× discount) CLI can also temporarily override: ```bash uv run web-agent "..." --provider openai --model gpt-5.5 ``` Provider auto-inference rules (`provider_from_model`): - `claude-*`, `anthropic/*` → anthropic - `gpt-*`, `o[1-5]-*`, `openai/*` → openai - `gemini-*` → gemini (not implemented client, reserved for extension) - others → default anthropic ## Behavioral Switches (environment variables) Grouped by category, default values suitable for most scenarios: ```bash # === Safety / pop-ups === WEB_AGENT_AUTO_APPROVE= # default empty (blocks send/pay/delete); set rule names CSV to allow (e.g. "send-or-pay") # or "*" to fully open (use with caution in production); see src/web_agent/safety.py WEB_AGENT_AUTO_DISMISS=true # automatically dismiss cookie/GDPR/notification pop-ups (V0.5.1; blacklist includes password/pay protection) # === Captcha takeover UX (W4-2) === WEB_AGENT_CAPTCHA_DISABLE= # default empty (enabled); true/1/yes degrades to V0.8.x light-DOM-only behavior WEB_AGENT_CAPTCHA_TIMEOUT_S=300 # user manual captcha solving timeout in seconds WEB_AGENT_CAPTCHA_POLL_S=3 # poll interval in seconds to check if captcha is cleared # === Perception (V0.12.0 W5-B) === WEB_AGENT_SOM_SHADOW=true # SoM uses open shadowRoot piercing; false degrades to V0.11.x light-DOM-only WEB_AGENT_AUTO_DISMISS=true # (see above) # === Notify (W4-3) === WEB_AGENT_NOTIFY_DISABLE= # default empty (enabled); true disables desktop notifications (CI/headless/do not disturb) # === Memory (V0.13.0 W5-D) === WEB_AGENT_MEMORY_DISABLE= # default empty (enabled); true disables cross-session task outcome persistence WEB_AGENT_MEMORY_DB=data/memory.db # custom memory db path (default data/memory.db) # === Reliability === WEB_AGENT_MAX_WALLCLOCK_S=300 # single task hard timeout (avoids SDK retry + perceive accumulation exceeding max_steps × average step consumption) WEB_AGENT_CDP_URL=http://127.0.0.1:9222 # taken-over Chrome debugging port # === Demo-specific === WEB_AGENT_TEST_RECIPIENT= # gmail_compose demo recipient (W3-C, strongly suggest sending to yourself) ``` ## Anti-detection Layer (upgrade as needed) W1 uses `playwright-stealth` 2.0.3 + V0.16.14 SwiftShader GL flags (`scripts/start_chrome.sh`). sannysoft.com tested ~72% pass rate. If encountering Cloudflare/Datadome/Akamai: 1. ~~Switch to patchright-python~~ — V0.16.14 spike test closure, see ARCHITECTURE §1.3 2. **Use residential proxy** (Chrome `--proxy-server=` flag, compatible with connect_over_cdp) — actual next-layer defense. IP reputation is CF's first gate, Chrome's real TLS fingerprint can't save DC IP segment 3. ~~curl_cffi TLS fingerprinting~~ — V0.16.15 NO-GO, Chrome already uses real BoringSSL JA3/JA4, curl_cffi useless in browsing path, see ARCHITECTURE §1.3 4. Do not bypass 2Captcha automatically (cross-line), use "pause → pop-up user solve → resume loop" UX (W4-2 V0.9.0 already implemented) ## Legal Boundaries - ✅ Operate on own accounts / own websites / public data / personal assistance - ❌ Violate ToS scraping / credential stuffing / ticket brushing / bypass reCAPTCHA scraping commercial sites ## Directory ``` src/web_agent/ __init__.py # __version__ browser.py # CDP takes over local Chrome + stealth perceiver.py # SoM JS injection + Shadow DOM piercing + auto-dismiss pop-ups actuator.py # human-like click/type/scroll (3rd-order Bezier + normal keystrokes) loop.py # ReAct + safety + captcha + reflection + anti-loop + trace trace.py # SQLite persistence + Step/Trace dataclass safety.py # W3-A authorization whitelist (send/pay/delete/sensitive fields) captcha.py # W4-2 Cloudflare/reCAPTCHA/hCaptcha detection + pause UX notify.py # W4-3 desktop notifications (osascript / notify-send) replay.py # W4-1/W4-1.1 replay HTML panel + index page memory.py # W5-D cross-session long-term memory (domain → past goals/results, SQLite) cli.py # combined root + web-agent / web-agent-replay / web-agent-memory entry llm/ # cross-provider Protocol (anthropic / openai / Kimi compatible) demos/ wikipedia_search.py # W1 github_search.py # W2-B gmail_summary.py # W3-B (read-only) gmail_compose.py # W3-C (write, safety blocks Send by default) scripts/ start_chrome.sh # launch 9222 debugging port independent Chrome (auto/xvfb/headed/headless) tests/ # 255 passed + 2 skipped, 23 files (including audit gap 6/6 + W5-D + W5-C + W5-C.2 spike + Anthropic/Kimi/GPT triple framework) conftest.py # vcr_config locks cassette filter (V0.15.3) cassettes/ # vcrpy yaml (V0.15.3, .bak gitignored, main yaml committed) docs/ roadmap.txt # original technical roadmap ARCHITECTURE.md # V0.16.22 architecture decisions / module boundaries / triple-track dual-layer defense / W5-C.2 spike 7-version closed-loop verdict data/ trace.db # SQLite trace (gitignored) memory.db # W5-D cross-session long-term memory (gitignored) screenshots/ # step-by screenshots (gitignored) replays/ # web-agent-replay output HTML (gitignored) ``` ## CHANGELOG See [`CHANGELOG.md`](CHANGELOG.md).