PhantomHand

# PhantomHand PhantomHand is a browser control engine for AI agents. It exposes the browser's perception and control capabilities to large language models through the [Chrome DevTools Protocol](https://developer.chrome.com/docs/devtools/) and the [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) standard, allowing agents to visually understand web pages and interact with them as humans do. > All commands default to the browser's **currently active tab**, but you can also specify an arbitrary tab using the explicit `tabId` parameter. ## Core Features - **Reuse a logged-in real browser**: Operate directly on your daily Chrome, preserving login state, plugins, and bookmarks - no need to open a separate isolated profile. - **True visual perception**: Set-of-Mark annotated screenshots can recognize canvas, video, and complex CSS-rendered content (which pure accessibility trees can't see). - **Stable ref references**: Each interactable element is assigned a stable `ref`, which is used to resolve its **current** coordinates during manipulation, even if the page is slightly rearranged. - **Human-like and anti-detection**: Bézier mouse trajectories, humanized typing rhythms, and pre-click aiming pauses; based on `chrome.debugger` rather than WebDriver, and automatically erases automation fingerprints in the page's main world. - **Information density prioritization**: Text snapshots, batch structured extraction, page smart summaries, and page identity API requests; large volume products are stored or summarized by default, without context reflux. - **Observability**: console, network requests, performance metrics, and Web Storage are all refluxed. ## Quick Start Register `phantom-hand` in your AI Agent's (Claude Desktop, Cursor, Windsurf, Antigravity, etc.) MCP configuration. Choose one of two deployment methods. ### A. Install from npm (released) ```json { "mcpServers": { "phantom-hand": { "command": "npx", "args": ["-y", "phantom-hand-mcp"] } } } ``` ### B. Run from local source code (development / unreleased) Clone the repository, build it, and register the product's **absolute path**: ```bash npm install && npm run build ``` ```json { "mcpServers": { "phantom-hand": { "command": "node", "args": ["<absolute path to project root>/dist/index.cjs"] } } } ``` ### Agent Configuration File Locations | Agent | Configuration File | |---|---| | Claude Desktop (Windows) | `%APPDATA%\Claude\claude_desktop_config.json` | | Claude Desktop (macOS) | `~/Library/Application Support/Claude/claude_desktop_config.json` | | Cursor | Workspace `.cursor/mcp.json` or MCP GUI in settings | | Windsurf | `~/.codeium/windsurf/mcp_config.json` | | Antigravity | `~/.gemini/config/mcp_config.json` | Save and restart the Agent, and the service will start in the background and listen on `ws://localhost:37210`. ## Install Browser Extension (Required) PhantomHand communicates with Chrome through a browser extension, which needs to be loaded manually: 1. Have your AI assistant call the `get_extension_info` tool to get the absolute path of the extension directory on your machine (located in `~/.npm/_npx/.../extension`). 2. Open `chrome://extensions/`, enable developer mode, click "Load unpacked", and select the directory from step 1. 3. Click the PhantomHand icon in the browser toolbar, keep the URL as `ws://localhost:37210`, click "Connect to Agent Server", and the indicator will turn green to indicate a successful connection. ## Tool List | Category | Tools | |---|---| | Meta Information | `get_extension_info` | | Perception | `browser_scan`, `browser_snapshot` (text snapshot, no image), `browser_screenshot`, `browser_get_page_info`, `browser_query_ref` (precise ref retrieval by selector) | | Mouse | `browser_click`, `browser_click_ref`, `browser_double_click`, `browser_triple_click`, `browser_right_click`, `browser_hover`, `browser_drag` | | Keyboard / Clipboard | `browser_type`, `browser_type_ref`, `browser_press_key`, `browser_keyboard_shortcut`, `browser_select_all`, `browser_clipboard_set`, `browser_clipboard_get` | | Scrolling | `browser_scroll`, `browser_scroll_to_element` | | DOM / JS | `browser_eval_js`, `browser_get_html`, `browser_extract_text`, `browser_get_attributes`, `browser_extract_structured` (batch → JSON), `browser_read_page` (smart page summary) | | Form | `browser_set_value`, `browser_clear_value`, `browser_select_option`, `browser_check`, `browser_focus`, `browser_upload_file` | | Tab / Navigation | `browser_navigate`, `browser_go_back`, `browser_go_forward`, `browser_reload`, `browser_tab_new`, `browser_tab_close`, `browser_tab_list`, `browser_tab_switch` | | Observability · debug | `browser_list_console_messages` | | Observability · network | `browser_list_network_requests`, `browser_get_network_request`, `browser_fetch` (API request with page identity), `browser_mock_request`, `browser_clear_mocks` | | Observability · performance | `browser_performance_start_trace`, `browser_performance_stop_trace` | | Storage | `browser_storage_get`, `browser_storage_set`, `browser_storage_remove` | | Capture / Simulation | `browser_print_to_pdf`, `browser_set_viewport`, `browser_clear_viewport`, `browser_screencast_start`, `browser_screencast_stop` | | Session | `browser_session_export`, `browser_session_import` | | Timing | `browser_wait`, `browser_wait_for_element` (precise waiting for element appearance/disappearance), `browser_fast_forward`, `browser_wait_for_download` | ## Core Paradigm ### Ref References (Recommended Element Manipulation Method) `browser_scan` / `browser_snapshot` assigns a stable ref to each interactable element (e.g., `[ref: 3]`). Prioritize using `browser_click_ref` / `browser_type_ref` by ref - it resolves the element's **current** coordinates, even if the page is slightly rearranged. If you know the target selector, you can directly use `browser_query_ref` to get the ref and skip the full-page scan. ### Penetrating Same-Origin iframes Perception recursively enters shadow DOM and **same-origin** iframes: elements within iframes are also assigned refs, and their coordinates are calculated to the main page viewport, allowing direct ref manipulation without considering iframe boundaries and coordinate offsets. Cross-origin iframes (e.g., third-party captchas) have inaccessible DOM and won't appear in perception results. ### Information Density To avoid context expansion, large volume products default to **storage or summarization**: - `browser_print_to_pdf` writes PDF to the download directory, returning only the path; - `browser_storage_get` returns only key name lists if no key is provided; - `browser_screencast_start` / `browser_screencast_stop` only count frames, without transmitting images; - `browser_scan` prompts "X / Y elements in viewport" on long pages, informing how many are not displayed. When collecting content in bulk, prioritize high-density tools: `browser_extract_structured` (batch structured), `browser_read_page` (page summary), `browser_fetch` (direct site API connection). ### Automatic Waiting Navigation and click/input default to auto-wait: navigation waits for page `complete`, interaction waits for network idle (`networkIdle`). For delayed rendering elements in SPA, use `browser_wait_for_element` to precisely wait for its appearance, attachment, or disappearance. ### Anti-Detection PhantomHand is based on `chrome.debugger` (CDP) to control the browser, without WebDriver fingerprints. After connecting to a tab, it automatically injects a lightweight patch into the page's **main world** (via `Page.addScriptToEvaluateOnNewDocument`, taking effect before any site scripts): - `navigator.webdriver` → `undefined` - Complete `window.chrome.runtime` - Regularize `navigator.languages` / `navigator.plugins` - Correct `permissions.query` leakage for `notifications` Injection is idempotent per tab, automatically cleaned up when the tab is closed, and enabled by default without configuration. Mouse movement uses Bézier curves with random jitter, typing differentiates pauses by character type (long pauses for sentence-ending punctuation, pauses for commas), and clicks have a short aiming pause. ## Configuration All configurations are driven by environment variables (see `.env.example`): | Variable | Default | Description | |---|---|---| | `PORT` | `37210` | WebSocket port | | `PHANTOM_HAND_TOKEN` | (empty) | WebSocket authentication token | | `PHANTOM_HAND_SCREENSHOT_FORMAT` | `jpeg` | Screenshot re-encoding format (jpeg/webp/png, requires `sharp`) | | `PHANTOM_HAND_SCREENSHOT_QUALITY` | `75` | Screenshot quality 1-100 | | `PHANTOM_HAND_SCREENSHOT_MAX_WIDTH` / `_HEIGHT` | `0` (unlimited) | Maximum screenshot size, scaled down if exceeded | | `PHANTOM_HAND_AUTO_WAIT_TIMEOUT` | `3000` | auto-wait timeout upper limit (ms), `0` to disable | | `PHANTOM_HAND_DOMAIN_ALLOWLIST` | (empty) | Domain whitelist (comma-separated), set to only allow these domains | | `PHANTOM_HAND_DOMAIN_BLOCKLIST` | (empty) | Domain blacklist (comma-separated) | | `PHANTOM_HAND_CATEGORY_DEBUG` / `_NETWORK` / `_PERFORMANCE` / `_STORAGE` | `true` | Set to `false` to hide entire tool categories, reducing agent selection | **Screenshot Compression (Optional)**: Default is very lightweight. After installing `npm i sharp`, screenshots are re-encoded with the above parameters (webp/jpeg significantly smaller than png), reducing context occupation; without installation, original images are returned, and functionality is unaffected. ## Security - The service listens on `ws://localhost` by default, accessible only locally. **Do not expose the service to the public or untrusted remote addresses** - tools like `browser_eval_js` have remote code execution privileges equivalent to the local browser. - Enable authentication: set `PHANTOM_HAND_TOKEN=<key>`, and append `?token=<key>` to the extension popup URL. The service will reject connections without the correct token. - Unless you have complete control over the remote host, do not fill in `wss://remote address` in the extension popup. ## Multi-Agent Concurrency (Proxy Broker) Multiple host agents (e.g., running Cursor and Claude Desktop simultaneously, each starting phantom-hand) can share the same browser extension: 1. The first process occupies the `37210` port, becoming the **Master**, hosting the WebSocket service and communicating with the extension. 2. Subsequent processes discover the occupied port, automatically acting as **Proxy**, forwarding commands. Each request/response is routed through a unique `requestId`, and concurrent calls do not interfere with each other. If you want multiple agents to control independent tabs, pass an explicit `tabId` in tool calls (retrievable via `browser_tab_list`). ## Development ```bash npm install # Install dependencies npm run build # esbuild package to dist/index.cjs npm run dev # Run src/index.ts directly with tsx npm start # Run compiled product npm test # Unit tests npm run lint # tsc --noEmit type checking ``` End-to-end functional verification is in [`test/smoke-test.md`](./test/smoke-test.md). ## FAQ 1. **No browser response after sending instructions?** Chrome prohibits extensions from executing debugging APIs on system protection pages (e.g., `chrome://newtab`, `chrome://settings`, `chrome://extensions`, Web Store). Ensure you focus on a regular webpage. 2. **Port occupied?** If `37210` is occupied, the service won't terminate the occupied process; instead, it continues working in Proxy mode (see above). To change ports: set `PORT=xxxx` and fill the corresponding port in the extension popup. 3. **Browser always shows "Debugging this browser" yellow bar?** This is a normal CDP debugging phenomenon. The extension automatically detaches the debugger when the tab is closed; if you need to clear it immediately, close the corresponding tab. 4. **Clipboard read occasionally fails?** Modern browsers require user gestures for clipboard reads. `browser_clipboard_get` may fail in gesture-less contexts - a browser security policy. 5. **Operation by ref reports "ref not found"?** Ref does not survive scans. Re-scan with `browser_scan` / `browser_snapshot` / `browser_query_ref` to get a new ref before operation.