Manos-MCP

# Manos A CLI [Model Context Protocol](https://modelcontextprotocol.io) server for **ad-hoc UI testing of Android & iOS apps** — purpose-built for the exploratory, test-free loop where an LLM agent pokes at an app and reacts to what it sees. It controls Android emulators/devices (`adb`) and iOS simulators (`xcrun simctl`, with `idb` for native UI interaction) and exposes 39 tools over stdio. manos gives an agent a tight **act → observe** loop, **device-condition control**, **crash/log capture**, **network capture** for debug builds, **OCR targeting** for off-tree elements, an **accessibility audit**, and **session recording** that promotes an ad-hoc exploration into a replayable regression test in one call. See [IMPROVEMENTS.md](IMPROVEMENTS.md) for the design rationale and roadmap. ## Install ```bash npm install -g manos-mcp # install the CLI, or run on demand with: npx -y manos-mcp manos doctor # check toolchain + list devices & capabilities ``` Requires **Node 20+**. Most MCP clients can launch manos on demand with `npx` — no global install needed; see [Register with an MCP client](#register-with-an-mcp-client). Working on manos itself? See [From source](#from-source). | Backend | Used for | Install | | --- | --- | --- | | `adb` | all Android control | Android platform-tools (auto-detected from `$ANDROID_HOME` or the default SDK path) | | `xcrun simctl` | iOS lifecycle, conditions, logs, screenshots, push | `xcode-select --install` | | `idb` | fast native iOS UI inspect/tap/type | `brew install idb-companion && pipx install fb-idb` | | `maestro` | under the hood: `run_flow`, the warm hierarchy engine, and the cross-platform inspect/interaction fallback | <https://maestro.dev> | iOS UI interaction works **without** `idb` by falling back to Maestro (slower; call `launch_app` first). Android UI inspection falls back from `uiautomator dump` to `maestro hierarchy` automatically when the on-device UiAutomation connection is contended. ## Register with an MCP client Each tab launches manos with `npx -y manos-mcp serve`, which fetches and runs the published package on demand — no clone or global install required. (If you installed globally with `npm install -g manos-mcp`, use a bare `manos serve` instead.) <details open> <summary><b>Claude Code</b></summary> ```bash claude mcp add manos -- npx -y manos-mcp serve claude mcp list # verify ``` </details> <details> <summary><b>Claude Desktop</b></summary> Edit `claude_desktop_config.json` (macOS `~/Library/Application Support/Claude/`, Windows `%APPDATA%\Claude\`) and restart the app: ```json { "mcpServers": { "manos": { "command": "npx", "args": ["-y", "manos-mcp", "serve"] } } } ``` </details> <details> <summary><b>Cursor</b></summary> `.cursor/mcp.json` (project) or `~/.cursor/mcp.json` (global), then enable **manos** under Settings → MCP: ```json { "mcpServers": { "manos": { "command": "npx", "args": ["-y", "manos-mcp", "serve"] } } } ``` </details> <details> <summary><b>VS Code (GitHub Copilot agent mode)</b></summary> `.vscode/mcp.json` — note the `servers` key and explicit `type`: ```json { "servers": { "manos": { "type": "stdio", "command": "npx", "args": ["-y", "manos-mcp", "serve"] } } } ``` </details> <details> <summary><b>Windsurf</b></summary> `~/.codeium/windsurf/mcp_config.json`, then hit **Refresh** in the Windsurf MCP panel: ```json { "mcpServers": { "manos": { "command": "npx", "args": ["-y", "manos-mcp", "serve"] } } } ``` </details> <details> <summary><b>Other / generic MCP client</b></summary> Any MCP-capable client speaks the same stdio protocol. Configure a server that runs: ``` command: npx args: ["-y", "manos-mcp", "serve"] transport: stdio ``` </details> ## From source For working on manos itself, or to pin a local build instead of the published package: ```bash git clone https://github.com/ryanperkins/Manos-MCP.git && cd Manos-MCP npm install # also builds via the prepare script npm run build # or rebuild after changes node dist/cli.js doctor ``` Register the local build by pointing your client's `command`/`args` at `node /ABS/PATH/to/Manos-MCP/dist/cli.js serve` instead of the `npx` form above. ## CLI ``` manos serve Start the MCP server on stdio (default) manos doctor Toolchain + connected devices + per-device capabilities manos devices List connected devices (tab-separated) manos --help ``` ## The tools Full reference and the **Android vs iOS comparison matrix** are in [docs/index.html](docs/index.html). Highlights: - **Core:** `list_devices`, `device_capabilities`, `inspect_screen`, `take_screenshot`. - **Authored flows:** `run_flow` runs a declarative flow locally; `cheat_sheet` gives the syntax. `export_flow` (below) turns a recorded session into one. - **Act + observe:** `tap`, `long_press`, `input_text`, `press_key`, `swipe` — each takes a selector (`id`/`text`/`resource_id`/`accessibility`) **or** coordinates and returns the resulting screen (`observe: screen | diff | screenshot | none`). - **Smart waits / assertions / search:** `wait_for`, `assert`, `find_elements`, and `find_text` (OCR the screenshot to locate text the accessibility tree misses — styled buttons, canvas/Flutter/game UIs, WebViews). Targeting falls back to OCR automatically when a text selector finds nothing in the tree, or force it with `tap{text, ocr:true}`. - **App state:** `launch_app`, `stop_app`, `clear_app_state`, `open_deeplink`, `set_permission`. - **Device conditions:** `set_appearance`, `set_orientation`, `set_locale`, `set_network`, `set_location`, `set_font_scale`, `set_status_bar`, `push_notification`, and `set_conditions` (apply many at once / named presets like `offline`, `accessibility`, `screenshot`). - **Diagnostics:** `get_logs` (with crash/ANR detection), `a11y_audit`. - **Network capture (debug apps):** `network_start`/`network_requests`/`network_clear`/`network_stop` — capture decrypted HTTP filtered to specific endpoints. Android hooks OkHttp via Frida (works through HTTP/2, pinning, proxy-bypass); iOS Simulator uses mitmproxy + a `simctl`-trusted CA. See [NETWORK.md](NETWORK.md). - **Network mocking (Android):** `network_mock` — manipulate live API responses to test hard-to-reproduce states: override `status`/`headers`, replace the `body`, **regex-`rewrite` a field in the live body** (keep the rest), or inject latency. Rule-based (URL regex + method), hot-reloads, no separate mock server or build change. Rides the Frida/OkHttp capture hook (per-process, no host-proxy disruption — `network_start` first). iOS response mocking is in development. See [NETWORK.md](NETWORK.md#mocking-responses-android--network_mock). - **Recording:** `start_recording` → act → `export_flow` (replayable Maestro flow) or `export_report` (self-contained HTML report: screenshot timeline + flow + logs + captured network). A typical loop: ``` list_devices → inspect_screen → tap{text:"Login", observe:"diff"} → input_text{...} → wait_for{text:"Welcome"} ``` ## How element targeting works `inspect_screen` returns a compact tree where every node has a **stable `id`** derived from its semantic identity (resource-id / accessibility / class + digit-normalized text), not its position. So a counter ticking from `5` to `6` keeps its id and shows up as a *changed* node in a diff, while a newly-appeared element is *added*. Act tools accept that `id`, a text/resource-id selector, or raw coordinates; when you use a selector, the recorded flow stores the selector (resilient replay) rather than brittle coordinates. ## Performance Hierarchy reads use a three-tier backend, chosen per device: 1. **adb `uiautomator dump`** (~2.5s, no extra process) — the default on Android. Tried first. 2. **Warm hierarchy engine** — when uiautomator can't reach UI-idle (e.g. apps with constant animations/watermarks, where `uiautomator dump` errors with `could not get idle state`), manos keeps **one long-lived `maestro mcp` child** resident (used under the hood) and reuses its connected driver. First call pays a one-time warm-up; subsequent inspects are **~150–300ms** — the payoff of reusing a resident engine instead of cold-starting the JVM per call. 3. **Cold `maestro hierarchy` CLI** — last resort if the warm session can't start. Once a device needs the warm session, manos remembers it (per-device) so it doesn't re-pay the uiautomator timeout on every inspect. Screen size/density are cached. The warm child (and its `simulator-server`) are killed via a process-tree cleanup on exit. Measured on a hard case (an app that never idles, so everything routes through the warm session): | | First inspect (one-time) | Steady-state inspect (median) | Full tap+observe loop | | --- | --- | --- | --- | | **manos** | ~11s | **~175ms** | **~3s** (was ~35s with cold per-call CLI) | So: on apps where uiautomator works, the adb path is fast with no extra process; on apps that force the fallback, the resident warm engine keeps steady-state inspect in the **~175ms** range, and the act+observe loop avoids the per-action JVM cold-start that made it slow before. The remaining first-call cost is the one-time uiautomator probe before switching to warm. ## Architecture ``` src/ cli.ts serve / doctor / devices server.ts McpServer wiring (stdio) tools/ register.ts all 39 tools context.ts shared state: resolveTarget + act/observe + last-screen cache drivers/ types.ts Driver contract + Capability model android.ts adb ios.ts simctl + idb (+ maestro fallback) registry.ts device → driver routing core/ hierarchy.ts compact JSON, stable ids, screen diff, search a11y.ts accessibility heuristics waits.ts condition polling session.ts action journal flow.ts Maestro-flow emitter maestro.ts maestro CLI passthrough (run_flow, cheat sheet) maestroDriver.ts cold maestro hierarchy + one-shot action fallbacks maestroSession.ts warm long-lived `maestro mcp` backend (fast hierarchy/actions) netcapture.ts network capture (Frida OkHttp / mitmproxy) — see NETWORK.md ocr.ts OCR fallback (Apple Vision / Tesseract) for off-tree elements assets/frida/ okhttp-capture.js + sidecar.py (injected by netcapture) util/ exec + toolchain discovery ``` Each driver method may throw a `CapabilityError`; tools surface it as an actionable message instead of an opaque subprocess failure. The capability model is reported live so an agent can check support before relying on a platform-specific action. ## Test ```bash npm test # unit tests for hierarchy/diff/a11y/flow (no device needed) ``` The hierarchy parsing, stable-id diffing, accessibility math, screenshot capture, log/crash scan, and the recording→`export_flow`→`maestro check-syntax` pipeline have also been verified end-to-end against a live Android emulator. ## License MIT