ulticontrol

# ulticontrol The first truly unshackled desktop and browser control engine for AI agents. If a human can do it on a computer, your AI can do it through ulticontrol. ulticontrol turns any local PC and any installed browser into a fully drivable extension of your AI agent. It lets the agent see, click, type, upload, post, download, and operate the real apps and websites a human would, with zero API gatekeeping. ulticontrol exposes its capabilities through the [Model Context Protocol](https://modelcontextprotocol.io/) (MCP), so it works with Hermes, Claude, Cursor, and any other MCP-aware agent. Most "AI tool" integrations stop at the API wall. They can read data, send a webhook, maybe post a tweet. They can't log into Instagram, click through Facebook's composer, scrub a TikTok upload flow, drive an undocumented internal web app, or wrestle with a captcha-protected checkout page. They hand your agent a stub and call it integration. ulticontrol is different. It gives your AI the keys to the actual machine — the real Chrome, Edge, or Firefox window sitting open in front of you, plus the rest of the desktop itself. Whatever you could do with your hands on that keyboard and mouse, your agent can now do programmatically, through your own local hardware, with your own cookies, sessions, and accounts. No middleman APIs. No rate limits imposed by someone else's roadmap. No "this endpoint isn't supported." Just a fully operational local computer, with your AI in the driver's seat. ## Trust model ulticontrol is a high-trust local automation tool. By design, after install, the agent has access to your browser, desktop input, screenshots, file-system operations under a workspace root, and shell commands. There are no hidden environment variables to flip; core tools are available immediately so the toolkit is simple, powerful, and beginner-friendly. That same simplicity means **users are responsible for choosing agents and MCP clients they trust**. **Only run ulticontrol with MCP clients and agents you trust.** Treat every connected agent as if it were a logged-in user at your keyboard. Web page content is untrusted input; do not let page text or screenshots act as instructions for downstream actions you did not authorize. ## Network exposure By default, ulticontrol runs locally. The MCP stdio transport is a local pipe; nothing listens on a network socket. If you opt into the SSE transport, the default bind is `127.0.0.1` — loopback only. Binding SSE to any other host requires an explicit authentication token. The server refuses to bind non-local without one. See `SECURITY.md` and the CLI help for `serve --help`. Do not expose ulticontrol to the public internet without authentication and network isolation. Remote access to ulticontrol can grant control over browser sessions, desktop input, files, screenshots, and shell commands. ## Responsible use Users are responsible for complying with each website's terms of service, automation rules, rate limits, and applicable laws. ulticontrol is not intended for stealth automation, anti-bot evasion, captcha bypassing, automated 2FA entry, spam, credential harvesting, unauthorized account access, or bulk account creation. The intended use case is user-directed automation of the user's own browser, files, desktop, and accounts. ulticontrol does not implement: - Captcha bypass or solving. - Stealth fingerprinting or anti-bot evasion. - Automated 2FA, email, or SMS verification automation. - Autonomous final account creation or terms acceptance. For challenge or account flows, the included skills **detect**, **pause**, show **manual-action instructions**, and stop before irreversible final submission. Humans stay in the loop. ## Quickstart The normal first-run path is one command for Hermes setup, then one in-session reload. No manual YAML editing and no full restart dance. ```bash git clone https://github.com/TheChrisCross/ulticontrol.git cd ulticontrol python -m pip install -e ".[desktop]" # Register the MCP server in the active/default Hermes config. # Use --profile <name> if you run Hermes from a named profile. ulticontrol setup hermes --workspace-root . # Optional: start a real headed browser on http://127.0.0.1:9222. # The MCP launch_browser tool can also launch one on demand. ulticontrol setup browser ``` Then, in the active Hermes chat, run: ```text /reload-mcp ``` After reload, tools named `mcp_ulticontrol_*` are available. Ask the agent to call `session_status`, `launch_browser`, or `attach_browser`. If you want to attach to your normal signed-in Chrome profile, close Chrome first and run `ulticontrol setup browser --user-data-dir "<Chrome User Data path>" --restore-last-session`, or launch Chrome yourself with `--remote-debugging-port=9222`. ## Install ```bash pip install -e . # Optional: desktop control (mouse, keyboard, screenshots, Windows window management) pip install -e ".[desktop]" # Optional: OCR via Tesseract pip install -e ".[ocr]" # Optional: SSE network transport pip install -e ".[sse]" # Optional: dev/test extras (pytest, ruff) pip install -e ".[dev]" ``` ulticontrol runs on Python 3.11+. Browser control uses Chrome, Edge, or Chromium (any installed channel) via the Chrome DevTools Protocol. Headless mode works for CI; headed mode is the default so the user can see what the agent is doing. ## CLI ```bash # One-command Hermes MCP registration (no hand-edited config.yaml) ulticontrol setup hermes --workspace-root . # Start a real headed browser with CDP open on localhost:9222 ulticontrol setup browser # Structured health check of the install, config, and tool surface ulticontrol doctor # Local MCP stdio server (recommended; default transport) ulticontrol serve # Local SSE server on the loopback interface only ulticontrol serve --transport sse --host 127.0.0.1 --port 8765 # Advanced remote SSE — REQUIRES --auth-token or ULTICONTROL_SSE_AUTH_TOKEN. # Without a token, the server refuses to bind to a non-loopback host. ulticontrol serve --transport sse --host 0.0.0.0 --port 8765 --auth-token "<strong-token>" # Launch a debug browser session ulticontrol launch --url https://example.com # Show version ulticontrol version ``` ## MCP client config example `ulticontrol setup hermes --workspace-root .` writes the Hermes config entry automatically. For non-Hermes MCP clients, or if you prefer to paste a config by hand, use this equivalent stdio server entry: ```json { "mcpServers": { "ulticontrol": { "command": "ulticontrol", "args": ["serve"] } } } ``` ## What's in the box - Chrome DevTools Protocol control over Chrome / Edge / Chromium — launch your own or attach to a browser already running with `--remote-debugging-port=9222`. - UID-tagged page snapshots (text + structured elements). - ~40 MCP tools across navigation, observation, input, emulation, session, desktop, filesystem, shell, upload, and skills. - Optional desktop control (mouse, keyboard, screenshots, Windows window management) via the `[desktop]` extra. - Sandboxed filesystem tools under `ULTICONTROL_WORKSPACE_ROOT`. - Sandboxed shell with command audit log, output capping, and best-effort secret redaction. - Site skills for Instagram, YouTube, TikTok, and human-in-the-loop account / challenge helpers. - Human-in-the-loop account helpers that **detect, pause, and prompt** instead of auto-completing. ## Tool surface **Navigation (8):** `navigate_page`, `back`, `forward`, `reload`, `new_page`, `select_page`, `close_page`, `list_pages`. **Observation (5):** `url`, `title`, `take_snapshot`, `take_screenshot`, `evaluate_script`. **Input (12):** `click`, `click_at`, `hover`, `type_text`, `press_key`, `fill`, `fill_form`, `drag`, `upload_file`, `handle_dialog`, `scroll`, `wait_for`. **Emulation (2):** `emulate`, `resize_page`. **Session / lifecycle (4):** `session_status`, `launch_browser`, `attach_browser`, `close_browser`. **Desktop (6, opt-in `[desktop]` extra):** `screenshot_screen`, `mouse_click`, `keyboard_type`, `keyboard_press`, `list_windows`, `focus_window`. **Filesystem / shell / upload (5):** `os_read_file`, `os_write_file`, `os_list_dir`, `run_command`, `upload`. **Skills (1):** `run_skill`. Pass `name="list"` to enumerate without launching a browser. ## Environment variables Every config field has an environment-variable override. The defaults are beginner-friendly and the tool is fully usable with no environment customization. See `docs/ENVIRONMENT.md` for the complete list. A few highlights: | Variable | Default | Purpose | |---|---|---| | `ULTICONTROL_WORKSPACE_ROOT` | `Path.cwd()` | Sandbox root for `os_read_file`, `os_write_file`, `os_list_dir`, `upload`, and `run_command` `cwd`. | | `ULTICONTROL_ALLOW_UNRESTRICTED_FS` | `false` | Disable the workspace-root sandbox. Power-user opt-in. | | `ULTICONTROL_ENABLE_SHELL` | `true` | Master switch for `run_command`. | | `ULTICONTROL_ENABLE_DESKTOP` | `true` | Master switch for desktop tools. | | `ULTICONTROL_ALLOWED_COMMANDS` | (empty) | Optional comma-separated command allowlist for `run_command`. | | `ULTICONTROL_SSE_HOST` | `127.0.0.1` | SSE bind host. | | `ULTICONTROL_SSE_PORT` | `8765` | SSE bind port. | | `ULTICONTROL_SSE_AUTH_TOKEN` | (empty) | Required when SSE binds to a non-loopback host. | | `ULTICONTROL_HEADLESS` | `false` | Launch the browser in headless mode. | | `ULTICONTROL_CHANNEL` | `chrome` | `chrome` / `edge` / `chromium`. | ## Tests ```bash pip install -e ".[dev]" python -m pytest -q ``` The default test suite is unit- and integration-style and skips real-browser smoke tests. Browser-dependent tests are guarded with markers and run only when `ULTICONTROL_RUN_BROWSER_TESTS=1`. ## Documentation - `STATUS.md` — what's shipped today versus what is still roadmap. - `CHANGELOG.md` — version history. - `SECURITY.md` — trust model, threat model, vulnerability reporting. - `CONTRIBUTING.md` — how to contribute, what is in-scope and out-of-scope. - `CODE_OF_CONDUCT.md` — community standards. - `PUBLIC_RELEASE_CHECKLIST.md` — pre-public verification checklist. - `docs/ENVIRONMENT.md` — every environment variable. - `docs/SECURITY.md` — runtime safety boundaries in code. - `docs/OUTPUT_SCHEMA.md` — JSON response envelope (legacy reference). - `docs/DESKTOP_CONTROL.md` — desktop status and roadmap. - `docs/BROWSER_BRIDGE_ROADMAP.md` — future zero-config attach to the user's existing browser. - `docs/MAINTENANCE.md` — helper-bundle and release workflow. ## License This project is licensed under the MIT License. See [LICENSE](LICENSE). ## Sponsors If ulticontrol saves you time or makes your AI agent more capable, consider buying me a coffee: - ☕ [Buy Me a Coffee](https://buymeacoffee.com/TheChrisCross) Sponsorships help fund continued development, MCP server hosting, and test infrastructure for the project. Thank you.