AndroidUse

<p align="center"> <img src="assets/androiduse-logo.png" alt="AndroidUse logo" width="160"> </p> # AndroidUse <div align="center"> <p> <a href="Cargo.toml"><img alt="Rust" src="https://img.shields.io/badge/Rust-2021-f46623?logo=rust&logoColor=white" /></a> <a href="Cargo.toml"><img alt="MIT License" src="https://img.shields.io/badge/License-MIT-green.svg" /></a> <a href="docs/guide/what-is-androiduse.md"><img alt="Docs" src="https://img.shields.io/badge/Docs-available-blue" /></a> <a href="CHANGELOG.md"><img alt="Changelog" src="https://img.shields.io/badge/Changelog-updated-8b5cf6" /></a> </p> </div> <p align="center"> <a href="#overview">Overview</a> · <a href="#quick-start">Quick Start</a> · <a href="#module-paths">Module Paths</a> · <a href="#简体中文速览">简体中文</a> </p> > **Research and Education Use Only** > > This project is intended for research and educational purposes only. > Any use for illegal data access, system interference, or unlawful activities > is strictly prohibited. AndroidUse is a rooted Android automation module. It exposes a secret-protected MCP Streamable HTTP server on the phone, provides compact UI/XML analysis for daily navigation, and hosts an Android Use Zygisk Module runtime for dynamic app-process adaptation. ## Overview AndroidUse combines three layers into one device-side automation stack: - a rooted module that runs the MCP server on-device - a Rust CLI that handles automation, transfer, and control flows - a companion Android app for setup, status, and user-facing controls The result is a device-local toolchain that can inspect UI state, manipulate apps, and expose controlled automation primitives to an MCP client. ## Quick Start ```bash adb forward tcp:8765 tcp:8765 export ANDROIDUSE_MCP_SECRET="$(adb shell 'su -M -c "cat /data/adb/modules/AndroidUse/.secret/mcp_secret"' | tr -d '\r\n')" codex mcp add androiduse --url http://127.0.0.1:8765/mcp --bearer-token-env-var ANDROIDUSE_MCP_SECRET ``` ## 简体中文速览 AndroidUse 是一个面向 Root Android 的自动化模块，把设备端 MCP 服务、UI 分析、文件传输、Shell 会话和 AUZM 运行时整合到一起。 - 适合本地设备自动化、移动端 agent 工具链、受控 UI 操作。 - 支持 KernelSU、APatch 和 Magisk。 - 强调“设备本地执行 + 口令保护 + 可审计控制”。 The project is built around three parts: - Root module: starts the MCP server and stores runtime config under the module directory. - Rust CLI: implements MCP tools, shell sessions, UI automation, file transfer, app install helpers, and AUZM control. - Companion Android app: settings entry, AUZM status page, and AndroidUse Outer IME. ## Capabilities - MCP Streamable HTTP endpoint with bearer-secret authentication. - Default local bind: `127.0.0.1:8765`. - User-editable host/port config. - Persistent Android shell sessions through `portable-pty`. - XML-first UI analysis with screenshot fallback. - Coordinate, keyevent, swipe, UI-element tap, wake, and unlock tools. - AndroidUse Outer IME for text injection without permanently replacing the user's preferred keyboard. - File station for upload/download without asking agents to invent Android paths. - APK and root-module install helpers. - Runtime App group registration for progressive MCP tool disclosure. - AUZM target control through runtime config, not hardcoded packages. - Kam/kamfw lifecycle support across KernelSU, APatch, and Magisk. ## Module Paths After installation, the active module root is: ```text /data/adb/modules/AndroidUse/ ``` Important installed paths: ```text /data/adb/modules/AndroidUse/cli /data/adb/modules/AndroidUse/bin/androiduse-cli /data/adb/modules/AndroidUse/.secret/mcp_secret /data/adb/modules/AndroidUse/.config/androiduse/mcp.env /data/adb/modules/AndroidUse/.config/androiduse/app-groups.json /data/adb/modules/AndroidUse/.config/androiduse/zygisk-target /data/adb/modules/AndroidUse/.config/androiduse/auzm.d/androiduse-runtime/payload.so /data/adb/modules/AndroidUse/zygisk/arm64-v8a.so ``` During a pending KernelSU/APatch update before reboot, files may temporarily exist under: ```text /data/adb/modules_update/AndroidUse/ ``` Runtime scripts should derive `MODDIR` from their own location and must not persist `modules_update` paths. ## Root Manager Lifecycle AndroidUse uses kamfw to keep runtime behavior unified while adapting install and lifecycle details per root manager: - KernelSU/APatch: use `boot-completed.sh`. - Magisk: package/install compatibility falls back to `service.sh`. - Runtime home is normalized to the active module directory. - Module `bin/` is placed before common Magisk/KSU/system paths in `PATH`. Business logic should use `$MODDIR`, not direct Magisk/KSU/APatch home assumptions. ## Secret The MCP server refuses unauthenticated requests. The secret is generated on first boot if missing: ```text /data/adb/modules/AndroidUse/.secret/mcp_secret ``` The secret directory and file use mode `700`. Refresh the secret manually: ```sh adb shell 'su -M -c "/data/adb/modules/AndroidUse/action.sh"' ``` `action.sh` prints the new secret, restarts MCP, and opens the AndroidUse app if the companion app is installed. Do not store the secret directly in project config. ## Connect From Host Forward the phone-local MCP port: ```sh adb forward tcp:8765 tcp:8765 ``` Export the secret in your host shell: ```sh export ANDROIDUSE_MCP_SECRET="$(adb shell 'su -M -c "cat /data/adb/modules/AndroidUse/.secret/mcp_secret"' | tr -d '\r\n')" ``` Register the MCP server: ```sh codex mcp add androiduse --url http://127.0.0.1:8765/mcp --bearer-token-env-var ANDROIDUSE_MCP_SECRET ``` ## Host And Port The listener is configured by: ```text /data/adb/modules/AndroidUse/.config/androiduse/mcp.env ``` Default: ```sh ANDROIDUSE_MCP_HOST=127.0.0.1 ANDROIDUSE_MCP_PORT=8765 ``` Use `127.0.0.1` for adb-forward-only access. Use `0.0.0.0` or a LAN address only on trusted networks. The bearer secret is still required, but exposing the port beyond adb forwarding increases risk. The AndroidUse companion app also exposes Host and Port fields in the MCP access section. Saving them writes `mcp.env` and restarts MCP. After editing `mcp.env`, restart MCP: ```sh adb shell 'su -M -c "sh /data/adb/modules/AndroidUse/boot-completed.sh"' ``` ## Core MCP Tools ### `mksh` Runs commands in an Android shell. By default, the server keeps a persistent PTY session per MCP HTTP session, so state such as `cd` and shell variables can survive across tool calls. Useful arguments: - `command`: command line to run. - `timeout_secs`: command timeout, default `30`. - `reset`: restart the persistent shell session before running. - `stateless`: run a fresh `/system/bin/sh -c` process instead of the PTY session. ### UI Control - `foreground_app`: current focus, focused package, and secure-window flag. - `ui_dump`: compact XML-derived UI analysis. - `tap_ui`: tap a UI element by text, resource-id, or content-desc. - `tap`: tap raw coordinates. - `swipe`: swipe raw coordinates. - `keyevent`: send an Android keyevent. - `wake`: wake the display. - `unlock_device`: wake, swipe, enter a numeric PIN, and press enter. - `take_over`: request user review for login, captcha, payment confirmation, or other cases where automation should pause for human approval. - `human_takeover`: hand control to the user for captcha, login, or complex pages, then continue after the user marks the takeover complete. AndroidUse also exposes AutoGLM-compatible generic action names for clients that expect that tool surface: `Launch`, `Tap`, `Type`, `Swipe`, `Back`, `Home`, `Long Press`, `Double Tap`, `Wait`, and `Take_over`. These are MCP aliases over the same device-control layer, not app-specific AUZM logic. `ui_dump` is the default navigation tool. It filters redundant XML containers and emits one core element per line: ```text meta count=... vision_fallback=false text=... id=... desc=... class=... pkg=... bounds=[x1,y1][x2,y2] clickable=true ``` Use `screenshot` only when visual context is needed or `ui_dump` reports a vision fallback, such as SurfaceView, TextureView, WebView, game/video content, or captcha-like screens. ### Text Input - `text_input`: inject text into the current focused editor. - `ime_action`: trigger editor actions such as `send`, `done`, or `search`. AndroidUse uses an outer IME design. The MCP server temporarily switches to AndroidUse Outer IME, commits text or an editor action, then restores the user's previous input method by default. This keeps automation input-method independent without permanently replacing Sogou, Gboard, or another preferred keyboard. ### Screenshot `screenshot` captures the screen into the module state directory and returns a protected HTTP URL such as: ```text http://127.0.0.1:8765/screenshots/shot-...png ``` Fetch that URL with the same MCP secret, using `Authorization: Bearer <secret>` or `X-AndroidUse-Secret`. For clients that cannot attach headers, `include_secret_in_url=true` returns an additional `url_with_secret`, but the default avoids putting the secret into model-visible text. `include_base64=true` is still available for compatibility, but URL transfer is preferred. AndroidUse tries direct framebuffer access first where available, then falls back to Android `screencap -p`. Normal Android screenshots cannot reliably capture `FLAG_SECURE` windows. See the CaptureSposed section below. ### File Station The phone side exposes a module-local file station: ```text /data/adb/modules/AndroidUse/.state/file-station/ ``` Tools: - `file_upload`: upload `data_base64` as `name`. - `file_download`: download `name` and return `data_base64`. - `file_list`: list station files and sizes. - `file_delete`: delete a station file. File names must be plain basenames. Path separators and `..` are rejected. ### Install Helpers - `install_app`: install an APK from the file station or an absolute phone path. - `install_root_module`: install a root module ZIP through auto-detected KernelSU, Magisk, or APatch command paths. ## AUZM Runtime AUZM means Android Use Zygisk Module. Do not use the generic term "Zygisk module" for AndroidUse's dynamic runtime modules. The active loader lives at: ```text /data/adb/modules/AndroidUse/zygisk/arm64-v8a.so ``` This loader is a **meta module**: it is the Zygisk module whose job is to load other AUZM modules. App-specific behavior belongs in separate AUZM `payload.so` files, not in the loader itself. AUZM modules are registered under: ```text /data/adb/modules/AndroidUse/.config/androiduse/auzm.d/<module-id>/ ``` Each registry directory uses small text files: ```text enabled name scope path payload.so ``` The default runtime payload lives at: ```text /data/adb/modules/AndroidUse/.config/androiduse/auzm.d/androiduse-runtime/payload.so ``` The loader reads the registry and loads every enabled AUZM whose `scope` matches the current package/process. `scope` may contain one match string per line. ### AUZM Template New AUZM modules should start from the Kam-compatible template under: ```text tmpl/auzm_template/ ``` Build all workspace members to regenerate the distributable template archive: ```sh kam build -a ``` This produces: ```text templates/auzm_template.tar.gz dist/templates.zip ``` `templates/auzm_template.tar.gz` is the single-template archive. `dist/templates.zip` is the release bundle consumed by Kam/setup-kam template import workflows. The compatibility target file is: ```text /data/adb/modules/AndroidUse/.config/androiduse/zygisk-target ``` `set_auzm_target` writes both `zygisk-target` and the default runtime module's `auzm.d/androiduse-runtime/scope`. Empty scope means the default AUZM is not loaded. Use MCP to set it: ```json { "name": "set_auzm_target", "arguments": { "use_foreground": true } } ``` Or set an explicit package: ```json { "name": "set_auzm_target", "arguments": { "package": "com.example.app" } } ``` Clear it: ```json { "name": "set_auzm_target", "arguments": { "clear": true } } ``` Restart the target app after changing the target. A device reboot may be needed when root-manager Zygisk scope changed. Inspect registered AUZM modules: ```json { "name": "auzm_status", "arguments": {} } ``` ## Dynamic MCP Tool Registry AndroidUse supports runtime MCP tool blueprints for owner-controlled app adaptation. Register a new dynamic tool with `generate_new_hook_tool`. It writes: ```text /data/adb/modules/AndroidUse/.config/androiduse/dynamic-tools/blueprints/<tool>.json /data/adb/modules/AndroidUse/.config/androiduse/dynamic-tools/sources/<tool>.java ``` Calling the tool writes a request: ```text /data/adb/modules/AndroidUse/.state/dynamic-tools/requests/<request_id>.json ``` The injected AUZM payload consumes the request through the app cache bridge and writes a response: ```text /data/adb/modules/AndroidUse/.state/dynamic-tools/responses/<request_id>.json ``` Use `dynamic_tool_status` to inspect blueprints, pending requests, responses, and bridge paths. ## Dynamic App Groups App-specific tools are not hardcoded into the Rust binary. Tool visibility is controlled by: ```text /data/adb/modules/AndroidUse/.config/androiduse/app-groups.json ``` Register a group for the current foreground app: ```json { "name": "register_app_group", "arguments": { "group_name": "example_app", "use_foreground": true } } ``` Register an explicit package and set AUZM target at the same time: ```json { "name": "register_app_group", "arguments": { "group_name": "example_app", "package": "com.example.app", "make_auzm_target": true } } ``` When the foreground package matches a configured group, MCP progressively discloses tools whose blueprint uses that `group_name`. The built-in `global` group is always visible. Use `app_group_status` to inspect the current config. Remove a runtime group: ```json { "name": "unregister_app_group", "arguments": { "group_name": "example_app" } } ``` ## Human Review `take_over` opens the AndroidUse companion app and asks the user to review an automation request. The dialog exposes four choices: - Allow - Always allow - Deny - Always deny `Allow` and `Always allow` require biometric authentication before the decision is written, so automation cannot approve itself by clicking the dialog. `Deny` and `Always deny` do not require biometric authentication. Example: ```json { "name": "take_over", "arguments": { "reason": "Login or captcha needs user confirmation.", "scope": "com.example.app", "timeout_secs": 300 } } ``` Decisions are written under: ```text /data/adb/modules/AndroidUse/.state/take-over/ ``` Remembered policies are stored at: ```text /data/adb/modules/AndroidUse/.config/androiduse/take-over-policy.json ``` ## Human Takeover `human_takeover` is a separate flow for complex screens where the user needs to operate the phone directly, such as captcha, login, or a page that automation cannot interpret reliably. Example: ```json { "name": "human_takeover", "arguments": { "reason": "Please complete the captcha and return to AndroidUse.", "scope": "com.example.app", "timeout_secs": 600 } } ``` The AndroidUse app shows a human takeover dialog. After the user completes the task on the phone, they tap `Done` / `完成接管`, and MCP receives `completed`. ## Companion App The companion app source is under: ```text app/androiduse-app/ ``` It provides: - AndroidUse settings entry. - Secret display/copy flow. - MCP Host/Port editor. - Human review dialog. - Human takeover dialog. - AUZM module cards with metadata and paths. - Light/dark and language toggles. - AndroidUse Outer IME service. Build: ```sh bash tools/build_androiduse_app.sh ``` Install: ```sh adb install -r target/androiduse-app/androiduse-app.apk ``` ## Companion Modules ### CaptureSposed CaptureSposed is only for secure-window screenshot support: ```text https://github.com/99keshav99/CaptureSposed ``` AndroidUse does not bundle it. Install and enable it separately if you need to inspect apps using `FLAG_SECURE`. CaptureSposed is not the AUZM target and should not be written into `zygisk-target` unless you are specifically testing CaptureSposed itself. ### liboemcrypto-disabler For rooted-device testing against DRM streaming apps, users may optionally install: ```text https://github.com/Anonym0usWork1221/liboemcrypto-disabler ``` This is not a hard dependency for AndroidUse. Install it only when the target test case needs rooted-device DRM compatibility. ## Build Build the Android CLI: ```sh PATH=/opt/android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin:$PATH \ cargo build --release --target aarch64-linux-android -p androiduse-cli ``` Build the Rust AUZM loader: ```sh tools/build_zygisk_module.sh ``` Build the Rust payload: ```sh tools/build_payload_module.sh ``` Package the root module: ```sh kam build ``` Build the root module and AUZM templates together: ```sh kam build -a ``` The pre-build hooks copy the release binary into: ```text src/AndroidUse/bin/androiduse-cli ``` and package: ```text src/AndroidUse/zygisk/arm64-v8a.so src/AndroidUse/.config/androiduse/auzm.d/androiduse-runtime/payload.so ``` Skip AUZM builds when needed: ```sh ANDROIDUSE_BUILD_ZYGISK=0 ANDROIDUSE_BUILD_PAYLOAD=0 kam build ``` ## Verification Local checks: ```sh cargo fmt -- --check cargo check -p androiduse-cli cargo check -p androiduse-payload bash tools/build_androiduse_app.sh tools/build_zygisk_module.sh tools/build_payload_module.sh kam build ``` Device checks: ```sh adb shell 'su -M -c "sh /data/adb/modules/AndroidUse/boot-completed.sh"' adb shell 'su -M -c "ss -lntp 2>/dev/null | grep 8765"' adb forward tcp:8765 tcp:8765 ``` Then call `tools/list` with the bearer secret and confirm core tools such as `mksh`, `ui_dump`, `text_input`, `register_app_group`, and `set_auzm_target` are present. ## Safety Notes - This project is intended for research and educational purposes only. Any use for illegal data access, system interference, or unlawful activities is strictly prohibited. - Keep the MCP listener on loopback unless you trust the network. - Do not print or commit MCP secrets. - Do not edit KSU, Vector, or LSPosed databases from AndroidUse automation. - Prefer `ui_dump` and `tap_ui` over blind coordinate taps. - Use screenshots only when XML is insufficient. - Dynamic hooks should be scoped to owner-controlled automation, compatibility, observability, and UI tooling. ## Acknowledgements AndroidUse's generic mobile action surface is inspired by AutoGLM-compatible mobile GUI agent workflows. We thank the AutoGLM and MobileRL authors for their work on autonomous mobile GUI agents. AndroidUse's AUZM loader and hot-swap workflow also reference [HanSoBored/Zygisk-Loader](https://github.com/HanSoBored/Zygisk-Loader), a Zygisk module project for dynamically loading native payload libraries into Android application processes without requiring a full device reboot. ```bibtex @article{liu2024autoglm, title={Autoglm: Autonomous foundation agents for guis}, author={Liu, Xiao and Qin, Bo and Liang, Dongzhu and Dong, Guang and Lai, Hanyu and Zhang, Hanchen and Zhao, Hanlin and Iong, Iat Long and Sun, Jiadai and Wang, Jiaqi and others}, journal={arXiv preprint arXiv:2411.00820}, year={2024} } @article{xu2025mobilerl, title={MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agents}, author={Xu, Yifan and Liu, Xiao and Liu, Xinghan and Fu, Jiaqi and Zhang, Hanchen and Jing, Bohao and Zhang, Shudan and Wang, Yuting and Zhao, Wenyi and Dong, Yuxiao}, journal={arXiv preprint arXiv:2509.18119}, year={2025} } ```