mobiler-use

# mobiler-usr MCP A mobile device control service based on Spring Boot and Appium, supporting operations on mobile devices via REST API and MCP protocol. ## Main Features - Encapsulates common Appium operations, providing a simple API - Supports device screen capture - Automatically records the mobile interface and sends it back to the computer - Based on MCP protocol, facilitates interaction with AI models, can be quickly integrated into cursor, or develop your own MCP client (a JAVA version will be open-sourced later) - Currently supports only Android ## Environment Requirements - JDK 21+ - Maven 3.6+ - Node.js and npm - Appium 2.0+ - Android SDK (for operating Android devices) Please refer to the documentation for related technologies. ## Quick Start ### 1. Prepare the Environment Install and run the Appium server: ```bash npm install -g appium appium driver install uiautomator2 # Android driver appium # Start the Appium server, note that there is a bug in the current built-in startup ``` Connect the mobile device: - Connect the Android device via USB and enable USB debugging - Or use an Android emulator - Note: Although the program has a built-in Appium startup feature, there is currently a bug ### 2. Configure the Application Copy and edit the configuration file: ```bash # Execute in the project root directory cd mobile cp src/main/resources/application-local.properties.example src/main/resources/application-local.properties ``` Edit the `application-local.properties` file, filling in the API key and other sensitive configurations: ```properties # OpenAI API configuration mainly used for image recognition, use gemini-2.0-flash for testing spring.ai.openai.api-key=your-actual-api-key-here spring.ai.openai.base-url=your-actual-base-url-here spring.ai.openai.chat.options.model=your-actual-model-name-here ``` Modify `src/main/resources/application.yml` to configure the Appium server and device information: ### 3. Run the Service ```bash com.flyiu.ai.mcp.mobile.MobileApplication ``` ## API Usage Instructions ### REST API Examples ``` # Connect to device POST /api/mobile/android/connect # Take a screenshot GET /api/mobile/screenshot?deviceName=default-android # Click an element POST /api/mobile/click Parameters: - deviceName: Device name - locatorType: Locator type (id/xpath, etc.) - locatorValue: Locator value ``` ### MCP Tool List This service is implemented based on the MCP protocol of Spring AI, providing the following tools for interaction with AI models: | Tool Name | Description | Parameters | |-------------------------|---------------------------------|--------------------------| | connectAndroidDevice | Connect to Android device | deviceName (optional) | | connectIOSDevice | Connect to iOS device | deviceName (optional) | | disconnectDevice | Disconnect device | deviceName | | takeScreenshot | Get device screen capture | deviceName, asBase64 (optional) | | clickElement | Click an element | deviceName, locatorType, locatorValue | | sendKeys | Input text into an element | deviceName, locatorType, locatorValue, text | | waitForElement | Wait for an element to appear | deviceName, locatorType, locatorValue, timeoutInSeconds (optional) | | swipeScreen | Swipe the screen | deviceName, direction or startX/startY/endX/endY | | launchApp | Launch an application | deviceName, appPackage, appActivity | ### Invocation Example LLM tool invocation example: ```json { "name": "takeScreenshot", "parameters": { "deviceName": "default-android", "asBase64": true } } ``` ## Using in Cursor ### Recommended Models - It is recommended to use Claude 3.7 or Gemini 2.5 ### Troubleshooting **Service Connection Issues** - Ensure the MCP service is running (`mvn spring-boot:run`) - Check if the port is correct (default 9080) - Confirm that no firewall is blocking **Device Connection Issues** - Run `adb devices` to confirm the device is connected - Ensure the Appium server is running - Check if USB debugging is enabled on the device **API Call Failures** - Check if the API key configuration is correct - View the MCP service logs for detailed error information ## Project Structure ``` com.flyiu.ai.mcp.mobile ├── config // Configuration classes ├── controller // REST API controllers ├── mcp // MCP tool implementations ├── service // Service layer │ ├── appium // Appium service encapsulation │ ├── device // Device management │ └── screenshot // Screenshot service ├── model // Data models ├── util // Utility classes └── MobileApplication // Application entry point ``` ## License MIT