mcp_robot

# Tool List ### I. Project Overview * **Project Name:** Agentic MCP-based Intelligent Mechanical Arm Control System  * **Project Goals:** 1. Construct a six-joint mechanical arm with virtual simulation and physical implementation. 2. Build an MCP server that allows external intelligent agents (various large language models) to control the mechanical arm through standardized interfaces. 3. Enable large language models to understand user natural language instructions, autonomously arrange tasks, and convert them into specific operation sequences for the mechanical arm. * **Project Background:** With the rapid development of robotics and artificial intelligence, the naturalness and intelligence of human-machine interaction have become key bottlenecks. Traditional robot control methods (such as programming and teaching) are highly professional and inefficient. Large language models have shown great potential in understanding complex instructions and planning tasks. This project explores the use of MCP on embedded hardware through a six-joint mechanical arm case. * **Practical Problems Solved:** 1. **Lower Robot Operation Threshold:** Make non-professional users able to interact with the mechanical arm through natural language and complete complex tasks. 2. **Improve Robot Task Programming Efficiency:** Utilize the planning ability of LLM to quickly generate and adjust robot job processes. 3. **Provide Safe and Low-Cost R&D and Debugging Environment:** Through a browser-available simulator, verify control algorithms and AI logic in a low-cost and non-contact manner. 4. **Explore LLM Applications in Embodied Intelligence:** Give LLM "hands" to interact with the physical world more directly. ### II. Work Description and Highlights Our work is a comprehensive mechanical arm control platform integrating 3D simulation, physical hardware control, MCP services, and AI intelligent agent interaction. * **Core Functions:** 1. **High-Fidelity 3D Simulator:** * Based on Three.js and URDFLoader, load and render standard URDF mechanical arm models. * Provide realistic lighting, shadows, and customizable ground textures (MuJoCo style). * Support users to freely control and observe the mechanical arm through a slider controller. 2. **Multi-Mode Control:** * **Keyboard Control:** Allow users to control the simulated mechanical arm in real-time through keyboard keys and display key states. * **Physical Mechanical Arm Control:** Integrate physical servos, connect and control the physical six-axis mechanical arm through Web Serial API, and achieve synchronous movement with the simulator. * **MCP Service Control:** Through the MCP bridge service implemented by WebSocket, allow external programs (self-made MCP Python client, VSCode AI Toolkit) to send instructions to control the mechanical arm joint angles. 3. **Agentic AI Interaction (Implemented through Python Client):** * Large language models (such as DeepSeek, chatgpt) use "mcp tools" to convert understood user natural language instructions (e.g., "Let the mechanical arm dance like seaweed" or "nod your head") into calls to predefined robot control tools. * The Python client receives LLM tool call requests, converts them into MCP-compatible JSON commands, and sends them to the MCP bridge server to control the simulator or physical mechanical arm. * The results of robot operations (success, failure, warning) are fed back to LLM for subsequent decision-making or reporting to users. * **Highlights and Characteristics:** 1. **End-to-End Intelligent Control Chain:** Achieve a complete closed loop from natural language input -> LLM understanding and planning -> MCP service -> simulator/physical mechanical arm execution -> result feedback to LLM. 2. **Virtual and Real Combination with Low-Cost Verification:** The simulator provides a safe and efficient environment for AI algorithm development and testing, and can seamlessly migrate to control physical hardware. 3. **User-Friendly Interaction Interface:** Provide a clear control panel, status display, and instant feedback to enhance user experience. 4. **Modularity and Scalability:** The system's components (simulator, controller, MCP service, AI client) are relatively independent, making it convenient for future functional expansion and technical upgrades. ### III. MCP Service and Client Construction * Server  * MCP Client  ### IV. Agentic AI Platform Framework and Intelligent Agent Construction * **Define Tools (MCP Tools for LLM):** 1. We predefine a series of "tools" related to robot control for LLM: * set_robot_servo_angle: Control a single servo through ID and angle. * set_robot_joint_angle: Control a single joint through URDF joint name and angle. * set_robot_all_servo_angles: Control multiple servos simultaneously. 2. Each tool description includes name, function description, and detailed parameter definitions (type, description, required). This enables LLM to understand the purpose and correct usage of each tool. * **LLM Interaction Process (Implemented in Python Client):** 1. **User Instruction Input:** Users input natural language instructions (e.g., "Let the mechanical arm's first joint turn up 30 degrees and repeat this swing motion twice") to the Python client. 2. **Call LLM API:** The Python script sends user instructions and predefined robot control tool lists to the DeepSeek API. We set tool_choice="auto" to allow LLM to judge when and how to use these tools. 3. **LLM Generates Tool Calls:** If LLM determines that robot operation is needed to complete user instructions, it returns one or more tool_calls objects in the API response. Each tool_call contains the function name (our defined tool name) and parameters (JSON string generated by LLM based on user instructions). 4. **Python Executes Tool Calls:** * The Python script parses tool_calls, obtains the function name and parameters. * Calls the execute_robot_tool_calls function, which converts LLM's abstract tool calls into specific MCP command JSON. 5. **Obtain Robot Operation Results:** The Python script waits for the operation receipt returned from the MCP bridge server (indicating success, failure, or warning). 6. **Feedback Results to LLM:** The Python script sends the robot operation result (formatted as a JSON string) as a role: "tool" message, along with previous chat history, back to the DeepSeek API. 7. **LLM Generates Final Response:** After receiving tool execution results, LLM generates a final response to user instructions, such as confirming operation completion, reporting errors, or requesting further instructions. * **Agentic Characteristics:** * **Perception-Reasoning-Action Cycle:** LLM receives user input (perception), plans through tool calls (reasoning), Python script executes tool calls and operates the robot (action), and robot operation results are fed back to LLM (perception again), forming a closed loop. * **Multi-Step Reasoning and Complex Task Decomposition:** For instructions like "repeat the swing motion three times," LLM can understand and continuously generate multiple tool calls to achieve the task. * **Support for Multiple MCP Servers:** Can add MCP servers for other services (e.g., weather), allowing users to ask the mechanical arm questions like "Will it rain in Shenzhen today? If so, nod your head," and LLM can automatically call multiple MCP servers based on process judgment. ### V. Technical Innovations 1. **WebSocket-Based Lightweight MCP Bridge Service:** Since the MCP server ultimately controls physical hardware, and the server can be deployed online but hardware communication and control are on the client side, we designed a simple and efficient WebSocket bridge service instead of using complex RPC frameworks or heavyweight message queues. This achieves low-latency, bidirectional communication between the Python AI client and the browser-side Three.js simulator. 2. **Dynamic Blob Message Processing:** During browser-side WebSocket message reception, we found that even if the server sends text frames, the browser sometimes identifies event.data as a Blob object. We ensured correct message parsing by asynchronously reading Blob.text() content, enhancing communication robustness. 3. **Virtual and Real Synchronization and Error Recovery Mechanism (for Physical Servos):** * When connecting physical servos, the system not only synchronizes instructions to hardware but also attempts to read initial positions from servos and uses them as references for subsequent control. * Implemented real-time UI feedback on servo communication status (idle, pending, success, warning, error). * When servo operation fails or errors occur, the system records the last safe position and attempts to restore servos to that position, enhancing physical operation safety. Error information is also fed back to the AI client through the MCP service. ### VI. UI/UX Optimization 1. **Responsive Control Panel:** Adopted fixed positioning and maximum height limitation to ensure good display on different screen sizes, and implemented content scrolling through overflow: auto. Native scrollbars were hidden for a cleaner interface. 2. **Collapsible Sections:** Organized different functional modules of the control panel (keyboard control, physical robot, MCP service) into collapsible sections. Users can expand or collapse sections as needed, maintaining interface cleanliness. Icons (▼/►) intuitively indicate expansion status. 3. **Real-Time Status Feedback:** * Visual highlighting of keyboard keys (key-pressed class) and control area highlighting (control-active class) when keys are pressed. * Real-time display of servo connection status, communication status (idle, pending, success, warning, error), and specific error information on the UI, distinguished by different colors. 4. **Instant Warnings:** For virtual joint limit violations or physical servo operation failures/errors, non-blocking, color-distinguished warning boxes (jointLimitAlert, servoLimitAlert) pop up at the top of the screen and automatically disappear after a few seconds. 5. **Clear Help Prompts:** Added help icons near key operation buttons (e.g., "Connect Physical Mechanical Arm"). Hovering over these icons displays operation instructions and precautions. 6. **Smooth Mechanical Arm Animation:** Joint movements use interpolation and easing functions, providing a smooth visual experience. 7. **Consistent Button Style:** Adopted a unified visual style for interaction elements like connection buttons, with color changes based on connection status. ### VII. Team Contributions * **[Zekhua Zhang] - Project Leader/Architecture Design/Backend Development:** * **[Yang Tang] - Frontend Development/Three.js Simulator:** * **[Qi Xiao] - AI Integration/Python Client Development:** * **[Kaijun Xiao, Yindong Li] - Physical Mechanical Arm Integration/Testing and Documentation:** Frontend code partially referenced from: https://github.com/timqian/bambot ### VIII. Future TODO This project lays a solid foundation for building a more intelligent and user-friendly robot interaction system. In the future, we plan to deepen and expand from the following aspects: 1. **Enhance AI Perception and Interaction Capabilities:** * **Integrate Visual Feedback:** Introduce cameras and computer vision algorithms (or utilize LLM's multimodal capabilities) to enable the mechanical arm to "see" the environment, perform object recognition, positioning, and grasping tasks based on vision, and adjust actions according to visual feedback. 2. **Improve Simulator Fidelity and Functionality:** * **Integrate Physics Engines:** Introduce physics engines like Bullet, Ammo.js, or Rapier to achieve more realistic collision detection, gravity, friction, and support more complex grasping and operation simulations. 3. **Optimize MCP Service and Multi-Agent Collaboration:** * **Richer MCP Instruction Set:** Expand the MCP protocol to support querying robot status (e.g., current joint angles, end-effector position), setting speed/acceleration, controlling grippers, and more detailed operations.