nvidia-omniverse-isaacsim-mcp-server

# Omniverse MCP Server AI-driven control of **NVIDIA Omniverse** and **Isaac Sim** via the [Model Context Protocol](https://modelcontextprotocol.io). Build entire 3D worlds, digital twins, physics simulations, and animations from natural language through Claude Code, Codex, or any MCP-compatible AI agent. ## Architecture ``` ┌─────────────────────┐ MCP/stdio ┌──────────────────────┐ │ Claude Code / Codex │ ◄──────────────────► │ MCP Server │ │ (AI Agent) │ │ (Python, uv) │ └─────────────────────┘ └──────────┬───────────┘ │ WebSocket │ localhost:8766 ┌──────────▼───────────┐ │ Kit Extension │ │ (inside Omniverse / │ │ Isaac Sim process) │ └──────────────────────┘ ``` The MCP server runs in your terminal and communicates with the AI agent via stdio. It forwards commands over WebSocket to a Kit extension running inside Omniverse, which executes USD/PhysX operations on the live stage. ## Quick Start ### 1. Install the MCP Server ```bash cd omniverse-mcp uv sync ``` ### 2. Install the Kit Extension Copy or symlink the `kit_extension/` directory into your Omniverse extensions folder: **Option A — Symlink:** ```bash # Isaac Sim mklink /D "C:\Users\<you>\isaacsim\IsaacSim\source\extensions\omni.mcp_extension" "C:\path\to\omniverse-mcp\kit_extension" # Or use a direct extension path when launching: isaac-sim --ext-path /path/to/omniverse-mcp/kit_extension --enable kit_extension ``` **Option B — Extension Manager:** 1. Open Omniverse / Isaac Sim 2. Create a parent folder that contains the extension under the name `omni.mcp_extension` 3. Example: make `C:\Users\<you>\omniverse-exts\omni.mcp_extension` point to this repo's `kit_extension` 4. Window > Extensions 5. Click the gear icon, add the parent folder such as `C:\Users\<you>\omniverse-exts` to search paths 6. Resync the registry, then search for `omni.mcp_extension` or `Omniverse MCP Extension` and enable it Important: - Do not add `kit_extension/` itself as an Extension Manager search folder. Search folders are scanned for subfolders that contain `extension.toml`. This repo's `kit_extension/` is already the extension root, so it should be used with `--ext-path` or placed under a parent folder as `omni.mcp_extension/`. - `--ext-path` makes the extension discoverable, but the extension still needs to be enabled. - If you pass the repo's `kit_extension/` directory directly with `--ext-path`, the extension name is `kit_extension` because Kit derives the extension id from the folder name. - If you place it under a parent folder as `omni.mcp_extension/`, then enable `omni.mcp_extension`. ### 3. Configure Claude Code Add to your project's `.mcp.json` (already included in this repo): ```json { "mcpServers": { "omniverse": { "command": "/path/to/omniverse-mcp/scripts/run_omniverse_mcp.sh", "args": [], "env": { "UV_PROJECT_ENVIRONMENT": "${UV_PROJECT_ENVIRONMENT:-}", "OMNIVERSE_WS_HOST": "${OMNIVERSE_WS_HOST:-}", "OMNIVERSE_WS_PORT": "${OMNIVERSE_WS_PORT:-8766}" } } } } ``` Or add it directly from the CLI: ```bash claude mcp add omniverse --scope project \ --env OMNIVERSE_WS_PORT=8766 \ -- /path/to/omniverse-mcp/scripts/run_omniverse_mcp.sh ``` The launcher auto-detects the Windows host IP when run from WSL. If needed, you can still override either variable explicitly: ```bash export OMNIVERSE_WS_HOST=10.255.255.254 export OMNIVERSE_MCP_COMMAND=/path/to/omniverse-mcp/.venv/bin/omniverse-mcp ``` ### 4. Start Using 1. Launch Omniverse / Isaac Sim (the Kit extension starts automatically) 2. Open Claude Code in this project directory 3. Ask Claude to build something: > "Create a warehouse that is 80m x 40m with 4 aisles of racking. Add industrial lighting and a concrete floor." ## Troubleshooting Claude Code If `/mcp` shows **Disconnected**, Claude Code is usually failing to start the local MCP process. In this project, the most reliable fix is to launch the repo's wrapper script instead of `uv run`. Checklist: - Run `uv sync` in this repo first so `.venv/bin/omniverse-mcp` exists. - Prefer `/path/to/omniverse-mcp/scripts/run_omniverse_mcp.sh` as the MCP command. - For a user-scoped or manually added server, use an absolute path instead of `.` so it works outside the repo root. - If you use WSL2 with Isaac Sim running on Windows, leave `OMNIVERSE_WS_HOST` empty or set it to the Windows host IP rather than `localhost`. - The launcher can override the executable path with `OMNIVERSE_MCP_COMMAND`. - If `uv sync` fails with `failed to remove file ... .venv\\lib64: Access is denied` on a repo under `C:\...` mounted into WSL, the repo `.venv` is usually left incomplete. In that case, remove the broken `.venv` and recreate the environment on the Linux filesystem instead: ```bash UV_PROJECT_ENVIRONMENT=/tmp/omniverse-mcp-venv uv sync export OMNIVERSE_MCP_COMMAND=/tmp/omniverse-mcp-venv/bin/omniverse-mcp ``` - You can also leave `UV_PROJECT_ENVIRONMENT` exported before starting Claude Code; the included `.mcp.json` and launcher now pass it through automatically. - Verify the server command itself starts: `timeout 5 /path/to/omniverse-mcp/scripts/run_omniverse_mcp.sh` - Check the saved Claude config with `claude mcp get omniverse` ### WSL2 to Windows If Claude Code runs inside WSL2 but Isaac Sim runs on Windows: - In mirrored networking mode on recent WSL, prefer `localhost`. - In default WSL2 NAT mode, connect to the Windows host IP instead. - The first `nameserver` entry in `/etc/resolv.conf` is only a fallback heuristic and may be a DNS endpoint rather than the correct Windows host address. The included launcher script handles the common WSL case automatically. It probes `localhost` first and falls back to the Windows-side address heuristic only if loopback is not reachable. The Kit extension in this repo listens on `0.0.0.0:8766` by default, so it is reachable from WSL as long as the correct host is selected and Windows firewall allows the connection. Important: Claude Code showing the MCP server as connected is separate from Omniverse connectivity. The MCP server can be connected in `/mcp` while tool calls still fail until the Omniverse / Isaac Sim extension is running and reachable on the configured host and port. ### Isaac Sim Python Dependency The Kit extension depends on the Python `websockets` package inside Isaac Sim's bundled Python environment. The extension manifest now declares that dependency via `omni.kit.pipapi`, so on a machine with internet access Isaac Sim should install it automatically before enabling the extension. If that automatic install fails, install it manually from the Isaac Sim root on Windows: ```bat python.bat -m pip install websockets==16.0 ``` Then relaunch Isaac Sim with the extension enabled. ## Tools (60+) ### Connection & Scene Info - `get_scene_info` — **Always call first.** Verifies connectivity, returns stage info. - `get_stage_hierarchy` — Nested tree view of the USD stage. ### Scene Management - `new_stage` — Create a fresh stage with optional templates (empty, ground_plane, indoor, outdoor). - `open_stage` / `save_stage` — Open/save USD files. - `add_reference` / `create_layer` — USD composition. ### Prim Creation & Manipulation - `create_prim` — Create Cube, Sphere, Cylinder, Cone, Capsule, Plane, Mesh, Xform, Scope. - `create_mesh` — Procedural geometry with explicit vertices/faces. - `modify_prim` / `delete_prim` / `duplicate_prim` — CRUD operations. - `group_prims` / `create_grid` — Organization and instancing. ### Transforms - `set_transform` / `get_transform` / `move_prim` — Position, rotate, scale. - `align_prims` / `distribute_prims` — Multi-prim alignment and distribution. ### Materials & Shaders - `create_material` — OmniPBR, OmniGlass, UsdPreviewSurface with color, roughness, metallic, textures. - `assign_material` / `modify_material` — Bind and edit materials. ### Lighting & Environment - `create_light` — DistantLight, DomeLight, SphereLight, RectLight, DiskLight, CylinderLight. - `set_environment` — HDRI sky dome. - `create_sky` — One-click realistic outdoor lighting (morning, noon, sunset, night). ### Cameras & Viewports - `create_camera` — Perspective/orthographic with look-at targeting. - `set_active_camera` / `capture_viewport` — Viewport control and screenshots. ### Physics & Collision - `create_physics_scene` — PhysX scene with gravity, solver, CCD, GPU dynamics. - `add_rigid_body` / `add_collider` / `add_ground_plane` — Physics setup. - `create_joint` — Fixed, Revolute, Prismatic, Spherical, Distance joints. - `add_force_field` — Wind, drag, noise force fields. - `overlap_test` / `raycast` — Physics queries. ### Animation & Timeline - `set_keyframe` / `create_animation_clip` — Keyframe animation. - `set_timeline` / `play_animation` / `stop_animation` — Timeline control. - `create_skeletal_animation` — Apply animation clips to rigged characters. ### Human Characters - `spawn_character` — Place rigged human characters. - `set_character_animation` — idle, walk, run, sit, wave, talk. - `create_crowd` — Scatter animated characters across an area. - `create_character_path` — Define waypoint navigation paths. ### Digital Twin Templates - `create_warehouse` — Complete warehouse with racks, aisles, lighting, docks. - `create_solar_farm` — Panel arrays with inverters, substation, terrain. - `create_wind_farm` — Turbines with animated rotors, access roads. - `create_factory_floor` — Workstations, conveyors, safety zones. - `create_data_center` — Server racks, hot/cold aisles, cooling. ### Simulation Control - `start_simulation` / `stop_simulation` / `step_simulation` — Physics playback. - `get_simulation_state` / `set_simulation_speed` — Monitoring and speed control. ### Asset Management - `list_assets` / `import_asset` — Browse and import USD, FBX, OBJ, glTF, STL files. - `search_nvidia_assets` — Search the NVIDIA Omniverse asset library. ### Scene Queries - `get_prim_info` / `find_prims` — Inspect and search the stage. - `get_bounding_box` / `measure_distance` — Spatial queries. ### Rendering - `set_render_mode` — RTX Realtime, Interactive, Accurate, Iray. - `set_render_settings` / `render_to_file` — Production-quality renders. ### Code Execution (Escape Hatch) - `execute_python` — Run arbitrary Python in the Kit process with full `omni.*` and `pxr` access. - `execute_python_file` — Execute a `.py` script file. ## Configuration | Environment Variable | Default | Description | |---|---|---| | `OMNIVERSE_WS_HOST` | `localhost` | WebSocket host for Kit extension | | `OMNIVERSE_WS_PORT` | `8766` | WebSocket port | The Kit extension reads its settings from the extension.toml or carb settings: - `/exts/omni.mcp_extension/ws_host` — Server bind address - `/exts/omni.mcp_extension/ws_port` — Server port ## Development ### Testing ```bash # Run unit tests (no Omniverse required) uv run pytest tests/ -v # Interactive MCP debugging uv run mcp dev mcp_server/server.py ``` ### Project Structure ``` omniverse-mcp/ ├── mcp_server/ # MCP server (runs in terminal) │ ├── server.py # Entry point │ ├── ws_client.py # WebSocket client │ └── tools/ # 16 tool modules ├── kit_extension/ # Omniverse Kit extension │ ├── config/ # extension.toml │ └── omni/mcp_extension/ │ ├── extension.py # on_startup / on_shutdown │ ├── ws_server.py # WebSocket server │ ├── command_router.py │ └── handlers/ # 16 command handlers ├── tests/ └── examples/ # Example prompts ``` ## License MIT Troubleshootings: cd /mnt/c/Users/mstef/omniverse/omni_isaacsim_mcp unset OMNIVERSE_MCP_COMMAND export UV_PROJECT_ENVIRONMENT="$HOME/.venvs/omniverse-mcp" rm -rf "$UV_PROJECT_ENVIRONMENT" uv venv "$UV_PROJECT_ENVIRONMENT" ls -l "$UV_PROJECT_ENVIRONMENT/bin/python" timeout 5 ./scripts/run_omniverse_mcp.sh Expected result: Then start claude from that same shell so UV_PROJECT_ENVIRONMENT is inherited, and run: claude mcp remove omniverse -s user claude mcp reset-project-choices claude mcp get omniverse