omniverse-agent-studio

# Omniverse Agent Studio A **web-based Agentic AI editor** for NVIDIA Isaac Sim. Chat with an LLM in your browser and watch it edit an OpenUSD scene live — while Isaac Sim streams the viewport via WebRTC, Blender round-trips assets via USD, and the agent drives both through MCP tools. ``` ┌────────────────────────────────────────────────────────────────────────┐ │ Your web browser │ │ ┌──────────────┐ ┌────────────────────┐ ┌─────────────────────┐ │ │ │ Scene graph │ │ Isaac Sim stream │ │ Chat with agent │ │ │ │ /World │ │ ┌──────────────┐ │ │ > add a red cube │ │ │ │ ├ Ground │ │ │ (WebRTC) │ │ │ > export to USD │ │ │ │ └ Robot │ │ └──────────────┘ │ │ │ │ │ └──────┬───────┘ └─────────┬──────────┘ └──────────┬──────────┘ │ └──────────┼─────────────────────┼─────────────────────────┼─────────────┘ │ │ │ ▼ HTTP ▼ WebRTC 49100/47998 ▼ SSE ┌─────────────────────────────────────────────────────────────┐ │ Agent (FastAPI + Claude) │ │ invokes MCP tools via streamable-http │ └──┬───────────────────────────┬──────────────────────┬───────┘ │ │ │ ┌─────▼──────────┐ ┌───────▼────────┐ ┌───────▼────────┐ │ usd-mcp :8001 │ │ blender-mcp │ │ Isaac Sim │ │ (13 tools on │ │ :8002 (11 │ │ runheadless │ │ usd-core) │ │ tools via │ │ --livestream │ │ │ │ socket bridge │ │ │ └───────┬────────┘ └────┬───────────┘ └────────┬───────┘ │ │ │ └──────────────────────┼──────────────────────────┘ ▼ shared ./stages/ volume (USD files) ``` ## What you need before you start Omniverse Agent Studio needs a real NVIDIA GPU to run Isaac Sim. **It cannot run on macOS, in a VM without GPU passthrough, or on a CPU-only machine.** If you only have a Mac, you can read the code and run tests, but you won't see the streaming viewport. ### Hardware | Component | Minimum | Recommended | | --- | --- | --- | | GPU | NVIDIA RTX 3070 (8 GB) — use Isaac Sim 4.5 fallback | **NVIDIA RTX 4080 (16 GB)** — required for Isaac Sim 6.0 | | RAM | 16 GB | 32 GB | | Disk | 30 GB free | 50 GB SSD | | CPU | 4 cores | 8+ cores | > **Not supported**: A100, H100, or any data-center GPU. These have no NVENC encoder, so WebRTC streaming silently fails. You need a consumer or workstation RTX card. ### Software | Platform | Supported | Notes | | --- | --- | --- | | Ubuntu 22.04 / 24.04 | Yes, first-class | Easiest path | | Windows 10 / 11 | Yes, via WSL 2 | Requires WSL 2 + Docker Desktop | | macOS | No (Isaac Sim will not start) | Code and unit tests still run | ### Accounts You need **two free accounts** before installing: 1. **NVIDIA NGC** — for pulling the Isaac Sim Docker image. Sign up at <https://ngc.nvidia.com/signup> and generate an API key at <https://org.ngc.nvidia.com/setup/api-keys>. 2. **Anthropic** (recommended) or **OpenAI** — for the LLM that drives the agent. Sign up at <https://console.anthropic.com> and create an API key. OpenAI works as a fallback. Keep both keys handy — you'll paste them into `.env` later. ## Step-by-step install (Ubuntu 22.04 / 24.04) If you're on Windows, skip to the [Windows section](#step-by-step-install-windows-10--11). On macOS you can only run the test suites — see [Run the tests](#run-the-tests-any-os). ### 1. Clone the repo ```bash git clone https://github.com/JinwangMok/omniverse-agent-studio.git cd omniverse-agent-studio ``` ### 2. Run the preflight check ```bash ./scripts/preflight-check.sh ``` This prints a `PASS` / `WARN` / `FAIL` table for your GPU, driver, Docker, and memory. If anything says `FAIL`, fix it before continuing. Common fixes: - **`nvidia-smi not found`** → install the NVIDIA driver: `sudo apt install nvidia-driver-550` (or newer) and reboot. - **`Docker not installed`** → let the installer handle this in the next step. - **`GPU VRAM < 8 GB`** → your card is too small. Stop here. ### 3. Run the installer ```bash ./scripts/install-linux.sh ``` The script is **interactive and idempotent** — it asks before each destructive step and skips anything already installed. It installs: - Docker Engine (via `get.docker.com`) - NVIDIA Container Toolkit (needed for `--gpus all`) - `uv` (fast Python package manager, for local development) - Node.js 20 (for the web UI) - Blender 4.2.10 LTS (native CLI, for scripting without Docker) - `usd-core` in `~/.venvs/openusd` (for running USD scripts in a terminal) At the end it will prompt for your NGC key and run `docker login nvcr.io`. ### 4. Configure your environment ```bash cp .env.example .env ``` Open `.env` in your editor and fill in **at least** these four values: ```bash NGC_API_KEY=nvapi-XXXXXXXXXXXXXXXXXXXXXXXXXXXX # from step: NVIDIA NGC ANTHROPIC_API_KEY=sk-ant-XXXXXXXXXXXXXXXXXXXXX # from Anthropic console ISAACSIM_HOST=192.168.1.50 # the IP of THIS machine # (use `hostname -I | awk '{print $1}'`) ACCEPT_EULA=Y # you accept NVIDIA's Isaac Sim EULA ``` > **Why a real IP for `ISAACSIM_HOST`?** WebRTC media goes over UDP on port 47998 and doesn't work through `127.0.0.1` when the browser and Isaac Sim are in different Docker network namespaces. Use the interface IP your browser will reach. ### 5. Start everything ```bash make up ``` This runs `docker compose up -d`. The first run downloads ~12 GB of Isaac Sim image — grab a coffee. Watch progress with: ```bash make logs # tails all service logs docker compose ps # checks health of each service ``` All five services should show `healthy` within 2–3 minutes of the first start. ### 6. Open the browser Go to **<http://localhost:3000>**. You should see a three-column layout: scene graph on the left, Isaac Sim stream in the middle, chat on the right. Wait 5–10 seconds for the stream to connect (you'll see "Connected" in green at the bottom of the viewer). Try saying in chat: > `Add a red cube named MyBox at position (0, 1, 0), then focus the camera on it.` You should see the agent call `create_mesh`, `set_transform`, and `emit_isaac_event` — and the cube should appear in the streamed viewport. ## Step-by-step install (Windows 10 / 11) Windows cannot run Isaac Sim natively, but WSL 2 with GPU passthrough works well. ### 1. Install WSL 2 and Docker Desktop Open PowerShell **as Administrator** and run: ```powershell git clone https://github.com/JinwangMok/omniverse-agent-studio.git cd omniverse-agent-studio ./scripts/install-windows.ps1 ``` This enables WSL 2, installs Ubuntu 22.04, installs Docker Desktop, and verifies your NVIDIA driver. **Reboot** when it asks. ### 2. Switch to WSL 2 After rebooting: ```powershell wsl -d Ubuntu-22.04 ``` Inside the WSL shell: ```bash cd /mnt/c/path/to/omniverse-agent-studio ./scripts/preflight-check.sh ./scripts/install-linux.sh ``` ### 3. Configure and start Same as steps 4–6 of the Linux guide above. > **Windows networking gotcha**: set `ISAACSIM_HOST=$(hostname -I | awk '{print $1}')` — the WSL 2 IP, not `localhost` or your Windows IP. ## Run the tests (any OS) The Python services and the web frontend have real test suites that don't need a GPU. Use these to verify your checkout or contribute: ```bash # USD MCP server (12 tests, runs on macOS/Linux/Windows) cd services/usd-mcp && python3.11 -m venv .venv && source .venv/bin/activate pip install -e . pytest && pytest -q # Blender MCP server (15 tests, mocks Blender — no install needed) cd services/blender-mcp && python3.11 -m venv .venv && source .venv/bin/activate pip install -e . pytest && pytest -q # Agent orchestrator (12 tests) cd services/agent && python3.11 -m venv .venv && source .venv/bin/activate pip install -e . pytest pytest-asyncio && pytest -q # Web frontend (type check) cd web && npm install --ignore-scripts && npx tsc --noEmit ``` You should see `12 passed`, `15 passed`, `12 passed`, and a clean TypeScript check. ## Services and ports | Service | Port | Description | Source | | --- | --- | --- | --- | | web | 3000 | Next.js 15 + AppStreamer viewer + chat | `web/` | | agent | 8080 | FastAPI + Claude tool-use loop, SSE | `services/agent/` | | usd-mcp | 8001 | OpenUSD MCP (13 tools, `usd-core` 26.3) | `services/usd-mcp/` | | blender-mcp | 8002 | Blender MCP (11 tools) | `services/blender-mcp/` | | blender-mcp bridge | 9876 | TCP socket to Blender addon (internal) | `services/blender-mcp/src/blender_mcp/addon.py` | | isaac-sim signaling | 49100/tcp | WebRTC signaling (host networking) | `services/isaac-sim/` | | isaac-sim media | 47998/udp | WebRTC media stream (host networking) | — | | isaac-sim viewer | 8210 | Built-in web viewer (debug) | — | ## What can the agent do? 24 tools in total — see [docs/AGENT-TOOLS.md](docs/AGENT-TOOLS.md) for full signatures. - **Scene editing (USD MCP, 13 tools)** — create/open/save stages, list prims, create Xforms and meshes (cube, sphere, plane, cylinder), set transforms, add references, manage variant sets. - **Modeling (Blender MCP, 11 tools)** — create primitives in Blender, assign materials, run arbitrary `bpy` code, take viewport screenshots, round-trip via `bpy.ops.wm.usd_export` / `usd_import`. - **Viewport control (Isaac Sim events)** — the agent can send `focus_prim`, `reload_stage`, etc. to the browser, which forwards them to Isaac Sim via `AppStreamer.sendMessage`. ## Common first-time issues | Symptom | Cause | Fix | | --- | --- | --- | | `docker pull nvcr.io/...` says "unauthorized" | Not logged in to NGC | `./scripts/login-ngc.sh` | | Viewer shows "Disconnected" forever | `ISAACSIM_HOST` is wrong or firewall blocks 49100/47998 | Set `ISAACSIM_HOST` to a reachable IP, open ports | | Agent chat replies "500 server error" | `ANTHROPIC_API_KEY` missing or invalid | Check `.env`, restart with `make down && make up` | | Isaac Sim container exits immediately | Missing `ACCEPT_EULA=Y` in `.env` | Add it, rerun `make up` | | `make up` says "nvidia-container-cli: requirement error" | `nvidia-container-toolkit` not configured | Run `./scripts/install-linux.sh` again or `sudo nvidia-ctk runtime configure --runtime=docker && sudo systemctl restart docker` | | Streaming works but looks blurry | Default resolution is 1280x720 | Add `&resolution=1920x1080&fps=60` to the AppStreamer URL | More help: [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md). ## Stop and clean up ```bash make down # stops all services make clean # also removes containers, networks, and volumes (!!!) ``` `make clean` deletes cached Isaac Sim shaders — the next `make up` will recompile them (~30 s). It does not touch your `stages/` folder. ## Further reading - [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — deep dive into data flow and design decisions. - [docs/INSTALL.md](docs/INSTALL.md) — detailed install reference. - [docs/USAGE.md](docs/USAGE.md) — example chat workflows. - [docs/AGENT-TOOLS.md](docs/AGENT-TOOLS.md) — every MCP tool with input schema. - [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) — extended troubleshooting. Research notes used to design the project: [`.omc/research/`](.omc/research/) (Isaac Sim WebRTC streaming, Blender MCP patterns, OpenUSD tooling). ## License MIT — see [LICENSE](LICENSE). Isaac Sim itself is governed by the NVIDIA Omniverse license; you accept it by setting `ACCEPT_EULA=Y`.