Content
# Tool List
Based on NVIDIA Isaac Sim 5.1.0 Docker, supporting WebRTC remote viewing. Code edited on the host machine, running in the Docker container.
## Directory Structure
```
code/
├── README.md # This file
├── restart.sh # One-click restart after code changes
├── start.sh # First startup
├── lib/
│ ├── config.py # Startup configuration, UI scaling defaults
│ ├── livestream.py # WebRTC streaming
│ ├── scene.py # Scene, camera, robot
│ ├── ui.py # Interface scaling
│ └── mcp.py # MCP Extension
├── demos/ # Simulation entry scripts (each file can run independently)
│ ├── franka_livestream.py # Default: Franka arm demonstration
│ └── _template.py # New script template
├── mcp/ # MCP (installed by setup_mcp.sh)
│ ├── .venv/ # MCP Server Python environment
│ └── isaacsim-mcp-server/ # Omniverse Extension source code
├── .cursor/
│ └── mcp.json # Cursor MCP configuration
└── scripts/
├── docker_run.sh # Docker startup logic
├── run_mcp_server.sh # MCP Server
└── setup_mcp.sh # MCP installation
```
Container path: `/workspace/` ↔ Host machine `/home/ubuntu/isaac-dexterous-lab/`
## Daily Development Process
```bash
# 1. Modify code in Cursor (lib/ or demos/)
# 2. Restart with scene name after saving
./restart.sh franka
# 3. WebRTC connection to 117.50.175.186
```
## Common Commands
| Command | Description |
|------|------|
| `./restart.sh` | Restart default scene (franka) |
| `./restart.sh franka` | Franka arm (default) |
| `./restart.sh kuka_allegro` | Kuka iiwa7 + Allegro dexterous hand (single arm) |
| `./restart.sh allegro_hand` | Allegro dexterous hand (only hand) + table + block |
| `./restart.sh shadow_hand` | Shadow Hand + table + block |
| `./restart.sh wuji_hand2` | Wuji Hand 2 + table + block (requires `./scripts/setup_wuji_hand2.sh` first) |
| `./restart.sh dual_kuka_allegro` | Dual Kuka+Allegro + table + block |
| `./restart.sh bimanual` | OpenArm dual arms + table + block |
| `./restart.sh list` | List all available scenes |
| `./start.sh warehouse` | First startup with specified scene |
| `./start.sh --follow-logs` | Start default scene and follow logs |
| `sudo docker logs -f isaac-sim-robot` | View running logs |
| `sudo docker stop isaac-sim-robot` | Stop simulation |
## Writing a New Scene
```bash
# 1. Copy from template (file name is scene name)
cp demos/_template.py demos/warehouse.py
# 2. Edit demos/warehouse.py
# 3. Start with scene name
./restart.sh warehouse
```
Scene name rule: `warehouse` → `demos/warehouse.py`, if it doesn't exist, try `demos/warehouse_livestream.py`.
### Adjusting WebRTC Interface Font Size
Default UI scaled **2.0 times**. Can be modified with environment variable during startup:
```bash
UI_DPI_SCALE=2.0 ./restart.sh franka # Scale up 2 times
UI_DPI_SCALE=1.0 ./restart.sh franka # Restore default size
```
Can also directly modify `UI_DPI_SCALE` default value in `lib/config.py`.
## MCP (Cursor Control Isaac Sim)
Using [isaacsim-mcp-server](https://github.com/whats2000/isaacsim-mcp-server) to let Cursor control simulation with natural language (loading robots, building scenes, stepping through debugging, etc.).
### Architecture
```
Cursor ←SSE 8767→ MCP Server (host machine, automatically started by restart.sh in the background)
↓ TCP 8766
Docker internal Isaac MCP Extension
```
### First Installation
```bash
./scripts/setup_mcp.sh
```
### Daily Use (one command)
```bash
./restart.sh franka # Isaac Sim + MCP Extension + MCP Server ready
```
Then enable `isaac-sim` in Cursor **Settings → MCP** (`.cursor/mcp.json` already configured `http://127.0.0.1:8767/sse`).
Manual MCP Server management:
```bash
./scripts/run_mcp_server.sh --status
./scripts/run_mcp_server.sh --stop
./scripts/run_mcp_server.sh --daemon # Start SSE in the background separately
```
### Cursor Configuration
Project includes `.cursor/mcp.json`. When **Remote SSH connected to this cloud server**, `isaac-sim` should be visible in Cursor Settings → MCP.
If Cursor is on **local computer** and Isaac Sim is on the cloud, SSH tunnel (forwarding MCP SSE port) is required:
```bash
ssh -L 8767:127.0.0.1:8767 ubuntu@117.50.175.186
```
Then configure MCP in local Cursor to connect to `http://127.0.0.1:8767/sse`.
### Verification
Call `get_scene_info` in Cursor to check if the connection is normal.
### Script Structure Convention
Each `demos/*.py` follows a fixed order:
```python
from isaacsim import SimulationApp
simulation_app = SimulationApp(...) # ① Must be the first
from lib.livestream import setup_livestream # ② Import other modules
from lib.scene import create_world_with_ground, load_robot, ...
setup_livestream(simulation_app) # ③ Configure WebRTC
world = create_world_with_ground() # ④ Build scene
while simulation_app._app.is_running(): # ⑤ Simulation loop
world.step(render=True)
```
### What's in lib/
| Module | Function |
|------|------|
| `lib/config.py` | Startup resolution, `UI_DPI_SCALE` default value |
| `lib/livestream.py` | `setup_livestream()` WebRTC streaming |
| `lib/scene.py` | `create_world_with_ground()`, `load_robot()`, etc. |
| `lib/mcp.py` | `setup_mcp()` Enable MCP Extension |
`load_robot()` built-in robots: `franka`, `ur10`, can be extended in `lib/scene.py`'s `ROBOT_ASSETS`.
## Environment Information
| Project | Value |
|------|-----|
| Isaac Sim | 5.1.0 |
| Docker image | `nvcr.io/nvidia/isaac-sim:5.1.0` |
| GPU | RTX 4090 |
| Public IP | `117.50.175.186` (in `/home/ubuntu/.env`) |
## WebRTC Connection
1. Download [WebRTC Streaming Client](https://docs.isaacsim.omniverse.nvidia.com/5.1.0/installation/manual_livestream_clients.html)
2. Server fill `117.50.175.186` (only fill IP)
3. Security group allow TCP `49100`, UDP `47998`
## Host Machine vs Container
| Where | What can be done |
|------|---------|
| Host machine `ubuntu@...` | Modify code, `./restart.sh`, `sudo docker ...` |
| Container `isaac-sim@...` | Only `./python.sh`, no docker/sudo |
## Default Empty Interface (without Python script)
```bash
cd /home/ubuntu && sudo docker compose up -d
```
## FAQ
**Can't see robot** → Confirm using `./restart.sh`, logs show `Franka arm loaded`
**Restart shows waiting timeout** → Container might still be loading (2-5 minutes), execute `sudo docker logs -f isaac-sim-robot` to wait
**Stopping container will restart cloud server?** → No, only stop Docker container
## Reference
- [Isaac Sim container installation](https://docs.isaacsim.omniverse.nvidia.com/5.1.0/installation/install_container.html)
- [Python standalone script](https://docs.isaacsim.omniverse.nvidia.com/5.1.0/python_scripting/manual_standalone_python.html)
