Content
# CarlaMCP
Natural language → executable autonomous-vehicle scenarios in the
[CARLA](https://carla.org/) simulator, exposed to any LLM through the
[Model Context Protocol](https://modelcontextprotocol.io/).
Built on the tool-based architecture of [IROSA](https://arxiv.org/abs/2603.03897),
validated against [CARLOS](https://github.com/ika-rwth-aachen/carlos) for
containerized deployment.
A prompt like *"an aggressive cut-in from the right at highway speed in heavy
rain at night"* becomes a validated, reproducible scenario that runs in CARLA —
and can be saved as a spec for replay.
---
## How it works
```
NL prompt ─▶ LLM picks tools ─▶ validated ScenarioManifest ─┬─▶ execute in CARLA ─▶ ScenarioResult
(Pydantic primitives) │
└─▶ save .json spec ─▶ replay / reuse
```
Three layers, each independently testable:
| Layer | Package | Responsibility |
|---|---|---|
| **Primitives** | `carlamcp/primitives/` | Pydantic models with LLM-facing descriptions, range constraints, and cross-field validators. The LLM never writes raw simulator code. |
| **Executors** | `carlamcp/executors/` | Translate validated primitives into `carla` API calls. The only layer that touches the live simulator. |
| **Server** | `carlamcp/server.py` | The MCP server exposing tools to the model. |
## Project structure
```
CarlaMCP/
├── carlamcp/
│ ├── config.py # connection + simulation/spawn constants
│ ├── enums.py # CARLAMap, Side, WeatherCondition
│ ├── results.py # ScenarioResult
│ ├── carla_client.py # CARLA binding import + get_client()
│ ├── world.py # map / actor / sync-mode / find_ego helpers
│ ├── storage.py # save / load scenario specs (export & replay)
│ ├── primitives/ # one file per primitive + the manifest
│ │ ├── cut_in.py pedestrian.py weather.py manifest.py
│ ├── executors/ # one executor per primitive + the orchestrator
│ │ ├── cut_in.py pedestrian.py weather.py manifest.py
│ └── server.py # FastMCP server + tools + main()
├── tests/ # pytest suite, runs WITHOUT a simulator
│ ├── fakes.py conftest.py test_executor.py test_primitives.py
├── pyproject.toml requirements.txt .gitignore LICENSE README.md
```
---
## Setup
### 1. Start CARLA (separate terminal)
```bash
~/carla/CarlaUE4.sh -RenderOffScreen -quality-level=Low &
```
### 2. Install CarlaMCP
```bash
cd CarlaMCP
pip install -e ".[dev]" # package + test deps (no simulator needed)
pip install -e ".[carla,dev]" # also installs the carla==0.9.15 binding
```
### 3. Verify the CARLA connection
```bash
python -c "from carlamcp.carla_client import get_client; print(get_client().get_server_version())"
```
### 4. Run the MCP server
```bash
python -m carlamcp.server # or just: carlamcp
```
### 5. Wire into Claude Desktop
Add to your `claude_desktop_config.json` (use the absolute path to *your* clone):
```json
{
"mcpServers": {
"carlamcp": {
"command": "python",
"args": ["-m", "carlamcp.server"],
"cwd": "<path-to-your-CarlaMCP-clone>"
}
}
}
```
Restart Claude Desktop; the CarlaMCP tools will appear.
---
## Usage
Once connected, talk to the model:
- *"Check if CARLA is running"* → `get_status`
- *"Run an aggressive cut-in from the right at 120 km/h with 1.2s TTC"* → `run_scenario`
- *"Pedestrian crosses 25 m ahead, occluded by a parked car, in heavy fog"*
- *"Set the weather to rain_night at intensity 0.8"* → `set_weather`
- *"Validate this manifest before running it: {...}"* → `validate_scenario`
### Tools
| Tool | Purpose |
|---|---|
| `get_status` | Check the simulator is reachable. |
| `list_maps` | List supported towns and their characteristics. |
| `run_scenario` | Build + validate + execute a scenario (optional `save_as`). |
| `set_weather` | Apply a weather preset only. |
| `validate_scenario` | Validate a manifest JSON without running it. |
| `save_scenario` | Validate and persist a manifest for later. |
| `replay_scenario` | Load a saved spec and execute it. |
### Scenario primitives
| Primitive | Key parameters | Default map |
|---|---|---|
| `CutInManeuver` | `speed_kmh`, `ttc_s`, `side`, `ego_speed_kmh` | Town04 (motorway) |
| `PedestrianCross` | `gap_acceptance_s`, `crossing_distance_m`, `occluded` | Town03 (urban) |
| `WeatherOverlay` | `condition`, `intensity` | Any |
### Export & replay
Any scenario can be persisted as a validated JSON spec and re-run later — useful
for regression sets, sharing, and scenario inpainting / augmentation pipelines:
```text
run_scenario("...", cut_in_speed_kmh=120, cut_in_ttc_s=1.2, cut_in_side="right",
save_as="rainy_cutin") # writes scenarios/rainy_cutin.json
replay_scenario("rainy_cutin") # re-executes the exact same spec
```
The export layer (`carlamcp/storage.py`) is deliberately decoupled from
execution, so generated specs can be consumed by downstream tooling without a
running simulator. See **Roadmap → OpenSCENARIO export**.
---
## Testing
The suite runs entirely offline — `tests/conftest.py` injects a fake `carla`
module so the executor logic (spawn geometry, velocities, ordering, error paths,
collision metric) is verified without a GPU or a running simulator. This makes
it CI-friendly.
```bash
pip install -e ".[dev]"
pytest -q
```
`tests/test_executor.py` doubles as the canonical example of the project's
Google-style (Sphinx [Napoleon](https://sphinxcontrib-napoleon.readthedocs.io/))
docstring convention used across every module.
---
## FAQ — why Python and not C++?
CARLA runs on Unreal Engine (C++), so it's a fair question. For CarlaMCP, C++
buys nothing: the `carla` PyPI package is a thin binding over the C++
`LibCarla` client, and everything done here (spawning actors, weather, the
traffic manager) is fully exposed in Python with negligible overhead at these
tick rates. C++ only becomes worthwhile for work that lives *inside the CarlaUE4
server build* — custom Unreal sensors/actors/map assets, per-frame
high-bandwidth sensor processing where the GIL is the bottleneck, or C++
ScenarioRunner extensions. None of those are on the roadmap, so CarlaMCP stays
pure Python.
---
## Roadmap
- [x] Phase 1 — Validated primitives + MCP server + CARLA execution
- [x] Phase 2 — Modular package, offline test suite, scenario export/replay
- [ ] Phase 3 — CARLOS Docker integration (`docker-compose.yml`)
- [ ] Phase 4 — CSR/ISR/TCR evaluation harness
- [ ] Phase 5 — OpenSCENARIO export for the CARLOS replay pipeline
- [ ] Paper — IEEE IV / ITSC submission
## Related work
- [IROSA](https://arxiv.org/abs/2603.03897) — tool-based LLM architecture for robot skill adaptation (DLR/TUM)
- [CARLOS](https://github.com/ika-rwth-aachen/carlos) — containerized CARLA simulation framework (ika RWTH Aachen)
- [SUMO-MCP](https://arxiv.org/abs/2506.03548) — MCP for traffic simulation (adjacent domain)
## License
[MIT](LICENSE).
