Content
# Hailo MCP Server for OpenClaw
MCP (Model Context Protocol) server that exposes Hailo-10H AI Accelerator tools to any MCP-compatible agent framework. All inference runs locally on the Hailo AI accelerator — zero cloud dependency.
## Architecture
```
Telegram / CLI ──► OpenClaw Gateway ──► MCP Server (stdio) ──► Python scripts ──► Hailo-10H AI Accelerator
│ │
│ mcp_server.py
│ │
│ ┌──────┼──────────┐──────────────┐
│ ▼ ▼ ▼ ▼
│ vlm_analyze whisper detect monitor_*
│ (Qwen2-VL) (Whisper) (YOLO) (YOLO+Telegram)
│ │ │ │
▼ ▼ ▼ ▼
LLM (cloud) Hailo-10H AI Accelerator (local)
```
OpenClaw spawns `mcp_server.py` as a child process using stdio transport. The MCP server exposes 6 tools that delegate to Python backend scripts, each running inference on the Hailo-10H AI Accelerator.
## Tools
| Tool | Description | Backend Script |
|------|-------------|----------------|
| `vlm_analyze` | Image analysis using Qwen2-VL-2B vision-language model | `vlm_analyze.py` |
| `whisper_transcribe` | Speech-to-text using Whisper | `whisper_transcribe.py` |
| `detect` | Object detection/counting using YOLO (single image or short video) | `detect.py` |
| `monitor_start` | Start background detection monitoring with Telegram alerts | `detect.py --monitor` |
| `monitor_stop` | Stop a running monitor session by session ID | — (PID management) |
| `monitor_list` | List active background monitoring sessions | — (PID directory scan) |
When registered with OpenClaw, tools appear with the `hailo__` prefix (e.g. `hailo__vlm_analyze`).
---
## Part 1: Prerequisites
### Hardware
- A PC or Raspberry Pi 5
- **Hailo-10H** accelerator (HAT+, M.2, or PCIe)
- A USB camera (or RPi camera module)
### Accounts & Subscriptions
- **LLM provider** - Any OpenClaw-supported provider works. This guide uses OpenAI as an example, but others are equally good.
- **Telegram account** for receiving alerts and chatting with the bot - Currently the MCP supports Telegram as the messaging interface.
## Part 2: OpenClaw Setup
### 2.1 Get Your Telegram ID
Get your numeric account ID & Save it.
### 2.2 Install OpenClaw
Follow [docs.openclaw.ai/install](https://docs.openclaw.ai/install).
### 2.3 Configure LLM
Use the **"Sign in with openai-codex auth"** option and select **GPT 5.5**: [docs.openclaw.ai/providers/openai](https://docs.openclaw.ai/providers/openai).
### 2.4 Configure Telegram Channel
When prompted to select a messaging platform during install, choose **Telegram**.
Create your bot via @BotFather (give it a meaningful name): [docs.openclaw.ai/channels/telegram](https://docs.openclaw.ai/channels/telegram).
In the OpenClaw `openclaw.json` config file Set the **allowlist** policy with your Telegram ID so the bot only responds to you:
```json
"dmPolicy": "allowlist",
"allowFrom": ["YOUR_TELEGRAM_ID"]
```
Save the **bot token** — you'll need it later for the MCP config.
### 2.5 Test OpenClaw Standalone
At this point, OpenClaw + Telegram should work as a normal LLM chat — no Hailo in the loop yet. Send a message to your bot and verify you get a response.
## Part 3: Hailo Setup
### 3.1 Install Hailo Software Stack
Install the full package: HailoRT (deb + whl), PCIe driver, Tappas (deb + whl).: https://github.com/hailo-ai/hailo-apps/blob/main/doc/user_guide/installation.md#manual-installation-advanced under "Install Hailo prerequisite packages" (reboot at the end)
Verify the device is recognized:
```bash
hailortcli fw-control identify
```
### 3.2 Install hailo-apps
Follow: [github.com/hailo-ai/hailo-apps — Install](https://github.com/hailo-ai/hailo-apps/tree/main#install-hailo-apps)
### 3.3 Test hailo-apps Standalone
Plug in the Hailo-10H and a USB camera, then run a basic detection pipeline to confirm everything works:
```bash
source setup_env.sh # activate the venv
hailo-detect --input /dev/video0 # assuming UB camera is mapped to 0
```
You should see a window with bounding boxes. Press Ctrl+C to stop.
### 3.4 Install Additional Dependencies
```bash
sudo apt install ffmpeg gstreamer1.0-tools gstreamer1.0-plugins-base \
gstreamer1.0-plugins-good gstreamer1.0-plugins-bad
```
## Part 4: Hailo MCP Integration
### Files to Deploy
```
<SCRIPTS_DIR>/
├── mcp_server.py — MCP protocol server (6 tools)
├── vlm_analyze.py — VLM inference wrapper
├── whisper_transcribe.py — Whisper speech-to-text wrapper
└── detect.py — YOLO detection + monitoring engine
```
### 4.1 Install MCP SDK
```bash
source setup_env.sh # activate the venv
pip install "mcp[cli]"
# Verify:
python -c "import mcp; print(f'mcp {mcp.__version__} OK')"
```
### 4.2 Deploy Scripts
```bash
mkdir -p ~/hailo-openclaw
# Copy all 4 .py files into ~/hailo-openclaw/
```
### 4.3 Configure OpenClaw
Explore your existing OpenClaw config at `~/.openclaw/openclaw.json` and compare it with the compact `openclaw.json` template included in this repo. Merge the Hailo MCP server block into your config, filling in every `TODO` field:
| Field | What to put |
|-------|-------------|
| `gateway.auth.token` | Run `openclaw onboard` to generate, or set any random hex string |
| `auth.profiles` key | Your email, e.g. `openai-codex:you@gmail.com` |
| `channels.telegram.botToken` | Token from @BotFather (format: `123456789:AABBcc...`) |
| `channels.telegram.allowFrom` | Your Telegram chat ID (numeric string) |
| `mcp.servers.hailo.command` | Full path to venv python, e.g. `/home/hailo/.../venv_hailo_apps/bin/python` |
| `mcp.servers.hailo.args` | Full path to mcp_server.py |
| All `env` paths | Actual paths on your system (see Environment Variables below) |
| `HAILO_TELEGRAM_BOT_TOKEN` | Same as `botToken` above |
| `HAILO_TELEGRAM_CHAT_ID` | Same as `allowFrom` above |
**Important:** Stop the gateway before replacing the config file — a running gateway auto-restores the old config on file change:
```bash
openclaw gateway stop
cp openclaw.json ~/.openclaw/openclaw.json
openclaw gateway start
```
### 4.4 Verify
```bash
openclaw config validate # Should print: Config valid
openclaw mcp list # Should show: hailo
```
Check logs for errors:
```bash
tail -5 /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log | \
python3 -c "import sys,json; [print(json.loads(l).get('message','')) for l in sys.stdin if l.strip()]"
```
No `failed to start server "hailo"` = success.
## Part 5: Test via Telegram
| # | Send to your bot | Expected tool |
|---|------------------|---------------|
| 1 | A photo + "what's in this image?" | `hailo__vlm_analyze` |
| 2 | A voice memo | `hailo__whisper_transcribe` |
| 3 | "count objects on usb camera for 5 seconds" | `hailo__detect` |
| 4 | "watch the usb camera for people" | `hailo__monitor_start` |
| 5 | "stop monitoring" | `hailo__monitor_stop` |
## Environment Variables
These are set in the `mcp.servers.hailo.env` block of `openclaw.json` and read by `mcp_server.py` at startup:
| Variable | Required | Description |
|----------|----------|-------------|
| `HAILO_APPS_ROOT` | Yes | Path to `hailo-apps` |
| `HAILO_VENV_PATH` | Yes | Path to Python venv with hailo-apps + MCP SDK |
| `HAILO_SCRIPTS_DIR` | Yes | Path to the directory containing `mcp_server.py` |
| `HAILO_VLM_HEF_PATH` | Yes | Absolute path to Qwen2-VL HEF model |
| `HAILO_WHISPER_HEF_PATH` | Yes | Absolute path to Whisper HEF model |
| `HAILO_TELEGRAM_BOT_TOKEN` | For monitoring | Telegram bot token for alert notifications |
| `HAILO_TELEGRAM_CHAT_ID` | For monitoring | Telegram chat ID for alert notifications |
| `HAILO_VLM_TIMEOUT_MS` | No | VLM timeout in ms (default: 60000) |
| `HAILO_WHISPER_TIMEOUT_MS` | No | Whisper timeout in ms (default: 300000) |
| `HAILO_DETECTION_TIMEOUT_MS` | No | Detection timeout in ms (default: 120000) |
**Note**: The `env` block in the MCP config is required because MCP servers are spawned as separate child processes and do not inherit the global `env` from `openclaw.json`.
## Common Issues
| Problem | Fix |
|---------|-----|
| `ModuleNotFoundError: No module named 'mcp'` | `command` must be the full venv python path, not `python3` |
| `PYTHONPATH is blocked for stdio startup safety` | Don't put `PYTHONPATH` in the MCP env block — `mcp_server.py` handles it internally |
| Tools don't appear in conversations | Set `tools.profile` to `"full"` in openclaw.json |
| Config reverts after editing | Stop gateway first, then edit, then start (see Step 3) |
| `missing api key for openai` | Use `openai-codex/gpt-5.5` (OAuth), not `openai/gpt-5.5` (needs API key) |
| Monitor window fails / "cannot open display" | Run `xhost +local:` on the Pi, or add to `~/.profile` |
