Content
# Live2D MCP Companion
Interactive desktop companion with Live2D model, controlled via MCP (Model Context Protocol). Works on Windows.
## Features
- **Live2D Character** — Animated Hiyori model on desktop (via PyQt6 + OpenGL)
- **TTS (Text-to-Speech)** — Voice output via Piper TTS (Russian voices: Irina, Denis, Ruslan)
- **STT (Speech-to-Text)** — Speech recognition via faster-whisper ⚠️ (temporarily not working)
- **MCP Server** — 36 tools for controlling the character via any MCP client
- **RAG (Retrieval-Augmented Generation)** — Semantic search by documents
- **Web Interface** — Control via browser at `http://localhost:8766`
- **Music Player** — MP3 playback via pygame
- **Alarm and Timer** — With MCP notifications
- **Screenshots** — Screen capture with character reaction
- **Game Modes** — Hide and seek, dance, emotions, etc.
- **Additional Models** — [Nagatoro Sprite](https://github.com/nukkerman2000/nagatoru-live2d) (sprite model in a separate repository)
## System Requirements
- Windows 10/11
- Python 3.12+
- Graphics card with OpenGL support
- Microphone (for STT)
## Installation
### 1. Clone the Repository
```bash
git clone https://github.com/nukkerman2000/live2d-mcp.git
cd live2d-mcp
```
### 2. Install Portable Python
Download [embeddable Python 3.12](https://www.python.org/downloads/windows/) and unpack into the `python\` folder.
Alternatively, use the system Python.
### 3. Install Dependencies
Run `setup_python.bat` or execute manually:
```bash
python\python.exe get-pip.py
python\Scripts\pip.exe install -r requirements.txt
```
### 4. Configure
Copy `config.example.json` to `config.json` and customize.
### 5. Download Voice Models
Download Piper voices into the `voices\` folder:
- [ru_RU-irina-medium](https://huggingface.co/rhasspy/piper-voices/resolve/main/ru/ru_RU/irina/medium/ru_RU-irina-medium.onnx)
- [ru_RU-denis-medium](https://huggingface.co/rhasspy/piper-voices/resolve/main/ru/ru_RU/denis/medium/ru_RU-denis-medium.onnx)
- [ru_RU-ruslan-medium](https://huggingface.co/rhasspy/piper-voices/resolve/main/ru/ru_RU/ruslan/medium/ru_RU-ruslan-medium.onnx)
### 6. Download Piper TTS Binaries
Download [Piper TTS for Windows](https://github.com/rhasspy/piper/releases) and unpack `piper.exe` and DLL into the `piper\` folder.
## Launch
```bash
start.bat
```
After launch:
- MCP Server: `http://127.0.0.1:8765/mcp`
- Web Interface: `http://127.0.0.1:8766`
## MCP Tools
Total 36 tools for controlling the character via MCP.
### Model Control
| Tool | Description |
|---|---|
| `switch_model` | Switch between models (hiyori, nagatoro_sprite) |
| `set_emotion` | Set emotion/ facial expression |
| `set_mouth_open` | Control mouth opening (lip sync) |
| `play_motion` | Play motion animation |
| `stop_all_motions` | Stop all animations |
| `list_motion_groups` | List available animation groups |
| `set_eye_position` | Control eye direction |
| `set_parameter` | Set any Live2D model parameter |
### Character Window
| Tool | Description |
|---|---|
| `show` / `hide` | Show/hide character window |
| `move_window` | Move window by coordinates |
| `resize_window` | Change window size |
| `center_window` | Center window on screen |
| `get_status` | Get character and window status |
### TTS / Voice
| Tool | Description |
|---|---|
| `speak` | Speak text via TTS (with lip sync) |
| `list_voices` | List available voices |
| `stop_speaking` | Stop speaking |
### Music
| Tool | Description |
|---|---|
| `list_tracks` | List MP3 tracks |
| `play_track` / `stop_music` | Play/stop track |
| `next_track` / `prev_track` | Next/previous track |
| `play_random_track` | Random track |
| `set_loop` / `set_autoplay` | Repeat and autoplay modes |
| `set_music_volume` | Music volume |
| `set_mcp_music` | Allow MCP to control music |
### Alarm and Timer
| Tool | Description |
|---|---|
| `set_alarm_text` | Set alarm (time HH:MM, text) |
| `set_timer` | Start timer (in minutes) |
| `get_alarm_timer_status` | Alarm and timer status |
### Other
| Tool | Description |
|---|---|
| `take_screenshot` | Take screen screenshot |
| `list_available_emotions` | List available emotions |
| `query_documents` | Search by documents (RAG) |
| `ingest_file` | Load file into RAG index |
| `list_documents` | List indexed documents |
| `rag_status` | RAG system status |
## Configuration
Main settings in `config.json`:
- TTS engine, voice, speed
- MCP and web interface ports
- Enable/disable STT, music, alarm
- Emotions and animations for different modes
- Phrases for interactive communication
## Project Structure
```
├── run.py # Entry point
├── mcp_server.py # MCP server
├── live2d_display.py # Live2D rendering (PyQt6 + OpenGL)
├── talk.py # Dialogue and interactivity logic
├── tts.py # TTS engine (Piper)
├── stt.py # Speech recognition
├── music_player.py # Music player
├── alarm_timer.py # Alarm and timer
├── voice_bridge.py # Voice bridge
├── web_ui.py # Web interface (Starlette + uvicorn)
├── mcp_client.py # MCP client
├── phrases.py / .json # Character phrases
├── ui.js # Web interface client-side
├── rag/ # RAG engine
├── play_modes/ # Game modes
└── assets/live2d/ # Live2D and sprite models (Hiyori, Nagatoro_sprite)
```
## Security
The application does not send data to the internet (except for loading STT/TTS models on the first launch). MCP server listens only on `127.0.0.1`.
