Content
# MiniCPM Vision MCP
For **single-modal large models like DeepSeek V4.0**, equip them with eyes and ears - a locally running visual + audio MCP server, based on **Ollama + MiniCPM-V 4.6 + faster-whisper**.
> Image description · Video frame analysis · Audio speech-to-text · Completely local · Data stays on device
## ✨ Why Choose This?
| | This Project | xkiranj/ollama-vision-mcp | MikeyBeez/mcp-vision |
|---|---|---|---|
| Image Recognition | ✅ | ✅ | ✅ |
| **Video Analysis** | ✅ Unique! | ❌ | ❌ |
| **Audio Transcription** | ✅ Unique! | ❌ | ❌ |
| Chinese Model | ✅ MiniCPM native Chinese | ❌ llava mainly English | ❌ |
| Installation | 1 file, just drop it in | Python venv + pip | macOS only |
| Windows | ✅ | ⚠️ | ❌ |
| Auto Cleanup | ✅ Automatically delete video after analysis | ❌ | ❌ |
## 🚀 Quick Start
### 1. Install Ollama and Pull the Model
```bash
ollama pull minicpm-v4.6
```
### 2. Install Dependencies
```bash
npm install @ffmpeg-installer/ffmpeg @modelcontextprotocol/sdk
pip install faster-whisper
```
### 3. Configure AI Agent
**Claude Desktop** (`claude_desktop_config.json`)：
```json
{
"mcpServers": {
"vision": {
"command": "node",
"args": ["/path/to/vision_mcp_server.mjs"]
}
}
}
```
**Reasonix** (`reasonix.toml`)：
```toml
[[plugins]]
name = "vision"
command = "node"
args = ["/path/to/vision_mcp_server.mjs"]
```
Restart the client to use.
## 🛠️ Tools
### `describe_image` — Describe Image
```
Input: Absolute path to image file
Output: Chinese image content description
```
### `describe_video` — Analyze Video
```
Input: Video file path or HTTP/HTTPS URL, interval (optional)
Output: Timeline frame description + AI summary + Auto cleanup
```
### `describe_audio` — Audio to Text 🆕
```
Input: Video/audio file path or URL
Output: Audio metadata + faster-whisper speech-to-text
```
## 📊 Video Analysis Process
```
URL/file → Download → FFmpeg extract key frames → MiniCPM-V frame-by-frame analysis → Summary → Auto cleanup
```
## 🔧 Dependencies
- [Ollama](https://ollama.com) + [MiniCPM-V 4.6](https://ollama.com/library/minicpm-v4.6)
- [FFmpeg](https://ffmpeg.org) via `@ffmpeg-installer/ffmpeg`
- [faster-whisper](https://github.com/SYSTRAN/faster-whisper) for audio transcription
- [@modelcontextprotocol/sdk](https://npmjs.com/package/@modelcontextprotocol/sdk)
## 🌍 Applicable Scenarios
- 📸 Let pure text models like DeepSeek understand screenshots and photos
- 🎬 Quickly understand video content without watching frame by frame
- 🎵 Extract speech from videos and transcribe to text
- 📱 Analyze videos from platforms like Douyin/Bilibili
- 🔒 Local processing of sensitive images/videos, data stays on device
- 🇨🇳 Chinese content understanding (MiniCPM native Chinese)
## 📄 License
MIT License
## 🙏 Acknowledgements
- [MiniCPM-V](https://github.com/OpenBMB/MiniCPM-V) · [Ollama](https://ollama.com) · [faster-whisper](https://github.com/SYSTRAN/faster-whisper) · [MCP](https://modelcontextprotocol.io)
