Content
# VLM MCP Server
[中文文档](README.zh-CN.md) | **English**
A Model Context Protocol (MCP) server providing vision & video analysis tools, **configurable with any model provider**.
This is a reverse-engineered and extended reimplementation of `@z_ai/mcp-server` (Apache-2.0, credit to Chao Gong, Lei Yuan / Z.AI). It introduces a **provider abstraction layer** so the same set of tools can run against any of three API families:
- **Chat Completions** — OpenAI-compatible `POST {base}/chat/completions` (OpenAI, Z.AI, Zhipu, OpenRouter, Together, Groq, DeepSeek, Moonshot, local Ollama / LM Studio, …)
- **Responses** — OpenAI `POST {base}/responses` (gpt-4o, o-series reasoning models)
- **Anthropic Messages** — `POST {base}/v1/messages` (Claude, and Anthropic-compatible gateways)
## Quick Start
```shell
npx -y @syntx-ai/vlm-mcp-server
```
That's it for the server side — it speaks MCP over stdio. You need to configure it in your MCP client. Pick your provider and set three environment variables:
| Provider | Environment variables |
|----------|----------------------|
| Chat Completions | `OPENAI_CHAT_COMPLETIONS_API_KEY` · `OPENAI_CHAT_COMPLETIONS_BASE_URL` · `OPENAI_CHAT_COMPLETIONS_MODEL` |
| Responses | `OPENAI_RESPONSES_API_KEY` · `OPENAI_RESPONSES_BASE_URL` · `OPENAI_RESPONSES_MODEL` |
| Anthropic | `OPENAI_ANTHROPIC_API_KEY` · `OPENAI_ANTHROPIC_BASE_URL` · `OPENAI_ANTHROPIC_MODEL` |
**Claude Code one-liner** (Chat Completions example — replace with your values):
```shell
claude mcp add -s user vlm-mcp-server \
--env OPENAI_CHAT_COMPLETIONS_API_KEY=sk-... \
OPENAI_CHAT_COMPLETIONS_BASE_URL=https://api.openai.com/v1/ \
OPENAI_CHAT_COMPLETIONS_MODEL=gpt-4o \
-- npx -y @syntx-ai/vlm-mcp-server
```
For other clients (Cline, OpenCode, Crush, Roo Code, …), see [Client Configuration](docs/clients.md).
## Available Tools
### Image Analysis
| Tool | Description |
|------|-------------|
| `ui_to_artifact` | Convert UI screenshots to code, prompts, specs, or descriptions |
| `extract_text_from_screenshot` | OCR — extract code, terminal output, or text from screenshots |
| `diagnose_error_screenshot` | Analyze error messages and stack traces, suggest fixes |
| `understand_technical_diagram` | Analyze architecture, flowchart, UML, ER, and sequence diagrams |
| `analyze_data_visualization` | Extract insights, trends, and anomalies from charts |
| `ui_diff_check` | Visual regression — compare expected vs actual UI, prioritize issues |
| `analyze_image` | General-purpose image analysis (fallback) |
### Video Analysis
| Tool | Description |
|------|-------------|
| `analyze_video` | Video content analysis (local files or URLs, ≤8MB, MP4/MOV/M4V) |
## Configuration
The server loads variables from a `.env` file at startup (real environment variables take precedence). Three layers are supported; precedence is **per-provider groups > generic > legacy**.
### Per-provider groups
Configure each API family independently. `auto` picks the first group with both a key and a base URL set.
| Variable group | API family |
|----------------|------------|
| `OPENAI_CHAT_COMPLETIONS_API_KEY` / `_BASE_URL` / `_MODEL` | Chat Completions |
| `OPENAI_RESPONSES_API_KEY` / `_BASE_URL` / `_MODEL` | Responses |
| `OPENAI_ANTHROPIC_API_KEY` / `_BASE_URL` / `_MODEL` | Anthropic Messages |
### Generic variables
| Variable | Description | Default |
|----------|-------------|---------|
| `VLM_API_KEY` | API key | *(required)* |
| `VLM_BASE_URL` | Provider API root | Zhipu default |
| `VLM_VISION_MODEL` | Model name | `glm-4.6v` |
| `VLM_PROVIDER` | Provider family: `auto` / `chat-completions` / `responses` / `anthropic` | `auto` |
| `VLM_VISION_MODEL_TEMPERATURE` | Sampling temperature | `0.8` |
| `VLM_VISION_MODEL_TOP_P` | Top-p | `0.6` |
| `VLM_VISION_MODEL_MAX_TOKENS` | Max output tokens | `32768` |
| `VLM_TIMEOUT` | Request timeout (ms) | `300000` |
| `VLM_RETRY_COUNT` | Retry attempts | `1` |
| `VLM_ENABLE_THINKING` | Enable provider-specific reasoning / thinking request fields. Off by default for broad OpenAI-compatible Chat Completions support. | `false` |
| `VLM_ANTHROPIC_VERSION` | `anthropic-version` header (Anthropic only) | `2023-06-01` |
| `VLM_LOG_PATH` | Custom log file path | `~/.vlm/vlm-mcp-YYYY-MM-DD.log` |
### Provider auto-detection
In `auto` mode (when no `OPENAI_*` group is set), the provider is inferred as follows:
- Base URL contains `anthropic`, or key starts with `sk-ant` → `anthropic`
- Otherwise → `chat-completions` (the most broadly compatible default)
## Usage Examples
Once the server is installed in your client, you can use it through conversation. For example, in Claude Code, type `describe this demo.png` — the MCP Server will process the image and return a description (the image must exist in the current directory).
> Outside Claude Code, pasting an image directly into the client will NOT invoke this MCP Server — the client encodes the image and calls the model API itself. **Best practice**: place images in a local directory and refer to them by name or path in conversation, e.g. `What does demo.png describe?`
## Troubleshooting
Run the server directly from the command line to verify it starts, isolating environment / permission issues:
```bash
# Linux / macOS
OPENAI_CHAT_COMPLETIONS_API_KEY=sk-... \
OPENAI_CHAT_COMPLETIONS_BASE_URL=https://api.openai.com/v1/ \
OPENAI_CHAT_COMPLETIONS_MODEL=gpt-4o \
npx -y @syntx-ai/vlm-mcp-server
# Windows CMD
set OPENAI_CHAT_COMPLETIONS_API_KEY=sk-... && set OPENAI_CHAT_COMPLETIONS_BASE_URL=https://api.openai.com/v1/ && set OPENAI_CHAT_COMPLETIONS_MODEL=gpt-4o && npx -y @syntx-ai/vlm-mcp-server
# Windows PowerShell
$env:OPENAI_CHAT_COMPLETIONS_API_KEY="sk-..."; $env:OPENAI_CHAT_COMPLETIONS_BASE_URL="https://api.openai.com/v1/"; $env:OPENAI_CHAT_COMPLETIONS_MODEL="gpt-4o"; npx -y @syntx-ai/vlm-mcp-server
```
- If it starts successfully, the environment is correct — the issue is likely in the client's MCP config; double-check it.
- If it fails, investigate the error message (pasting it to an LLM for analysis is recommended).
### Common issues
**Connection failure**
1. Ensure Node.js 18 or newer is installed.
2. Run `node -v` and `npx -v` to confirm the runtime is available.
3. Verify the environment variables (`OPENAI_*` triple or `VLM_*`) are set correctly.
**Invalid API Key**
1. Confirm the API Key was copied correctly.
2. Check that the API Key is activated.
3. Ensure the selected provider family matches the API Key (Chat Completions / Responses / Anthropic).
4. Check that the API Key has sufficient balance.
**Connection timeout**
1. Check your network connection.
2. Check firewall settings.
3. Try switching to a different provider family or base URL.
4. Increase the timeout (`VLM_TIMEOUT`, default 300000ms).
## Architecture
```
src/
├── index.ts # Entry point: starts the MCP server, registers all tools
├── types/ # Error types (McpError, ApiError, ValidationError, …)
├── core/
│ ├── environment.ts # Env config (VLM_* + OPENAI_* groups), URL resolution
│ ├── chat-service.ts # Delegates to the active VisionProvider
│ ├── file-service.ts # File validation + base64 encoding (image/video)
│ ├── base-image-service.ts # Shared image-processing logic for all image tools
│ ├── api-common.ts # Message builders, response helpers, retry wrapper
│ ├── error-handler.ts # Error hierarchy + handling/recovery strategies
│ └── logger.ts # stderr + file logger (keeps stdout JSON-clean)
├── providers/ # Pluggable model-provider abstraction
│ ├── types.ts # VisionProvider interface, ChatMessage, postJson helper
│ ├── chat-completions.ts # OpenAI-compatible Chat Completions
│ ├── responses.ts # OpenAI Responses API
│ ├── anthropic.ts # Anthropic Messages API
│ └── index.ts # Provider selection (VLM_PROVIDER / auto-infer)
├── prompts/ # System prompts for each specialized tool
└── tools/ # 8 tool registrations (7 image + 1 video)
```
The **provider layer** (`src/providers/`) is the key extension. Each provider implements a `VisionProvider` interface that takes normalized `ChatMessage[]` (the OpenAI Chat Completions content-part format as internal lingua franca) and translates it to the provider's wire format. `chat-service.ts` simply delegates to the resolved provider, so none of the tool code needed to change.
## License
Apache-2.0
