VisionMCP

 # VisionMCP **Vision tools for AI agents** VisionMCP is an MCP server for working with images, long screenshots, OCR, PDFs, Office files, and web pages. For text-only models, it can act as the visual layer. For multimodal models, it is useful when native vision is not enough — for example long screenshots, small text, document rendering, web interaction, or a second visual check. > 中文简介：VisionMCP 是一个给 AI Agent 使用的视觉工具。纯文本模型可以把它当视觉入口；已经支持多模态的模型，则可以在长图、小字、OCR、PDF / Office、网页和局部精读等场景下调用它做进一步处理。 --- ## What it can do - **Images / screenshots** — inspect an image and focus on task-related visual information - **OCR** — read text, code, error messages, numbers, and tables - **Zoom into regions** — locate a useful area and crop from the original image for a closer look - **Long screenshots** — split dense or very long images for OCR when needed - **PDF** — render pages and pass them through the same visual pipeline - **Office files** — render DOCX / PPTX / XLSX pages before visual analysis - **Web pages** — open and render live pages with a browser - **Web interaction** — click, fill, select, scroll, switch tabs, and re-observe the page - **Structured output** — return task-focused fields such as `summary`, `critical_text`, and `evidence` - **Telemetry** — record model route, token usage, latency, cache hits, zoom / OCR activity, and errors --- ## Text-only vs multimodal models VisionMCP is designed to work with both. ### Text-only models Use `vision` whenever the task depends on visual information. Typical flow: ```text image / PDF / Office / webpage ↓ VisionMCP ↓ visual model / OCR ↓ structured evidence ↓ text-only host continues reasoning ``` ### Multimodal models For normal images, use the host model's native vision first. Call VisionMCP when you need: - precise OCR - long screenshot handling - small-text reading - region zoom / crop - PDF or Office rendering - web rendering or interaction - a second visual check when the native answer is uncertain The MCP tool does not try to auto-detect the host model type. The host simply follows the usage pattern that matches its own capability. --- ## How VisionMCP looks at an image One of the core ideas is simple: ```text look at the whole image ↓ find what matters for the current task ↓ if the detail is unclear ↓ crop the original image ↓ zoom / OCR / stronger visual pass ↓ return only useful evidence ``` This avoids treating every image as a full-image transcription task. The current task routes are: - `inspect` — screenshots, UI states, terminals, general visual checks - `ocr` — text extraction, code, numbers, small text - `analyze` — charts, diagrams, flows, spatial relationships - `auto` — select a route from the task --- ## Why use it if models already support vision? Native multimodal models are already good enough for many normal images. VisionMCP is not meant to replace that. It is mainly useful when the task needs extra handling around vision: | Task | Native vision | VisionMCP | |---|---|---| | Normal image | Usually enough | Usually unnecessary for multimodal hosts | | Long screenshot / small text | Can miss details | Crop, zoom, OCR, or tile when needed | | PDF / Office | Depends on model/API support | Render pages before visual analysis | | Web page | Can understand a screenshot | Can open and interact with the live page | | Text-only host | No native image input | Acts as the visual layer | | Provider compatibility | Uses the current model/provider | DeepSeek and Qwen adapters have been validated through a common interface | The main runtime currently still defaults to Qwen-based routes through Alibaba Cloud Bailian. The DeepSeek / Qwen provider adapters under `benchmark_v2/providers/` are compatibility-validation code, not production cross-provider auto-routing yet. --- ## What I tested I used a small controlled benchmark to check where native vision works well and where extra visual handling may help. ### Pilot In a 10-case pilot: - DeepSeek Vision: **9/10** - Qwen Vision: **9/10** - both failed on the same 5000px terminal screenshot On that failed screenshot, a separate ROI crop + OCR experiment recovered the target: ```text ERROR-5123 ``` This suggests that local re-reading can help on dense long screenshots, but that experiment was a simulated crop/OCR check rather than proof that the real pipeline automatically triggered zoom. ### Real pipeline check I also ran the real `vision_server.vision()` entry point on five closed-loop cases: - long screenshot - small text - chart - scanned PDF - webpage rendering All five completed successfully. For the long-terminal case, the real pipeline read `ERROR-5123` directly with `fast-inspect`; zoom / OCR was not triggered in that run. So the current benchmark supports two separate conclusions: 1. the real VisionMCP pipeline works end to end across several input types; 2. local crop + OCR can recover at least one native-vision failure, but more real zoom/OCR cases are still needed before claiming an accuracy uplift. See [`docs/benchmark.md`](docs/benchmark.md) and `benchmark_v2/` for details. --- ## Provider support VisionMCP currently has: - a stable Qwen-based core runtime - a unified provider interface used in benchmark / compatibility validation - validated DeepSeek and Qwen adapters Current status: ```text core runtime └── Qwen-based routes benchmark / compatibility validation ├── DeepSeek adapter └── Qwen adapter ``` Automatic production routing across providers is not part of the core runtime yet. --- ## Repository layout ```text vision_server.py MCP server main vision_telemetry_report.py usage / telemetry report config.example.json config template docs/ architecture.md system structure routing.md inspect / ocr / analyze + document/web routing design-decisions.md design choices and trade-offs benchmark.md real pipeline verification host-configuration.md how to configure text-only vs multimodal hosts examples/ long-screenshot.md long terminal screenshot example pdf.md scanned PDF example office.md DOCX / PPTX / XLSX rendering web-agent.md open, click, fill, re-observe benchmark_v2/ benchmark/runner.py real pipeline benchmark runner cases/ controlled test inputs providers/ DeepSeek / Qwen compatibility adapters dual_mode_tests/ text-only / multimodal host behavior tests web_smoke/ browser interaction smoke tests results/ reports and raw benchmark data ``` --- ## Installation Requires Python 3.10+. ```bash pip install pillow pymupdf playwright ``` `playwright install` is usually **not** needed — the script uses a system browser (e.g. Chrome / Edge) automatically when one is installed. Then: 1. Copy `config.example.json` to `config.json` 2. Create `apikey.txt` next to `vision_server.py` 3. Put your Alibaba Cloud Bailian API key in `apikey.txt` 4. Register `vision_server.py` as a stdio MCP server in your MCP client Detailed setup: [`deepseek-visionmcp.md`](deepseek-visionmcp.md) The default runtime uses Qwen-based routes through Bailian. > 如果你用的是纯文本模型（如 DeepSeek）作为宿主：安装时只需提供一个视觉 API Key（阿里云百炼，qwen-vl 系列识图模型，按张计费、很便宜）。宿主负责推理，VisionMCP 负责"看"——不需要为了识图额外准备多模态大模型。 --- ## Using with text-only vs multimodal hosts VisionMCP works with both, configured per workspace (not per model name): - **text-only host** — `vision` is your default visual entry; call it proactively on visual tasks - **multimodal host** — use native vision for ordinary images; call `vision` for long screenshots, OCR, PDF/Office, web interaction, or verification Declare `Current host vision mode: multimodal | text-only` in the workspace `AGENTS.md` (template: [`AGENTS.example.md`](AGENTS.example.md)). Details: [`docs/host-configuration.md`](docs/host-configuration.md) --- ## Notes - `apikey.txt` is ignored by Git and should never be committed - image recognition uses paid cloud models - `vision_server.py` contains the current stable core pipeline - when reporting issues, include the relevant error and telemetry output, but do not expose API keys or private paths Telemetry report: ```bash python vision_telemetry_report.py --log .vision_telemetry.jsonl ``` --- ## Next - make `intent="verify"` trigger a real second-pass visual check instead of only being recorded as metadata - add more real cases where zoom / OCR is actually required - decide whether DeepSeek / Qwen cross-provider routing is useful enough to move into the main runtime - improve evidence provenance for page / region / source tracking