OcularAudio-MCP

# OcularAudio MCP  An asynchronous Model Context Protocol (MCP) server that gives AI models "eyes and ears" to process web videos. It extracts transcripts and captures screenshots from YouTube and other video platforms. ## Features - **Hybrid transcript extraction**: Fetches YouTube captions instantly, falls back to local Whisper ASR - **On-demand video screenshots**: Captures frames at any timestamp without downloading the full video - **OCR on screenshots**: Extract visible text from frames using Tesseract (optional, `--ocr` flag) - **Cookie authentication**: Supports age-restricted and private videos via cookies.txt - **Local caching**: Processed videos are cached for instant subsequent lookups - **Async architecture**: Non-blocking design keeps MCP clients responsive - **Flexible output**: Clipboard, stdout, file, or JSON — your choice ## Benchmark See [BENCHMARK.md](BENCHMARK.md) for performance benchmarks and a deep competitive analysis against all major video transcript, screenshot, and OCR tools in the MCP and CLI ecosystem. ## Requirements - **Python 3.9+** (required for `list[int]` type hints) - **FFmpeg** (required by yt-dlp and OpenCV) - **Node.js 18+** (only for the CLI wrapper) - **Tesseract** (optional, only for `--ocr` flag) ## Installation ### 1. Install system dependencies **macOS:** ```bash brew install ffmpeg python3 # Optional (for OCR): brew install tesseract ``` **Windows:** ```bash choco install ffmpeg python # Optional (for OCR): choco install tesseract ``` **Linux:** ```bash sudo apt update && sudo apt install ffmpeg python3 python3-pip # Optional (for OCR): sudo apt install tesseract-ocr ``` ### 2. Install Python packages ```bash pip install -r requirements.txt ``` Or manually: ```bash pip install mcp youtube-transcript-api yt-dlp opencv-python-headless faster-whisper requests pytesseract ``` ### 3. Install Node.js CLI (optional) ```bash npm install ``` ## Usage ### Option A: MCP Server (Recommended) The MCP server gives AI models direct access to video transcripts and screenshots. **Quick Install** — No installation needed. Just add the config to your MCP client below. #### Claude Desktop ```json { "mcpServers": { "ocular-audio-mcp": { "command": "npx", "args": ["ocular-audio-mcp"] } } } ``` Config: `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows) #### Cursor ```json { "mcpServers": { "ocular-audio-mcp": { "command": "npx", "args": ["ocular-audio-mcp"] } } } ``` Config: `.cursor/mcp.json` (project) or `~/.cursor/mcp.json` (global) #### Claude Code ```bash claude mcp add ocular-audio-mcp -- npx ocular-audio-mcp ``` #### Codex CLI (OpenAI) ```bash codex mcp add ocular-audio-mcp -- npx ocular-audio-mcp ``` #### Gemini CLI ```bash gemini mcp add ocular-audio-mcp npx ocular-audio-mcp --scope user ``` #### Windsurf ```json { "mcpServers": { "ocular-audio-mcp": { "command": "npx", "args": ["ocular-audio-mcp"] } } } ``` Config: `~/.codeium/windsurf/mcp_config.json` #### Zed ```json { "context_servers": { "ocular-audio-mcp": { "command": "npx", "args": ["ocular-audio-mcp"] } } } ``` Config: `~/.config/zed/settings.json` #### VS Code (GitHub Copilot) ```json { "servers": { "ocular-audio-mcp": { "type": "stdio", "command": "npx", "args": ["ocular-audio-mcp"] } } } ``` Config: `.vscode/mcp.json` #### OpenCode ```json { "mcpServers": { "ocular-audio-mcp": { "command": "npx", "args": ["ocular-audio-mcp"] } } } ``` Config: `~/.opencode/config.json` #### Cline (VS Code Extension) ```json { "mcpServers": { "ocular-audio-mcp": { "command": "npx", "args": ["ocular-audio-mcp"] } } } ``` #### Local Development (from source) ```json { "mcpServers": { "ocular-audio-mcp": { "command": "python", "args": ["/path/to/ocular_audio_mcp.py"] } } } ``` ### Option B: CLI ```bash npx ocular-audio "https://www.youtube.com/watch?v=dQw4w9WgXcQ" ``` By default, the transcript is printed to stdout and copied to your clipboard. Paste it into Claude Web, ChatGPT, or any AI chat. #### CLI Options | Flag | Description | |------|-------------| | `-h`, `--help` | Show help message | | `-v`, `--version` | Show version number | | `--stdout` | Print transcript to stdout only (no clipboard, no file) | | `--no-clipboard` | Skip clipboard copy | | `--output <file>` | Write context to a specific file path | | `--json` | Output raw JSON (metadata + transcript) for programmatic use | | `--detail <level>` | Screenshot capture mode: `overview`, `balanced`, `deep`, `auto` (default: `auto`) | | `--ocr` | Extract text from screenshots using Tesseract OCR | | `--force` | Bypass cache and re-process the video | | `--verbose` | Show detailed progress information | | `--quiet` | Suppress summary and status messages | | `--check` | Check system dependencies (Python, FFmpeg, Whisper, Tesseract) | | `--list-cached` | List all cached videos with titles | | `--cache-info` | Show cache statistics (count, size, oldest/newest) | | `--clear-cache` | Delete all cached transcripts and screenshots | #### Examples ```bash # Basic usage — prints to stdout + copies to clipboard npx ocular-audio "https://www.youtube.com/watch?v=dQw4w9WgXcQ" # Stdout only — great for piping to other tools npx ocular-audio --stdout "https://www.youtube.com/watch?v=dQw4w9WgXcQ" | head -50 # Write to a specific file npx ocular-audio --output transcript.txt "https://www.youtube.com/watch?v=dQw4w9WgXcQ" # Raw JSON for programmatic consumption npx ocular-audio --json "https://www.youtube.com/watch?v=dQw4w9WgXcQ" | jq .metadata.title # Transcript only, no screenshots npx ocular-audio --detail overview "https://www.youtube.com/watch?v=dQw4w9WgXcQ" # Maximum screenshots npx ocular-audio --detail deep "https://www.youtube.com/watch?v=dQw4w9WgXcQ" # No clipboard copy, just print to terminal npx ocular-audio --no-clipboard "https://www.youtube.com/watch?v=dQw4w9WgXcQ" # Screenshots with OCR — extract visible text from frames npx ocular-audio --ocr "https://www.youtube.com/watch?v=dQw4w9WgXcQ" # JSON output with OCR npx ocular-audio --json --ocr "https://www.youtube.com/watch?v=dQw4w9WgXcQ" # Force re-process (bypass cache) npx ocular-audio --force "https://www.youtube.com/watch?v=dQw4w9WgXcQ" # Verbose mode — see all progress details npx ocular-audio --verbose "https://www.youtube.com/watch?v=dQw4w9WgXcQ" # Quiet mode — minimal output npx ocular-audio --quiet --stdout "https://www.youtube.com/watch?v=dQw4w9WgXcQ" # Check system capabilities npx ocular-audio --check # List cached videos npx ocular-audio --list-cached # Show cache stats npx ocular-audio --cache-info # Clear all cached data npx ocular-audio --clear-cache ``` ## Cookie Setup (for age-restricted/private videos) YouTube may block transcript access for age-restricted or private videos. To fix this, export your browser cookies: 1. Install a browser extension like "Get cookies.txt LOCALLY" (Chrome/Firefox) 2. Go to youtube.com while logged in 3. Export cookies to a file named `cookies.txt` 4. Place the file in one of these locations: - `~/.cache/ocular_audio_mcp/cookies.txt` - `~/.config/ocular_audio_mcp/cookies.txt` - `./cookies.txt` (in the project directory) The server will automatically detect and use the cookies file. ## MCP Tools ### `get_ocular_audio_capabilities` Returns system capabilities and dependency status. Use this to check what features are available. **Parameters:** None **Returns:** System info including Python version, FFmpeg, Whisper, Tesseract, OpenCV, and cookie status. ### `get_ocular_audio_metadata` Extracts only video metadata (title, creator, duration, views, chapters) without transcript. Much faster than getting the full transcript. **Parameters:** - `url` (string): Video URL ### `get_ocular_audio_transcript` Extracts the complete transcript, video chapters, and metadata from a video. **Parameters:** - `url` (string): Video URL - `use_local_whisper` (boolean, default: true): Enable Whisper fallback if captions unavailable ### `get_ocular_audio_chapters` Extracts only video chapters with timestamps. Returns chapter titles with start times in [MM:SS] format. **Parameters:** - `url` (string): Video URL ### `get_ocular_audio_video_screenshots` Captures screenshots at specific timestamps. **Parameters:** - `url` (string): Video URL - `timestamps_secs` (array of integers): Timestamps to capture (e.g., `[45, 120, 300]`) - `enable_ocr` (boolean, default: false): If true, run OCR on each captured frame to extract visible text ### `get_ocular_audio_video_context` Extracts transcript, metadata, and intelligent screenshots in one call. Automatically analyzes the transcript to find visually important moments and captures screenshots at those timestamps. **Parameters:** - `url` (string): Video URL - `detail_level` (string, default: "auto"): Controls screenshot capture mode: - `"auto"` - Adapts to video length and content importance - `"overview"` - Transcript and metadata only, no screenshots (fastest) - `"balanced"` - Screenshots only at visually important moments (strong signals) - `"deep"` - Screenshots at every visually significant moment (all signals) - `use_local_whisper` (boolean, default: true): Enable Whisper fallback if captions unavailable - `enable_ocr` (boolean, default: false): If true, run OCR on captured screenshots to extract visible text ### `list_ocular_audio_cache` Lists all cached videos with their metadata (title, uploader, duration, when cached). **Parameters:** None ### `clear_ocular_audio_cache` Clears cached video data. **Parameters:** - `video_id` (string, optional): Video ID to clear specific video. If empty, clears all cache. ## Cache Management Processed videos are cached in `~/.cache/ocular_audio_mcp/` for 7 days. Use the CLI flags to manage the cache: ```bash npx ocular-audio --list-cached # See what's cached npx ocular-audio --cache-info # Storage stats npx ocular-audio --clear-cache # Wipe everything ``` Or manually: ```bash rm -rf ~/.cache/ocular_audio_mcp/*.json ``` ## Troubleshooting ### "No local ASR engines found" Install a Whisper engine: ```bash pip install faster-whisper ``` ### "Audio track download failed" - Check your network connection - For age-restricted videos, add a cookies.txt file (see Cookie Setup above) - Ensure FFmpeg is installed: `ffmpeg -version` ### "Failed to extract a playable video stream" - The video may be private or geo-blocked - Try adding cookies.txt - Check if the video is still available ### Python not found on Windows Ensure Python is in your PATH. Try: ```bash python --version ``` If not found, reinstall Python from python.org and check "Add Python to PATH" during installation. ### MCP server not connecting - Verify the path in your MCP client config is correct - Test the server manually: `python /path/to/ocular_audio_mcp.py` - Check that all dependencies are installed: `pip list | grep -E "mcp|whisper|yt-dlp"` ## License MIT