mcp-gateway

# Proxymic ("DataDog" for AI Agents) Proxymic is a transparent MCP gateway that makes agent-to-server, agent-to-tool, and client-to-server communication effortless — while caching, securing, and giving you full observability into every tool call your AI agents make. Slash API costs by 40-70%. Kill agentic loops before they burn tokens. Detect anomalies, hallucinations, and prompt injections in real time. All without modifying a single line of agent or server code. **Drop it between any agent (Claude, Cursor, custom) and any MCP server. It just works.** [](https://mcp-gateway-production-c0f5.up.railway.app/) [](https://goreportcard.com/report/github.com/Arnab758/mcp-gateway) [](LICENSE) --- ## Why Every AI agent today talks to tools — file systems, databases, APIs, browsers, terminals. But the protocol between agents and these tools (MCP over stdio) is brittle: - **Identical prompts cost the same every time.** An agent asking "list files in /src" pays full LLM cost on every single call — even when the directory hasn't changed. - **No built-in security.** Any tool the server exposes, the agent can call — including `exec_command`, `bash`, `write_file`. One prompt injection and your filesystem is gone. - **Agents get stuck in loops.** A misconfigured agent will hammer the same tool thousands of times before anyone notices. - **A single hung tool call takes down the whole agent.** No timeouts. No circuit breakers. No recovery. These aren't hypotheticals. If you've built with AI agents, you've hit every one of these. --- ## Background I follow AI infrastructure closely. Two of my friends were building AI startups, and their biggest pain point was the same: LLM API bills were eating them alive. Every repeated prompt cost money. Every agent loop burned tokens. I read about startups hitting their yearly API budget in a month. Microsoft capped employee token usage. Salesforce had the same problem. So I built an AI Gateway — semantic caching, multi-strategy matching, adaptive thresholds. It saved 40-70% on LLM costs. But the deeper I went, the more I realized: the real problem isn't just API costs. It's the entire MCP layer. AI agents call tools hundreds of times a day. Each call is slow, unsecured, uncached. MCP has no Kong, no Cloudflare, no API Gateway. So I built one. Proxymic is that layer. One binary. Zero dependencies. It caches, secures, and hardens every tool call between your AI agent and its MCP servers. Same way Kong became essential for microservices — Proxymic becomes essential for AI agents. > *What Cloudflare and Datadog did for web infrastructure, Proxymic does for AI agent infrastructure.* --- ## Try It Now [**Open the live playground →**](https://mcp-gateway-production-c0f5.up.railway.app/) No signup. No install. It's a live demo with an interactive "Try It" console — type a prompt and see exactly how caching, rate limiting, and the agent health system respond in real time. The demo page includes: - An interactive tool call console - Live Agent Health dashboard showing every tool call, anomaly detection, and savings estimates - Cache hit/miss indicators - Real-time metrics and agent behavior analysis --- ## How It Works Proxymic intercepts every JSON-RPC message flowing between an agent and its MCP server. It passes through initialization and discovery transparently, but intercepts every `tools/call` request to apply seven layers of hardening: ``` Agent (Claude, Cursor, custom) │ stdin/stdout (JSON-RPC over MCP) ▼ ┌──────────────────────────────────────────────────┐ │ Proxymic Gateway │ │ │ │ ┌──────────── Request Pipeline ────────────┐ │ │ │ │ │ │ │ 1. Security Blocklist │ │ │ │ ├── Blocked? → Error │ │ │ │ └── Allowed? → Continue │ │ │ │ │ │ │ │ 2. Rate Limiter │ │ │ │ ├── Exceeded? → Error │ │ │ │ └── OK? → Continue │ │ │ │ │ │ │ │ 3. Circuit Breaker │ │ │ │ ├── Open? → Error │ │ │ │ └── Closed? → Continue │ │ │ │ │ │ │ │ 4. Cache Lookup (multi-strategy) │ │ │ │ ├── Semantic HIT → <3ms return │ │ │ │ ├── Exact HIT → <1ms return │ │ │ │ └── MISS → Forward │ │ │ │ │ │ │ │ 5. Response Deduplication │ │ │ │ ├── 3rd+ identical → Placeholder │ │ │ │ └── Unique → Store + pass │ │ │ │ │ │ │ │ 6. Timeout Guard + Auto-Restart │ │ │ │ ├── OK → Cache + return │ │ │ │ └── Timeout → Kill + restart │ │ │ │ │ │ │ │ 7. Agent Health (observability) │ │ │ │ └── Every call logged, analyzed, │ │ │ │ displayed in real-time │ │ │ └────────────────────────────────────────────┘ │ │ │ │ ┌───────── Background Systems ──────────┐ │ │ │ │ │ │ │ • Health checks every 15s │ │ │ │ • Process monitoring + auto-restart │ │ │ │ • Agentic loop detection & kills │ │ │ │ • Metrics server (:9090/metrics) │ │ │ │ • Structured JSON logging │ │ │ └────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────┘ │ stdin/stdout (JSON-RPC over MCP) ▼ MCP Server (filesystem, database, browser, terminal, ...) ``` --- ## Core Capabilities ### Smart Caching Not just exact-match. Proxymic uses a cascade of five matching strategies to catch semantically identical prompts even when they differ in phrasing: | Strategy | What It Catches | Typical Hit Latency | |----------|----------------|-------------------| | **Exact hash** | Identical prompts (O(1)) | <1ms | | **L1 hot cache** | Frequently accessed prompts | <1ms | | **HNSW vector search** | Semantically similar prompts (paraphrases) | ~3ms | | **Jaccard + n-gram** | Partial overlaps, rephrased queries | ~2ms | | **Template matching** | Dynamic inputs ("weather in {city}") | <1ms | The cache self-tunes — adaptive threshold adjustment means it learns what "similar enough" means for your particular workload. ### Response Deduplication When an agent asks the same question 100 times in parallel (they do this), Proxymic forwards only one request to upstream. The other 99 get a placeholder response. Saves context window space and eliminates redundant compute. ### Agentic Loop Killer Detects runaway agents before they burn through your token budget. Three stages of escalation: 1. **Monitor** — Track frequency and content similarity per agent session 2. **Throttle** — Insert progressive delays as loop score rises 3. **Block** — Return a clear error with `Retry-After` header when the loop is confirmed The detection uses a combination of request frequency, content similarity, and repetition analysis to distinguish genuine usage from buggy loops. ### Circuit Breaker If the upstream server starts failing, Proxymic opens the circuit — all subsequent calls fail fast without touching the server. After a recovery timeout, it tries a single probe request. If that succeeds, the circuit closes. If not, it waits longer. Prevents cascading failures. ### Security Blocklist Define which tools are allowed and which are blocked at the proxy level. Not at the agent level, not at the server level — at the network boundary. A prompt-injected agent cannot call `exec_command` or `bash` even if it tries to. ### Agent Health — Live Observability Every tool call is captured, analyzed, and displayed on a live dashboard: - **Tool call trace** — See every call, its arguments, result, and latency - **Anomaly detection** — Blocked calls, repeated identical calls, slow responses are flagged automatically - **Savings estimates** — See exactly how many tokens and how much latency each cache hit saved - **Performance breakdown** — Cache hit rate, circuit breaker state, rate limit hits, all in real time --- ## Quick Start ```bash # Clone git clone https://github.com/Arnab758/mcp-gateway.git cd mcp-gateway # Build (single static binary, zero runtime dependencies) go build -o proxymic . # Configure # Edit gateway.yaml with your MCP server command # Run ./proxymic --demo # Demo mode (synthetic traffic + live dashboard) ``` **Or with Docker:** ```bash docker run -v $(pwd)/gateway.yaml:/app/gateway.yaml arnab758/proxymic ``` --- ## Configuration A single YAML file controls everything: ```yaml server: command: "node" args: ["server.js"] timeout_seconds: 30 max_restarts: 5 cache: ttl_seconds: 300 similarity_threshold: 0.95 security: allowed_tools: ["list_files", "read_file", "write_file"] blocked_tools: ["exec_command", "bash", "rm"] rate_limit: enabled: true max_requests: 60 window_seconds: 60 circuit_breaker: enabled: true failure_threshold: 3 recovery_timeout_seconds: 30 loop_killer: enabled: true window_seconds: 30 block_seconds: 120 ``` --- ## Project Structure ``` proxymic/ ├── main.go # Entry point, HTTP server, route registration ├── proxy.go # MCP stdio proxy + tool call pipeline ├── gateway.yaml # Configuration ├── cache.go # Multi-strategy semantic cache ├── advanced_cache.go # HNSW + L1 + Redis + template cascade ├── fuzzy_cache.go # Jaccard + n-gram + Levenshtein matching ├── embedding.go # Local embedding generation (no external API) ├── circuit_breaker.go # Failure threshold + recovery probe ├── dedup.go # Response deduplication tracker ├── agent_loop.go # Loop detection + throttle + block ├── metrics.go # Metrics server + landing pages ├── tracer.go # Tool call tracing + anomaly detection ├── observer.go # Silent observation mode ├── hardening.go # Panic recovery, input validation ├── template_matcher.go # Dynamic template extraction ├── config.go # YAML config loader └── cmd/ ├── mcp-client/ # Test client for the gateway └── mcp-demo-server/ # Synthetic MCP server for demos ``` Built entirely in Go. No runtime dependencies. Single static binary. --- ## Technical Details - **Transport:** Intercepts stdio-based MCP protocol (JSON-RPC 2.0) - **Caching:** HNSW vector index (local) + optional Redis (distributed) + L1 hot cache - **Embeddings:** Local — no external embedding API needed. Uses a lightweight TF-IDF + hash-based embedding - **Loop detection:** Sliding window with configurable timeframes per stage - **Deduplication:** SHA-256 response hashing with session-scoped tracking - **Metrics:** Prometheus `/metrics` endpoint + JSON structured logs - **Binary size:** ~15MB static binary --- ## License MIT --- <div align="center"> <a href="https://mcp-gateway-production-c0f5.up.railway.app/">Live Demo</a> · <a href="https://github.com/Arnab758/mcp-gateway">GitHub</a> · <a href="https://github.com/Arnab758/mcp-gateway/issues">Issues</a> </div>