Content
# MCP-webfetch
MCP server for web content fetching through SOCKS5 proxy. Based on opencode official webfetch implementation with added proxy support.
## Features
- Fetch web pages through `127.0.0.1:10808` SOCKS5 proxy
- Supported formats: markdown (default), text, html
- HTML→Markdown conversion (TurndownService)
- HTML→Text extraction (htmlparser2)
- Automatic redirect following (anti-loop detection)
- Cloudflare anti-scraping detection + automatic retry
- MIME type verification (images/binary automatically rejected)
- Body size limit (5MB)
## Installation
```bash
# 1. Clone or copy this directory
cd D:\workplace\MCP-webfetch
# 2. Install dependencies (htmlparser2 + turndown)
npm install
```
## Configure opencode
Add the following to `opencode.json`:
```json
{
"mcp": {
"webfetch-proxy": {
"type": "local",
"command": ["node", "D:\\workplace\\MCP-webfetch\\webfetch-proxy.js"],
"enabled": true
}
},
"permission": {
"webfetch": "deny"
}
}
```
- **Project-level**: `opencode.json` in the current directory
- **Global-level**: `~/.config/opencode/opencode.json`
### Disable built-in webfetch (optional)
Set `webfetch: "deny"` in `permission` to restrict AI to use only this MCP tool.
```json
{
"permission": {
"webfetch": "deny"
}
}
```
## Configure proxy address
Edit the top of `webfetch-proxy.js`:
```js
const PROXY_HOST = "127.0.0.1";
const PROXY_PORT = 10808;
```
## Restart opencode
Restart opencode after modifying configurations.
## Testing
```bash
node -e "
const { spawn } = require('child_process');
const proc = spawn('node', ['D:\\\\workplace\\\\MCP-webfetch\\\\webfetch-proxy.js']);
proc.stdin.write(JSON.stringify({jsonrpc:'2.0',id:1,method:'initialize',params:{protocolVersion:'2024-11-05',capabilities:{},clientInfo:{name:'t',version:'1'}}})+'\n');
setTimeout(() => {
proc.stdin.write(JSON.stringify({jsonrpc:'2.0',id:2,method:'tools/call',params:{name:'webfetch',arguments:{url:'https://httpbin.org/get',format:'text',timeout:5}}})+'\n');
}, 500);
setTimeout(() => { proc.kill(); process.exit(); }, 5000);
proc.stdout.on('data', (d) => { console.log(d.toString()); });
"
```
## Dependencies
- `effect` — Functional runtime (HttpClient, Stream, Effect)
- `htmlparser2` — HTML parsing (text extraction)
- `socks-proxy-agent` — SOCKS5 proxy client
- `turndown` — HTML→Markdown conversion
