Content
# aggregate-search-mcp
[](https://opensource.org/licenses/MIT)
**aggregate-search-mcp** – A lightweight, API key-free MCP server that provides multi-engine search capabilities for clients like LM Studio, aggregating results from Baidu, Bing, and Sogou. It features built-in caching, anti-scraping strategies, and result deduplication and sorting, and also supports directly fetching the content of any URL.
## ✨ Features
- 🔍 **Multi-engine search** – Simultaneously fetch results from Baidu, Bing, and Sogou, merge and sort results by relevance
- 🚫 **No API Key required** – Simulate real browser requests, directly fetch public search results
- 📄 **Content fetching** – Automatically fetch the content of search result pages, providing rich context for models
- 🌐 **Direct URL fetching** – Provide a `fetch_url` tool to fetch the content of any webpage
- 🧩 **Dynamic rendering fallback** – When static fetching fails, automatically enable Puppeteer headless browser to greatly improve success rate
- ⚡ **Dual caching** – Search result cache (5 minutes) + content cache (1 hour), reduce repeated requests
- 🛡️ **Anti-scraping strategy** – User-Agent rotation + request retry backoff + optional proxy
- 💾 **Low resource occupancy** – Only depend on Node.js runtime, memory resident about 30–50 MB (Puppeteer started on demand)
- 🔧 **Flexible and adjustable** – Configurable cache time, maximum result number, timeout, fetch engine and other parameters
- 🧩 **Compatible with LM Studio** – Perfectly compatible with MCP protocol, plug and play
## 🚀 Quick Start
### 1. Clone and install dependencies
```bash
git clone https://github.com/your-username/aggregate-search-mcp.git
cd aggregate-search-mcp
npm install
# Note: Chromium will be downloaded during installation (about 150MB),
# the first installation is slightly slower, but will greatly improve the success rate of dynamic page fetching.
```
2. Configure LM Studio
In LM Studio's MCP settings, edit `mcp.json` (usually located at `~/.lmstudio/mcp.json` or in the settings interface), add:
```json
{
"mcpServers": {
"multi-search": {
"command": "node",
"args": ["/your/absolute/path/aggregate-search-mcp/index.js"],
"env": {}
}
}
}
```
Save and restart LM Studio's MCP service.
3. Use in conversation
Load any model that supports tool calls (such as Qwen, DeepSeek, etc.), input:
```
Search today's tech news
```
The model will automatically call the `search` tool, return comprehensive results from multiple engines, and attach a content summary.
You can also directly let the model fetch a specific webpage:
```
Fetch https://www.baidu.com
```
The model will call the `fetch_url` tool, return the content of the page.
### Search Process
- Query input – User or model initiates a search request
- Cache check – If the same query is within TTL, directly return cached results
- Parallel fetching – Simultaneously send HTTP requests to Baidu, Bing, and Sogou (simulate browser)
- Parsing and extraction – Use cheerio to parse HTML, extract title, link, summary
- Deduplication and sorting – Deduplicate by link, sort by keyword hit count + source weight
- Content fetching – Fetch content of top `maxResults` results (Axios + Puppeteer fallback)
- Return results – Return structured results with content summary to client
### URL Direct Fetching
- Call `fetch_url` tool, pass URL
- Try Axios static fetching → If failed or content too short, automatically fallback to Puppeteer headless browser
- Extract page content (use Readability or common container selector)
- Return truncated content
🧩 Available Tools
### search
- Description: Multi-engine search + automatic content fetching
- Parameters:
- `query` (string, required) – Search keyword
- `max_results` (number, optional) – Return result number (1-10, default 5)
### fetch_url
- Description: Directly fetch content of specified URL
- Parameters:
- `url` (string, required) – Full webpage address (including protocol)
- `max_length` (number, optional) – Maximum return content length (default 500)
⚠️ Notes
- Anti-scraping risk – Frequent requests may trigger search engine's CAPTCHA, recommend reasonable control of call frequency.
- Page structure change – If Baidu/Bing/Sogou modify HTML layout, parsing may fail. Update corresponding cheerio selector (see `fetchBaidu`, `fetchBing`, `fetchSogou` functions).
- Network environment – Ensure server can normally access above search engines (domestic network needs to directly connect Baidu, Sogou, Bing may need proxy).
- Puppeteer resource – First startup will download Chromium, subsequent startups will start headless browser on demand, occupying extra memory (about 100-200 MB), but only triggered when Axios fetching fails.
🤝 Acknowledgements and References
This project references many excellent open-source projects in MCP community, main inspirations come from:
📚 Baidu Search MCP Server
- caiyili/baidu-search-mcp – Core Baidu search function inspiration
- iflow-mcp/baidu-search-mcp – Provide project structure reference
- @alex.ss/mcp-server-baidu-search – Different implementation ideas
- Evilran/baidu-mcp-server – Provide webpage search and content fetching capability
🔍 Multi-engine Search MCP Server
- dlmufei/go-web-search-mcp – Core idea of “multi-engine aggregation”
- MemoryClear/claude-web-search-mcp – “Parallel search” and “intelligent deduplication”
- MetaSearchMCP – Multi-provider aggregation and result deduplication
- pranavms13/web-search-mcp – Headless browser fetching idea
- tamb/simple-web-search-mcp – Zero-configuration concept
🧩 MCP SDK and Examples
- @modelcontextprotocol/sdk – Project foundation
- Model Context Protocol official documentation – Protocol design basis
💡 Other Inspirations
- web-research-mcp – No API key, parallel multi-query
- lc-mcp-server – Simulation mode
- Tencent/WebSearchMCP – Enterprise-level MCP service implementation
📌 Developer Notes
- This MCP server's compilation and development process received assistance from DeepSeek, but has been personally tested to be operational.
- All configurations and codes are assisted by DeepSeek, if defects are found during use, welcome to submit Issue or Pull Request.
- If you do not trust this project's capability, you can choose not to clone and use, thank you for reading this README.
Finally, thank you for downloading and using this project! 🎉
If you find it useful, welcome to Star ⭐ support, let more people benefit.
