Content
# MCP Deep Web Research Server (v0.3.0)
[](https://nodejs.org/)
[](https://www.typescriptlang.org/)
[](https://opensource.org/licenses/MIT)
A Model Context Protocol (MCP) server for advanced web research.
<a href="https://glama.ai/mcp/servers/5afpizjl6x"><img width="380" height="200" src="https://glama.ai/mcp/servers/5afpizjl6x/badge" alt="Web Research Server MCP server" /></a>
## Latest Changes
- Added visit_page tool for direct webpage content extraction
- Optimized performance to work within MCP timeout limits
* Reduced default maxDepth and maxBranching parameters
* Improved page loading efficiency
* Added timeout checks throughout the process
* Enhanced error handling for timeouts
> This project is a fork of [mcp-webresearch](https://github.com/mzxrai/mcp-webresearch) by [mzxrai](https://github.com/mzxrai), enhanced with additional features for deep web research capabilities. We're grateful to the original creators for their foundational work.
Bring real-time info into Claude with intelligent search queuing, enhanced content extraction, and deep research capabilities.
## Features
- Intelligent Search Queue System
- Batch search operations with rate limiting
- Queue management with progress tracking
- Error recovery and automatic retries
- Search result deduplication
- Enhanced Content Extraction
- TF-IDF based relevance scoring
- Keyword proximity analysis
- Content section weighting
- Readability scoring
- Improved HTML structure parsing
- Structured data extraction
- Better content cleaning and formatting
- Core Features
- Google search integration
- Webpage content extraction
- Research session tracking
- Markdown conversion with improved formatting
## Prerequisites
- [Node.js](https://nodejs.org/) >= 18 (includes `npm` and `npx`)
- [Claude Desktop app](https://claude.ai/download)
## Installation
### Global Installation (Recommended)
```bash
# Install globally using npm
npm install -g mcp-deepwebresearch
# Or using yarn
yarn global add mcp-deepwebresearch
# Or using pnpm
pnpm add -g mcp-deepwebresearch
```
### Local Project Installation
```bash
# Using npm
npm install mcp-deepwebresearch
# Using yarn
yarn add mcp-deepwebresearch
# Using pnpm
pnpm add mcp-deepwebresearch
```
### Claude Desktop Integration
After installing the package, add this entry to your `claude_desktop_config.json`:
#### Windows
```json
{
"mcpServers": {
"deepwebresearch": {
"command": "mcp-deepwebresearch",
"args": []
}
}
}
```
Location: `%APPDATA%\Claude\claude_desktop_config.json`
#### macOS
```json
{
"mcpServers": {
"deepwebresearch": {
"command": "mcp-deepwebresearch",
"args": []
}
}
}
```
Location: `~/Library/Application Support/Claude/claude_desktop_config.json`
This config allows Claude Desktop to automatically start the web research MCP server when needed.
### First-time Setup
After installation, run this command to install required browser dependencies:
```bash
npx playwright install chromium
```
## Usage
Simply start a chat with Claude and send a prompt that would benefit from web research. If you'd like a prebuilt prompt customized for deeper web research, you can use the `agentic-research` prompt that we provide through this package. Access that prompt in Claude Desktop by clicking the Paperclip icon in the chat input and then selecting `Choose an integration` → `deepwebresearch` → `agentic-research`.
### Tools
1. `deep_research`
- Performs comprehensive research with content analysis
- Arguments:
```typescript
{
topic: string;
maxDepth?: number; // default: 2
maxBranching?: number; // default: 3
timeout?: number; // default: 55000 (55 seconds)
minRelevanceScore?: number; // default: 0.7
}
```
- Returns:
```typescript
{
findings: {
mainTopics: Array<{name: string, importance: number}>;
keyInsights: Array<{text: string, confidence: number}>;
sources: Array<{url: string, credibilityScore: number}>;
};
progress: {
completedSteps: number;
totalSteps: number;
processedUrls: number;
};
timing: {
started: string;
completed?: string;
duration?: number;
operations?: {
parallelSearch?: number;
deduplication?: number;
topResultsProcessing?: number;
remainingResultsProcessing?: number;
total?: number;
};
};
}
```
2. `parallel_search`
- Performs multiple Google searches in parallel with intelligent queuing
- Arguments: `{ queries: string[], maxParallel?: number }`
- Note: maxParallel is limited to 5 to ensure reliable performance
3. `visit_page`
- Visit a webpage and extract its content
- Arguments: `{ url: string }`
- Returns:
```typescript
{
url: string;
title: string;
content: string; // Markdown formatted content
}
```
### Prompts
#### `agentic-research`
A guided research prompt that helps Claude conduct thorough web research. The prompt instructs Claude to:
- Start with broad searches to understand the topic landscape
- Prioritize high-quality, authoritative sources
- Iteratively refine the research direction based on findings
- Keep you informed and let you guide the research interactively
- Always cite sources with URLs
## Configuration Options
The server can be configured through environment variables:
- `MAX_PARALLEL_SEARCHES`: Maximum number of concurrent searches (default: 5)
- `SEARCH_DELAY_MS`: Delay between searches in milliseconds (default: 200)
- `MAX_RETRIES`: Number of retry attempts for failed requests (default: 3)
- `TIMEOUT_MS`: Request timeout in milliseconds (default: 55000)
- `LOG_LEVEL`: Logging level (default: 'info')
## Error Handling
### Common Issues
1. Rate Limiting
- Symptom: "Too many requests" error
- Solution: Increase `SEARCH_DELAY_MS` or decrease `MAX_PARALLEL_SEARCHES`
2. Network Timeouts
- Symptom: "Request timed out" error
- Solution: Ensure requests complete within the 60-second MCP timeout
3. Browser Issues
- Symptom: "Browser failed to launch" error
- Solution: Ensure Playwright is properly installed (`npx playwright install`)
### Debugging
This is beta software. If you run into issues:
1. Check Claude Desktop's MCP logs:
```bash
# On macOS
tail -n 20 -f ~/Library/Logs/Claude/mcp*.log
# On Windows
Get-Content -Path "$env:APPDATA\Claude\logs\mcp*.log" -Tail 20 -Wait
```
2. Enable debug logging:
```bash
export LOG_LEVEL=debug
```
## Development
### Setup
```bash
# Install dependencies
pnpm install
# Build the project
pnpm build
# Watch for changes
pnpm watch
# Run in development mode
pnpm dev
```
### Testing
```bash
# Run all tests
pnpm test
# Run tests in watch mode
pnpm test:watch
# Run tests with coverage
pnpm test:coverage
```
### Code Quality
```bash
# Run linter
pnpm lint
# Fix linting issues
pnpm lint:fix
# Type check
pnpm type-check
```
## Contributing
1. Fork the repository
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add some amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request
### Coding Standards
- Follow TypeScript best practices
- Maintain test coverage above 80%
- Document new features and APIs
- Update CHANGELOG.md for significant changes
- Follow semantic versioning
### Performance Considerations
- Use batch operations where possible
- Implement proper error handling and retries
- Consider memory usage with large datasets
- Cache results when appropriate
- Use streaming for large content
## Requirements
- Node.js >= 18
- Playwright (automatically installed as a dependency)
## Verified Platforms
- [x] macOS
- [x] Windows
- [ ] Linux
## License
MIT
## Credits
This project builds upon the excellent work of [mcp-webresearch](https://github.com/mzxrai/mcp-webresearch) by [mzxrai](https://github.com/mzxrai). The original codebase provided the foundation for our enhanced features and capabilities.
## Author
[qpd-v](https://github.com/qpd-v)
Connection Info
You Might Also Like
No recommendations available