Content
# MagpieMcp
A web search and web fetch MCP (Model Context Protocol) server built with .NET 10, using SearXNG for web search functionality.
## Overview
MagpieMcp is a Model Context Protocol (MCP) server that provides web search and content fetching capabilities through a secure, isolated environment. It uses SearXNG as the search backend, offering privacy-focused web search with configurable settings.
## Features
- **MCP Server**: Implements MCP protocol for AI assistants
- **SearXNG Integration**: Privacy-focused web search backend
- **Web Fetch Tool**: Secure content fetching with URL validation and sanitization
- **Modular Architecture**: Configurable tool pipeline with service composition
- **Secure Containerization**: Docker-based deployment with security hardening
- **Resource Isolation**: Network-level isolation between services
- **.NET 10**: Built with the latest .NET framework
## Architecture
```
┌─────────────┐
│ AI Client │
└──────┬──────┘
│ MCP Request
↓
┌─────────────┐
│ MagpieMcp │
│ (API) │
└──────┬──────┘
│ HTTP
↓
┌─────────────┐
│ SearXNG │
│ (Search) │
└─────────────┘
```
## Prerequisites
- Docker\Podman (for containerized deployment)
- .NET 10 SDK (for local development)
## Installation
### Setup
* Change secret_key in searxng/settings.yml.
* Ports and IP addresses can be changes based on your preferences.
### Using Docker Compose
1. Clone the repository:
```bash
git clone <repository-url>
cd MagpieMcp
```
2. Start the services:
```bash
docker-compose -f docker-compose.yml up -d --build
```
or
```bash
podman-compose -f podman-compose.yml up -d --build
```
This will start:
- **mcp-api**: The MCP server on port 5066
- **searxng**: The search backend (internal only)
### Local Development
1. Restore dependencies:
```bash
dotnet restore
```
2. Build the project:
```bash
dotnet build
```
3. Run the API:
```bash
dotnet run --project MagpieMcp.Api
```
4. Run searxng from compose (use docker-compose.override.yml to expose on localhost):
```bash
./run_searxng_compose.sh
```
## Configuration
### MCP Client Configuration
Configure your MCP client (e.g., Claude Desktop) to connect to MagpieMcp:
```json
{
"mcpServers": {
"magpiemcp": {
"type": "http",
"url": "http://localhost:5066"
}
}
}
```
### Tool Configuration
The server provides two main tools configured in `appsettings.json` with the following service types:
**Supported Service Types:**
- **OnlyExternalUrlService**: Validates external URLs and blocks private/internal hosts
- **HttpService**: Makes HTTP requests to URLs (supports UrlTemplate and Headers configuration)
- **HtmlSanitizationService**: Sanitizes HTML content
- **HtmlReadabilityExtractionService**: Extracts readable content from HTML
- **HtmlToMarkdownService**: Converts HTML to Markdown format
- **SearXngJsonToMarkdownService**: Processes SearXNG JSON search results into Markdown
- **InjectionSanitizationService**: Sanitizes against prompt injection attacks
**HttpService Configuration Parameters:**
- **UrlTemplate**: Template for constructing HTTP requests (supports {0} placeholder for input)
- **Headers**: Dictionary of HTTP headers to include in requests
Note: Other services in the pipeline do not require configuration parameters - they operate with default behavior based on their function.
**web_search**: Performs web searches using the configured backend
```json
{
"Name": "web_search",
"PreProcessSteps": [
{
"ServiceType": "HttpService",
"UrlTemplate": "http://localhost:8080/search?q={0}&format=json"
}
],
"PostProcessSteps": [
{
"ServiceType": "SearXngJsonToMarkdownService"
},
{
"ServiceType": "InjectionSanitizationService"
}
]
}
```
**web_fetch**: Fetches web content with security validation and processing
```json
{
"Name": "web_fetch",
"PreProcessSteps": [
{
"ServiceType": "OnlyExternalUrlService"
},
{
"ServiceType": "HttpService",
"UrlTemplate": "{0}"
}
],
"PostProcessSteps": [
{
"ServiceType": "HtmlSanitizationService"
},
{
"ServiceType": "HtmlReadabilityExtractionService"
},
{
"ServiceType": "HtmlToMarkdownService"
},
{
"ServiceType": "InjectionSanitizationService"
}
]
}
```
### SearXNG Settings
Edit `searxng/settings.yml` to customize search behavior:
```yaml
use_default_settings: true
server:
secret_key: "your-secret-key" # Change this for production
search:
safe_search: 1
default_lang: "en"
formats:
- html
- json
```
## Security Features
The Docker configuration includes several security measures:
- **Read-only filesystem**: `/app` and `/etc` mounted as read-only
- **Non-root user**: Runs as non-root user
- **Capability dropping**: `cap_drop: ALL`
- **No new privileges**: `security_opt: no-new-privileges:true`
- **Resource limits**: CPU and memory constraints
- **Network isolation**: Internal network for API, separate network for SearXNG egress
## Usage
Once running, you can use the MCP server through your AI assistant:
1. Start the MCP server
2. Query your AI assistant with search or fetch requests
3. The assistant will use MagpieMcp to perform web searches or content fetching
## Development
### Project Structure
```
MagpieMcp/
├── MagpieMcp.Api/ # Main API project
├── MagpieMcp.Api.Tests/ # Test project
├── searxng/ # SearXNG configuration
├── docker-compose.yml # Docker services
├── Dockerfile # Build configuration
└── .mcp.json # MCP client configuration
```
The modular architecture uses a pipeline of tool services:
- **Tool Services**: Configurable processing steps (URL validation, HTTP fetching, HTML sanitization, etc.)
- **Configuration**: Tools defined with pre- and post-processing steps
- **Service Composition**: Each tool can have multiple processing stages using different service types
### Running Tests
```bash
dotnet test --project MagpieMcp.Api.Tests
```
The test suite includes:
- **Integration tests** for the modular tool services pipeline
- **Search integration tests** for web search functionality
## Running Scripts
Several helper scripts are provided:
- `run_config_compose.sh`: Check configuration
- `run_full_compose.sh`: Start full stack
- `run_searxng_compose.sh`: Start SearXNG only for debugging
