Content
# chomp-mcp
[English Documentation](./README.md)
> Generated by SCSE.
## How This Project Was Generated
The project was generated from the requirements stage with the project description shown below. The local parent directory has been redacted.
```bash
PROJECT_DESCRIPTION="$(cat <<'EOF'
<exact project description shown below>
EOF
)"
scse generate "$PROJECT_DESCRIPTION" --from requirements --output "<REDACTED_PARENT_DIR>/generated-web-research-mcp"
```
**Prompt (Project Description):**
> # Project Description: scse-chomp-mcp (MCP server with digest capability)
>
> ## Project Positioning
> Provide "out-of-the-box, digest-enabled" web research capabilities for scse-agent (and any LLM client that follows the MCP protocol). Solve the "information cocoon" problem for LLM agents: when the user's needs involve external frameworks, open-source projects, dependency libraries, APIs, or cutting-edge technologies that the agent is unfamiliar with, the agent can conduct web research through this MCP server and directly obtain structured and refined conclusions, rather than a pile of raw web links.
>
> Core philosophy: **the caller always receives a refined digest, never raw web page content**, to prevent long content from polluting the agent's context.
>
> ## Technology Stack
> - Language: TypeScript (Node.js 18+), ESM module
> - MCP protocol: @modelcontextprotocol/sdk, stdio transport (JSON-RPC over stdin/stdout)
> - No paid third-party API dependencies; zero API key requirements (except for the final LLM digest reuse of the caller's configured LLM)
>
> ## Architecture: Search → Scrape → Deduplicate → LLM-Synthesize Four-Stage Pipeline
>
> The core of this MCP server is a reusable research pipeline, and all external tools are built on top of this pipeline:
>
> 1. **Search (Search)** : Call the SearXNG meta-search engine (aggregating results from multiple engines like Google/Bing/DuckDuckGo) to obtain candidate results (title, URL, snippet). SearXNG requires no API key, is free, and is privacy-friendly.
>
> 2. **Scrape (Scrape)** : Concurrently fetch web page content from candidate URLs. Primary path: HTTP request + cheerio HTML parsing + @mozilla/readability content extraction (same algorithm as Firefox reading mode). For pages that require JavaScript rendering (like some SPA documentation sites), use Puppeteer headless browser rendering and then extract. Each page's content is truncated to a configurable limit (default 8,000 characters per page) to prevent overly long content.
>
> 3. **Deduplicate (Deduplication)** : Perform URL normalization (remove tracking parameters, unify protocol, remove fragments) and then content-level deduplication (based on paragraph fingerprints, such as segmenting the content and taking a hash set for similarity comparison). Near-duplicates are merged, retaining the highest-scored item. Deduplication is crucial for ensuring the quality of the subsequent digest.
>
> 4. **LLM-Synthesize (LLM Digest)** : Feed the deduplicated content to the LLM to produce a structured digest (Answer + Key Points + Sources with URLs). This is the soul of the entire pipeline: the LLM does not simply concatenate content but comprehensively synthesizes multiple sources, removes false information, and extracts key facts, core usage, integration methods, and precautions related to the query. The digest format is structured text, including: direct answers (Answer), key points (Key Points), and source citations (Sources, including URLs). **LLM configuration is injected through environment variables** (see configuration below), and the MCP server internally calls the LLM, so the caller does not need to worry about it.
>
> ## External Tools (MCP Tools)
>
> This MCP server exposes two tools externally, both of which have built-in complete or partial pipelines and built-in LLM Synthesize. The caller always receives a refined result:
>
> ### Tool 1: web_search
> - Purpose: Conduct a complete web search for a query and return a structured digest. This is the main tool, covering scenarios where "I don't understand X, help me find out about it."
> - Parameters:
> - `query` (string, required): The research query, e.g., "How to implement an MCP server using TypeScript" or "How to integrate MyBatis-Plus with RuoYi-Vue3's latest version?"
> - `max_results` (integer, optional, default 5): The maximum number of candidate results to fetch from SearXNG and enter the pipeline (affects time consumption and digest quality; recommended: 3-8)
> - `focus` (string, optional): Research focus hint, guiding the digest to focus on a certain aspect, e.g., "integration steps" / "API usage" / "version compatibility"
> - Returns: Structured digest text (Answer + Key Points + Sources), not raw web content.
>
> ### Tool 2: web_fetch
> - Purpose: Fetch and digest a specific known URL, returning key information about the page. Covers scenarios where "I already know I need to look at this document/repo/page, help me extract key points."
> - Parameters:
> - `url` (string, required): Target URL
> - `focus` (string, optional): Extraction focus, e.g., "installation method" / "configuration parameters" / "CHANGELOG"
> - Returns: Structured digest text for the page, not raw content.
>
> ## Module Division
>
> This project is divided into the following modules based on responsibilities (for further refinement of User Stories during the requirements phase):
>
> ### Module SR (search-retrieval): Search Retrieval
> Responsible for calling SearXNG, parsing the returned JSON, and organizing it into a unified candidate result structure (title/url/snippet/score). Handles errors, timeouts, and retries when the SearXNG public instance is unreachable. Supports switching to a self-hosted SearXNG instance via environment variables.
>
> ### Module SC (scrape-extract): Scrape and Extract
> Responsible for converting URLs into plain text content. Primary path: cheerio + readability; fallback path: Puppeteer. Handles HTTP errors (404/403/timeout), encoding recognition, content length truncation, and special handling for non-HTML resources (like API documentation that is directly JSON/XML).
>
> ### Module DD (deduplicate): Deduplication
> Responsible for URL normalization and content-level deduplication. URL normalization: remove utm_* tracking parameters, unify case, remove fragments, and unify trailing slashes. Content deduplication: segment the content, calculate paragraph fingerprints, compare similarity, and merge near-duplicates.
>
> ### Module SY (synthesize): LLM Digest
> Responsible for interacting with the LLM to produce a structured digest. Encapsulates the LLM client (OpenAI compatible interface), manages digest prompts (requires producing Answer/Key Points/Sources structure, requires citing source URLs, and requires conciseness), and handles LLM call failures, timeouts, and token limits (digest in batches and then merge).
>
> ### Module PL (pipeline-orchestration): Orchestration Layer
> Responsible for orchestrating the four stages into a complete web_search tool process and the sub-process of web_fetch (Scrape → Synthesize). Manages concurrency limits for concurrent fetching, overall timeouts, and error isolation (a single URL failure does not affect the overall process).
>
> ### Module MC (mcp-server): MCP Server Entry
> Responsible for the lifecycle of the MCP server: listening to stdio transport, processing JSON-RPC messages, registering tool definitions (inputSchema for web_search/web_fetch), dispatching tool calls to the PL orchestration layer, and formatting tool responses. This is the entry point for scse-agent (or any MCP client) connections.
>
> ## Cross-Module Conventions
>
> **Configuration (all through environment variables, zero configuration files)**
> - `SEARXNG_URL` (optional, defaults to a public SearXNG instance): SearXNG search endpoint, can point to a self-hosted instance
> - `LLM_API_KEY` (required): LLM service API key (used in the Synthesize stage)
> - `LLM_BASE_URL` (optional, defaults to OpenAI official): OpenAI compatible LLM interface address (supports docking with any compatible service)
> - `LLM_MODEL` (optional, defaults to gpt-4o-mini): Model name used for digest
> - `SCRAPE_TIMEOUT_MS` (optional, defaults to 15000): Single-page fetch timeout
> - `MAX_CONTENT_CHARS` (optional, defaults to 8000): Single-page content truncation limit
> - `MAX_CONCURRENCY` (optional, defaults to 3): Concurrent fetch count
>
> **Error Handling**
> - All external calls (SearXNG, web scraping, LLM) must have timeouts and retries, with single-point failures not affecting the overall pipeline
> - Tool return error messages must be friendly and readable, including failure reasons and suggestions (e.g., when SearXNG is unreachable, prompt "Please check SEARXNG_URL or switch to a self-hosted instance")
> - When LLM digest fails, degrade gracefully to a structured concatenation (truncation) of the fetched content as a backup, rather than overall failure
>
> **Observability**
> - Critical steps output stderr logs (do not affect stdio JSON-RPC communication): number of search results, number of pages successfully fetched, number of pages after deduplication, and digest token count
>
> **MCP Tool Response Specifications**
> - Responses must be MCP standard content arrays (type: text), with the text content being structured digests
> - The digest must include Sources (list of URLs) at the end, allowing the caller to trace back
>
> **Project Deliverables**
> - A directly executable MCP server with `node dist/index.js`
> - package.json declares bin, supporting `npx chomp-mcp`
> - docker-compose.yml (optional): provides a minimal deployment for self-hosted SearXNG, making it easy for users to upgrade to a more stable/private search backend
> - README.md: includes installation, configuration, methods for integrating with scse-agent (mcpServers configuration examples), and methods for self-hosting SearXNG
>
> ## Example of Accessing in MCP Client (README should include)
>
> Add the following to the MCP client's configuration file:
> ```json
> {
> "mcpServers": {
> "chomp-mcp": {
> "command": "node",
> "args": ["/absolute/path/to/chomp-mcp/dist/index.js"],
> "env": {
> "LLM_API_KEY": "sk-xxx",
> "LLM_BASE_URL": "https://api.openai.com/v1",
> "LLM_MODEL": "gpt-4o-mini"
> }
> }
> }
> }
> ```
> After accessing, scse-agent's agent loop will automatically discover the `mcp__web-research__web_search` and `mcp__web-research__web_fetch` tools and call them as needed.
>
> ## Quality Requirements
> - TypeScript strict mode compilation passes with zero errors (tsc --noEmit)
> - MCP server can be normally connected and discover two tools via @modelcontextprotocol/sdk's StdioClientTransport
> - web_search returns a structured digest (non-empty, with Sources) for real queries
> - web_fetch returns a structured digest for real URLs
> - Single web_search (max_results=5) end-to-end time target < 30 seconds (search + fetch + digest)
>
> ## Why This Design (Background for Requirements Generation)
> - No dependence on paid APIs like Tavily: allows any user to use it at zero cost and without restrictions from a single service provider
> - Synthesize inside MCP: avoids raw web content flowing through stdio and polluting the caller's context, which is the core value of "digest capability"
> - SearXNG public instance by default: out-of-the-box usability without users needing to deploy anything; while retaining the option to upgrade to a self-hosted instance
> - Four-stage pipeline modularization: each stage can be tested and optimized independently, and web_fetch can reuse the latter two stages
---
## Project Overview
**chomp-mcp** is an MCP (Model Context Protocol) server that provides "out-of-the-box, digest-enabled" web research capabilities for LLM agents and any MCP-compatible client. It solves the "information cocoon" problem: when a user's needs involve external frameworks, open-source projects, dependency libraries, APIs, or cutting-edge technologies the agent is unfamiliar with, the agent can research the web through this MCP server and receive structured, refined conclusions — not a pile of raw web links.
**Core philosophy:** The caller always receives a refined digest, never raw web page content, preventing long content from polluting the agent's context window.
### Architecture: Search → Scrape → Deduplicate → LLM-Synthesize
The server's core is a reusable research pipeline. Both exposed tools are built on this pipeline:
1. **Search** — Queries the SearXNG meta-search engine (aggregating Google/Bing/DuckDuckGo results) to obtain candidate results (title, URL, snippet). SearXNG requires no API key, is free, and privacy-friendly.
2. **Scrape** — Concurrently fetches page content from candidate URLs. Primary path: HTTP fetch + jsdom + @mozilla/readability. Fallback: Puppeteer headless browser for JavaScript-rendered pages. Content is truncated to a configurable limit (default 8,000 chars/page).
3. **Deduplicate** — URL normalization (strip tracking params, unify scheme/fragment) followed by content-level deduplication (paragraph fingerprinting + Jaccard similarity comparison). Near-duplicates are merged, retaining the highest-scored item.
4. **LLM-Synthesize** — Feeds deduplicated content to an LLM to produce a structured digest (Answer + Key Points + Sources with URLs). Handles token overflow via batch-and-merge. On LLM failure, degrades gracefully to truncated raw content.
### How Modules Interact
```
MCP Client (stdio JSON-RPC)
│
▼
┌─────────────────────────────────┐
│ MC (mcp-server) │ Tool dispatch, validation, response formatting
│ web_search / web_fetch │
└──────────────┬──────────────────┘
│
▼
┌─────────────────────────────────┐
│ PL (pipeline-orchestration) │ Concurrency, timeout, error isolation
│ runWebSearch / runWebFetch │
└──┬──────┬──────┬──────┬─────────┘
│ │ │ │
▼ ▼ ▼ ▼
SR SC DD SY
Search Scrape Dedup Synthesize
```
- **web_search** flows through all four stages: SR → SC → DD → SY
- **web_fetch** flows through two stages: SC → SY
## Project Structure
```
chomp-mcp/
├── package.json # Project manifest, scripts, dependencies
├── tsconfig.json # TypeScript strict-mode compiler config
├── tsconfig.test-check.json # TypeScript config for test type-checking
├── README.md # This file (English)
├── README.zh-CN.md # Chinese documentation
├── .gitignore
│
├── src/
│ ├── index.ts # Entry point — bootstraps MCP server
│ │
│ ├── shared/ # Cross-module shared code
│ │ ├── types/
│ │ │ ├── index.ts # Barrel re-exports for all shared types
│ │ │ ├── search.ts # SearchResult, SearchOptions
│ │ │ ├── scrape.ts # ScrapeResult, ExtractionMethod
│ │ │ ├── content.ts # ContentItem (search result + scraped content)
│ │ │ ├── deduplicate.ts # DeduplicatedItem, DDConfig
│ │ │ └── digest.ts # DigestResult, SourceRef
│ │ ├── utils/
│ │ │ ├── errors.ts # AppError base class + ErrorCategory
│ │ │ ├── logger.ts # stderr-only structured logger
│ │ │ ├── env.ts # Env var parsing with defaults
│ │ │ ├── semaphore.ts # Async concurrency limiter
│ │ │ └── url-utils.ts # URL validation, SSRF check helpers
│ │ └── config/
│ │ └── index.ts # Centralized config singleton
│ │
│ └── modules/
│ ├── mcp-server/ # MC — MCP protocol entry point
│ │ ├── index.ts # Public exports: createServer(), startServer()
│ │ ├── server.ts # Server lifecycle, StdioServerTransport
│ │ ├── tool-definitions.ts # web_search + web_fetch tool schemas
│ │ ├── tool-handlers.ts # Tool-call dispatch to PL functions
│ │ └── validation.ts # Zod-based parameter validation
│ │
│ ├── pipeline-orchestration/ # PL — end-to-end pipeline coordinator
│ │ ├── index.ts # Public exports: runWebSearch(), runWebFetch()
│ │ ├── orchestrator.ts # Top-level pipeline entry + config loader
│ │ ├── web-search-pipeline.ts # Search → Scrape → Dedup → Synthesize
│ │ ├── web-fetch-pipeline.ts # Scrape → Synthesize
│ │ ├── concurrency-limiter.ts # Semaphore-based MAX_CONCURRENCY
│ │ └── timeout-guard.ts # Pipeline timeout with AbortController
│ │
│ ├── search-retrieval/ # SR — SearXNG meta-search client
│ │ ├── index.ts # Public exports: search(), errors
│ │ ├── searxng-client.ts # HTTP GET to SearXNG /search
│ │ ├── response-parser.ts # JSON response → normalized results
│ │ ├── result-normalizer.ts # Score normalization, dedup, truncation
│ │ ├── retry-failover.ts # Exponential backoff + multi-instance failover
│ │ ├── instance-pool.ts # Public instance list + rotation
│ │ └── errors.ts # SearchError, SearchTimeoutError, etc.
│ │
│ ├── scrape-extract/ # SC — URL → clean text content
│ │ ├── index.ts # Public exports: scrape()
│ │ ├── scraper.ts # Main scrape orchestrator (primary → fallback)
│ │ ├── http-fetcher.ts # Native fetch with redirect, timeout, retry
│ │ ├── content-extractor.ts # jsdom + Readability extraction
│ │ ├── puppeteer-renderer.ts # Headless-browser fallback for JS pages
│ │ ├── encoding-detector.ts # Charset detection + iconv-lite conversion
│ │ ├── content-type-router.ts # Route by Content-Type (HTML/JSON/XML/text)
│ │ ├── truncator.ts # Word-boundary truncation at MAX_CONTENT_CHARS
│ │ ├── url-validator.ts # URL format + SSRF private-IP validation
│ │ └── types.ts # ScrapeOptions, ScrapeConfig, internal types
│ │
│ ├── deduplicate/ # DD — Pure dedup/denoise transformation
│ │ ├── index.ts # Public exports: deduplicate()
│ │ ├── deduplicator.ts # Full pipeline entry (URL dedup → content dedup)
│ │ ├── url-normalizer.ts # Tracking-param strip, scheme/port/fragment unify
│ │ ├── paragraph-segmenter.ts # Split text → paragraph segments
│ │ ├── fingerprinter.ts # FNV-1a 32-bit hash → fingerprint sets
│ │ ├── similarity-computer.ts # Jaccard similarity on fingerprint sets
│ │ ├── content-merger.ts # Group + merge near-duplicates, retain best
│ │ ├── stats.ts # DedupStats accumulator + stderr summary
│ │ └── types.ts # DDConfig, DedupStats, FingerprintedItem
│ │
│ └── synthesize/ # SY — LLM synthesis → structured digest
│ ├── index.ts # Public exports: synthesize(), synthesizeSingle()
│ ├── synthesizer.ts # Main synthesis orchestrator
│ ├── llm-client.ts # OpenAI-compatible client init + retry
│ ├── prompt-builder.ts # System + user prompt construction
│ ├── response-parser.ts # Multi-strategy digest parsing from LLM text
│ ├── batch-merger.ts # Token-overflow batch-and-merge logic
│ ├── degradation-handler.ts # Fallback DigestResult from raw content
│ ├── token-estimator.ts # Heuristic token counting (EN/CJK aware)
│ └── types.ts # SynthesizeConfig, LLMResponse, error classes
│
└── tests/ # Test files (mirror src/ structure)
├── shared/
└── modules/
├── mcp-server/
├── pipeline-orchestration/
├── search-retrieval/
├── scrape-extract/
├── deduplicate/
└── synthesize/
```
---
## Getting Started
### Prerequisites
- **Node.js** 18 LTS or later
- **npm** (bundled with Node.js)
- An **LLM API key** (OpenAI-compatible) — required for the synthesis stage
### 1. Install Dependencies
```bash
npm install
```
### 2. Build the Project
```bash
npm run build
```
This runs `tsc` and outputs compiled JavaScript to `dist/`.
### 3. Run the MCP Server
#### Production (compiled)
```bash
LLM_API_KEY="sk-your-api-key" node dist/index.js
```
#### Development (with tsx, no build step)
```bash
LLM_API_KEY="sk-your-api-key" npm run dev
```
### 4. Type-Check (no emit)
```bash
npm run typecheck
```
### 5. Run Tests
```bash
npm test
```
### Environment Variables
All configuration is via environment variables — zero config files.
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `LLM_API_KEY` | **Yes** | — | API key for the LLM service (OpenAI-compatible) |
| `LLM_BASE_URL` | No | `https://api.openai.com/v1` | LLM API base URL |
| `LLM_MODEL` | No | `gpt-4o-mini` | Model name for synthesis |
| `SEARXNG_URL` | No | Public instances | SearXNG search endpoint (self-hosted URL) |
| `SCRAPE_TIMEOUT_MS` | No | `15000` | Per-page scrape timeout (ms) |
| `MAX_CONTENT_CHARS` | No | `8000` | Max content chars per page |
| `MAX_CONCURRENCY` | No | `3` | Concurrent scrape limit |
| `DEDUP_SIMILARITY_THRESHOLD` | No | `0.85` | Jaccard threshold for near-duplicate detection |
| `MIN_PARAGRAPH_CHARS` | No | `20` | Min paragraph length for fingerprinting |
| `DEDUP_EXTRA_TRACKING_PARAMS` | No | (empty) | Comma-separated extra tracking params to strip |
### Self-host SearXNG
`web_search` works best with a self-hosted SearXNG instance. The example below starts SearXNG on `127.0.0.1:8080`, enables JSON responses, and prefers Bing because some public engines may be slow or unavailable in restricted network environments.
Create a local SearXNG directory:
```bash
mkdir -p ./searxng-local
cd ./searxng-local
```
Create `settings.yml`:
```yaml
use_default_settings: true
server:
secret_key: "<REPLACE_WITH_RANDOM_SECRET>"
limiter: false
image_proxy: false
search:
safe_search: 0
autocomplete: ""
formats:
- html
- json
ui:
static_use_hash: true
outgoing:
request_timeout: 10.0
max_request_timeout: 15.0
engines:
- name: bing
engine: bing
shortcut: bi
disabled: false
base_url: https://cn.bing.com
- name: google
disabled: true
- name: duckduckgo
disabled: true
- name: brave
disabled: true
- name: startpage
disabled: true
- name: wikipedia
disabled: true
- name: wikidata
disabled: true
```
Create `docker-compose.yml`:
```yaml
services:
searxng:
image: ghcr.io/searxng/searxng:latest
container_name: searxng-local
restart: unless-stopped
ports:
- "127.0.0.1:8080:8080"
volumes:
- ./settings.yml:/etc/searxng/settings.yml:ro
environment:
- SEARXNG_BASE_URL=http://127.0.0.1:8080/
```
If pulling from `ghcr.io` is slow in your network, replace the image with a trusted mirror and keep the same container configuration.
Start SearXNG:
```bash
docker compose up -d
```
Verify JSON search:
```bash
curl 'http://127.0.0.1:8080/search?q=Model%20Context%20Protocol&format=json'
```
Use this value in chomp-mcp:
```bash
SEARXNG_URL="http://127.0.0.1:8080"
```
Do not include `/search` in `SEARXNG_URL`; chomp-mcp appends `/search` internally.
### Connecting from an MCP Client
Add the server to your MCP client configuration, such as Trae or Claude Desktop. The example below is redacted: replace paths, API keys, base URLs, and model names with your own values.
```json
{
"mcpServers": {
"chomp-mcp": {
"command": "node",
"args": [
"<ABSOLUTE_PATH_TO_CHOMP_MCP>/dist/index.js"
],
"env": {
"LLM_API_KEY": "<REDACTED_LLM_API_KEY>",
"LLM_BASE_URL": "<OPENAI_COMPATIBLE_BASE_URL>",
"LLM_MODEL": "<MODEL_NAME>",
"LLM_TIMEOUT_MS": "120000",
"LLM_MAX_CONTEXT_TOKENS": "1000000",
"SEARXNG_URL": "http://127.0.0.1:8080",
"SEARXNG_TIMEOUT_MS": "15000",
"SCRAPE_TIMEOUT_MS": "30000",
"MAX_CONTENT_CHARS": "30000",
"MAX_CONCURRENCY": "2",
"PIPELINE_TIMEOUT_MS": "240000"
}
}
}
}
```
After connecting, the client will discover `web_search` and `web_fetch` tools.
Example prompts:
```text
Use chomp-mcp to search "Model Context Protocol" and summarize the technical details.
```
```text
Use chomp-mcp to fetch https://modelcontextprotocol.io/docs/getting-started/intro and summarize the core concepts.
```
## API Overview
This server exposes two MCP tools via stdio JSON-RPC transport. There are no HTTP REST endpoints — all communication is over stdin/stdout using the MCP protocol.
### Tool: `web_search`
Performs a full web research pipeline (Search → Scrape → Deduplicate → Synthesize) and returns a structured digest.
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `query` | string | Yes | — | Research query (max 500 chars) |
| `max_results` | integer | No | `5` | Max candidate results from SearXNG (recommended 3–8) |
| `focus` | string | No | — | Research focus hint to guide the digest (max 500 chars) |
**Returns:** Structured digest text containing:
- **Answer** — Direct synthesized answer to the query
- **Key Points** — Bullet list of key findings
- **Sources** — Referenced URLs with titles
### Tool: `web_fetch`
Fetches a specific URL and synthesizes a structured digest of its content (Scrape → Synthesize).
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `url` | string | Yes | — | Target URL to fetch and summarize |
| `focus` | string | No | — | Extraction focus hint (max 500 chars) |
**Returns:** Structured digest text (same format as `web_search`).
### Response Format
All tool responses are MCP-standard `content` arrays with `type: "text"`. The text content is the structured digest. Error responses use `isError: true` with a human-readable error message. Responses exceeding 25,000 characters are truncated with a marker.
## Module Overview
### MC — MCP Server (`src/modules/mcp-server/`)
The MCP protocol entry point. Manages server lifecycle, stdio transport, tool registration, parameter validation (Zod), and tool-call dispatch to the pipeline-orchestration layer. Wraps all handlers in try/catch to ensure server stability.
**Key files:** `server.ts` (lifecycle), `tool-definitions.ts` (tool schemas), `tool-handlers.ts` (dispatch), `validation.ts` (Zod validation)
### PL — Pipeline Orchestration (`src/modules/pipeline-orchestration/`)
The end-to-end pipeline coordinator. Orchestrates the Search → Scrape → Dedup → Synthesize flow for `web_search` and the Scrape → Synthesize flow for `web_fetch`. Manages concurrency limiting (semaphore-based), overall pipeline timeout (AbortController), and per-URL error isolation.
**Key files:** `orchestrator.ts` (entry + config), `web-search-pipeline.ts`, `web-fetch-pipeline.ts`, `concurrency-limiter.ts`, `timeout-guard.ts`
### SR — Search Retrieval (`src/modules/search-retrieval/`)
SearXNG meta-search client. Sends HTTP GET requests to SearXNG, parses JSON responses, normalizes results (score normalization, URL dedup, field truncation). Supports self-hosted and public instance modes with automatic failover and exponential backoff retry.
**Key files:** `searxng-client.ts` (HTTP client), `response-parser.ts` (JSON parsing), `result-normalizer.ts` (normalization), `retry-failover.ts` (retry logic), `instance-pool.ts` (instance rotation), `errors.ts` (error hierarchy)
### SC — Scrape Extract (`src/modules/scrape-extract/`)
URL-to-clean-text extraction. Primary path: native `fetch` + jsdom + @mozilla/readability. Fallback: Puppeteer headless browser for JavaScript-rendered pages. Handles HTTP redirects, timeouts, charset detection (iconv-lite), content-type routing (HTML/JSON/XML/text/binary), word-boundary truncation, and SSRF protection (private-IP blocking).
**Key files:** `scraper.ts` (orchestrator), `http-fetcher.ts` (HTTP with retry), `content-extractor.ts` (Readability), `puppeteer-renderer.ts` (JS rendering), `content-type-router.ts`, `encoding-detector.ts`, `truncator.ts`, `url-validator.ts`
### DD — Deduplicate (`src/modules/deduplicate/`)
Pure deduplication and denoising transformation. URL normalization strips tracking parameters (utm_*, etc.), unifies scheme/port/fragment. Content-level dedup uses paragraph segmentation, FNV-1a 32-bit fingerprinting, and Jaccard similarity comparison. Near-duplicates are merged via Union-Find, retaining the highest-scored item. Never throws — all errors are caught and logged.
**Key files:** `deduplicator.ts` (pipeline entry), `url-normalizer.ts`, `paragraph-segmenter.ts`, `fingerprinter.ts`, `similarity-computer.ts`, `content-merger.ts`, `stats.ts`
### SY — Synthesize (`src/modules/synthesize/`)
LLM-powered synthesis producing structured digests. Initializes an OpenAI-compatible client, constructs system/user prompts, calls the LLM with retry (exponential backoff for transient errors, separate timeout retries). Handles token overflow via batch-and-merge. On unrecoverable failure, degrades to a fallback digest from raw content. Never rejects a Promise to the caller.
**Key files:** `synthesizer.ts` (orchestrator), `llm-client.ts` (client + retry), `prompt-builder.ts` (prompts), `response-parser.ts` (multi-strategy parsing), `batch-merger.ts` (token overflow), `degradation-handler.ts` (fallback), `token-estimator.ts` (EN/CJK aware)
### Shared (`src/shared/`)
Cross-module type definitions (`SearchResult`, `ScrapeResult`, `ContentItem`, `DeduplicatedItem`, `DigestResult`, `SourceRef`), the `AppError` base class with `ErrorCategory` discriminator, and utility functions (logger, env parsing, semaphore, URL helpers).
## License
MIT
MCP Config
Below is the configuration for this MCP Server. You can copy it directly to Cursor or other MCP clients.
mcp.json
Connection Info
You Might Also Like
No recommendations available