Content
# PDF MCP Module
[](https://opensource.org/licenses/MIT)
[](https://www.rust-lang.org/)
[](https://modelcontextprotocol.io/)
**AI-native knowledge compilation engine** — Compile PDF documents into structured knowledge bases, providing long-term memory and reasoning backend for AI clients like Claude, Cursor.
[English](./README.en.md) |
📖 **Online Documentation**: [smile9493.github.io/Compendium](https://smile9493.github.io/Compendium/)（MkDocs, see [docs/PUBLISHING_DOCS.md](docs/PUBLISHING_DOCS.md)）
## ✨ Features
- 🔥 **Karpathy Compiler Mode** — Pre-compile PDFs into structured Markdown, accumulative knowledge, explainable, supporting L0→L1→L2→L3 knowledge pyramid
- 🧠 **Cognitive Index Layer** — Tantivy full-text search + petgraph knowledge graph + TF-IDF vector embedding, three-way retrieval fusion
- 🚀 **Pure Rust Implementation** — Single binary deployment, zero external service dependencies, high-performance FFI levee
- 🔄 **Incremental Compilation** — Merkle hash detection, compile only changed PDFs
- 🖼️ **VLM Visual Understanding** — Conditional OCR fallback for scanned/image PDFs
- 🌐 **Dual-protocol Service** — stdio (MCP) + HTTP (Wiki browsing), oneshot signal startup
- 🦀 **Rust 2024 Edition** — Written in Rust 2024, requires rustc 1.91+
- 🎯 **53 MCP Tools** (full mode) — or **Code Mode** (2 tools + TypeScript API resources, see [Code Mode](docs/CODE_MODE.md))
## 📦 Installation
### One-click Installation
```bash
curl -fsSL https://raw.githubusercontent.com/smile9493/Compendium/main/install.sh | bash
```
### Docker
```bash
docker pull smile9493/pdf-mcp:latest
```
### Build from Source
```bash
git clone https://github.com/smile9493/Compendium.git
cd Compendium/pdf-module-rs
cargo build --release --bin pdf-mcp
```
## 🚀 Quick Start
### 1. Configure AI Client
**Cursor** (`~/.cursor/mcp.json`):
```json
{
"mcpServers": {
"pdf-mcp": {
"command": "/opt/pdf-module/pdf-mcp",
"env": {
"PDFIUM_LIB_PATH": "/opt/pdf-module/lib/libpdfium.so"
}
}
}
}
```
**Code Mode** (reduce tool schema overhead in context, recommended for multi-step ingest/query tasks):
```json
{
"mcpServers": {
"pdf-mcp": {
"command": "/opt/pdf-module/pdf-mcp",
"env": {
"COMPENDIUM_MCP_MODE": "code",
"KNOWLEDGE_BASE_PATH": "/home/user/my-kb"
}
}
}
}
```
See [docs/CODE_MODE.md](docs/CODE_MODE.md) for details.
**Claude Desktop** (`claude_desktop_config.json`):
```json
{
"mcpServers": {
"pdf-mcp": {
"command": "/opt/pdf-module/pdf-mcp"
}
}
}
```
### 2. Initialize Knowledge Base (Karpathy Template)
```bash
compendium kb init ~/my-kb
```
### 3. Compile PDF to Knowledge Base
```
User: Help me compile /path/to/paper.pdf into knowledge base
AI: [Invoke compile_to_wiki tool]
Compiled PDF into knowledge base:
- Raw extraction: raw/paper.md
- Compilation prompt: raw/paper.compile_prompt.md
Please read extraction content, distill core concepts, and create atomic entries...
```
### 4. Search Knowledge Base
```
User: Search for knowledge about HTTP/2
AI: [Invoke search_knowledge tool]
Found 3 relevant knowledge:
1. [IT] HTTP/2 Multiplexing (score: 0.92)
2. [IT] HTTP/2 Header Compression (score: 0.85)
3. [Network] HTTP/2 vs HTTP/1.1 Comparison (score: 0.78)
```
## 🛠️ MCP Tools (53)
### PDF Extraction (6)
| Tool | Description |
|------|-------------|
| `extract_text` | Extract PDF plain text |
| `extract_structured` | Extract structured data (page text + bbox) |
| `get_page_count` | Get PDF page count |
| `search_keywords` | Search keywords in PDF (supports regex) |
| `extrude_to_server_wiki` | Extract to server-side Wiki |
| `extrude_to_agent_payload` | Return Markdown payload to conversation |
### Knowledge Compilation (10)
| Tool | Description |
|------|-------------|
| `init_knowledge_base` | Initialize empty knowledge base (Karpathy template) |
| `compile_to_wiki` | PDF → knowledge base compilation entry |
| `incremental_compile` | Incremental compilation (Merkle hash detection) |
| `save_wiki_entry` | Create or update wiki entry (with front matter) |
| `recompile_entry` | Recompile single entry + version backup |
| `aggregate_entries` | L1→L2 aggregation candidate discovery |
| `check_quality` | Wiki quality scan (drift/contradiction detection) |
| `micro_compile` | Instant extraction (not persisted) |
| `hypothesis_test` | Contradiction pair discovery + debate framework generation |
| `lint_wiki` | Karpathy aggregation lint (contradiction/orphan/concept/broken link) |
| `archive_answer` | QA conversation result written as overview page |
| `complete_compile_job` | Complete compilation job: rebuild index + quality gate |
### Cognitive Index (6)
| Tool | Description |
|------|-------------|
| `search_knowledge` | Multimodal search (keyword/semantic/hybrid/wiki_first) |
| `rebuild_index` | Rebuild all indexes |
| `get_entry_context` | N-hop neighbor discovery (knowledge graph traversal) |
| `find_orphans` | Orphan entry detection |
| `suggest_links` | Link suggestions (Jaccard similarity) |
| `export_concept_map` | Mermaid.js concept map export |
### Management (16)
| Tool | Description |
|------|-------------|
| `get_config` | Get current configuration |
| `set_config` | Update configuration item |
| `get_health_report` | System health report (engine/index/cache status) |
| `trigger_incremental_compile` | Trigger batch incremental compilation |
| `get_compile_status` | Query compilation task status |
| `list_quality_issues` | List pending quality issues |
| `fix_suggest` | Generate fix suggestions for quality issues |
| `apply_quality_gate` | Apply quality gate (block/degrade/warning) |
| `show_wiki_browser` | Show Wiki browser entry |
| `list_workspaces` | List all workspaces |
| `set_active_workspace` | Set current workspace |
| `register_workspace` | Register new workspace |
| `sync_push` / `sync_pull` / `sync_status` | Git-style sync |
| `submit_patch_proposal` / `list_patch_proposals` | Patch proposal |
| `list_extraction_plugins` / `probe_extraction` | Plugin management and probing |
### Resources (2)
| Resource | Description |
|----------|-------------|
| `rust-pdf://dashboard` | Embedded system dashboard (rust_embed compiled) |
| `rust-pdf://wiki-browser` | Embedded knowledge base browser |
## 🏗️ Architecture
```
┌──────────────────────────────────────────────────┐
│ AI Client (Claude / Cursor) │
│ 53 MCP tools via JSON-RPC │
└──────────────┬───────────────┬───────────────────┘
│ stdio │ HTTP
▼ ▼
┌──────────────────────┐ ┌──────────────────┐
│ pdf-mcp (server) │ │ Wiki HTTP │
│ JSON-RPC dispatch │ │ axum + embed │
├──────────────────────┤ └──────────────────┘
│ Extraction │ Compile│
│ Indexing │ Manage │
└──────────────┬───────┘
│
┌──────────┴───────────┐
▼ ▼
┌──────────────┐ ┌──────────────────┐
│ PdfiumEngine │ │ VlmVisualGateway │
│ (FFI levee) │ │ (conditional OCR) │
└──────────────┘ └──────────────────┘
│ │
▼ ▼
┌──────────────────────────────────────┐
│ Knowledge Engine │
│ Tantivy │ petgraph │ TF-IDF Vector │
│ hash_cache │ cache_db │ bincode │
└──────────────────────────────────────┘
```
### Levee Layering
```
Facade Layer: pdf-mcp (MCP protocol), vlm-visual-gateway (HTTP facade)
Core Layer: pdf-core (extraction, knowledge, parallel), pdf-common (shared)
Infra Layer: pdf-macros (derive macros), pdf-wasm (WASM target)
```
## 📁 Knowledge Base Structure
```
knowledge_base/
├── raw/ # Raw PDF extraction (YAML)
├── wiki/ # Compiled knowledge base
│ ├── index.md # Global navigation
│ ├── log.md # Operation log
│ ├── .versions/ # Recompilation backup
│ └── <domain>/ # Domain entries
├── schema/ # Compilation instructions
├── .hash_cache/ # Merkle hash cache (JSON)
├── .rsut_index/ # Rebuildable index
│ ├── tantivy/ # Full-text search index
│ └── graph.bin # Knowledge graph persistence (bincode)
└── .cache_db/ # Entry cache
```
## 📝 Entry Format
```yaml
---
entry_type: concept # concept | entity | source-summary | comparison | overview
confidence: high # high | medium | low
title: "HTTP/2 Multiplexing"
domain: "IT"
source: "raw/rfc7540.pdf"
page: 12
tags: ["http", "networking", "protocol"]
level: L1
status: compiled
quality_score: 0.85
version: 1
contradictions: []
related: ["wiki/it/http1.md"]
created: 2026-05-04T00:00:00Z
updated: 2026-05-04T00:00:00Z
---
# HTTP/2 Multiplexing
Content...
```
## 🗺️ Knowledge Pyramid
```
L3 Domain Map (domain navigation, one per domain)
↑ aggregated from
L2 Aggregation (overview, multiple L1 merged)
↑ aggregated from
L1 Atomic Concept (atomic concept, core knowledge unit)
↑ compiled from
L0 Raw Extraction (raw extraction, PDF → text)
```
## ⚙️ Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `PDFIUM_LIB_PATH` | PDFium dynamic library path | Auto-detect |
| `VLM_API_KEY` | VLM API key | - |
| `VLM_MODEL` | Model name | `glm-4v-flash` |
| `VLM_ENDPOINT` | API endpoint | ZhiPu API |
| `MCP_HTTP_PORT` | HTTP Wiki service port | - (disabled) |
| `KB_PATH` | Knowledge base root path | `./knowledge_base` |
## 📁 Project Structure
```
pdf-module/
├── pdf-module-rs/ # Rust workspace (core engine)
│ ├── crates/
│ │ ├── pdf-common/ # Shared types/errors/DTO
│ │ ├── pdf-core/ # Extraction/knowledge/parallel engine
│ │ ├── pdf-mcp/ # MCP protocol server
│ │ ├── pdf-wasm/ # WASM compilation target
│ │ ├── pdf-web/ # Deprecated: management API sidecar — use pdf-mcp (ADR-004)
│ │ ├── pdf-cli/ # CLI tool
│ │ ├── pdf-dashboard/ # Dashboard service
│ │ ├── pdf-macros/ # Derive macros
│ │ └── vlm-visual-gateway/ # VLM gateway
│ ├── ARCHITECTURE.md # Architecture documentation
│ └── CHANGELOG.md # Changelog
├── scripts/ # Test/tool scripts
├── docs/ # User documentation
│ ├── API_REFERENCE.md # MCP tool reference
│ ├── VLM_INTEGRATION.md # VLM integration guide
│ └── ...
├── CONTRIBUTING.md # Contribution guide
├── plugins/ # Third-party MCP plugins
├── pdf-mcp-installer/ # Installer
├── deploy/ # Deployment configuration
├── docker/ # Docker image
├── Dockerfile
└── docker-compose.yml
```
## 📄 License
[MIT](LICENSE)
