mcp-rag-agent

# Tool List **English** | [English](README.en.md) > Transforming papers into a retrievable and traceable knowledge base using Retrieval-Augmented Generation (RAG) agents: A hybrid approach combining retrieval, knowledge graph multi-hop reasoning, and multi-agent orchestration, providing tools via the MCP protocol and evaluated using RAGAS.    ## What is This Given a PDF paper, the system can: - Parse the paper into structured text (separating charts, formulas, and titles) - Build a vector database, knowledge graph, and hierarchical summary index - Answer questions about the paper (facts, relationships, summaries) with source page numbers and sections - Perform multi-hop reasoning ("What is the relationship between A and B?") - Understand paper charts (multimodal) - Self-check answer credibility (reflection + token-level hallucination detection) Unlike directly querying an LLM, this system does not rely on the LLM to memorize information but instead **retrieves evidence from the paper and then generates answers**, making answers traceable and hallucinations controllable. ## Effects End-to-end question-answering was tested on four publicly available papers from different fields, using the gold standard keyword hit rate to calculate F1: | Paper | Field | F1 | |---|---|---| | bge_paper | Embedding model | 91 ~ 100% | | MiniMax-M1 technical report | LLM | 83% | | Attention Is All You Need | NLP classic | 100% | | ResNet | CV classic | 94% | Evaluated using ragas 0.2.15 (LLM-as-judge, not custom metrics) for RAGAS standard metrics: | Metric | Meaning | Overall | |---|---|---| | **Context Recall** | Proportion of gold standard facts retrieved | **97.9%** | | **Faithfulness** | Proportion of answer assertions supported by evidence (higher means lower hallucination) | **95.2%** |    ## Quick Start ### 1. Environment Python 3.12, recommend creating a virtual environment: ```bash python3.12 -m venv env source env/bin/activate # Windows: env\Scripts\activate ``` ### 2. Install Dependencies ```bash pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple ``` (Requirements.txt also lists Alibaba Cloud, Huawei Cloud, and Douban sources, feel free to choose.) ### 3. Download Models Two models need to be downloaded from HuggingFace (large in size, so not included in the repository): ```bash # Embedding model bge-m3 git clone https://huggingface.co/BAAI/bge-m3 models/Xorbits/bge-m3 # Re-ranking model bge-reranker-v2-m3 git clone https://huggingface.co/BAAI/bge-reranker-v2-m3 models/bge-reranker-v2-m3 ``` For slow HF access, use ModelScope: ```bash pip install modelscope modelscope download --model BAAI/bge-m3 --local_dir models/Xorbits/bge-m3 modelscope download --model BAAI/bge-reranker-v2-m3 --local_dir models/bge-reranker-v2-m3 ``` ### 4. Configure API Key Create a `.env` file in the project root directory (already in `.gitignore`, won't be uploaded): ```ini MINIMAX_API_KEY=your_MiniMax_API_Key MINIMAX_API_HOST=https://api.minimaxi.com/v1 ``` MiniMax API Key can be registered and obtained at https://platform.minimaxi.com. The LLM call in the code uses the OpenAI-compatible interface of MiniMax-M3. ### 5. (Optional) Start Neo4j The knowledge graph can run with the built-in in-memory graph, no need to install Neo4j. If desired: ```bash docker-compose up -d ``` Default connection `bolt://localhost:7687`, account and password can be modified in `docker-compose.yml` and `.env`. ### 6. Run ```bash # ①入库：parse data/ PDFs → chunk → vectorize and store in ChromaDB python script/run_embed_all.py # ② Build knowledge graph (example with bge_paper) python script/08_kg_extractor.py data/bge_paper.pdf python script/09_kg_builder.py output/kg_triples/bge_paper.pdf.json # ③ Direct RAG question-answering python script/06_rag_query.py "what is bge-m3" # ④ Agent orchestration question-answering (recommended, full route → retrieval → reasoning → reflection) python script/16_agent_orchestrator.py "what does MiniMax-M1 use" # ⑤ Run evaluation python script/17_eval_framework.py # ⑥ Start web interface python script/19_web_app.py ``` Each script supports `--help` to view parameters. Each `test_XX_accuracy.py` can verify the accuracy of the corresponding module. ## System Architecture  The system consists of five layers: - **User Layer**: Gradio web interface / FastAPI / knowledge graph visualization - **MCP Protocol Layer**: FastMCP encapsulates capabilities into standard tools, directly callable by MCP clients like Claude - **Agent Orchestration Layer**: Routing → retrieval → reasoning → reflection, Corrective RAG paradigm - **Retrieval Layer**: Vector + knowledge graph + hierarchical summary three-way recall, Cross-Encoder two-stage re-ranking - **Data Layer**: ChromaDB / in-memory graph (can switch to Neo4j) / bge-m3 / bge-reranker ## Core Technologies ### 1. Hybrid Retrieval: Vector + BM25 + RRF + HyDE Pure vector retrieval has issues with proper nouns and numbers. The solution is to stack strategies: ``` Pure vector (semantic recall) → +BM25 (exact matching) + RRF fusion → +HyDE (hypothetical answer retrieval) → +Cross-Encoder re-ranking ``` - **RRF** (Reciprocal Rank Fusion): Vectors have cosine scores (0~1), BM25 scores are non-normalized, with different units. RRF only looks at rankings, not absolute scores: `score(d) = Σ 1/(k + rank)`, k = 60. - **HyDE** (Hypothetical Document Embeddings): Queries are too short, chunks are too long, direct similarity calculation is unfair. First, let the LLM generate a hypothetical answer, and use the vector of the hypothetical answer to retrieve – it and the chunk are both "answer-like", semantically closer. ### 2. Cross-Encoder Two-Stage Re-ranking Bi-Encoder: Query and doc are encoded separately for cosine similarity, fast but with limited accuracy, no interaction between vectors. Cross-Encoder: Query and doc are concatenated and fed into a transformer for cross-attention, outputting a relevance score. Accuracy is higher by one order of magnitude, but slower (need to pass through the model each time). So, use two stages: ``` Bi-Encoder coarse recall top-N → Cross-Encoder (bge-reranker-v2-m3) fine ranking top-5 (fast, can be precomputed) (accurate, real-time scoring) ``` ### 3. GraphRAG: Enabling RAG for Multi-Hop Reasoning Vector retrieval can only find "similar fragments", but questions like "What other technologies has A used?" have answers scattered across multiple paragraphs, which similarity can't find. The solution is to extract entity relationships from the paper into a graph and use graph traversal for multi-hop reasoning: ``` Paper text → LLM extracts triples → entity disambiguation → build graph → text2cypher query ``` **Entity disambiguation** is the most challenging part ("MiniMax-M1", "M1", "the model" might refer to the same entity), using a three-layer strategy: substring inclusion → similarity threshold → LLM arbitration. **text2cypher** (similar to text2SQL): LLM translates natural language into graph query statements. When Neo4j is not connected, a built-in in-memory graph backend is used as a fallback. ### 4. Multi-Agent Orchestration (Corrective RAG) ``` Router (12) → Retriever (13) → Reasoner (14) → Reflector (15) ``` - **Router**: Determine question type (fact / relationship / summary), tools to use (vector / graph / summary), and hops. Rules are prioritized, and if rules are uncertain, ask the LLM for arbitration. - **Retriever**: Multi-route recall according to routing strategy → RRF fusion → MMR diversity deduplication → cross-encoder re-ranking. - **Reasoner**: Extraction-then-generation – first, let the LLM enumerate all relevant terms in the fragment, and then write the answer, retaining key terms verbatim. Multiple sampling takes the one with the highest faithfulness (self-consistency). - **Reflector**: Break down the answer into atomic assertions, and for each, find evidence in retrieval fragments. There's a "subject co-reference tolerance" detail: in the answer, "it used lightning attention" and in the fragment, "MiniMax-M1 used lightning attention", "it" = MiniMax-M1, shouldn't be judged as hallucination. ### 5. MCP Protocol Tool Layer Encapsulate retrieval / graph / multimodal / internet search into standard MCP (Model Context Protocol) tools. Any MCP-supported client (Claude Desktop, other agents) can directly call without handwritten API integration. ## Project Structure ``` . ├── script/ # All code (numbered in pipeline order) │ ├── 01_config.py # Global configuration │ ├── 02_pdf_parser.py # PDF parsing (multi-column / charts / formulas / titles) │ ├── 03_chunker.py # Text chunking + section attribution │ ├── 04_embedder.py # bge-m3 vectorization + ChromaDB │ ├── 05_llm_client.py # MiniMax-M3 client encapsulation │ ├── 06_rag_query.py # Hybrid retrieval (vector+BM25+RRF+HyDE+rerank) │ ├── 07_mcp_server.py # MCP tool service │ ├── 08_kg_extractor.py # LLM triple extraction │ ├── 09_kg_builder.py # Entity disambiguation + graph building │ ├── 10_kg_query.py # text2cypher + graph query │ ├── 11_summary_indexer.py # Hierarchical summary (document/section/paragraph) │ ├── 12_router_agent.py # Routing agent │ ├── 13_retriever_agent.py # Retrieval agent (multi-route + MMR + rerank) │ ├── 14_reasoning_agent.py # Reasoning agent (self-consistency) │ ├── 15_reflection_agent.py # Reflection agent (hallucination detection) │ ├── 16_agent_orchestrator.py # Agent orchestrator │ ├── 17_eval_framework.py # RAGAS evaluation framework │ ├── 17b_ragas_official.py # ragas original (LLM-as-judge对照) │ ├── 18_eval_dataset.json # Evaluation dataset │ ├── 19_web_app.py # Gradio web interface │ ├── 20_api_server.py # FastAPI │ ├── 21_kg_visualizer.py # Knowledge graph visualization │ ├── _reranker.py # Cross-Encoder re-ranking module │ ├── _memory_graph.py # In-memory graph backend (Neo4j fallback) │ ├── _kg_gold.py # Test gold standard data │ ├── _eval_helpers.py # Evaluation tool functions │ ├── run_embed_all.py # One-click入库 script │ └── test_*.py # Module accuracy tests ├── data/ # 4 test paper PDFs ├── images/ # Architecture and result diagrams ├── requirements.txt ├── docker-compose.yml # Neo4j (optional) ├── LICENSE └── README.md ``` ## Core Module Quick Check | Module | Key Function | Effect | |---|---|---| | `02_pdf_parser` | `parse_pdf`, `detect_all_figures`, `is_real_table` | Three-layer chart detection (bitmap/vector/annotation) + fake table filtering | | `06_rag_query` | `retrieve`, `rrf_fuse`, `generate_hyde`, `rerank_with_llm` | Hybrid retrieval main process | | `09_kg_builder` | `disambiguate_entities`, `normalize_triples` | Three-layer entity disambiguation + relationship normalization + citation filtering | | `10_kg_query` | `nl_to_cypher`, `standardize_projection`, `graph_search` | Natural language → Cypher → execution → answer projection | | `11_summary_indexer` | `extractive_summary`, `retrieve_hierarchical` | Key sentence extraction summary + three-layer RRF retrieval | | `13_retriever_agent` | `retrieve`, `mmr_rerank` | Multi-route recall + MMR diversity re-ranking | | `15_reflection_agent` | `verify_answer`, `_claim_supported` | Token-level hallucination detection (named entity/number strict) | | `16_agent_orchestrator` | `answer` | End-to-end orchestration: routing → retrieval → reasoning → reflection | ## Tech Stack | Component | Selection | Used in | |---|---|---| | LLM | MiniMax-M3 | Reasoning / triple extraction / text2cypher | | Embedding | bge-m3 | Text vectorization | | Reranker | bge-reranker-v2-m3 | Cross-Encoder re-ranking | | Vector Database | ChromaDB | Semantic retrieval | | Knowledge Graph | NetworkX (in-memory graph) / Neo4j | Multi-hop reasoning | | Tool Protocol | FastMCP | Standardized tool calling | | Interface | Gradio / FastAPI | Web / REST API | | Evaluation | ragas 0.2.15 | RAGAS standard metrics | ## Contributors - **Apageoflove** — Independent design and implementation ## License [MIT](LICENSE)