kb-rag

# Tool List > **One sentence**: Throw PDF/Word/EPUB/HTML documents into it, and let LLM automatically extract entities and relationships to build a knowledge graph that supports natural language retrieval. Supports Obsidian dual-chain notes, MCP direct connection to Claude/Cursor, and Streamlit visual browsing. **Project positioning**: Not just another naive RAG toy, but a production-grade knowledge base pipeline designed for individual developers/small teams. Core solutions to "heterogeneous document parsing → accurate entity extraction → high-quality hybrid retrieval → multi-terminal access" full-chain problems have been verified in multiple scenarios such as philosophical works, academic papers, and novels. Currently, three-layer pipeline + intelligent PDF routing + hybrid retrieval (vector + BM25 + RRF) + semantic cache + Obsidian entity isolation export have been implemented. **Features go far beyond these** - visual retrieval path, adaptive block strategy, graph front-end canvas, multi-modal expansion, etc. are being planned and prototyped. Welcome to define and implement the next stage together. [](https://www.python.org/) [](https://github.com/astral-sh/uv) [](https://github.com/HKUDS/LightRAG) [](https://opensource.org/licenses/MIT) ## Why choose kb-rag instead of directly using LightRAG / LlamaIndex / AnythingLLM? - **Intelligent PDF classification and routing** (`importers/classifier.py` + `router.py`): Use PyMuPDF to pre-scan, statistically analyze **formula density** (character spacing anomaly line ratio ≥2%), **citation markers** (`[1]`, `et al.` and other regular hits ≥3), and **academic keywords** (Abstract/Introduction/References). Automatically decide to use free and fast PyMuPDF or high-precision MinerU API (formula → LaTeX, table → HTML, scan piece OCR). **Underlying logic**: Avoids entity recall rate crash caused by PyMuPDF text layer missing in math/academic documents, and avoids unnecessary API cost and delay of full MinerU. - **Production-grade hybrid retrieval pipeline** (`knowledge_graph/lightrag_engine.py`): Three-stage late fusion 1. LLM understands the problem and generates query 2. LightRAG vector retrieval (entities_vdb + relationships_vdb, low-level + high-level keywords) + jieba + BM25Okapi keyword retrieval 3. **RRF (Reciprocal Rank Fusion)** re-ranking (only re-ranking position, not stuffing BM25 noise block into prompt) **Solving pain points**: Pure vector easily loses "short reference block", pure BM25 easily ranks unrelated high-frequency word blocks in front, leading to LLM hallucinations. RRF lets the most relevant block rank in front of the prompt, significantly improving answer quality (see development log "Tang Junyi's six kinds of reason"). - **Semantic cache + lazy initialization**: After query embedding, cosine similarity > 0.92 (configurable) directly hits the cache (persisted to `kb_data/query_cache.json`, LRU eliminates the oldest 1000). Synonymous questions ("What are the six kinds of reasons?" vs "What are the characteristics of the six kinds of reasons?") return in milliseconds. **LightRAG Engine only initializes at the first operation**, avoiding large VDB/KV memory allocation + disk I/O peak at startup. - **Obsidian deep integration**: Entity nodes write `source_doc` attributes, isolated export as independent Markdown + [[wikilink]] dual chain. Cross-document same-name entities do not pollute each other's vault. `--vault` parameter one-click synchronization. - **Multi-interface unified backend** (same LightRAG instance): - `kb serve` → FastMCP Server (SSE, 4 tools: query / list_documents / get_entities / get_document), perfectly connected to Claude Desktop / Cursor / Continue.dev - `kb ui` → Streamlit 4 pages (import / retrieval / document browsing / entity graph) - `kb query` → CLI rapid testing - **Robustness verified by blood and tears**: Fixed embedding worker event loop closed (async/threading mixed use classic pit), Streamlit asyncio.Lock cross-event loop failure, LLM no timeout card death, jieba first load stuck, BM25 noise injection, etc. **Each fix touches GIL, event loop binding, cache consistency, and other computer underlying principles**. ## Quick start (5 minutes to run through) ```bash # 1. Clone + environment (recommended uv, never pip) git clone https://github.com/q1ngn1ng-web/kb-rag.git cd kb-rag uv python pin 3.12 uv sync source .venv/bin/activate # 2. Configuration (recommended environment variables, never hardcoding) cp config.yaml.example config.yaml # Optional, prioritize env export DEEPSEEK_API_KEY="sk-..." export DASHSCOPE_API_KEY="sk-..." # text-embedding-v4 export MINERU_API_KEY="..." # If you need to process papers/scan pieces # 3. Verify LLM + Embedding connectivity uv run python -m pytest tests/scripts/verify_llm.py -x -q # 4. Import documents (automatic classification + convert to Markdown) kb import ./my_papers/ # Only conversion kb import --index ./my_papers/ # Conversion + immediate indexing kb import --vault ./my_vault ./my_books/ # Conversion + indexing + Obsidian entity note synchronization # 5. Query kb query "What are the six kinds of reasons Tang Junyi divides them into? What are the characteristics?" kb serve # Start MCP Server, port 8000 kb ui # Start Streamlit local interface http://localhost:8501 ``` **Recommended workflow**: First `kb import --index` → `kb serve` lets Claude directly use your knowledge base → `kb ui` visual debugging retrieval path. ## Architecture (three-layer pipeline) ```mermaid graph TB subgraph Import["Layer 1: Import (kb import)"] CLASSIFIER["PDF Classifier<br/>Formula density/citation/keywords"] ROUTER["ImportRouter"] PYMUPDF["PyMuPDF (ordinary documents)"] MINERU["MinerU API (papers/scan pieces)"] DOCX["DOCX/EPUB/HTML/TXT"] end subgraph Index["Layer 2: Indexing (kb index)"] LIGHTRAG["LightRAG Engine (lazy initialization)"] CHUNK["Chunking strategy (F: fixed / R: recursive)"] LLM["LLM Provider (DeepSeek/Ollama)"] EMBED["Embedding (DashScope v4)"] BM25["BM25 keyword indexing (jieba)"] CACHE["Semantic query cache (0.92 threshold)"] OBSIDIAN["Obsidian Vault synchronization<br/>source_doc isolation"] end subgraph Query["Layer 3: Query (unified backend)"] MCP["FastMCP Server (4 tools)"] STREAMLIT["Streamlit UI"] CLI["kb query CLI"] end CLASSIFIER --> ROUTER --> PYMUPDF & MINERU DOCX & EPUB & HTML & TXT --> MD_WRITER["Markdown + Frontmatter"] MD_WRITER --> LIGHTRAG LIGHTRAG --> CHUNK & LLM & EMBED & BM25 & CACHE LIGHTRAG --> OBSIDIAN MCP & STREAMLIT & CLI --> LIGHTRAG ``` **Three-stage path during query** (`mode=mix`): 1. LLM generates query → vector retrieval entities/relationships 2. jieba word segmentation + BM25 keyword recall 3. RRF fusion re-ranking → LLM generates final answer with context (with `asyncio.wait_for` timeout protection) ## Configuration instructions All sensitive information is preferentially read from environment variables, and `config.yaml` only serves as a default value. Key adjustable parameters (`config.yaml` or env): - `knowledge_graph.chunk_size` / `chunk_overlap`: Directly affect entity extraction granularity and prompt token budget (philosophical text recommends 1500-2500, narrative text can be larger) - `knowledge_graph.cache_threshold`: Default 0.92 (too high hit rate is low, too low misjudgment) - `pdf.routing`: Formula density threshold, citation threshold (can fine-tune classifier) - `llm.timeout`: Single LLM call maximum waiting seconds Complete configuration see `config.yaml.example`. ## Examples and real use cases (reserved, welcome to write) We intentionally leave a lot of blank space here - because **the best examples should come from real user scenarios**. ### Example 1: Philosophical works multi-hop reasoning (strongly recommended) - Data: Tang Junyi's "The Original Theory of Chinese Philosophy" first six chapters - Typical questions: "How does Tang Junyi distinguish these six kinds of reasons?" / "What is the essential difference between empty reason and sexual reason?" - Expected effect: RRF lets the most relevant original block rank first, LLM gives structured answer with precise reference; second synonymous query directly cache hit (<1s) - **To be supplemented**: Complete import log + query log + cache hit screenshot + Obsidian entity note screenshot (welcome PR) **Example 1 Tang Junyi's six kinds of reasons query results**  **Example 1 Tang Junyi's six kinds of reasons semantic cache hit example** Question1: How does Tang Junyi distinguish six kinds of reasons?  Question2: What are the characteristics of the six kinds of reasons proposed by Tang Junyi?  It can be seen that under cache hit, the time consumption is only 0.3 seconds, and these two questions are semantically similar, hit by semantic cache, will not be considered as a new question and then retrieve in the vector library, and the results of the two are the same. **Example 1 Tang Junyi's six kinds of reasons query background log**  ### Example 2: Academic paper formula + table precise QA - Data: Gesture recognition related papers (including YOLO improvement, SPP formula, etc.) - Question: "What are the confidence formula and loss function of HGDR-Net?" - Expected effect: MinerU automatic routing, formula retains LaTeX, table structured, accurate reference formula number - **To be supplemented**: Classifier hit log + formula extraction effect comparison (PyMuPDF vs MinerU) ### Example 3: Novel character relationship graph + Obsidian dual-chain browsing - Data: Any long novel EPUB - Effect: Character, location, event entity automatic extraction, export to Obsidian to see [[leaf Wenjie]] → [[Three-Body]] dual-chain network - **To be supplemented**: Different chunk strategy impact on narrative coherence experiment **Invitation**: If you have public or private test documents (especially with gold standard entity/relationship), please submit to `examples/` or `tests/fixtures/`, and attach reproduction commands. We will put high-quality cases into README. ## Testing, benchmark, and quality gate Current test entry: ```bash pytest tests/ -q uv run python -m pytest tests/scripts/verify_llm.py ``` **Welcome to contribute the following content** (these are the most lacking and valuable directions for the project): - Entity/relationship extraction F1 benchmark (suggest using HotpotQA dev.json or self-annotated philosophical/technical documents, establish gold_standard) - Retrieval quality ablation experiment: pure vector vs BM25 vs RRF recall@K / precision@K / human evaluation of answer quality - Chunking strategy benchmark: different chunk_size / overlap / strategy (F vs R) impact on cache hit rate, embedding cost, and final answer quality - Concurrency security test: multi-thread batch import + concurrent query data consistency, deadlock, event loop pollution - End-to-end performance analysis: each stage time-consuming breakdown (classifier / chunking / LLM entity extraction / embedding / RRF / LLM response) All new tests follow the `openspec/` change management process and attach benchmark data and flame graph (if possible) in PR. ## Roadmap (far beyond current implementation) The project is evolving from "can run" to "extreme engineering". Completed V3 (BM25 only sorting + semantic cache + entity isolation), next plan: - [ ] Document type adaptive chunking (philosophical/technical small chunk to keep entity granularity, narrative large chunk to keep coherence) - [ ] Retrieval path visualization + user explicit feedback closed loop ("this block is useful" → reinforcement ranking model) - [ ] Graph visualization front-end (Streamlit or independent Canvas) - [ ] More importer (PPTX, scan piece enhancement, Markdown enhancement frontmatter processing) - [ ] Local priority support (Ollama + BGE / text2vec, local embedding batch memory optimization) - [ ] Incremental update and knowledge graph versioning **These are not just talk** - many ideas are scattered in `Individual-knowledge-base.md` and `openspec/changes/`, lacking implementers. ## Contribution guide (sincere invitation) This project is **absolutely not something one person can complete**. It involves: - Bottom: memory locality (chunk design), cache consistency (semantic cache + RRF), asynchronous model (event loop + threading mixed use), I/O optimization (lazy initialization + batch embedding) - Middle: LLM prompt engineering (entity extraction quality), retrieval algorithm (RRF weight tuning) - Top: user experience (Streamlit page, Obsidian integration), multi-format parsing robustness **No matter who you are**: - Experienced Python engineer who has stepped into async/embedding worker pitfalls - Algorithm engineer with deep understanding of Chinese LLM + knowledge graph - Daily heavy Obsidian user, want to make knowledge base extreme, power user - Developer who wants to contribute real tools for personal/small team knowledge management **All are welcome**. **How to start contributing**: 1. Read `AGENTS.md`, `onboarding.md` (in the repository root directory) and `Individual-knowledge-base.md` to understand evolution history and pits 2. Run `uv sync && pytest tests/` 3. Choose a direction you feel most comfortable with (example, test, importer optimization, cache strategy tuning, etc.), open issue or direct PR 4. Follow `openspec/` workflow (/opsx:explore → /opsx:propose → /opsx:apply) We do not accept "can run" PR - each change must explain the impact on cache hit rate, token efficiency, concurrency security, entity extraction precision. ## Acknowledgment and history This project started from nano-graphrag experiment, evolved to LightRAG + custom hybrid retrieval pipeline, stepped through event loop closed, cache threshold misjudgment, chunk boundary cut concept, BM25 noise, etc. Thanks to all commits during development. Special thanks to LightRAG, MinerU, PyMuPDF, jieba, rank-bm25 and other excellent open-source projects. ## License MIT License. Commercial use, modification, redistribution are welcome, but please keep the original author signature. --- **Last sentence of tongue-in-cheek advice**: > Don't make another "can run" knowledge base wheel. > What the world lacks is **tools that can do cache locality, asynchronous boundary, retrieval fusion, and entity isolation to the extreme at a personal scale**. > kb-rag is moving in this direction. > **Come on, let's make this wheel into a diamond.** If you have any questions, ideas, or complaints, welcome to open issues or directly @ project maintainers. Let's make personal knowledge bases truly usable, truly professional, and truly open. (This README is a refactored version, reserved a lot of example and test expansion positions. The original README is too brief and can't bear the current technical depth and community invitation intention.)