android-knowledge-base-rag-mcp

# Tool List A local-first knowledge base project for Android learning, interview review, and knowledge maintenance. The project uses Markdown to save knowledge content, provides a searchable documentation website through VitePress, offers semantic search through ChromaDB and embedding models, and connects to AI clients like Codex and Claude Desktop through MCP Server to enable retrieval, creation, update, and deletion of knowledge. ## Key Capabilities - Browse Markdown knowledge base with VitePress, supporting sidebar, full-text search, dark mode, and hot update during development. - Use ChromaDB to save local vector indexes, which can be rebuilt from Markdown at any time. - Generate semantic vectors locally with `Qwen/Qwen3-Embedding-0.6B` by default, without requiring an OpenAI API Key. - Optionally use OpenAI Embeddings or compatible interfaces as cloud embedding services. - Provide MCP tools and resources for AI clients to retrieve and securely edit the knowledge base on demand. - Support full reindexing, directory import, metadata filtering, tag filtering, and cosine similarity search. - Ensure consistency protection for document writing and vector updates; recover original files and indexes if updates fail. ## System Architecture ```mermaid flowchart LR G["GitHub Repository"] -->|"clone / pull"| F["Markdown Knowledge Base"] F --> V["VitePress Static Frontend"] V --> B["Local Browser"] A["Codex / Claude Desktop"] --> M["MCP Server"] M --> S["KnowledgeService"] S --> F S --> E["Qwen3 / OpenAI Embeddings"] E --> C["ChromaDB Local Vector Index"] C --> S ``` The project follows these data principles: - Markdown files in `knowledge-base/` are the source of truth and should be committed to Git. - `.chroma/` contains local derived indexes and should not be committed to Git; it can be regenerated after cloning the repository or pulling updates. - `.model-cache/`, `.venv/`, and `node_modules/` are local dependencies or caches and should not be committed to Git. - `.env` saves local configurations or secrets and should not be committed to Git; the repository only provides `.env.example`. - VitePress page search and RAG vector retrieval are independent capabilities; the browsing frontend does not require generating vectors beforehand. ## Tech Stack | Module | Technology | |---|---| | Documentation & Frontend | Markdown, VitePress, Vue | | MCP Server | Python, MCP Python SDK | | Vector Database | ChromaDB | | Default Embedding Model | `Qwen/Qwen3-Embedding-0.6B` | | Optional Embedding Service | OpenAI Embeddings / OpenAI Compatible Interface | | AI Client | Codex, Claude Desktop, or other MCP clients | ## Environment Requirements - Git - Node.js 18 or higher - npm - Python 3.10 or higher, recommended Python 3.12 - Access to model repository on first use of local embedding model - Local model, Python dependencies, and vector index require several GB of available disk space The project is primarily used and verified on macOS. Linux and Windows can adjust commands according to local Python, PyTorch, and MCP client path rules; `LOCAL_EMBEDDING_DEVICE=auto` will choose available MPS, CUDA, or CPU. ## Quick Start ### 1. Clone Repository ```bash git clone https://github.com/timeisthe/android-knowledge-base-rag-mcp.git cd android-knowledge-base-rag-mcp ``` Subsequent commands assume execution in the repository root directory. ### 2. Start Knowledge Base Website Only If you only need to read and edit Markdown files, without RAG or MCP: ```bash npm ci --cache ./work/npm-cache npm run docs:dev -- --open ``` Development server default address: ```text http://localhost:5173/ ``` After modifying Markdown files in `knowledge-base/`, the page will automatically refresh. Stop the server with `Control + C`. If the default port is occupied: ```bash npm run docs:dev -- --port 5174 --open ``` ### 3. Install RAG and MCP Environment Create a Python virtual environment: ```bash python3.12 -m venv .venv source .venv/bin/activate ``` If the local command name is not `python3.12`, use `python3` that meets the version requirement. For Windows PowerShell: ```powershell py -3.12 -m venv .venv .\.venv\Scripts\Activate.ps1 ``` Install MCP Server, local models, and development dependencies: ```bash python -m pip install --upgrade pip python -m pip install -e './mcp-server[dev,local-models]' ``` Verify CLI installation: ```bash android-kb-mcp --help ``` ### 4. Create Local Configuration macOS or Linux: ```bash cp .env.example .env ``` Windows PowerShell: ```powershell Copy-Item .env.example .env ``` The default template configures local Qwen3 embedding mode; usually, no OpenAI API Key is required. Key configurations: ```dotenv EMBEDDING_PROVIDER=sentence-transformers LOCAL_EMBEDDING_MODEL=Qwen/Qwen3-Embedding-0.6B LOCAL_EMBEDDING_CACHE_PATH=.model-cache/huggingface LOCAL_EMBEDDING_DEVICE=auto LOCAL_EMBEDDING_BATCH_SIZE=8 HF_HUB_OFFLINE=0 TRANSFORMERS_OFFLINE=0 KNOWLEDGE_BASE_PATH=knowledge-base CHROMA_PERSIST_PATH=.chroma CHROMA_COLLECTION=android_knowledge_qwen3_06b MCP_TRANSPORT=stdio ``` `.env` is excluded by `.gitignore`. Do not commit API keys, internal addresses, or sensitive configurations to the repository. ### 5. Establish Vector Index ```bash android-kb-mcp reindex ``` On the first execution, it will download `Qwen/Qwen3-Embedding-0.6B`. After downloading, the command will: 1. Scan knowledge documents under `knowledge-base/`; 2. Parse Markdown frontmatter, titles, tags, and body; 3. Generate document vectors; 4. Write vectors and metadata to `.chroma/`; 5. Remove outdated vectors without corresponding Markdown files. Check document and index counts: ```bash android-kb-mcp metadata ``` Perform a semantic search: ```bash android-kb-mcp search "Activity configuration change" --top-k 3 ``` If the local model is successfully downloaded and you want to load it completely offline, modify `.env` to: ```dotenv HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 ``` ## Pull Updates and Re-vectorize GitHub only synchronizes Markdown and program source code, not `.chroma/`. After pulling knowledge base updates from the remote repository, it is recommended to execute: ```bash git pull --rebase source .venv/bin/activate android-kb-mcp reindex android-kb-mcp metadata ``` For Windows PowerShell, replace the activation command with: ```powershell .\.venv\Scripts\Activate.ps1 ``` `reindex` is the most reliable synchronization method. It recalculates vectors for all current knowledge documents and cleans up outdated vectors left by deleted or renamed documents. ## Local Frontend and Static Build Start development server: ```bash npm run docs:dev -- --open ``` Generate production version: ```bash npm run docs:build ``` Build output located at: ```text knowledge-base/.vitepress/dist/ ``` Preview production build: ```bash npm run docs:preview -- --open ``` The frontend directly reads Markdown from VitePress. Even without installing Python, downloading embedding models, or generating `.chroma/`, you can still browse the website and use VitePress's local full-text search. ### Deploy to GitHub Pages The current repository provides VitePress static build capability but does not preset GitHub Pages workflow. Deploying to GitHub Pages requires additional configuration: - GitHub Actions build and release process; - VitePress `base` path used by the project site; - Pages release source in Repository Settings. GitHub Pages can only host static frontends and cannot run the project's Python MCP Server, embedding models, or ChromaDB. RAG and MCP still run on the user's local machine by default. ## Connect to MCP Client MCP clients usually automatically start the Server through `stdio`. Normal usage does not require running a separate long-term service. ### 1. Get Repository Absolute Path macOS or Linux: ```bash pwd ``` Windows PowerShell: ```powershell (Get-Location).Path ``` ### 2. Use Configuration Template in Repository - Codex: [mcp-server/examples/codex-config.toml](./mcp-server/examples/codex-config.toml) - Claude Desktop: [mcp-server/examples/claude_desktop_config.json](./mcp-server/examples/claude_desktop_config.json) Replace `/ABSOLUTE/PATH/TO/REPOSITORY` in the template with the actual absolute path of the repository, then merge it into the corresponding client configuration. Key requirements in the configuration: - `command` must point to the Python in the current repository's virtual environment; - `cwd` must be the repository root directory; - `KNOWLEDGE_BASE_PATH`, `CHROMA_PERSIST_PATH`, and model cache path recommend using absolute paths; - The model and Collection used by the MCP client should be consistent with `.env` when creating the index; - Save the configuration and completely exit and restart the MCP client. After configuration, let the client call `search_knowledge` to verify the connection. Manually diagnose the Server: ```bash android-kb-mcp serve ``` After the `stdio` Server starts, it will wait for client messages, and the terminal does not have continuous output, which is a normal phenomenon. Press `Control + C` to stop. When normally started by the MCP client, do not run another manual Server simultaneously. ## MCP Capability ### Tools | Tool | Effect | |---|---| | `search_knowledge` | Semantic search and return relevant document summaries | | `get_document` | Read a single complete Markdown | | `create_document` | Create a document and automatically generate a vector | | `update_document` | Update a document and automatically rebuild the vector | | `delete_document` | Delete a document and the corresponding vector | | `append_to_section` | Append content to a specified Markdown section | | `list_documents` | List document metadata by category, difficulty, title, or tag | | `bulk_ingest` | Batch import a specified directory | | `reindex_all` | Fully calibrate Markdown and vector index | ### Resources | URI | Content | |---|---| | `knowledge://metadata` | Document count, index count, category, and difficulty statistics | | `knowledge://tags` | All tags in the knowledge base | | `knowledge://documents/{category}` | Document metadata under a specified category | | `knowledge://document/{file_path}` | Complete content of a specified document | See [USAGE.md](./USAGE.md) for complete parameters, filters, and AI editing examples. ## CLI Commands | Command | Effect | |---|---| | `android-kb-mcp serve` | Start MCP Server | | `android-kb-mcp ingest [directory]` | Vectorize a specified directory | | `android-kb-mcp reindex` | Fully synchronize Markdown and ChromaDB | | `android-kb-mcp metadata` | Output document and index statistics | | `android-kb-mcp search QUERY` | Perform semantic search | If the virtual environment is not activated, macOS or Linux can directly use: ```bash .venv/bin/android-kb-mcp --help ``` ## Embedding Model Configuration ### Default: Qwen3 Local Model ```dotenv EMBEDDING_PROVIDER=sentence-transformers LOCAL_EMBEDDING_MODEL=Qwen/Qwen3-Embedding-0.6B LOCAL_EMBEDDING_DEVICE=auto LOCAL_EMBEDDING_BATCH_SIZE=8 CHROMA_COLLECTION=android_knowledge_qwen3_06b ``` If memory is insufficient, reduce the batch size or force using CPU: ```dotenv LOCAL_EMBEDDING_BATCH_SIZE=2 LOCAL_EMBEDDING_DEVICE=cpu ``` ### Optional: OpenAI Embeddings ```dotenv EMBEDDING_PROVIDER=openai OPENAI_API_KEY=<YOUR_OPENAI_API_KEY> OPENAI_EMBEDDING_MODEL=text-embedding-3-small CHROMA_COLLECTION=android_knowledge_openai_small ``` After modifying the provider, model, or vector dimension, a new Chroma Collection or cleanup of old indexes is required, then execute: ```bash android-kb-mcp reindex ``` Query vectors and document vectors must come from the same model and have the same dimension. Vectors generated by different models cannot be placed in the same Collection. ## Knowledge Document Convention Knowledge documents are located in `knowledge-base/`, using Markdown and YAML frontmatter: ```markdown --- title: Activity Lifecycle tags: [Android, Framework, High Frequency] difficulty: medium category: android-framework subcategory: four-components --- # Activity Lifecycle ## Overview Body content. ## Reference - [Android Developers](https://developer.android.com/) ``` The index text consists of the title, tags, and body. Files in `index.md` and `.vitepress/` are not written to ChromaDB as RAG knowledge documents. ## Project Structure ```text . ├── knowledge-base/ │ ├── .vitepress/ # VitePress configuration, theme, and sidebar │ ├── index.md # Knowledge base homepage │ └── **/*.md # Knowledge documents ├── mcp-server/ │ ├── src/android_kb_mcp/ # Repository, RAG, ChromaDB, and MCP │ ├── tests/ # Python tests │ └── examples/ # MCP client configuration templates ├── .env.example # Environment variable template ├── package.json # VitePress scripts and dependencies ├── USAGE.md # MCP usage guide ├── COSTS.md # Embedding cost and scale suggestions └── README.md # Project entry document ``` ## Development and Verification Run Python tests after installing development dependencies: ```bash python -m pytest mcp-server/tests ``` Build the frontend: ```bash npm run docs:build ``` Complete local acceptance process: ```bash npm run docs:build python -m pytest mcp-server/tests android-kb-mcp reindex android-kb-mcp metadata android-kb-mcp search "Activity Lifecycle" --top-k 3 ``` Test coverage includes document path security, CRUD, filtering, chapter appending, failure rollback, stale vector cleanup, ChromaDB read/write, embedding parameters, and MCP registration contracts. ## Frequently Asked Questions ### `android-kb-mcp: command not found` Confirm that the virtual environment is activated: ```bash source .venv/bin/activate android-kb-mcp --help ``` Alternatively, use `.venv/bin/android-kb-mcp` directly. ### Missing `sentence-transformers` or `torch` ```bash source .venv/bin/activate python -m pip install -e './mcp-server[dev,local-models]' ``` ### Qwen3 out of memory on MPS Reduce the batch size in `.env`: ```dotenv LOCAL_EMBEDDING_BATCH_SIZE=2 ``` Switch to CPU if still failing: ```dotenv LOCAL_EMBEDDING_DEVICE=cpu ``` ### Vector dimension mismatch after model switching Recommend setting a new `CHROMA_COLLECTION` for the new model. Alternatively, backup the local index and rebuild: ```bash mv .chroma ".chroma.backup.$(date +%Y%m%d-%H%M%S)" android-kb-mcp reindex ``` `.chroma` is only a derived index and does not affect the Markdown source. ### MCP client cannot discover Server Check in sequence: 1. If `command` points to the current repository's virtual environment Python; 2. If `cwd` and data paths are real absolute paths; 3. If TOML or JSON configuration syntax is valid; 4. If MCP client configuration uses the same model and Collection as `.env`; 5. If the client is fully restarted after saving the configuration; 6. If running `android-kb-mcp serve` immediately reports an error. ## Security and Version Control Run before committing: ```bash git status ``` Confirm that the following are not included in the commit: ```text .env .chroma/ .model-cache/ .venv/ node_modules/ knowledge-base/.vitepress/cache/ knowledge-base/.vitepress/dist/ ``` If a key has been submitted, even if the file is later deleted, it should be immediately revoked and re-generated on the service provider; simply creating a delete commit cannot remove the old key from Git history. Also, check for personal information, internal data, and third-party content reproduction and authorization in knowledge documents before making the repository public. ## Other Documents - [USAGE.md](./USAGE.md): MCP tools, resources, filters, and AI editing examples. - [COSTS.md](./COSTS.md): Local model, OpenAI embedding costs, and scale growth suggestions. - [mcp-server/README.md](./mcp-server/README.md): Python MCP Server independent development guide. ## License This project uses a dual licensing model, with license scope distinguished by file type: - Program source code, frontend implementation, MCP Server, and project configuration are under the [MIT License](./LICENSE). - Original articles, descriptions, charts, and images in `knowledge-base/` are under [CC BY-SA 4.0](./knowledge-base/LICENSE.md). Third-party references, external links, trademarks, and content with explicitly marked other rights are not re-licensed under this project's license and remain under their original terms.