zoho-docs-index

# Local docs RAG (`zoho-docs-index`) Fully **offline retrieval-augmented generation** over a crawled technical corpus: crawl → normalize → chunk → embed → vector store → grounded answers with **URL citations**. Built as a **DE / applied AI** project: ingestion pipeline, pluggable embeddings + stores, local inference, and an MCP/API consumption layer. The current corpus is official Zoho documentation (convenient for Kinetic Bridge work); the architecture is corpus-agnostic. **No cloud LLMs.** Generation goes through LM Studio’s local OpenAI-compatible server. Primary UX: the `zoho-docs` MCP tools (`search` / `answer`). **Repo:** https://github.com/blakeallard/zoho-docs-index ## What I built - **Docs crawler** with Playwright for JS-rendered product pages - **Chunking + embedding pipeline** (`build_index.py`) with incremental modes - **Pluggable backends** — LM Studio or sentence-transformers embeddings; Chroma or numpy vector store - **Grounded answer API** (FastAPI) returning answers + citations + used context - **MCP server** for Claude Code / Neovim MCPHub over the local API ## Architecture ```mermaid flowchart LR A[Crawl corpus] --> B[Normalize + chunk] B --> C[Embed] C --> D[Vector store] E[Question] --> F[Embed query] F --> D D --> G[Retrieve chunks] G --> H[Grounded prompt] H --> I[Local LM Studio] I --> J[Answer + citations] ``` ``` question ──▶ FastAPI /zoho-docs/answer │ 1. embed query │ 2. vector search (Chroma | numpy) ◀── built by build_index.py │ 3. build grounded prompt w/ chunk URLs └─ 4. LM Studio chat ──▶ answer + citations ``` ## How to evaluate (60 seconds) 1. This README + diagram 2. `build_index.py` / `zoho_docs_index/` package layout 3. `mcp_server.py` + FastAPI answer path 4. `.env.example` for backend knobs Related: [`kinetic-bridge-ops`](https://github.com/blakeallard/kinetic-bridge-ops), [`kinetic-bridge-email-intel`](https://github.com/blakeallard/kinetic-bridge-email-intel). --- ## 1. Prerequisites - **Python 3.10+**. The venv here is built with **3.13** — the Homebrew 3.14.6 on this machine has a broken `pyexpat` (libexpat symbol mismatch) that breaks pip and XML parsing, so we use 3.13.5 instead. Recreate with: `python3.13 -m venv .venv`. - **LM Studio** running its local server at `http://127.0.0.1:1234/v1` with: - a chat model loaded: **`qwen/qwen3-14b`** (or another instruct model you prefer — set `LMSTUDIO_MODEL` in `.env` to the exact LM Studio id) - an **embedding model loaded** (if using the default `lmstudio` embedding backend), e.g. `text-embedding-nomic-embed-text-v1.5`. In LM Studio: *Developer → load an embedding model*; it serves on the same `/v1/embeddings` endpoint. - **Playwright Chromium** (only for JS-rendered Zoho docs — analytics, writer, sign, meeting, campaigns, vault, …): `pip install playwright && playwright install chromium` ### Install ```bash cd ~/bevco/automations/zoho-docs-index python3.13 -m venv .venv && source .venv/bin/activate pip install -r requirements.txt playwright install chromium # if you will crawl js:true sources cp .env.example .env # then edit if needed ``` > **Embedding model name:** set `LMSTUDIO_EMBEDDING_MODEL` in `.env` to match the > exact id LM Studio shows for your loaded embedding model. ### Backend choices (important on newer Python) This project keeps the two heavy pieces **pluggable** so it runs anywhere: | Concern | Default (`.env`) | Fallback / alternative | |------------|-----------------------------|-------------------------------------------------| | Embeddings | `EMBEDDING_BACKEND=lmstudio` (no torch) | `sentence-transformers` (all-MiniLM-L6-v2, needs torch) | | Vectors | `VECTOR_STORE=chroma` | `VECTOR_STORE=numpy` (SQLite + numpy, zero heavy deps) | If `chromadb` won't install on your Python version, set `VECTOR_STORE=numpy` — brute-force cosine search is plenty fast for a docs-sized corpus and needs only numpy. --- ## 2. Build the index Crawl roots come from (first match wins): `ZOHO_SOURCES_FILE` env, `sources.txt` / `sources.json`, then **`zoho_sources.json`** at the repo root (includes `"js": true` flags for Playwright products), else built-in defaults. ```bash # Full pipeline: crawl Zoho docs → normalize → chunk → embed → store python3.13 build_index.py ``` Other modes: ```bash python3.13 build_index.py --incremental # add only new URLs / new chunks python3.13 build_index.py --crawl-only # crawl + normalize only python3.13 build_index.py --no-crawl # re-embed from existing normalized.jsonl # Tools catalog (committed copy — app,tool_name,purpose) python3.13 build_index.py --tools-csv catalog/zoho_mcp_tools.csv # Local narrative text (e.g. MCP tool writeups); product=zoho-mcp, no URL python3.13 build_index.py --local-text /path/to/zoho_mcp_documentation.txt ``` ### Playwright / JS docs Sources with `"js": true` in `zoho_sources.json` are fetched with headless Chromium so SPA doc sites yield full text (not empty shells). The corpus exporter can resume and force-refresh those products: ```bash python3.13 crawl_zoho_docs.py \ --sources zoho_sources.json \ --resume \ --max-pages-per-root 500 \ --output-csv data/zoho_docs_all.csv \ --output-jsonl data/zoho_docs_all.jsonl ``` `--resume` skips already-crawled **static** URLs but **re-crawls JS products** (strips their prior rows first) so you can fix excerpt-only exports. The crawler is **polite**: respects `robots.txt`, stays under each configured root, throttles requests (`CRAWL_DELAY`, default 1s/host), and caps pages per root (`MAX_PAGES_PER_ROOT`). Outputs land in `data/` (gitignored): - `data/normalized.jsonl` — one JSON record per page (`id,url,product,title,clean_text`). - `data/chroma/` or `data/vectors.sqlite3` — the vector store. - `catalog/zoho_mcp_tools.csv` — tools catalog checked into git for reproducibility. --- ## 3. Start the API ```bash uvicorn zoho_docs_index.api:app --host 127.0.0.1 --port 8008 ``` Sanity check it's alive and how many chunks are indexed: ```bash curl -s http://127.0.0.1:8008/health | python3 -m json.tool ``` --- ## 4. Quick sanity checks **Search** (raw chunks + URLs): ```bash curl -s -X POST http://127.0.0.1:8008/zoho-docs/search \ -H 'Content-Type: application/json' \ -d '{"query": "How do I insert records via the Zoho CRM API?", "top_k": 5}' \ | python3 -m json.tool ``` **Answer** (grounded generation via LM Studio + citations): ```bash curl -s -X POST http://127.0.0.1:8008/zoho-docs/answer \ -H 'Content-Type: application/json' \ -d '{"question": "What OAuth scope is needed to create a contact in Zoho CRM?", "top_k": 8}' \ | python3 -m json.tool ``` Response shape: ```json { "answer": "…grounded answer with [1][2] citations…", "citations": ["https://www.zoho.com/crm/developer/docs/api/v8/...", "..."], "used_context": [{"text": "…", "url": "…", "product": "crm"}] } ``` --- ## 5. Use it from Claude Code (MCP) — primary The MCP server (`mcp_server.py`) exposes two tools over stdio: `zoho_docs_search` and `zoho_docs_answer`. It calls the FastAPI service, so **keep the API on :8008 running** (launchd agent `com.bevco.zoho-docs-api` does this automatically when LM Studio is up). Already registered in `~/.claude/mcp.json` as `zoho-docs`. To re-add: ```json { "mcpServers": { "zoho-docs": { "command": "/Users/blakeallard/bevco/automations/zoho-docs-index/.venv/bin/python", "args": ["/Users/blakeallard/bevco/automations/zoho-docs-index/mcp_server.py"], "env": { "ZOHO_API_URL": "http://127.0.0.1:8008" } } } } ``` In a Claude Code session, call `zoho_docs_search` / `zoho_docs_answer` for any Zoho product or API question. ### Optional — Neovim (CodeCompanion + MCPHub) Same MCP server works with MCPHub. In `~/.config/mcphub/servers.json`: ```jsonc { "mcpServers": { "zoho-docs": { "command": "/Users/blakeallard/bevco/automations/zoho-docs-index/.venv/bin/python", "args": ["/Users/blakeallard/bevco/automations/zoho-docs-index/mcp_server.py"], "env": { "ZOHO_API_URL": "http://127.0.0.1:8008" } } } } ``` > Use the venv's Python so the `mcp` + `requests` deps are importable. After > editing, reload MCPHub (`:MCPHub` → restart the server, or restart Neovim). CodeCompanion already consumes MCPHub tools. Suggested system prompt nudge: ```lua -- Whenever a question concerns Zoho products or APIs, call zoho_docs_search -- or zoho_docs_answer and answer using only the returned documentation. ``` ### Optional — direct HTTP tool from CodeCompanion If you'd rather skip MCP, define a CodeCompanion tool that POSTs to the API. Example custom tool (adapt to your CodeCompanion version's tool API): ```lua -- A CodeCompanion tool that calls the local Zoho docs service over HTTP. local zoho_docs_tool = { name = "zoho_docs_search", description = "Search official Zoho documentation and return content + URLs.", cmds = { function(_, args) local body = vim.fn.json_encode({ query = args.query, top_k = args.top_k or 5 }) local out = vim.fn.system({ "curl", "-s", "-X", "POST", "http://127.0.0.1:8008/zoho-docs/search", "-H", "Content-Type: application/json", "-d", body, }) return { status = "success", data = out } end, }, schema = { type = "function", ["function"] = { name = "zoho_docs_search", parameters = { type = "object", properties = { query = { type = "string" }, top_k = { type = "integer", default = 5 }, }, required = { "query" }, }, }, }, } -- Register zoho_docs_tool with your CodeCompanion `strategies.chat.tools` table. ``` Suggested system prompt: *"Use the zoho_docs_search tool whenever a question concerns Zoho products. Answer using only the returned docs and always include URL citations."* --- ## 6. Project layout ``` zoho-docs-index/ ├── build_index.py # CLI: crawl → chunk → embed → store ├── mcp_server.py # MCP stdio server (tools for MCPHub/CodeCompanion) ├── requirements.txt ├── .env.example ├── sources.example.json # optional crawl-scope override ├── zoho_docs_index/ │ ├── settings.py # all config (env-overridable) + crawl sources │ ├── crawler.py # polite scoped crawler + HTML→clean text │ ├── chunking.py # token-aware overlapping chunker │ ├── embeddings.py # LM Studio | sentence-transformers (pluggable) │ ├── store.py # ChromaStore | NumpyStore (pluggable) │ ├── indexer.py # pipeline orchestration + shared search() │ ├── llm.py # LM Studio client + grounded prompt │ └── api.py # FastAPI app (search/answer/health) └── data/ # crawled docs + vectors (gitignored) ``` --- ## 7. Maintenance & re-indexing - **Rebuild from scratch:** `python build_index.py` (wipes + re-embeds). - **Add new pages only:** `python build_index.py --incremental`. - **Re-tune chunking** without re-crawling: edit `CHUNK_*` in `.env`, then `python build_index.py --no-crawl`. - **Narrow/expand scope:** edit `DEFAULT_SOURCES` in `settings.py` or set `ZOHO_SOURCES_FILE` to a JSON list of `{"product","root"}` objects. ### Design decisions (the "why") - **Embeddings default to LM Studio** so the whole stack stays in one local server with no torch dependency; `sentence-transformers` remains a one-flag alternative. - **Chunk size ~750 tokens, 150 overlap** — API docs are dense and cross- referencing; this keeps passages focused while preserving context across boundaries. Token counts use a cheap ~4-chars/token estimator (we're sizing passages, not billing tokens). - **Cosine similarity** throughout; embeddings are L2-normalized in the numpy store and Chroma uses `hnsw:space=cosine`. - **Retrieval lives in one place** (`indexer.search`) shared by the API and MCP server, so behavior never diverges. ``` --- ## 8. Fine-tune Qwen2.5-Coder-14B on Zoho docs (fully local, no API limits) Instead of querying at inference time, bake the Zoho knowledge directly into the model weights via **LoRA fine-tuning** on Apple Silicon with [mlx-lm](https://github.com/ml-explore/mlx-examples/tree/main/llms/mlx_lm). The result loads back into LM Studio as a drop-in GGUF replacement. ``` normalized.jsonl ──▶ generate_dataset.py ──▶ pairs_raw.jsonl │ ▼ prepare_splits.py train.jsonl + valid.jsonl │ ▼ run_finetune.sh (mlx-lm LoRA) adapters/ │ ▼ merge_export.sh (llama.cpp GGUF) qwen2.5-coder-14b-zoho-Q4_K_M.gguf (load in LM Studio → done) ``` ### Step 1 — Generate Q&A training pairs (~2-4h, unattended) ```bash # LM Studio must be running with qwen2.5-coder-14b-instruct loaded cd ~/bevco/automations/zoho-docs-index && source .venv/bin/activate python3 finetune/generate_dataset.py # Safe to interrupt and resume later: python3 finetune/generate_dataset.py --resume ``` Produces `finetune/data/pairs_raw.jsonl` — MLX-LM instruct-format examples like `{"messages": [{"role":"system","content":"..."}, {"role":"user",...}, {"role":"assistant",...}]}`. ### Step 2 — Split into train / valid ```bash python3 finetune/prepare_splits.py ``` ### Step 3 — Download base model (HuggingFace safetensors, not GGUF) ```bash pip install huggingface_hub[cli] huggingface-cli download Qwen/Qwen2.5-Coder-14B-Instruct \ --local-dir ~/models/Qwen2.5-Coder-14B-Instruct ``` ### Step 4 — LoRA fine-tune on Apple Silicon (~4-6h on M1 Pro 32GB) ```bash pip install mlx-lm bash finetune/run_finetune.sh ``` Saves adapter checkpoints to `finetune/adapters/` every 200 steps. ### Step 5 — Merge adapters + export GGUF for LM Studio ```bash bash finetune/merge_export.sh # Output: ~/models/qwen2.5-coder-14b-zoho-Q4_K_M.gguf ``` Open LM Studio → My Models → Load model file → select the `.gguf` above. ### Why this over RAG? | | RAG (this project's API) | Fine-tuned model | |---|---|---| | Answers from docs | Yes (retrieval-grounded) | Yes (baked into weights) | | Needs LM Studio running | Yes | Yes | | Needs API at :8008 | Yes | No | | OpenAI / Claude API limits | No (local) | No (local) | | Zoho knowledge | Retrieval window | All of it, always | | Model size | Base Qwen 14B | Same (LoRA = small delta) | Both approaches are complementary — the fine-tuned model knows more about Zoho, and the RAG layer can still retrieve specific page text when you need exact citations. --- ## 9. Corpus export (all Zoho docs → CSV/JSONL) Separate from the RAG index, you can crawl **all** Zoho product docs (not just MCP) into a flat, RAG-friendly corpus — one row per page — for embedding/indexing with any external tool. This reuses the same polite crawler + extractor; it just shapes the output differently. **Output columns** (CSV header / JSONL keys): `product, category, url, title, section, raw_html_length, text_length, excerpt` where `category ∈ {api_reference, guides, mcp_docs, other}` and `excerpt` is the first ~800 chars of clean text collapsed to a single line. **Seeds:** copy `zoho_sources.example.json` to `zoho_sources.json` and edit. It's a plain `[{"product","root"}]` list covering the API-/MCP-enabled apps (CRM, Books, Inventory, Desk, Analytics, Billing, Bigin, WorkDrive, Mail, Cliq, …). Add or remove products there — no code changes needed. ```bash # Small smoke test: CRM only, 100 pages, CSV out python crawl_zoho_docs.py \ --sources zoho_sources.json \ --max-pages-per-root 100 \ --output-csv data/zoho_docs_all.csv # Full run: CSV + JSONL python crawl_zoho_docs.py \ --sources zoho_sources.json \ --max-pages-per-root 500 \ --request-delay 0.5 \ --output-csv data/zoho_docs_all.csv \ --output-jsonl data/zoho_docs_all.jsonl ``` Flags override env vars (`ZDOCS_SOURCES_FILE`, `ZDOCS_MAX_PAGES_PER_ROOT`, `ZDOCS_REQUEST_DELAY_SEC`). The crawler stays on-host and under each root's path, honors `robots.txt`, throttles per host, and skips non-HTML (images/CSS/JS/PDFs). > **Note on coverage:** some Zoho doc sites (e.g. People, Projects, Recruit, > SalesIQ) reject automated User-Agents (HTTP 403) or are JS single-page apps > (e.g. Desk) that a static crawler can't fully read. Those roots will yield few > or no rows; the crawler skips them gracefully rather than failing.