vault-search

# Obsidian Vault Search — a local-first semantic search trio > **English** · [繁體中文](./README.zh-TW.md) Three tools that turn your Obsidian vault into a **semantically searchable, AI-chattable knowledge base** — running entirely on your own machine. No cloud indexing, no notes leaving your computer for search, no per-query API bill for retrieval. Built by a physician to study from a vault of thousands of clinical notes, but **domain-agnostic**: point it at any vault of markdown notes (research, law, engineering, a personal wiki) and it works.  ### Two ways to use it (one shared index) This repo ships **two front-ends over the same local LanceDB index + retrieval core** — use either or both: - **`vault-search-obsidian`** — the Obsidian plugin (Search / Related Notes / Chat in the sidebar), talking to a small local API server. → [Quickstart](#vault-search-obsidian--the-obsidian-plugin) - **`vault-search-mcp`** — an MCP server exposing the same search as tools (`vault_search`, `vault_similar`, `vault_stats`, `textbook_search`) to Claude Code or any MCP client. No Obsidian required. → [Quickstart](#vault-search-mcp--use-it-from-claude-code-or-any-mcp-client) Both reuse the same `server/` core (indexer · scoring · Personalized-PageRank · knowledge graph), so you index once and query from wherever you like. --- ## The three-piece set (三套組) | Tool | What it does | Backed by | |---|---|---| | 🔍 **Vault Search** | Type a question in natural language (any language) and get the most *semantically* relevant note sections — not keyword matches. | LanceDB vector search + local embeddings | | 🔗 **Related Notes** | A live sidebar that, as you read or select text, surfaces related notes you forgot you wrote — surfacing hidden links across your vault. | Same index, "find similar" mode | | 💬 **Vault Chat** | Chat with your notes (RAG). Three modes: **Vault** (notes only, 100% local & free), **Hybrid** (notes + AI knowledge), **Free** (open chat + literature search). | Local LLM *or* your Claude subscription | All three live in the Obsidian sidebar. The same index also exposes an **MCP server**, so coding agents like Claude Code can search your vault as a tool. ### Screenshots | 🔍 Vault Search | 🔗 Related Notes | |---|---| |  |  | | 💬 Vault Chat | 👁 Context preview | |---|---| |  |  | --- ## Why it exists Built-in Obsidian search is lexical — it finds the words you typed. But you rarely remember the exact words in a note you wrote months ago. Semantic search finds notes by *meaning*: search `"how to manage a flaccid bladder after spinal injury"` and it finds your note titled `Neurogenic Bladder` even if those exact words never appear. The design priority is **local-first and private**: - **Embeddings run locally** via [Ollama](https://ollama.com) (`bge-m3`, bilingual, CPU-friendly). Your note text is never sent anywhere to be indexed or searched. - **Vault-mode chat is fully local and free** — a local LLM answers using retrieved notes, zero tokens billed. - **Hybrid/Free chat optionally** shells out to the `claude` CLI so you can use a Claude subscription you already pay for, instead of wiring up (and paying per-token for) a raw API key. --- ## How it works ``` ┌─────────────────────────────────────────┐ Obsidian ◄────► │ Plugin (sidebar): Search · Related · Chat│ └───────────────────┬─────────────────────┘ │ HTTP (localhost:3789) ▼ ┌─────────────────────────────────────────┐ │ api_server.py (FastAPI) │ │ /search /similar /chat /reindex │ └──────┬───────────────────────┬───────────┘ │ │ embed query RAG context ▼ ▼ ┌────────────────────────┐ ┌──────────────────────────┐ │ Ollama (bge-m3) │ │ Vault mode → local LLM │ │ → vector │ │ Hybrid/Free → `claude` CLI │ └───────────┬────────────┘ └──────────────────────────┘ ▼ ┌────────────────────────┐ │ LanceDB vector index │ ◄── indexer.py (chunks notes by heading, │ (~/.vault-search) │ embeds, stores; incremental updates) └────────────────────────┘ Same index + scoring is also served to coding agents via mcp_server.py (MCP). ``` - **`indexer.py`** walks your vault, splits each note into chunks by heading, embeds them with `bge-m3`, and stores them in a local **LanceDB** table. Re-runs are incremental (content-hash cache). - **`scoring.py`** reranks raw vector hits with optional **path weighting** (trust some folders more), **recency** (newer notes rank higher), and **relation** boosts. - **`ppr.py`** ranks notes by **Personalized PageRank** over the `[[wiki-link]]` graph — a HippoRAG / LinearRAG-style random-walk-with-restart. It powers **Related Notes** and multi-hop **query expansion** (seeding the walk on the top semantic hits), surfacing relevant notes that pure vector similarity misses. On the author's vault this lifted sparse-bridge recall@10 from ~60% to ~84% versus naive 1-hop / shared-link expansion. - **`api_server.py`** serves the plugin: semantic search, find-similar, single-note reindex, and chat (with retrieval-augmented context). - **`mcp_server.py`** exposes the same search to MCP clients (e.g. Claude Code) as `vault_search`, `vault_similar`, `vault_stats`. - **Optional add-ons:** a second long-form **reference corpus** (`textbook_indexer.py`, parent-child chunking) and a **knowledge graph** of `[[wikilinks]]` + entities (`graph_builder.py`). --- ## Prerequisites | Requirement | Notes | |---|---| | **Python 3.10+** | For the server. | | **[Ollama](https://ollama.com)** running locally | Provides embeddings (and, optionally, local chat). `bge-m3` runs fine on **CPU** — a GPU only speeds up bulk indexing. | | **Obsidian** | To use the sidebar plugin. The server can also be used headless via the MCP/HTTP API. | | **Claude CLI** *(optional)* | Only for Hybrid/Free chat. Vault-mode chat uses a local model instead. | | **A machine that stays on** *(optional)* | If you want search/chat available to your phone or other devices, run the server on an always-on PC/NAS. For single-machine use, just start it when you open Obsidian. | Pull the embedding model once: ```bash ollama pull bge-m3 # optional, for zero-cost local Vault-mode chat: ollama pull gemma2:9b ``` --- ## Installation ### Step 0 — shared core (do this once, for either front-end) ```bash git clone https://github.com/drpwchen/vault-search.git cd vault-search pip install -r requirements.txt cp env.example .env # then edit .env — set VAULT_PATH to your vault cd server python indexer.py # build the index (incremental on later runs) ``` `ollama pull bge-m3` first if you haven't (see [Prerequisites](#prerequisites)). Now pick one or both front-ends below. ### `vault-search-obsidian` — the Obsidian plugin Start the local API server, then install the plugin: ```bash python server/api_server.py # serves http://localhost:3789 cp -r plugin "<YOUR_VAULT>/.obsidian/plugins/vault-search-plugin" ``` In Obsidian: **Settings → Community plugins → enable "Vault Semantic Search"**. Open its settings and confirm **API Server URL** is `http://localhost:3789`. Three ribbon icons appear: 🔍 Search, 💬 Chat, 🔗 Related Notes. > The plugin ships a `data.json.example`. Obsidian writes its real `data.json` on first run; never commit a `data.json` that contains an API key. ### `vault-search-mcp` — use it from Claude Code (or any MCP client) No Obsidian, no API server — just register the MCP server against the index you built in Step 0: ```bash claude mcp add vault-search -- python "/abs/path/to/server/mcp_server.py" ``` Now Claude Code (or any MCP client) can call `vault_search`, `vault_similar`, `vault_stats`, and — if you indexed a reference corpus — `textbook_search`, all backed by the same local index, Personalized-PageRank expansion, and reranking. --- ## Configuration Everything is driven by environment variables (see **`env.example`** for the full annotated list). The only required one is `VAULT_PATH`. Highlights: | Variable | Default | Purpose | |---|---|---| | `VAULT_PATH` | — *(required)* | Absolute path to your vault. | | `VAULT_SEARCH_DATA_DIR` | `~/.vault-search` | Where indexes/caches/logs live. | | `OLLAMA_HOST` | `http://localhost:11434` | Ollama endpoint. | | `VAULT_SEARCH_EMBEDDING_MODEL` | `bge-m3` | Vault embedding model. | | `VAULT_SEARCH_API_PORT` | `3789` | API server port. | | `CLAUDE_CMD` | `claude` | CLI used for Hybrid/Free chat. | | `VAULT_SEARCH_CHAT_MODEL` | `gemma2:9b` | Local model for Vault-mode chat. | | `VAULT_SEARCH_PERSONA` / `_LANGUAGE` | generic | Customize the assistant's role & reply language. | | `VAULT_SEARCH_PATH_WEIGHTS` | `{}` | Boost/demote folders in ranking. | | `VAULT_SEARCH_EXCLUDE_PATTERNS` | *(none)* | Hide derivative notes (drafts, scratch) from search by default. | --- ## Optional add-ons <details> <summary><b>Secondary reference corpus (textbooks / manuals)</b></summary> If you keep a large corpus of long-form reference docs (converted PDFs, manuals) separate from your notes, index it with parent-child chunking for precise hits *with* surrounding context: ```bash export VAULT_SEARCH_TEXTBOOK_PATH=/path/to/reference-md ollama pull qwen3-embedding:0.6b cd server && python textbook_indexer.py ``` Adds a `textbook_search` tool (MCP + HTTP). Optionally rank trusted sources higher with a boost file — see `examples/source_boost.example.json` and `VAULT_SEARCH_SOURCE_BOOST`. </details> <details> <summary><b>Knowledge graph (wikilinks + entities)</b></summary> ```bash cd server python graph_builder.py # parse [[wikilinks]] into an adjacency graph python graph_builder.py --ner # optional: scispaCy/NER entity extraction ``` When present, the wiki-link graph drives the **Personalized PageRank** retrieval in `ppr.py` (Related Notes + query expansion), and search results are enriched with linked notes and extracted relations. The NER step needs `scispacy` (see `requirements.txt`). PPR itself is pure-Python and needs only the wiki-link graph (Phase A) — no NER required. </details> --- ## Adapting it to your setup (alternatives) This project was built around an **always-on PC with a GPU + a Claude subscription**, but none of that is required. Pick the row that matches you: | You don't have… | Do this instead | |---|---| | **A GPU** | Nothing — `bge-m3` embeds on CPU. Indexing is slower for huge vaults but search is instant. Smaller models (`nomic-embed-text`) are even lighter; set `VAULT_SEARCH_EMBEDDING_MODEL` (and `_DIM`) accordingly and re-index. | | **A Claude subscription** | Use **Vault mode** for chat — it runs entirely on a local Ollama model (`VAULT_SEARCH_CHAT_MODEL`), zero cost. Set `CLAUDE_CMD=` empty to disable Claude-backed modes. | | **An always-on machine** | Just start `api_server.py` when you open Obsidian (or wrap it in a login item / `systemd --user` / Task Scheduler entry). Search only needs the server while you're using it. | | **A second device to reach it** | Run the server on a home PC or NAS, set the plugin's **API Server URL** to that host, and set an **API key** (`VAULT_API_KEY` env on the server + matching key in plugin settings) so only you can query it. | | **Want to avoid running a server at all** | Use just the **MCP server** with Claude Code from the terminal — no FastAPI process, no plugin. | Want a cloud embedder instead of Ollama? The embedding call is isolated in `indexer.py` / `api_server.py` (`client.embed(...)`); swap it for any provider and keep the same LanceDB pipeline. --- ## Security notes - The API server supports an **API key** (`X-API-Key`). Set `VAULT_API_KEY` (env) or `~/.vault-search/api_key.txt` on the server and the matching key in plugin settings. **Always** set one if the server is reachable beyond `localhost`. - `.env`, `plugin/data.json`, and the whole `~/.vault-search/` data dir are git-ignored. Never commit secrets or your index. - Hybrid/Free chat runs the `claude` CLI as a subprocess with a **deny-list of tools** (no file write, no shell) — it can only read the context you pass it and call read-only literature tools. --- ## Repository layout ``` server/ indexer · scoring · ppr · api_server · mcp_server (core three-piece set) textbook_indexer · graph_builder (optional add-ons) config.py (all settings, env-driven) plugin/ main.js · manifest.json · styles.css (Obsidian plugin) examples/ source_boost.example.json docs/images/ screenshots env.example copy to .env ``` ## Author Built by **Dr. P.W. Chen** — a physician who needed to actually find things in his own notes. 🌐 [drpwchen.com](https://drpwchen.com) · 🐙 [github.com/drpwchen](https://github.com/drpwchen) ## License MIT — see [LICENSE](./LICENSE). Contributions and issues welcome.