ltm-vault

# ltm-vault > A source-grounded MCP memory runtime and Obsidian-compatible vault template for > LLM agents. > > `ltm-vault` helps agents keep long-running project memory without silently > overwriting uncertainty: immutable raw sources, source summaries, decision > records, contradiction tracking, confidence-aware retrieval, and human review > queues are first-class parts of the system. [](LICENSE) [](https://www.python.org/) Read this first: [English](#what-is-this) ## What is this? `ltm-vault` combines an Obsidian-compatible Markdown vault, source-grounded archival rules, decision records, contradiction preservation, and an MCP memory runtime backed by DuckDB, BM25, optional Ollama embeddings, and graph expansion. The core assumption is simple: **LLM output can be wrong by default**. Raw evidence, citations, uncertainty labels, preserved contradictions, and human review queues are therefore not optional extras. They are the operating model. Agents such as Claude, Cursor, Antigravity, and Codex should read [AGENTS.md](AGENTS.md) before writing to the vault. The full mental model lives in [00_System/Second Brain Operating Model.md](00_System/Second%20Brain%20Operating%20Model.md). ## Start From A Clean Private Vault This repository is a framework skeleton plus seed/example knowledge. If you are turning it into your own private second brain, keep the framework files and delete the existing seed documents before adding personal memory. Keep these: - `00_System/`, `90_Engine/`, `docs/`, `scripts/` - root docs such as `README.md`, `SETUP.md`, `AGENTS.md`, `LICENSE` - `.mcp.json.example`, `requirements.txt`, `.gitignore` - layer `README.md` files and `.gitkeep` placeholders, if present Delete seed/user-specific content from the knowledge layers: ```bash find 10_MOC 20_Concepts 30_Projects 40_Decisions \ 50_Source_Summaries 60_Open_Questions 70_Contradictions 80_Reviews \ -type f -name '*.md' ! -name 'README.md' -delete find 05_Inbox 06_Raw \ -type f ! -name 'README.md' ! -name '.gitkeep' -delete ``` Then rebuild the local cache: ```bash python3 90_Engine/indexer.py --force --embed --report ``` If Ollama is not running yet, omit `--embed` for a BM25-only first pass. See [SETUP.md](SETUP.md) for MCP client setup and the private/public split. See [examples/mini-vault](examples/mini-vault) for a tiny source-to-decision demo. ## Layers | Layer | Path | Role | |------|------|------| | **Archive** | `05_Inbox/`, `06_Raw/` | Unprocessed inbox (05_Inbox, excluding indexing) → Immutable raw sources (06_Raw, for full-text search only · graph excluded). | | **Wiki/Knowledge** | `20_Concepts/`, `50_Source_Summaries/`, `10_MOC/` | Source summaries and durable conceptual knowledge, exploration maps. | | **Project** | `30_Projects/` | Active project dashboards. | | **Decisions** | `40_Decisions/` | Important choices and rationales. Existing decisions are immutable (supersede-style). | | **Reviews** | `60_Open_Questions/`, `70_Contradictions/`, `80_Reviews/` | Open questions, contradiction preservation, human review queues. | | **Runtime** | `90_Engine/` | DuckDB indexing, hybrid search, MCP server. | Data flow: `05_Inbox/` → `06_Raw/` (immutable) → `50_Source_Summaries/` → interpretation layer updates → review/contradiction/question routing → `90_Engine/` reindexing. For a detailed diagram, see [Second Brain Operating Model](00_System/Second%20Brain%20Operating%20Model.md). ## Architecture **Markdown is the source of truth; DuckDB is a derived, regenerable cache.** Clients write Markdown through MCP tools, the indexer compiles it into a DuckDB graph cache, and the retriever serves layer/confidence-aware hybrid search back to the clients. The cache (`ltm_cache.db`) is gitignored and rebuilt per machine — only Markdown syncs across devices. ```mermaid flowchart LR CLI["LLM clients<br/>Claude · Codex · Antigravity"] MD["Markdown vault<br/>source of truth · 9-predicate links"] IDX["indexer.py<br/>incremental compile · MD5 · orphan prune"] DB[("DuckDB cache · ltm_cache.db<br/>nodes · edges · embeddings")] RET["retriever.py<br/>BM25 + dense + graph expansion"] MCP["mcp_server.py<br/>MCP tools"] OLL["Ollama · bge-m3"] GIT["git remote<br/>cross-machine sync"] CLI -->|"create / update / delete"| MD MD <-->|"pull / push"| GIT MD --> IDX OLL -.->|"embed"| IDX IDX --> DB DB --> RET OLL -.->|"query embed"| RET RET --> MCP MCP --> CLI ``` ## Daemon (always-on & startup) The daemon (`vault_daemon.py`) **automatically starts on the first MCP client request** and resides idle-free (`DAEMON_IDLE_SHUTDOWN` defaults to off). That is, once you've opened a client, the daemon stays running, and no extra env vars (`USE_DAEMON`, etc.) are needed. `mcp_server.py` is a thin proxy for the daemon. **Always-on / startup registration.** To ensure the daemon runs and performs periodic git syncs even if no clients have been opened since reboot, register it to run at login: - **macOS** — [`scripts/launchd/com.ltm-vault-daemon.plist.example`](scripts/launchd/com.ltm-vault-daemon.plist.example) (RunAtLoad + KeepAlive) - **Linux** — [`scripts/systemd/ltm-vault-daemon.service.example`](scripts/systemd/ltm-vault-daemon.service.example) (Restart=always) - **Windows** — add a shortcut to `pythonw.exe vault_daemon.py` in `shell:startup`, or a Task Scheduler 'on logon' trigger. For step-by-step commands, substitutions, and `SYNC_ENABLED` (which also handles git syncs), see the 'Daemon' section in [SETUP.md](SETUP.md). ## Template ↔ Instance Sync (public ↔ private) This project operates as **public template** (`ltm-vault`, upstream) and **private instance** (`ltm-vault-private`, origin) repositories. The framework (engine, docs, policies, scripts) is shared between the two, and actual knowledge resides only in the private instance (see [SETUP.md](SETUP.md) §8). The two repositories **selectively sync only framework paths** — in both directions, only specific paths are transferred, not the entire `git merge`. | Direction | Command | Transferred content | |------|------|-----------| | private → public | [`scripts/sync-template.sh`](scripts/sync-template.sh) | Curated framework improvements only (allowlist guard) | | public → private | [`scripts/pull-framework.sh`](scripts/pull-framework.sh) | Upstream framework updates (`90_Engine` · `docs` · `scripts` · `00_System` · root docs) | ## Getting Started - Installation, initial indexing, and MCP client setup: [SETUP.md](SETUP.md) - MCP tool specifications and AI memory write flow: [docs/MCP_TOOLS.md](docs/MCP_TOOLS.md) - Antigravity-specific connection guide: [docs/ANTIGRAVITY.md](docs/ANTIGRAVITY.md) - Why this structure is necessary: [docs/WHY_LTM_VAULT.md](docs/WHY_LTM_VAULT.md) - Tiny demo vault: [examples/mini-vault](examples/mini-vault) - Cross-device git sync and optional automatic sync: [SETUP.md](SETUP.md#automatic-sync-optional) The minimal flow: 1. Install Python dependencies 2. Prepare Ollama and the `bge-m3` model 3. Compile Markdown into DuckDB cache with `90_Engine/indexer.py` 4. Register `90_Engine/mcp_server.py` in MCP client setup For detailed commands and OS-specific setup examples, refer to [SETUP.md](SETUP.md). ## Folder Structure ```text ltm-vault/ ├── README.md ├── AGENTS.md # LLM Agent Operating Guidelines ├── SETUP.md ├── docs/ │ ├── ANTIGRAVITY.md │ ├── MCP_TOOLS.md │ └── WHY_LTM_VAULT.md ├── examples/ │ └── mini-vault/ # small runnable source→summary→decision example ├── 00_System/ # Ontology + Second Brain Policy │ ├── Ontology Specification.md │ ├── Second Brain Operating Model.md │ ├── Source Policy.md │ ├── Ingest Policy.md │ ├── Review Policy.md │ └── Naming Convention.md ├── 05_Inbox/ # Unprocessed Inbox (excluding indexing) ├── 06_Raw/ # Immutable Originals = Source of Truth (for full-text search only, excluding graph) ├── 10_MOC/ # Map of Content ├── 20_Concepts/ # Durable Concept Knowledge ├── 30_Projects/ # Active Project Dashboard ├── 40_Decisions/ # Decision Records ├── 50_Source_Summaries/ # Source Compression Understanding ├── 60_Open_Questions/ # Unresolved Questions ├── 70_Contradictions/ # Contradictions and Outdated Assumptions ├── 80_Reviews/ # Human Verification Queue └── 90_Engine/ # Runtime/Indexing/MCP ├── indexer.py ├── retriever.py ├── mcp_server.py # MCP stdio server (thin proxy for daemon) ├── vault_daemon.py # Single-owner daemon (standard; FastAPI/localhost, auto-start) ├── daemon_client.py # Daemon discovery + lightweight HTTP client ├── vault_sync.py # Daemon-driven git synchronization helper (event-driven) └── ltm_cache.db # locally generated cache, ignored by git ``` The numeric prefixes are for Obsidian's natural sorting and reading order. The key rules are the 9 predicate constitutions in `00_System/Ontology Specification.md` and the second brain policy documents in `00_System/`. ## Daily Workflow Nodes can be created by humans directly in Markdown or generated using MCP write tools. The standard ingest procedure follows [Ingest Policy](00_System/Ingest%20Policy.md). - After human editing, update the cache by running `sync_vault()` or the indexer. - When agents save memories, they check existing titles using `list_nodes()`, then use `create_node()`, `update_node()`, or `upsert_edge()`. - If dangling edges occur, automatic reconciliation handles them, or use `reconcile_graph()` for immediate cleanup. - **Not all sources become concept nodes.** The default destination is `50_Source_Summaries/`, and only durable knowledge is promoted to `20_Concepts/`. - Search is **hierarchical and confidence-aware**. `06_Raw/` is for full-text search only and is demoted, and low-confidence and deprecated items are also demoted and marked; review layers (`60/70/80`) are excluded from default search (`retrieve_knowledge(..., include_reviews=true)`). Review queue inspection is done with `review_queue()`. See: [[2026-06-18-layer-and-confidence-aware-retrieval]]. When creating nodes, only the following 9 edge predicates are allowed (used for stable knowledge relationships): `requires` · `utilizes` · `implemented_by` · `extends` · `abstracts` · `causes` · `contradicts` · `replaces` · `defines` ## Learn More - Agent guidelines: [AGENTS.md](AGENTS.md) - Design rationale: [docs/WHY_LTM_VAULT.md](docs/WHY_LTM_VAULT.md) - Operating model: [00_System/Second Brain Operating Model.md](00_System/Second%20Brain%20Operating%20Model.md) - Policies: [Source Policy](00_System/Source%20Policy.md) · [Ingest Policy](00_System/Ingest%20Policy.md) · [Review Policy](00_System/Review%20Policy.md) · [Naming Convention](00_System/Naming%20Convention.md) - Ontology constitution: [00_System/Ontology Specification.md](00_System/Ontology%20Specification.md) - Initial corpus (LLM mental model): [10_MOC/Karpathy LLM Framework MOC.md](10_MOC/Karpathy%20LLM%20Framework%20MOC.md) - Setup and operation: [SETUP.md](SETUP.md) · MCP tools: [docs/MCP_TOOLS.md](docs/MCP_TOOLS.md) ## License MIT License. See [LICENSE](LICENSE) for details. ## Sources - [Andrej Karpathy](https://karpathy.ai) - [Intro to Large Language Models](https://www.youtube.com/watch?v=zjkBMFhNj_g) - [Let's build the GPT Tokenizer](https://www.youtube.com/watch?v=zduSFxRajkE) - [Software 2.0 Essay](https://karpathy.medium.com/software-2-0-a64152b37c35) - [llm.c GitHub](https://github.com/karpathy/llm.c) - [nanoGPT GitHub](https://github.com/karpathy/nanoGPT)