yantrikdb-server

# YantrikDB > ⚠ **Correction notice (2026-04-19):** Phase 3 benchmark writeups committed > earlier today (`docs/phase3a/`, `docs/phase3b/`, `docs/phase3c/`, > `docs/phase3d/`) used a Python simulator for the "structured memory" > condition — NOT the actual yantrikdb engine. Rerun with real yantrikdb > is in [`docs/phase3e/`](docs/phase3e/). Full audit trail, preliminary > corrected numbers, and the exact publication gate for a findings post > are in [`CORRECTIONS.md`](CORRECTIONS.md#whats-pending). **A memory database that forgets, consolidates, and detects contradictions.** Vector databases store memories. They don't manage them. After 10,000 memories, recall quality degrades because there's no consolidation, no forgetting, no conflict resolution. Your AI agent just gets noisier. YantrikDB is different. It's a **cognitive memory engine** — embed it, run it as a server, or connect via MCP. It thinks about what it stores. **Shortest path to try it:** [MCP setup for Claude Code / Cursor / Windsurf →](#get-started-in-60-seconds--claude-code--cursor--windsurf) (one `pip install`, one config block). > **The bigger picture:** YantrikDB is the memory layer being built on the road to [YantrikOS](https://github.com/yantrikos) — an AI-native operating system where agents are first-class primitives, not apps on top. Memory was the bottleneck, so we're shipping it first. [](https://crates.io/crates/yantrikdb-server) [](https://pypi.org/project/yantrikdb/) [](https://github.com/yantrikos/yantrikdb-server/pkgs/container/yantrikdb) [](https://doi.org/10.5281/zenodo.20128887) [](LICENSE) > 📄 **New paper (May 2026):** [*Skill as Memory, Not Document*](https://doi.org/10.5281/zenodo.20128887) — measuring three failure modes when agent skill catalogs scale to 5,000 skills. Code + raw CSVs reproducible in [`benchmarks/skill_recall/`](benchmarks/skill_recall/).  ## 99.9% token savings vs file-based memory | Memories | File-Based (CLAUDE.md) | YantrikDB | Token Savings | Recall Precision | |---|---|---|---|---| | 100 | 1,770 tokens | 69 tokens | **96%** | 66% | | 500 | 9,807 tokens | 72 tokens | **99.3%** | 77% | | 1,000 | 19,988 tokens | 72 tokens | **99.6%** | 84% | | 5,000 | 101,739 tokens | 53 tokens | **99.9%** | 88% | At 500 memories, file-based memory exceeds 32K context. At 5,000, it doesn't fit in any model — not even 200K. YantrikDB stays at ~70 tokens per query. **Precision improves with more data** — the opposite of context stuffing. Reproduce: the script lives in [yantrikdb-mcp](https://github.com/yantrikos/yantrikdb-mcp/blob/main/benchmarks/bench_token_savings.py), not this repo -- `git clone https://github.com/yantrikos/yantrikdb-mcp && cd yantrikdb-mcp && python benchmarks/bench_token_savings.py` --- ## Three things no other database does ### 1. It forgets ```python db.record("read the SLA doc by Friday", importance=0.4, half_life=86400) # 1 day # 24 hours later, this memory's relevance score has decayed # 7 days later, recall stops surfacing it unless explicitly queried ``` ### 2. It consolidates ```python # 20 similar memories about the same meeting for note in meeting_notes: db.record(note, namespace="standup-2026-04-12") db.think() # → {"consolidation_count": 5} # collapsed 20 fragments into 5 canonical memories ``` ### 3. It detects contradictions ```python db.record("Acme is based in Boston.") db.record("Acme is based in Denver.") # added later, in another conversation db.think() # → {"conflicts_found": 1, ...} db.get_conflicts() # → [{"kind": "identity_fact", ...}] ``` Detection runs over the **relations the extractor recognises** — here a location relation, single-valued at any moment — not over arbitrary sentences. Two limits are worth knowing before you rely on it: - **Claims are keyed by subject.** `"Alice is the CEO of Acme"` and `"Bob is the CEO of Acme"` have *different* subjects with one value each, so nothing compares. The constraint here is inverse-functional — it lives on the object ("CEO of Acme" admits one filler) — and that direction is not modelled. - **Copular attribute claims are off by default.** `"The logo is blue"` → `"The logo is green"` needs `ThinkConfig.extract_attribute_claims`; without it the copular path never runs. Both are tracked in [#95](https://github.com/yantrikos/yantrikdb-server/issues/95). Plus: temporal decay with configurable half-life, entity graph with relationship edges, personality derivation from memory patterns, session-aware context surfacing, multi-signal scoring (recency × importance × similarity × graph proximity). --- ## What makes this different YantrikDB isn't just storage with operations. The engine has a layer that makes agents feel less reactive: - **Proactive triggers** — the system surfaces what needs attention: pending conflicts, decaying important memories, approaching deadlines, patterns across domains. Agents don't have to ask what they should care about. The memory tells them. - **Derived personality** — stable tendencies extracted from memory patterns over time. "This user prefers X, reacts to Y, values Z." Informs default agent behavior across sessions. - **Procedural memory** — strategies that worked before get recorded and reinforced. Agents learn what to do, not just what they know. - **Temporal awareness** — `stale` surfaces important memories that haven't been touched recently. `upcoming` surfaces memories with approaching deadlines. Full cognitive architecture lives in the [standalone engine repo](https://github.com/yantrikos/yantrikdb). This server repo focuses on deployment, HTTP API, and cluster operations. --- ## Get started in 60 seconds — Claude Code / Cursor / Windsurf **The fastest adoption path.** One `pip install`, one config block, and your agent gets persistent memory that auto-recalls on conversation start, auto-remembers decisions, and flags contradictions — without you prompting it. ```bash pip install yantrikdb-mcp ``` Add this to your MCP client config — typically `~/.claude.json` or `.mcp.json` in your project for Claude Code, and the equivalent `mcp` block in settings for Cursor/Windsurf ([Claude Code](https://docs.claude.com/en/docs/claude-code/mcp) · [Cursor](https://docs.cursor.com/context/model-context-protocol) · [Windsurf](https://docs.windsurf.com/mcp)): ```json { "mcpServers": { "yantrikdb": { "command": "yantrikdb-mcp" } } } ``` That's it. No env vars. Uses a local SQLite memory file at `~/.yantrikdb/memory.db`. First call auto-initializes the schema. Restart your client — the `yantrikdb` MCP server will show up with 15 memory tools (see below). **Want a shared memory across machines or teammates?** Point at a YantrikDB cluster instead of local SQLite: ```json { "mcpServers": { "yantrikdb": { "command": "yantrikdb-mcp", "env": { "YANTRIKDB_SERVER_URL": "http://node1:7438,http://node2:7438", "YANTRIKDB_TOKEN": "ydb_your_database_token" } } } } ``` **Want it over HTTP/SSE instead of stdio?** (For IDE integrations that don't support stdio MCP servers.) ```json { "mcpServers": { "yantrikdb": { "type": "sse", "url": "http://your-server:8420/sse", "headers": { "Authorization": "Bearer YOUR_API_KEY" } } } } ``` Then start: `yantrikdb-mcp --transport sse --port 8420`. ### The 15 MCP tools your agent gets | | Tools | |---|---| | **Core memory** | `remember` · `recall` · `forget` · `correct` | | **Cognition** | `think` (consolidate + conflict-detect) · `memory` · `trigger` (proactive insights) | | **Knowledge graph** | `graph` (entities + relations) · `category` | | **Conflicts & corrections** | `conflict` (list + resolve contradictions) | | **Time** | `session` · `temporal` (stale/upcoming queries) | | **Behavior** | `procedure` (strategies) · `personality` (derived traits) | | **Ops** | `stats` (engine health + diagnostics) | Full tool reference and agent-integration patterns: **[yantrikdb-mcp →](https://github.com/yantrikos/yantrikdb-mcp)** · [docs](https://yantrikdb.com/guides/mcp/) --- ## Other ways to use it ### As a network server (binary API + HA cluster) ```bash docker run -p 7438:7438 ghcr.io/yantrikos/yantrikdb:latest curl -X POST http://localhost:7438/v1/remember -d '{"text":"hello"}' ``` Single Rust binary. HTTP + binary wire protocol. 2-voter + 1-witness HA cluster via Docker Compose or Kubernetes. Per-tenant quotas, Prometheus metrics, AES-256-GCM at-rest encryption, runtime deadlock detection. See [docker-compose.cluster.yml](deploy/docker-compose.cluster.yml) and [k8s manifests](deploy/kubernetes/). ### As an embedded library (Python or Rust) ```bash pip install yantrikdb # or cargo add yantrikdb ``` ```python import yantrikdb db = yantrikdb.YantrikDB("memory.db", embedding_dim=384) db.set_embedder(SentenceTransformer("all-MiniLM-L6-v2")) db.record("Alice leads engineering", importance=0.8) db.recall("who leads the team?", top_k=3) db.think() # consolidate, detect conflicts, derive personality ``` --- ## Performance The numbers previously published here (1,689 memories, 112ms recall p50, 76 writes/sec) were from an April-2026 snapshot that didn't trace to any benchmark checked into this repo. They've been removed rather than repeated. The most recent throughput investigation actually in-repo is a 2-core LXC reliability bench from 2026-05-20 ([`benchmarks/throughput_2core_lxc/results_2026-05-20.md`](benchmarks/throughput_2core_lxc/results_2026-05-20.md)). It found fresh installs wedging permanently after 256 writes over HTTP, and read throughput collapsing under write load. The fix shipped the same day in v0.19.1's ancestor release (then v0.8.18, see [CHANGELOG.md](CHANGELOG.md)): | Behavior — 2-core LXC, fresh install | Pre-fix | Post-fix (v0.8.18) | |---|---|---| | Sustained write throughput, HTTP, concurrency 4, pre-computed embeddings | locks at 256 writes | **381/s** | | Engine ceiling, direct call (no HTTP), 32 concurrent writers | n/a — compactor wasn't spawned | **1,115/s** | That's the newest throughput data checked into the repo — it hasn't been re-run since (the server is 20+ releases past it now), and the read-collapses-under-write-load issue that same bench documented is still open. See [`benchmarks/`](benchmarks/) for raw results, harnesses, and caveats across every run; performance work is ongoing. --- ## Status **v0.20.0** — in production use. The embeddable engine has been used in production by the YantrikOS ecosystem since early 2026; the network server runs live on a multi-node Proxmox cluster with multiple tenants. Since the early alpha line, the server has moved from a raft-lite prototype to **openraft-based HA clustering** (2-voter + 1-witness), added **mounted read-only memory packs** (`/v1/packs`, `/v1/pack-context`), reached **HTTP route parity** with the embedded engine (RFC 032), and picked up the engine's **claim grounding + temporal validity windows** on recalled claims and its more precise entity/relation extraction. Operational hardening from the original hardening sprint — `parking_lot` mutexes with runtime deadlock detection, per-handler Prometheus metrics, chaos-tested failover, per-tenant quotas — remains in place and has kept pace release-to-release. Full per-release detail, including engine-pin changes and test-suite results for each version, is in [CHANGELOG.md](CHANGELOG.md). Read the maturity notes: https://yantrikdb.com/server/quickstart/#maturity ## The Problem Current AI memory is: > Store everything → Embed → Retrieve top-k → Inject into context → Hope it helps. That's not memory. That's a search engine with extra steps. Real memory is hierarchical, compressed, contextual, self-updating, emotionally weighted, time-aware, and predictive. YantrikDB is built for that. ## Why Not Existing Solutions? | Solution | What it does | What it lacks | |----------|-------------|---------------| | **Vector DBs** (Pinecone, Weaviate) | Nearest-neighbor lookup | No decay, no causality, no self-organization | | **Knowledge Graphs** (Neo4j) | Structured relations | Poor for fuzzy memory, not adaptive | | **Memory Frameworks** (LangChain, Mem0) | Retrieval wrappers | Not a memory architecture — just middleware | | **File-based** (CLAUDE.md, memory files) | Dump everything into context | O(n) token cost, no relevance filtering | ### Benchmark: Selective Recall vs. File-Based Memory | Memories | File-Based | YantrikDB | Token Savings | Precision | |----------|-----------|-----------|---------------|-----------| | 100 | 1,770 tokens | 69 tokens | **96%** | 66% | | 500 | 9,807 tokens | 72 tokens | **99.3%** | 77% | | 1,000 | 19,988 tokens | 72 tokens | **99.6%** | 84% | | 5,000 | 101,739 tokens | 53 tokens | **99.9%** | 88% | At 500 memories, file-based exceeds 32K context windows. At 5,000, it doesn't fit in any context window — not even 200K. YantrikDB stays at ~70 tokens per query. Precision *improves* with more data — the opposite of context stuffing. ## Architecture ### Design Principles - **Embedded, not client-server** — single file, no server process (like SQLite) - **Local-first, sync-native** — works offline, syncs when connected - **Cognitive operations, not SQL** — `record()`, `recall()`, `relate()`, not `SELECT` - **Living system, not passive store** — does work between conversations - **Thread-safe** — `Send + Sync` with internal Mutex/RwLock, safe for concurrent access ### Five Indexes, One Engine ``` ┌──────────────────────────────────────────────────────┐ │ YantrikDB Engine │ │ │ │ ┌──────────┬──────────┬──────────┬──────────┐ │ │ │ Vector │ Graph │ Temporal │ Decay │ │ │ │ (HNSW) │(Entities)│ (Events) │ (Heap) │ │ │ └──────────┴──────────┴──────────┴──────────┘ │ │ ┌──────────┐ │ │ │ Key-Value│ WAL + Replication Log (CRDT) │ │ └──────────┘ │ └──────────────────────────────────────────────────────┘ ``` 1. **Vector Index (HNSW)** — semantic similarity search across memories 2. **Graph Index** — entity relationships, profile aggregation, bridge detection 3. **Temporal Index** — time-aware queries ("what happened Tuesday", "upcoming deadlines") 4. **Decay Heap** — importance scores that degrade over time, like human memory 5. **Key-Value Store** — fast facts, session state, scoring weights ### Memory Types (Tulving's Taxonomy) | Type | What it stores | Example | |------|---------------|---------| | **Semantic** | Facts, knowledge | "User is a software engineer at Meta" | | **Episodic** | Events with context | "Had a rough day at work on Feb 20" | | **Procedural** | Strategies, what worked | "Deploy with blue-green, not rolling update" | All memories carry **importance**, **valence** (emotional tone), **domain**, **source**, **certainty**, and **timestamps** — used in a multi-signal scoring function that goes far beyond cosine similarity. ## Key Capabilities ### Relevance-Conditioned Scoring Not just vector similarity. Every recall combines: - **Semantic similarity** (HNSW) — what's topically related - **Temporal decay** — recent memories score higher - **Importance weighting** — critical decisions beat trivia - **Graph proximity** — entity relationships boost connected memories - **Retrieval feedback** — learns from past recall quality Weights are tuned automatically from usage patterns. ### Conflict Detection & Resolution When memories contradict, YantrikDB doesn't guess — it creates a conflict segment: ``` "works at Google" (recorded Jan 15) vs. "works at Meta" (recorded Mar 1) → Conflict: identity_fact, priority: high, strategy: ask_user ``` Resolution is conversational: the AI asks naturally, not programmatically. ### Semantic Consolidation After many conversations, memories pile up. `think()` runs: 1. **Consolidation** — merge similar memories, extract patterns 2. **Conflict scan** — find contradictions across the knowledge base 3. **Pattern mining** — cross-domain discovery ("work stress correlates with health entries") 4. **Trigger evaluation** — proactive insights worth surfacing ### Proactive Triggers The engine generates triggers when it detects something worth reaching out about: - Memory conflicts needing resolution - Approaching deadlines (temporal awareness) - Patterns detected across domains - High-importance memories about to decay - Goal tracking ("how's the marathon training?") Every trigger is grounded in real memory data — not engagement farming. ### Multi-Device Sync (CRDT) Local-first with append-only replication log: - **CRDT merging** — graph edges, memories, and metadata merge without conflicts - **Vector indexes rebuild locally** — raw memories sync, each device rebuilds HNSW - **Forget propagation** — tombstones ensure forgotten memories stay forgotten - **Conflict detection** — contradictions across devices are flagged for resolution ### Sessions & Temporal Awareness ```python sid = db.session_start("default", "claude-code") db.record("decided to use PostgreSQL") # auto-linked to session db.record("Alice suggested Redis for caching") db.session_end(sid) # → computes: memory_count, avg_valence, topics, duration db.stale(days=14) # high-importance memories not accessed recently db.upcoming(days=7) # memories with approaching deadlines ``` ## Full API | Operation | Methods | |-----------|---------| | **Core** | `record`, `record_batch`, `recall`, `recall_with_response`, `recall_refine`, `forget`, `correct` | | **Knowledge Graph** | `relate`, `get_edges`, `search_entities`, `entity_profile`, `relationship_depth`, `link_memory_entity` | | **Cognition** | `think`, `get_patterns`, `scan_conflicts`, `resolve_conflict`, `derive_personality` | | **Triggers** | `get_pending_triggers`, `acknowledge_trigger`, `deliver_trigger`, `act_on_trigger`, `dismiss_trigger` | | **Sessions** | `session_start`, `session_end`, `session_history`, `active_session`, `session_abandon_stale` | | **Temporal** | `stale`, `upcoming` | | **Procedural** | `record_procedural`, `surface_procedural`, `reinforce_procedural` | | **Lifecycle** | `archive`, `hydrate`, `decay`, `evict`, `list_memories`, `stats` | | **Sync** | `extract_ops_since`, `apply_ops`, `get_peer_watermark`, `set_peer_watermark` | | **Maintenance** | `rebuild_vec_index`, `rebuild_graph_index`, `learned_weights` | ## Technical Decisions | Decision | Choice | Rationale | |----------|--------|-----------| | **Core language** | Rust | Memory safety, no GC, ideal for embedded engines | | **Architecture** | Embedded (like SQLite) | No server overhead, sub-ms reads, single-tenant | | **Bindings** | Python (PyO3), TypeScript | Agent/AI layer integration | | **Storage** | Single file per user | Portable, backupable, no infrastructure | | **Sync** | CRDTs + append-only log | Conflict-free for most operations, deterministic | | **Thread safety** | Mutex/RwLock, Send+Sync | Safe concurrent access from multiple threads | | **Query interface** | Cognitive operations API | Not SQL — designed for how agents think | ## Ecosystem | Package | What | Install | Source | |---------|------|---------|--------| | **[yantrikdb-mcp](https://github.com/yantrikos/yantrikdb-mcp)** | **MCP server for Claude Code / Cursor / Windsurf — start here** | `pip install yantrikdb-mcp` | [GitHub](https://github.com/yantrikos/yantrikdb-mcp) · [PyPI](https://pypi.org/project/yantrikdb-mcp/) | | [yantrikdb-client](https://github.com/yantrikos/yantrikdb-client) | Python client SDK for the HTTP server (typed memory, reflect, character primitives) | `pip install yantrikdb-client` | [GitHub](https://github.com/yantrikos/yantrikdb-client) · [PyPI](https://pypi.org/project/yantrikdb-client/) | | [yantrikdb-server](https://github.com/yantrikos/yantrikdb-server) | HTTP + wire-protocol server, HA cluster | `docker run ghcr.io/yantrikos/yantrikdb` | [GitHub](https://github.com/yantrikos/yantrikdb-server) | | [yantrikdb (Rust)](https://crates.io/crates/yantrikdb) | Embedded Rust engine | `cargo add yantrikdb` | [GitHub](https://github.com/yantrikos/yantrikdb) | | [yantrikdb (Python)](https://pypi.org/project/yantrikdb/) | Python bindings via PyO3 | `pip install yantrikdb` | [GitHub](https://github.com/yantrikos/yantrikdb) | | [langchain-yantrikdb](https://github.com/yantrikos/langchain-yantrikdb) | LangChain `VectorStore` + `ChatMessageHistory` with explainable recall | `pip install langchain-yantrikdb` | [GitHub](https://github.com/yantrikos/langchain-yantrikdb) · [PyPI](https://pypi.org/project/langchain-yantrikdb/) | | [yantrikdb-hermes-plugin](https://github.com/yantrikos/yantrikdb-hermes-plugin) | Memory provider for NousResearch/hermes-agent, with a reproducible recall benchmark | `pip install yantrikdb-hermes-plugin` | [GitHub](https://github.com/yantrikos/yantrikdb-hermes-plugin) · [PyPI](https://pypi.org/project/yantrikdb-hermes-plugin/) | | [yantrik-memory](https://github.com/yantrikos/yantrik-memory) | Framework-agnostic Python memory layer — traits, bond evolution, knowledge graph | `pip install yantrik-memory` | [GitHub](https://github.com/yantrikos/yantrik-memory) · [PyPI](https://pypi.org/project/yantrik-memory/) | | [openclaw-memory-yantrikdb](https://github.com/yantrikos/openclaw-memory-yantrikdb) | OpenClaw memory plugin — conflict-aware session briefings, recall with retrieval reasons, auto-capture | `openclaw plugins install npm:@yantrikos/openclaw-memory-yantrikdb` | [GitHub](https://github.com/yantrikos/openclaw-memory-yantrikdb) | ## Roadmap - [x] **V0** — Embedded engine, core memory model (record, recall, relate, consolidate, decay) - [x] **V1** — Replication log, CRDT-based sync between devices - [x] **V2** — Conflict resolution with human-in-the-loop - [x] **V3** — Proactive cognition loop, pattern detection, trigger system - [x] **V4** — Sessions, temporal awareness, cross-domain pattern mining, entity profiles - [ ] **V5** — Multi-agent shared memory, federated learning across users ## Research & Publications ### 📄 Skill as Memory, Not Document (May 2026) **[Sarkar, P. (2026). *Skill as Memory, Not Document: A Database-Native Substrate for Agent Skill Catalogs*. Zenodo.](https://doi.org/10.5281/zenodo.20128887)**  A measurement paper on what happens when LLM agent skill catalogs scale. On a 5,000-skill corpus: - **Token cost:** full-catalog disclosure consumes 919,200 tokens (exceeds Claude 3.7's 200K window). YantrikDB's indexed top-K disclosure: 369 tokens, constant in catalog size. The honest ratio against an indexed filesystem baseline is 1.49× — an ablation pinpoints the entire gap as YAML frontmatter overhead. - **Latency:** p50 = 87.3 ms, p95 = 106.3 ms at 5,000-skill scale, single-node. - **Invalid-skill admission:** YantrikDB rejects 70/70 adversarially-malformed skills (0%) at write time; a document-only baseline admits 68/70 (97%). Three failure modes for filesystem-shaped skill catalogs (token burn, slowdown, invalid-skill admission). One unifying framing: skill catalogs for autonomous learning aren't documents; they're memory. Code + scripts + raw CSVs are reproducible at [`benchmarks/skill_recall/`](benchmarks/skill_recall/). Full deposit (PDF + source + data + scripts) on Zenodo. Companion blog post: [yantrikdb.com/papers/skill-substrate](https://yantrikdb.com/papers/skill-substrate/). ### Earlier work - **U.S. Patent Application 19/573,392** (March 2026): "Cognitive Memory Database System with Relevance-Conditioned Scoring and Autonomous Knowledge Management" - **Zenodo (software):** [YantrikDB: A Cognitive Memory Engine for Persistent AI Systems](https://doi.org/10.5281/zenodo.18793952) ## Author **Pranab Sarkar** — [ORCID](https://orcid.org/0009-0009-8683-1481) · [LinkedIn](https://www.linkedin.com/in/pranab-sarkar-b0511160/) · developer@pranab.co.in ## License Apache-2.0. See [LICENSE](LICENSE) for the full text. The [MCP server](https://github.com/yantrikos/yantrikdb-mcp) is MIT-licensed.