memex

# memex [](https://github.com/ayushagrawal288/memex/releases) [](https://www.python.org) [](https://fastapi.tiangolo.com) [](https://modelcontextprotocol.io) [](LICENSE) [](https://glama.ai/mcp/servers/ayushagrawal288/memex) [](https://glama.ai/mcp/servers/ayushagrawal288/memex) A production-grade persistent memory service for AI agents. Agents forget everything between sessions by default — memex fixes that. It stores, retrieves, and ranks conversation memory using semantic search with recency decay, so agents surface what's relevant *and* recent, not just what's semantically closest. ``` POST /v1/memories → store a memory, embed it, persist to Postgres POST /v1/memories/search → retrieve top-k memories ranked by similarity + recency DELETE /v1/memories/{id} → forget a specific memory GET /v1/memories/count → how many memories does this agent/user have GET /health → liveness + DB connectivity check GET /metrics → Prometheus metrics ``` --- ## Architecture ``` caller (agent / app) │ ▼ FastAPI (async) │ ┌────┴────┐ │ │ embeddings asyncpg pool (min=5, max=20) (fastembed │ ONNX, ▼ local) PostgreSQL 16 pgvector extension ivfflat index (cosine) ``` **Write path:** content → fastembed ONNX inference (local, ~12 ms CPU, `BAAI/bge-small-en-v1.5`) → INSERT with 384-dim vector → return memory ID. **Read path:** query → embed → pgvector cosine search (top_k × 3 candidates) → re-rank with recency decay in Python → return top_k results with scores. --- ## Design decisions ### 1. Recency decay on top of semantic search Pure vector similarity returns the most semantically similar memories, not the most useful ones. A fact from 90 days ago that's a 0.95 similarity match is often less useful than a 0.80 match from yesterday. Score formula: ``` score = α × cosine_similarity + (1 − α) × exp(−λ × age_days) ``` Where `λ = ln(2) / half_life_days` (default: 30 days, so a 30-day-old memory has 50% recency weight). `α` is configurable per request (default 0.7). Task-focused agents use higher α (semantic dominates). Conversational agents use lower α (recency matters more). ### 2. Fetch 3× candidates, re-rank in Python The pgvector query returns `top_k × 3` candidates sorted by pure similarity. Python re-ranks with the decay formula and slices to `top_k`. This prevents recency decay from starving high-similarity older memories — they're still in the candidate pool. At 10× scale (>1M memories per agent): push the scoring into a Postgres function using `pg_proc` to eliminate the Python re-ranking round-trip. ### 3. asyncpg + explicit pool sizing over SQLAlchemy async SQLAlchemy adds ORM overhead on every query. The hot retrieval path — embed, query, re-rank — needs to be tight. asyncpg gives direct control over pool min/max (same instinct as tuning HikariCP in Java). pgvector queries require raw SQL for the `<=>` operator anyway. Pool defaults: `min=5, max=20`. Right-size for a single-instance deployment. Override via `DB_MAX_POOL_SIZE` env var. ### 4. Rate limiting in Postgres, not Redis Sliding window counter via upsert. One fewer dependency. Correct under concurrent requests (transactional upsert). At 10× scale with distributed deployments: replace with Redis `INCR + EXPIRE` — atomic operations, no lock contention. ### 5. ivfflat index, not HNSW `ivfflat` has lower build cost and lower memory footprint — the right tradeoff at small-to-medium scale (<1M vectors). `lists=100` works well up to ~1M rows. At 10× scale: switch to HNSW (`m=16, ef_construction=64`) for better recall at the cost of higher memory and build time. --- ## Running locally **Prerequisites:** Docker and Docker Compose. No API keys required — the entire stack runs locally. ```bash git clone https://github.com/ayushagrawal288/memex cd memex docker compose up ``` The API is live at `http://localhost:8000`. Interactive docs at `http://localhost:8000/docs`. --- ## API reference ### Store a memory ```bash curl -X POST http://localhost:8000/v1/memories \ -H "Content-Type: application/json" \ -d '{ "agent_id": "my-agent", "user_id": "user-123", "content": "User prefers concise responses and dislikes verbose explanations.", "memory_type": "semantic", "importance": 1.2 }' ``` ```json { "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6", "agent_id": "my-agent", "user_id": "user-123", "content": "User prefers concise responses and dislikes verbose explanations.", "importance": 1.2, "memory_type": "semantic", "created_at": "2026-05-26T10:30:00Z", "score": null } ``` ### Search memories ```bash curl -X POST http://localhost:8000/v1/memories/search \ -H "Content-Type: application/json" \ -d '{ "agent_id": "my-agent", "user_id": "user-123", "query": "how does this user like to communicate", "top_k": 5, "alpha": 0.7 }' ``` ```json { "results": [ { "id": "3fa85f64-...", "content": "User prefers concise responses and dislikes verbose explanations.", "memory_type": "semantic", "created_at": "2026-05-26T10:30:00Z", "score": 0.8921 } ], "query": "how does this user like to communicate", "total": 1 } ``` ### Memory types | Type | Use for | |---|---| | `episodic` | Specific events, past conversations | | `semantic` | Facts, preferences, general knowledge | | `procedural` | Workflows, how-to instructions | --- ## Load test results Run on a MacBook M-series, Docker Desktop, single Postgres instance: ```bash locust -f scripts/load_test.py --host=http://localhost:8000 \ --headless -u 50 -r 10 -t 60s ``` **Realistic load** (50 users, 100–300 ms think time — models actual agent traffic): | Endpoint | RPS | p50 (ms) | p95 (ms) | p99 (ms) | Error rate | |---|---|---|---|---|---| | POST /v1/memories (write) | 27 | 160 | 270 | 330 | 0% | | POST /v1/memories/search | 83 | 110 | 200 | 250 | 0% | | Aggregated | **113** | 120 | 230 | 300 | **0%** | **Saturation test** (500 users, minimal think time — finds the throughput ceiling): | Endpoint | RPS (plateau) | p50 (ms) | p99 (ms) | Error rate | |---|---|---|---|---| | POST /v1/memories (write) | 28 | 3,900 | 6,100 | **0%** | | POST /v1/memories/search | 91 | 3,600 | 5,800 | **0%** | | Aggregated | **~120** | 3,700 | 5,900 | **0%** | > Run on MacBook M-series, Docker Desktop (4 CPUs), 4 uvicorn workers, 16 threads/worker. > Embeddings: local ONNX (`BAAI/bge-small-en-v1.5`) — zero external API calls, zero cost. **Why the ceiling is ~120 RPS:** Every write and every search requires one ONNX inference (~10–15 ms on CPU). With 4 Docker CPUs: `4 cores / 12 ms ≈ 333 embeddings/s` theoretical max. After Python overhead, DB queries, and asyncio scheduling: ~120 RPS actual. **Path to higher throughput:** | Approach | Expected gain | Complexity | |---|---|---| | Embedding cache (Redis, key = SHA256 of text) | 2–3× (40–60% hit rate on repeated agent queries) | Low | | Horizontal scaling (N replicas behind a load balancer) | N× linear | Medium | | GPU inference (swap ONNX runtime → CUDA) | 10–50× | Medium | | Voyage-3 API (offload to Anthropic's inference fleet) | Scales to thousands of RPS, limited by API quota | Low code change | --- ## Project structure ``` memex/ ├── app/ │ ├── main.py # REST API — FastAPI, lifespan, router registration │ ├── mcp_server.py # MCP server — single-worker FastAPI on port 8001 │ ├── core/ │ │ └── config.py # All settings, loaded from env │ ├── db/ │ │ └── pool.py # asyncpg pool, migrations │ ├── models/ │ │ └── schemas.py # Pydantic request/response models │ ├── services/ │ │ ├── embeddings.py # fastembed ONNX inference (local, zero API calls) │ │ ├── local_summarizer.py # Extractive summariser — Jaccard dedup + TF scoring │ │ ├── memory.py # Core write/search/scoring logic │ │ ├── metrics.py # Prometheus metric definitions │ │ ├── summarizer.py # Background summarisation job │ │ └── rate_limit.py # Sliding window rate limiter │ └── api/routes/ │ ├── memories.py # Memory endpoints │ ├── health.py # Health + readiness │ └── mcp_tools.py # MCP tool definitions (store, search, delete, count) ├── scripts/ │ └── load_test.py # Locust load test ├── docker-compose.yml ├── Dockerfile └── requirements.txt ``` --- ## Observability `docker compose up` starts Prometheus and Grafana alongside the API: | Service | URL | Credentials | |---|---|---| | REST API docs | http://localhost:8000/docs | — | | MCP server | http://localhost:8001/mcp/ | — | | Prometheus | http://localhost:9090 | — | | Grafana | http://localhost:3000 | admin / admin | The Grafana dashboard is provisioned automatically. Panels: - **HTTP request rate + latency p50/p99** — from `prometheus-fastapi-instrumentator` - **Embedding API latency p50/p99** — per-attempt histogram by operation (`embed` / `embed_batch`) - **Memory operations/s** — create, search, delete throughput - **DB pool utilisation** — active vs idle connections (update interval: 15 s) - **Summariser activity** — memories condensed per hour, run outcomes - **Embedding errors/min** — by operation and error type Custom metrics are in `app/services/metrics.py` and exposed on `/metrics` alongside the standard FastAPI instrumentator metrics. --- ## MCP endpoint memex exposes itself as an [MCP](https://modelcontextprotocol.io) server so any MCP-aware agent (Claude Desktop, Claude Code, custom agents) can store and retrieve memories without custom HTTP integration. **Transport:** Streamable HTTP (MCP 2024-11-05 spec). Single-worker process on port 8001 — session state is in-process, so a separate service avoids sticky-session complexity while keeping the REST API's multi-worker throughput. **Tools:** | Tool | Description | |---|---| | `store_memory` | Embed + persist a memory (type, importance configurable) | | `search_memories` | Semantic + recency ranked retrieval with configurable alpha | | `delete_memory` | Forget a specific memory by UUID | | `count_memories` | How many memories an agent/user pair has | ### Connect from Claude Desktop Add to `~/.config/claude/claude_desktop_config.json`: ```json { "mcpServers": { "memex": { "type": "streamable-http", "url": "http://localhost:8001/mcp/" } } } ``` ### Connect from Claude Code ```bash claude mcp add --transport http memex http://localhost:8001/mcp/ ``` ### Design: why a separate service The MCP Streamable HTTP transport is session-stateful — `initialize`, `tools/list`, and `tools/call` must all reach the **same server process**. The REST API runs 4 uvicorn workers with round-robin routing; routing different MCP requests to different workers breaks session state. Running a dedicated single-worker MCP service on port 8001 avoids sticky-session infrastructure (nginx `ip_hash`, Redis session store) while keeping the REST API fully multi-worker. --- ## Memory summarisation Runs as a background asyncio task on a configurable interval (default: every 5 minutes). Finds any `(agent_id, user_id)` pair where episodic memory count exceeds a threshold, condenses the oldest batch into a single `semantic` memory, then deletes the originals. **Fully local — no LLM API calls.** **How it summarises:** Pure Python extractive algorithm. Sentences are deduplicated by Jaccard similarity (≥ 0.7 threshold), scored by word frequency (TF), and the top-N are returned in original order. ~1 ms per summarisation, zero dependencies beyond the standard library. **Why episodic-only:** Episodic memories are conversation events with natural time-based obsolescence. Semantic and procedural memories encode facts and skills — silently condensing them risks precision loss; they age out via recency decay instead. **Concurrency safety:** Uses `pg_try_advisory_xact_lock` keyed on `hashtext(agent_id|user_id)`. The lock is held only during the DB write transaction, not during the embedding call. Tune via env vars: | Var | Default | Description | |---|---|---| | `SUMMARIZATION_ENABLED` | `true` | Toggle the background job | | `SUMMARIZATION_THRESHOLD` | `100` | Episodic count to trigger per pair | | `SUMMARIZATION_BATCH_SIZE` | `50` | Oldest N memories to condense per run | | `SUMMARIZATION_INTERVAL_SECONDS` | `300` | How often the job wakes up | --- ## What's next - [x] **Memory summarisation** — background job to condense old episodic memories (local extractive algorithm, zero API calls) when count exceeds threshold - [x] **Prometheus + Grafana** — p50/p99 latency dashboards, embedding API call duration, pool saturation - [x] **MCP-compatible endpoint** — Streamable HTTP server on port 8001; 4 tools (store, search, delete, count); connects to Claude Desktop and Claude Code - [ ] **HNSW index option** — flag to switch from ivfflat to HNSW for deployments with >1M vectors - [ ] **Importance-weighted retrieval** — factor `importance` score into ranking formula alongside similarity and recency --- ## Tech stack | Layer | Choice | Why | |---|---|---| | API | FastAPI + uvicorn | Async-first, fast, excellent OpenAPI generation | | Embeddings | fastembed ONNX (`BAAI/bge-small-en-v1.5`) | Local, zero API calls, ~12 ms CPU inference, 384-dim | | Database | PostgreSQL 16 + pgvector | Relational + vector in one system, no extra infra | | Vector index | ivfflat | Lower build cost than HNSW at this scale | | Pool | asyncpg | Direct control, zero ORM overhead | | Summariser | Pure Python extractive | Jaccard dedup + TF scoring, zero ML deps, ~1 ms | | Retry | tenacity | Jitter-based backoff on transient errors | | Metrics | Prometheus + prometheus-fastapi-instrumentator | Standard observability | | Load testing | Locust | Python-native, realistic user simulation |