mem

# mem > **A portable, self-hosted memory plane for AI agents.** `mem` keeps files, metadata, and embeddings under your control and exposes the same core through an HTTP API, a command-line client, an MCP server, and a web interface. **mem** is a user-owned, portable Memory Plane: users store files, photos, recordings, and notes, and agents can write back task states, observations, decisions, and outputs. Switching agents (e.g., Claude Code, Codex) or moving to a different computer, any authorized agent can retrieve context with provenance and continue working. The AI file system is mem's trust interface for humans. Currently, the web interface allows browsing/downloading original files, natural language search, viewing structured memory and immutable task checkpoints, feedback, archiving, restoring, and forgetting, and exporting or recovering the same data via Workspace Transfer. Immutable correction/supersede, incremental sync, and merge restore are on the roadmap. APIs, MCP, and CLI provide the same capabilities for agents and automation. The project's goal, migration scope, and current gaps are in **[GOAL.md](GOAL.md)**. mem **is not**: - A chat product; answers are handled by upper-layer agents, while mem provides traceable context. - An agent runtime or workflow orchestrator; mem does not manage planning, tool cycles, or model execution. - A generic knowledge base that only uploads PDFs and chats; original multimodal assets, long-term relationships, and agent-written data are equally important. ```mermaid flowchart LR H["Human"] --> UI["AI File System UI"] A["Any Agent"] --> MCP["MCP / API"] O["Automation & Ops"] --> CLI["CLI"] UI --> MEM["mem Memory Plane"] MCP --> MEM CLI --> MEM MEM --> SRC["Original Assets & Provenance"] MEM --> IDX["Structured Memory, Entities, Relations & Index"] SRC --> CTX["Evidence-based Context Pack"] IDX --> CTX ``` > [!WARNING] > `mem` is experimental. Interfaces, storage schemas, and release artifacts may > change without notice. Do not use it as the only copy of important data. ## What is in this repository? | Component | Purpose | | --- | --- | | `server/` | Go HTTP service (`memd`), CLI (`mem`), and stdio MCP server (`mem-mcp`) | | `worker/` | Python gRPC worker for extraction, embeddings, and model providers | | `web/` | React/Vite web interface | | `docker-compose.yml` | Local PostgreSQL/pgvector, Redis, and MinIO dependencies | The current implementation includes file and folder operations, token-based access, search and retrieval surfaces, an extensible processing worker, and MCP tools backed by the same service API. See [SPEC.md](SPEC.md) for the evolving product and architecture contract. ## Tool List | Scenario | What mem provides | |---|---| | **Agent Handover / Device Continuity** | Retrieve previous work, key decisions, and their basis, allowing new agents or devices to continue work | | **Precise Personal Evidence Search** | Find specific information (e.g., amounts, dates, original text) in contracts, invoices, documents, and notes | | **Meeting & Communication Sedimentation** | Convert recordings, minutes, and chats into decisions, action items, entities, and time relations | | **Preference & Constraint Reuse** | Store user-confirmed preferences, long-term constraints, and correction records | | **Multimedia Life Memory Retrieval** | Search for photos, audio, and videos by person, time, location, and event | ## Memory Control Loop mem's core is not a one-time RAG question but a five-stage control loop: 1. **Write**: Save original assets, sources, timestamps, agents/sessions, and access scopes. 2. **Consolidate**: Parse, chunk, extract entities and timestamps, and identify duplicates, conflicts, and version changes. 3. **Recall**: Combine semantic, keyword, entity, timestamp, path, and relation searches, then rerank. 4. **Use**: Return original text fragments, provenance, and confidence levels based on context budgets, allowing upper-layer agents to decide. 5. **Feedback**: Record adoption, corrections, pinning, and forgetting, driving subsequent consolidation and ranking. The current implementation has a model-agnostic `remember → lexical context → feedback / archive / restore / forget` control loop. Automatic consolidation, correction/supersede relations, and evaluated versioned ranking are not yet completed. ## API / MCP / CLI / UI Division of Labor | Interface | Primary Responsibilities | |---|---| | **API** | Sole capability kernel and stable contract; all clients ultimately call it | | **MCP** | Shortest path for agents: write, recall context, read original files, and feedback/lifecycle control | | **CLI** | Batch import, script automation, index rebuild, debugging, evaluation, export, and maintenance | | **UI** | AI file system and trust interface: browse original files, understand recall, correct errors, manage permissions, and forget | `mem_remember` writes structured memory with provenance; `mem_search` returns candidate assets; `mem_context` assembles evidence-based context from files and structured memory. `mem_context` does not handle chat personas or agent reasoning. Simple permission contracts: `remember` requires `write`, `search/context` requires `search`, reading/list memory or original files requires `read`; feedback, archiving, and restoring require `read + write`; forgetting requires `delete` and a workspace role that allows deletion. ## Development Environment Local development dependencies: - Go 1.25 - Python 3.11+ and [`uv`](https://docs.astral.sh/uv/) - Node.js 24 and npm - `protoc` 34.1, `protoc-gen-go` v1.36.11, and `protoc-gen-go-grpc` v1.6.2 when changing protobuf definitions - Docker with Compose ## Why Agent-Native? ```bash # Human or script searches for candidate assets (default CLIP currently uses English as the accepted baseline) mem search "a golden retriever standing on green grass" --format json # Agent host starts an independent stdio MCP adapter MEM_TOKEN=mem_... ./bin/mem-mcp # Agent writes an idempotent, traceable decision mem remember "Contract review prioritizes automatic renewal clauses" \ --kind decision --path /Contracts \ --idempotency-key contract-review-renewal-v1 --agent-id codex # Prepare context for the next task instead of starting a chat in mem mem context "Continue the previous contract review" --source memory --scope /Contracts ``` Each core capability should have a single semantic meaning in the API and be exposed to MCP, CLI, and UI as needed. Agents are not passive file readers but frequent memory users and writers with permission constraints. ## Boundaries with Adjacent Projects | Project | Core Object | What mem borrows | Different choices in mem | |---|---|---|---| | [Tencent/WeKnora](https://github.com/Tencent/WeKnora) | Document knowledge platform, RAG, agent, and auto Wiki | Parsing, hybrid search, citation, evaluation, and agent-first CLI/MCP | Focuses on personal long-term data and cross-agent writing, not knowledge base Q&A or agent runtime | | [mem0ai/mem0](https://github.com/mem0ai/mem0) | Universal agent memory layer | Structured memory, cross-session recall, and feedback | Saves browsable original files, photos, recordings, and AI file system experience | | [letta-ai/letta](https://github.com/letta-ai/letta) | Stateful agent runtime | Long-term state and context management | Does not run agents; acts as an external memory backend for any agent | | [khoj-ai/khoj](https://github.com/khoj-ai/khoj) | Personal AI, chat, and agent | Private personalized knowledge experience | No chat entry or agent personality binding; prioritizes open protocols and data portability | | [nextcloud/server](https://github.com/nextcloud/server) | File sync & collaboration | User-controlled original files and mature file experience | Adds multimodal understanding, relation recall, and agent-native read/write on top of files | mem's moat is not tied to a specific large model but long-term accumulated, user-correctable multimodal original assets, time-person-event relations, cross-agent writing, and real usage feedback-based personalized recall. ## Quick Start The project is currently in Phase 1 MVP. Current development experience: ```bash git clone https://github.com/fullstack-ai-infra/mem.git cd mem ./scripts/dev_up.sh mem auth login mem put ~/Photos --recursive mem search "a golden retriever standing on green grass" mem remember "Photo import completed" --kind task_state --path /Photos \ --idempotency-key photos-imported-v1 mem context "Where did photo import get to?" --source memory --scope /Photos # One agent hands over, another agent/new session resumes with a stable task key mem checkpoint --input handoff.json --idempotency-key photos-handoff-v1 mem resume photos/import # Save a complete, verifiable workspace package across mem deployments or computers mem workspace export --output agent-workspace.membundle ``` Deployment and current capabilities are based on the documentation and code in the repository; a one-click complete startup with `docker compose up` is the target experience. The actual visual model's English/Chinese boundary is in [Visual Search Baseline](docs/acceptance/VISUAL_SEARCH_BASELINE.md). ## Roadmap 1. **Agent Handover**: Versioned contracts, CAS/idempotent persistence, and API/CLI/MCP are completed; continue expanding real host verification and adaptation experience. 2. **Device Continuity**: Bundle v1 and API/CLI/Web `fresh` export/import are completed; next steps are `merge_conservative`, incremental packages, checkpoint upload, and local sync. 3. **Natural Language Image Search**: Continuously enhance visual vectors, captions, entities, time, location, and event-based hybrid recall. 4. **Agent Memory Protocol**: `remember / context / feedback / forget` control loop is completed; next steps are immutable correction/supersede, consolidation, and evaluated versioned ranking. 5. **Visibility & Controllability**: Structured memory, task handover, resume, and workspace transfer views are completed; continue adding correction/supersede, import history, and complete permission management. 6. **Trustworthy Recall**: Continue improving indexing, relations, precise citation, rerank, and context budget. The current text vector records the actual provider; old vectors without provenance are marked as `legacy:unknown` and require explicit rebuilding via `mem provider reindex`, never guessing the model identity. The project does not expand to become a chat frontend, agent runtime, universal workflow platform, or connector collection. ## Design Documents - Project goal and migration target: [GOAL.md](GOAL.md) - Product, architecture, and current implementation specification: [SPEC.md](SPEC.md) - Agent memory product direction: [docs/AGENT_MEMORY_DIRECTION.md](docs/AGENT_MEMORY_DIRECTION.md) - Structured memory records decisions: [docs/adr/0001-agent-memory-records.md](docs/adr/0001-agent-memory-records.md) - MCP access and tool conventions: [docs/mcp.md](docs/mcp.md) - North Star repeatable verification: [docs/acceptance/NORTH_STAR.md](docs/acceptance/NORTH_STAR.md) - Workspace bundle decision: [docs/adr/0004-workspace-bundle.md](docs/adr/0004-workspace-bundle.md) - Local running and verification: [docs/RUN_LOCAL.md](docs/RUN_LOCAL.md) - Project development boundary: [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) - Testing environment and regression threshold: [docs/TESTING.md](docs/TESTING.md) - Participation contribution: [CONTRIBUTING.md](https://github.com/fullstack-ai-infra/.github/blob/main/CONTRIBUTING.md) and [mem development contract](docs/DEVELOPMENT.md) - Version changes: [CHANGELOG.md](CHANGELOG.md) ## Repository Structure ```text mem/ ├── GOAL.md ← Project goal, migration boundary, and acceptance scenarios ├── SPEC.md ← Current product, architecture, and interface specification ├── server/ ← Go main service (API / CLI / MCP Server) ├── worker/ ← Python AI Worker (Processor + Provider) ├── web/ ← AI file system UI ├── docs/ ← Product direction, development, testing, and access documentation ├── docker-compose.yml ← Local development infrastructure └── docker-compose.test.yml ← One-time PostgreSQL regression environment ``` Complete local startup, worker, web, and smoke test steps are in [docs/RUN_LOCAL.md](docs/RUN_LOCAL.md). Web development mode can be started separately: ```bash cd web npm ci npm run dev ``` The service requires a development user and token before protected operations can be used. Follow [docs/RUN_LOCAL.md](docs/RUN_LOCAL.md) for the complete local setup and smoke test. For agent integration, build `mem-mcp` and follow the [MCP setup guide](docs/mcp.md). ## Verify a change CI applies the following checks to pull requests. Run the applicable groups locally before opening one; [the mem development contract](docs/DEVELOPMENT.md) and [testing guide](docs/TESTING.md) define the evidence expected when a check cannot be run. Go integration tests require an explicitly disposable PostgreSQL database whose name ends in `_test`. With the Docker Compose services running, this creates `mem_test` if needed: ```bash make up docker compose exec -T postgres sh -c \ "psql -U mem -d postgres -tAc \"SELECT 1 FROM pg_database WHERE datname='mem_test'\" | grep -q 1 || createdb -U mem mem_test" ``` Run the Go protobuf, formatting, vet, migration, race/coverage, and build checks: ```bash go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.36.11 go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.6.2 PATH="$(go env GOPATH)/bin:${PATH}" export PATH make proto-go git diff --exit-code -- server/internal/workerpb cd server test -z "$(gofmt -l .)" go vet ./... export MEM_TEST_DB=postgres://mem:mem@localhost:5432/mem_test?sslmode=disable go run github.com/pressly/goose/v3/cmd/goose@v3.22.1 \ -dir internal/db/migrations postgres "${MEM_TEST_DB}" up go test -race -p 1 -coverpkg=./... -covermode=atomic \ -coverprofile=coverage.out ./... go tool cover -func=coverage.out cd .. make build ``` Run the Worker protobuf, coverage, and package checks: ```bash cd worker uv sync --frozen --extra test --extra dev make proto git diff --exit-code -- mem_worker/proto uv run pytest --cov=mem_worker --cov-report=term-missing \ --cov-report=xml:coverage.xml uv build ``` Run the Web checks: ```bash cd ../web npm ci npm audit --omit=dev --audit-level=high # advisory; findings do not gate CI npm run lint npm run typecheck npm run build ``` Go and Python tests report coverage in CI. The web package currently uses linting, type checking, and a production build as its required baseline. Production dependency auditing is advisory so that a newly published external advisory cannot block an unrelated pull request. ## Contributing All changes follow an issue-first, pull-request-only workflow: 1. Open or select an issue and record its type, evidence, scope, and acceptance criteria; bugs also receive an impact severity. 2. Develop on a branch linked to that issue. 3. Add tests and verification evidence with the change. 4. Open a pull request that closes or references the issue. 5. Obtain an independent review and pass required CI checks before merge. Read the [organization contribution baseline](https://github.com/fullstack-ai-infra/.github/blob/main/CONTRIBUTING.md) and the [mem-specific development contract](docs/DEVELOPMENT.md), then use [docs/maintainers/triage.md](docs/maintainers/triage.md) for the issue taxonomy. Security reports must follow [SECURITY.md](SECURITY.md), not a public issue. Release maintainers should use [docs/maintainers/releasing.md](docs/maintainers/releasing.md). ## Project status `mem` is in active experimental development. The repository is establishing a stable contribution, test, and release baseline before committing to broad distribution channels or compatibility guarantees. ## License [Apache License 2.0](LICENSE) Copyright © 2026 mem contributors.