memory-harness-kit

# memory-harness-kit Portable, file-based **memory system** for AI coding agents (Claude Code and other harnesses): a CLI (`memoryctl`) plus an MCP server that turn raw session/history sources into normalized records, atomic facts, embeddings, searchable indexes, and review-gated summaries. > Status: **engineering preview (`yellow`)**. The pipeline below is implemented and unit-tested (289 tests, green on Linux CI 3.11/3.12 and local Windows). What is *not* yet proven is broad cross-host portability and the Windows-first bootstrap installer — see [What is proven vs not](#what-is-proven-vs-not-yet). Apache-2.0. --- ## Tool List `memory-harness-kit` (package `memoryctl`) is a **portable file-based memory for AI agents**. It converts raw sources (Claude Code session transcripts, git history, project `STATUS/HANDOFF` docs, daily notes, telegram uploads) into structured, searchable memory — and serves it back to the agent through MCP at the start of each session. All memory is stored as plain files in a single instance directory (`state/`, `runtime/`, `logs/`, `quarantine/`): it's inspectable, backup-friendly, and versionable, with no external database. ### What it Can Do - **Collect** raw material from multiple sources: session transcripts, git commits, project docs, daily notes. - **Normalize** it into deterministic records (`records.jsonl`) with canonical references and filters for "secret-like" values. - **Extract atomics**: events, entities, relationships, **decisions**, **outcomes** (e.g., "test passed/failed"), code anchors — with **linking** (solution ↔ outcome) and **grounding** (validity checking). - **Index for search**: chunking → embeddings → compact, vector, BM25, and graph indexes. - **Search semantically**: hybrid search (vector + BM25), `recall` with fallback, local embedding model without external APIs. - **Summarize**: daily/weekly/monthly summaries; separate path through cron-subagent Claude Code. - **Review loop**: detection of recurring patterns and graph review queues — **without auto-promotion**, all through explicit review packages. - **Integration with Claude Code**: MCP server starts automatically, lifecycle hooks (SessionStart/PreCompact/Stop) silently drain the queue and rebuild the index. ### Pipelines 1. **Foundation** — `sources → normalize → atomics`. Collection of sources, normalization, extraction of atomics (events/entities/relationships/decisions/outcomes + linking + grounding). 2. **Retrieval** — `chunk → embed → index → search`. Embeddings (fastembed ONNX **or** local EmbeddingGemma GGUF), vector + BM25 + graph indexes, hybrid search and recall. 3. **Summarization** — daily/weekly/monthly summaries from normalized and atomic layers, plus subagent provider (LLM summarization through cron-subagent). 4. **Sidecars / review** — health check of retrieval stack, recurrence review, graph review queues; all review-gated, nothing promoted to "truth" automatically. 5. **Integration / lifecycle** — MCP server for Claude Code, lifecycle hooks, scheduled tasks, and one-command installer (`scripts/bootstrap.ps1`, Windows-first). ### What Has Been Tested A total of **289 tests**, green locally (Windows) and in CI on Ubuntu (Python 3.11/3.12). - **Foundation**: collectors (5), session transcripts (29), decisions (16), outcomes (17), linking (20), grounding (17), normalization, and atomics. - **Retrieval**: embeddings (29), indexes (14), BM25 (5), graph (5), search (6). Live run of **real fastembed vector recall** in a separate job in CI on Linux. - **Summarization**: aggregator (13), subagent provider (22). - **Sidecars / MCP**: sidecars (5), MCP server (12). - **Packaging**: install from wheel/sdist proven locally and in GitHub Actions (`make ci-artifact-proof`). What **has not been proven yet**: broad cross-host portability beyond Linux CI and Windows development; Windows-first bootstrap installer (manual, without automated testing); scheduler binding; full replacement of the original OpenClaw runtime. --- ## What it Does `memory-harness-kit` turns raw agent history into a searchable, file-based memory and serves it back through MCP. Everything lives as plain files under one instance root (`state/`, `runtime/`, `logs/`, `quarantine/`): inspectable, backup-friendly, no external database. **Pipelines:** | Pipeline | Flow | Key Modules | |---|---|---| | **Foundation** | sources → normalize → atomics | `collectors/`, `normalize/`, `atomics/` (events, entities, relations, decisions, outcomes, linking, grounding) | | **Retrieval** | chunk → embed → index → search | `embeddings/` (fastembed ONNX / local EmbeddingGemma GGUF), `indexes/` (compact, vector, bm25, hybrid, graph), `search/` | | **Summarization** | records/atomics → daily/weekly/monthly + subagent | `summarize/` (aggregator, subagent provider) | | **Sidecars / review** | health + recurrence + review queues | `sidecars/`, `indexes/review_queues` (review-gated, no auto-promotion) | | **Integration / lifecycle** | MCP + hooks + scheduler + bootstrap | `mcp_server.py`, `scripts/hooks/`, `scripts/bootstrap.ps1` | ## What is Proven vs Not Yet **Proven (implemented + tested):** - 289 unit/contract tests green on local Windows and GitHub Actions Ubuntu (Python 3.11 / 3.12); - packaging: wheel + sdist install proof, locally and in CI (`make ci-artifact-proof`); - **live** retrieval: a dedicated CI job installs the `[embeddings]` extra and runs real fastembed vector recall on Linux; - the full foundation + retrieval + summarization + sidecar pipeline runs end-to-end on a real instance. **Not proven yet:** - broad cross-host portability beyond the current Linux CI + Windows dev evidence; - the Windows-first one-command bootstrap installer (`scripts/bootstrap.ps1`) — it works in practice but has no automated proof; - scheduler / Task Scheduler orchestration wiring; - full drop-in parity replacement of the original OpenClaw runtime. ## CLI All commands take `--root <instance>` and support `--json`. **Setup & validation** - `memoryctl doctor` — basic environment summary (not a readiness validator) - `memoryctl init --root <path>` — create a generic instance scaffold - `memoryctl init-openclaw --root <path> --workspace-root <path>` — OpenClaw-bound scaffold (`install-openclaw` is a deprecated alias) - `memoryctl validate --root <path>` — validate config, scaffold layout, adapter binding **Foundation pipeline** - `memoryctl inventory-sources` — collect the bounded source inventory + manifests - `memoryctl inventory-git [--repo <path> ...]` — collect git commit history → `git_commits.jsonl` - `memoryctl ingest-session --session-id <id>` — store a raw session transcript - `memoryctl list-raw-sessions` — list stored transcripts - `memoryctl normalize-records` — normalize inventory into deterministic `records.jsonl` - `memoryctl build-foundation` — inventory → normalize → atomics in one pass **Retrieval pipeline** - `memoryctl build-retrieval` — chunk + embed + compact/vector indexes (`embed-atomics` is an alias; provider from `memoryctl.yaml`) - `memoryctl build-index` — BM25 keyword index over chunks - `memoryctl build-graph-index` — relation/graph index + review-queue candidates - `memoryctl search --query <q> [--mode vector|bm25|hybrid]` — hybrid retrieval over embedded chunks - `memoryctl recall --query <q>` — compact index first, vector fallback **Summarization pipeline** - `memoryctl build-summaries` — daily / weekly / monthly summaries - `memoryctl summary-requests [--pending]` — list LLM summary request packets (subagent path) - `memoryctl apply-summary --request <packet>` — write a subagent-produced summary back **Sidecars & integration** - `memoryctl health-check` — diagnose foundation/embeddings/index layers → `ok|degraded|missing` - `memoryctl recurrence-review` — detect recurring patterns → review packet (no auto-promotion) - `memoryctl recurrence-status` — narrow OpenClaw recurrence status slice - `memoryctl mcp-serve` — start the MCP stdio server (auto-loaded by Claude Code) ## Install ### Manual (portable core) ```bash cd /path/to/memory-harness-kit python3 -m venv .venv . .venv/bin/activate # Windows: .venv\Scripts\activate pip install -e . # core only (PyYAML) pip install -e ".[embeddings]" # + fastembed ONNX vector retrieval pip install -e ".[gguf]" # + in-process EmbeddingGemma GGUF provider memoryctl doctor --json ``` Generic instance smoke: ```bash memoryctl init --root /tmp/memory-instance memoryctl validate --root /tmp/memory-instance --json # -> status: ok memoryctl build-foundation --root /tmp/memory-instance --json ``` ### One-command bootstrap (Windows-first) `scripts/bootstrap.ps1` takes a fresh clone to a working install (venv, embeddings backend, config, MCP registration, scheduled tasks, lifecycle hooks). It is idempotent and supports `-DryRun`. See [docs/Bootstrap_install.md](docs/Bootstrap_install.md). ```powershell powershell -ExecutionPolicy Bypass -File scripts\bootstrap.ps1 -DryRun ``` ### Claude Code integration Copy `.claude/settings.example.json` → `.claude/settings.json`, set `<REPO_ROOT>` and `<INSTANCE_ROOT>`. The `memory` MCP server then auto-loads; see [CLAUDE.md](CLAUDE.md) for the session protocol. ## CI `make ci-smoke` (isolated venv, editable install, unit/contract suite, `init` + `validate` smoke) and `make ci-artifact-proof` (wheel/sdist build + install + CLI proof). GitHub Actions runs both on Ubuntu 3.11/3.12 plus a live fastembed retrieval job — see [.github/workflows/ci-smoke.yml](.github/workflows/ci-smoke.yml). ## Release posture - package version `0.1.0`, no public release tags yet; - license: Apache-2.0 via top-level [LICENSE](LICENSE); - verdict: `yellow` / engineering preview — treat as pre-release until a wider-share release gate is intentionally cut; - version/tag/changelog discipline: [docs/Release_policy.md](docs/Release_policy.md), [CHANGELOG.md](CHANGELOG.md). ## Docs map - [docs/Start_here.md](docs/Start_here.md) — canonical onboarding map and doc precedence - [docs/Architecture.md](docs/Architecture.md) — layers (portable core / adapters / orchestration) - [docs/Config_contract.md](docs/Config_contract.md) — `memoryctl.yaml` shape - [docs/Install_semantics.md](docs/Install_semantics.md) — exact command promises and boundaries - [docs/Operator_install_guide.md](docs/Operator_install_guide.md) — operator install/validation path - [docs/DevOps_readiness_checklist.md](docs/DevOps_readiness_checklist.md) — proven vs not-yet-proven ladder - [AGENTS.md](AGENTS.md) / [CLAUDE.md](CLAUDE.md) — agent-facing entrypoints