Content
# Noema
Noema is a local-first, non-vector memory system for coding agents.
It stores durable agent memory as inspectable files, keeps new memories behind a
review queue, recalls context with lexical signals instead of embeddings, and is
designed to work across hosts such as zode, Codex, Claude Code, and MCP-capable
agent runtimes.
> Status: early implementation. The P0-P6 architecture is represented in this
> repository, but public APIs, file formats, and integration contracts may still
> change before a stable release.
## Why Noema
Most agent memory systems are built around vector databases. That is useful for
semantic search, but it can make memory opaque, hard to audit, hard to delete,
and difficult to run safely in enterprise environments.
Noema takes a different path:
- **No vectors or embeddings**: recall is lexical, explainable, and reproducible.
- **Local-first storage**: the default root is `~/.agent-memory/`, with an
optional `NOEMA_ROOT` override for tests and isolated workspaces.
- **Human review by default**: new candidates enter a hippocampus-style queue
before becoming durable cortex memory.
- **Agent-neutral protocol**: the same memory store can be used by the CLI,
zode, MCP clients, and future host adapters.
- **Enterprise-aware from the start**: tenant boundaries, sensitivity levels,
audit events, ACL policy hooks, KMS metadata policy, and S3-compatible cold
offload are part of the design rather than later patches.
## Features
- Tenant-scoped file layout under `~/.agent-memory/tenants/<tenant>/`.
- Markdown memory records with structured JSON frontmatter.
- Event-sourced review queue using `hippocampus/inbox.jsonl` and
`hippocampus/decisions.jsonl`.
- Candidate decisions: accept, reject, edit, and merge.
- Lexical recall that returns a `MemoryPack` with scores and source metadata.
- `explain` support for understanding why a memory matched a query.
- Automatic rejection of `secret` candidates before review.
- Personal-mode sensitivity cap: `public` and `internal` only.
- Payload-free audit events for review and memory lifecycle operations.
- Vacuum support for compacting review history.
- Forget/tombstone flow with restore-time deletion manifest checks.
- S3-compatible cold offload scaffolding with local fake-store tests.
- MCP stdio tool surface.
- Minimal enterprise server boundary for signed principals and policy checks.
## Repository Layout
```text
.
+-- crates/
| +-- noema-core/ # protocol types, paths, storage, recall, review, policy
| +-- noema-cli/ # `noema` command-line interface
| +-- noema-mcp/ # MCP stdio tool surface
| +-- noema-server/ # enterprise boundary and status service
+-- tests/ # CLI, MCP, enterprise, and offload contract tests
+-- Cargo.toml
+-- README.md
```
## Install From Source
Noema is currently distributed from source.
Requirements:
- Rust stable toolchain
- Git
```bash
git clone https://github.com/ZSeven-W/noema.git
cd noema
cargo build --workspace
```
Run the CLI from the workspace:
```bash
cargo run -p noema -- --help
```
Install it into your Cargo bin directory:
```bash
cargo install --path crates/noema-cli
```
## Quick Start
Use a temporary root while trying Noema:
```bash
export NOEMA_ROOT="$(mktemp -d)"
cargo run -p noema -- init
cargo run -p noema -- remember "Prefer Rust for Noema implementation work."
cargo run -p noema -- review
```
If `remember` auto-accepts the candidate, it will print an accepted memory id.
If it queues the candidate, accept it from the review queue:
```bash
cargo run -p noema -- accept cand_xxxxx
```
Search and inspect recall:
```bash
cargo run -p noema -- search "Rust implementation"
cargo run -p noema -- explain mem_xxxxx --query "Rust implementation"
```
For normal use, omit `NOEMA_ROOT` and Noema will use:
```text
~/.agent-memory/
```
## CLI Overview
| Command | Purpose |
| --- | --- |
| `noema init` | Create the local memory layout and config. |
| `noema remember <text>` | Create a memory candidate. |
| `noema review` | List pending candidates. |
| `noema edit <candidate> --body ... --reason ...` | Revise a pending candidate before acceptance. |
| `noema merge <candidate> <memory> --reason ...` | Drop a duplicate candidate in favor of an existing memory. |
| `noema accept <candidate>` | Persist a pending candidate as memory. |
| `noema reject <candidate> --reason ...` | Reject a pending candidate. |
| `noema search <query>` | Recall matching memories. |
| `noema explain <memory> --query ...` | Explain a recall score. |
| `noema vacuum` | Compact terminal review events into snapshots and archives. |
| `noema sleep [--llm]` | Move stale or low-utility memory toward deeper storage. |
| `noema bench` | Run a synthetic recall benchmark and print README-ready tables. |
| `noema bench --mem0-targets` | Print the Mem0 benchmark scores Noema must exceed. |
| `noema bench --mem0-result <file>` | Summarize a Mem0 benchmark result JSON file. |
| `noema bench --locomo-dataset <file>` | Summarize a LOCOMO dataset file. |
| `noema bench --locomo-evidence <file>` | Run Noema LOCOMO evidence retrieval. Use `--locomo-memory-source raw\|observation\|raw-plus-observation\|fact-layer\|raw-plus-fact-layer` to compare layers. |
| `noema bench --locomo-evidence <file> --locomo-predict-output <file>` | Write a mem0-like predict/proxy JSON file with retrieval contexts and evidence-hit cutoff results. |
| `noema bench --locomo-predict-input <file>` | Continue from an existing LOCOMO predict JSON without recomputing retrieval. |
| `noema bench --locomo-evidence <file> --locomo-answer-tasks-output <file>` | Write host-LLM answer-generation tasks from the same top-k LOCOMO retrieval context. |
| `noema bench --locomo-predict-input <file> --locomo-retry-answer-tasks-output <file>` | Write only missing, empty, or host-failed LOCOMO answer tasks. |
| `noema bench --locomo-predict-input <file> --locomo-answer-tasks-input <file> --locomo-retention-output <file>` | Audit budgeted answer-task prompt retention against the full predict retrieval context. |
| `noema bench --locomo-evidence <file> --locomo-judge-tasks-output <file>` | Convert answer results into judge tasks for LOCOMO correctness grading. |
| `noema bench --locomo-predict-input <file> --locomo-retry-judge-tasks-output <file>` | Write only missing, malformed, or host-failed LOCOMO judge tasks. |
| `noema bench --locomo-evidence <file> --locomo-final-output <file>` | Combine predict, answer, and judge results into final LOCOMO metrics. |
| `noema bench --locomo-predict-input <file> --locomo-target-output <file>` | Compare a judged LOCOMO final result against the Mem0 LoCoMo target. |
| `noema bench --locomo-predict-input <file> --locomo-require-beats-mem0` | Exit non-zero unless the judged LOCOMO score exceeds Mem0's LoCoMo score. |
| `noema bench --locomo-predict-input <file> --locomo-status-output <file>` | Summarize answer/judge completeness for an offline LOCOMO run. |
| `noema bench --locomo-predict-input <file> --locomo-report-output <file>` | Combine proxy, prompt retention, and answer/judge readiness into one run report. |
| `noema bench --locomo-predict-input <file> --locomo-host-manifest-input <file> --locomo-report-output <file>` | Embed zode host-run provenance in the LOCOMO run report. |
| `noema bench --locomo-predict-input <file> --locomo-fail-if-incomplete` | Exit non-zero until the LOCOMO answer and judge artifacts are ready for final scoring. |
| `noema forget <memory> [--hard]` | Tombstone or hard-erase a memory. |
| `noema offload status` | Show cold-offload status. |
| `noema offload run` | Run cold-offload processing. |
| `noema restore <snapshot-or-id>` | Restore while applying deletion manifests. |
| `noema doctor` | Check local store health. |
| `noema reindex` | Rebuild local indexes. |
## Storage Model
Noema keeps storage boring on purpose. The hot path is made of directories,
Markdown files, JSONL event logs, and lock files that are easy to inspect and
back up.
```text
~/.agent-memory/
+-- config.toml
+-- manifests/
+-- tenants/
+-- personal/
+-- tenant.lock
+-- audit.jsonl
+-- users/
| +-- <user-id>/
| +-- memories/
+-- projects/
| +-- <project-id>/
| +-- memories/
+-- hippocampus/
| +-- inbox.jsonl
| +-- decisions.jsonl
| +-- snapshots/
| +-- archive/
+-- indexes/
+-- trash/
+-- cold/
```
The default personal tenant is intentionally simple. Enterprise deployments can
add signed principals, stronger tenant boundaries, ACL policy, and external cold
storage without changing the local memory protocol.
## Memory Lifecycle
```text
candidate
-> pending review
-> accepted memory
-> recalled cortex memory
-> deep memory when stale
-> resurrected for one request when relevant
-> deep again unless confirmed
-> forgotten or hard-erased when required
```
Noema separates capture from persistence. Hosts can submit candidates
automatically, but the review queue remains the standard control point. A
configuration can make review more automatic, while sensitivity ceilings and
secret rejection still apply.
## Security And Privacy
Noema treats memory as potentially sensitive infrastructure.
- `secret` candidates are rejected before they enter review.
- Personal mode rejects `confidential` and `restricted` writes, so users do not
create memory they cannot recall.
- Recall filters memory above the principal's clearance.
- Audit events avoid storing raw memory payloads.
- Redacted or summary-only recall requires precomputed safe variants; Noema does
not rely on runtime prose redaction for sensitive data.
- Sensitive full-text indexing is isolated from normal turn recall.
- Hard erasure must cascade across hot memory, deep memory, local stubs, indexes,
cold payload metadata, and restore deletion manifests.
- Restores apply deletion manifests so old snapshots do not silently resurrect
erased records.
These controls are intentionally conservative. The personal CLI is useful on its
own, but the file protocol and policy model are built for multi-tenant
enterprise use.
## Integrations
### CLI
`noema` is the reference interface and the best place to inspect behavior while
the project is early.
### MCP
`noema-mcp` exposes a stdio JSON-RPC surface for MCP-capable hosts:
```bash
cargo run -p noema-mcp
```
### zode
Noema is designed to be consumed by zode as a native memory backend while
remaining a standalone repository. zode can depend on it through a submodule,
workspace dependency, or package release without making memory storage
zode-specific.
### Enterprise Server
`noema-server` contains the beginning of the enterprise trust boundary: signed
principal claims, tenant-aware policy checks, and service status endpoints.
## S3-Compatible Cold Offload
Noema's primary store is local, but cold memory can be offloaded to an
S3-compatible backend to control disk growth. The cold-offload design keeps local
metadata and restore safeguards, including deletion manifests for erased records.
Current implementation includes the storage abstraction and fake filesystem
backend used by tests. Production S3 wiring should be treated as an integration
target while APIs are still unstable.
## Benchmarks
### Mem0 Benchmark Targets
Noema is being built against the public
[mem0ai/memory-benchmarks](https://github.com/mem0ai/memory-benchmarks/tree/4b61c5d31b9c668a12b4f5e78064248a02c82d2b)
suite at commit `4b61c5d31b9c668a12b4f5e78064248a02c82d2b`.
That suite evaluates memory systems with an `Ingest -> Search -> Evaluate`
pipeline across LOCOMO, LongMemEval, and BEAM. The target is not to reproduce
Mem0's architecture; Noema intentionally stays local-first and non-vector. The
target is to beat the published scores while keeping Noema's auditability,
deletion, tenant isolation, and sensitivity controls intact.
Current external reference scores:
```bash
cargo run -p noema -- bench --mem0-targets
```
| Benchmark | Metric | Mem0 score | Noema must exceed | Notes |
| --- | --- | ---: | ---: | --- |
| LoCoMo | overall score | 92.5 | > 92.5 | multi-session dialogue memory |
| LongMemEval | overall score | 94.4 | > 94.4 | long-term memory questions |
| BEAM 1M | average score | 64.1 | > 64.1 | 1M-token memory ability benchmark |
| BEAM 10M | average score | 48.6 | > 48.6 | 10M-token memory ability benchmark |
These are external Mem0 baselines. Current judged Noema results:
| Benchmark | Cutoff | Runner | Noema score | Mem0 score | Margin | Correct / total | Status |
| --- | --- | --- | ---: | ---: | ---: | ---: | --- |
| LoCoMo | top_200 | zode host runner, v7 96k prompts | 92.6623 | 92.5 | +0.1623 | 1427 / 1540 | exceeds Mem0 and Noema target |
| LongMemEval | - | - | - | 94.4 | - | - | not yet run |
| BEAM 1M | - | - | - | 64.1 | - | - | not yet run |
| BEAM 10M | - | - | - | 48.6 | - | - | not yet run |
The LoCoMo run used the Mem0 benchmark data at commit
`4b61c5d31b9c668a12b4f5e78064248a02c82d2b`, Noema's
`raw-plus-fact-layer` top-200 retrieval export, a 96k-character answer prompt
budget, and zode as the offline answerer/judge host runner. Noema is considered
competitive for LoCoMo top-200 only; LongMemEval and BEAM still need comparable
runs before the broader benchmark goal is complete.
Noema can also summarize checked-in Mem0 result JSON files from that repository:
```bash
cargo run -p noema -- bench --mem0-result results/platform/beam_1m_results.json
```
Example summary for the Mem0 BEAM 1M result at the referenced commit:
| Cutoff | Metric | Score | Total |
| --- | --- | ---: | ---: |
| top_200 | avg_score | 64.1 | 700 |
Noema's benchmark adapter should map the mem0 suite as follows:
- `Ingest`: convert benchmark conversations into Noema memory candidates and
durable memory records, preserving user/session/time metadata.
- `Search`: run Noema lexical recall at the same top-k cutoffs used by the mem0
suite, especially top 50 and top 200.
- `Evaluate`: support predict-only retrieval first, then answerer/judge mode
through a host LLM so Noema can report comparable accuracy/pass-rate scores.
- `Report`: group results by LOCOMO category, LongMemEval question type, and
BEAM memory ability type.
Current LOCOMO adapter status:
```bash
cargo run -p noema -- bench --locomo-dataset /tmp/locomo10.json
```
| Dataset | Conversations | Sessions | Turns | Questions | Evaluable questions |
| --- | ---: | ---: | ---: | ---: | ---: |
| LOCOMO-10 | 10 | 272 | 5882 | 1986 | 1540 |
LOCOMO category counts for the mem0-evaluated set:
| Category | Questions |
| --- | ---: |
| multi-hop | 282 |
| open-domain | 96 |
| single-hop | 841 |
| temporal | 321 |
Noema can now run an evidence-retrieval proxy over LOCOMO. The raw mode uses
original conversation turns with previous/next turn context plus session-level
hippocampal episode memories; the fact-layer mode adds observation-derived fact
memories plus speaker-level summary memories with provenance back to the source
`dia_id`s.
```bash
cargo run -p noema -- bench --locomo-evidence /tmp/locomo10.json --top-k 200 --locomo-memory-source raw-plus-fact-layer
```
Current proxy result:
| Source | Top K | Any evidence hit | All evidence hit | Questions with resolved evidence |
| --- | ---: | ---: | ---: | ---: |
| raw | 50 | 99.7% | 97.4% | 1536 |
| raw | 200 | 99.9% | 99.9% | 1536 |
| raw-plus-fact-layer | 50 | 99.9% | 99.0% | 1536 |
| raw-plus-fact-layer | 200 | 100.0% | 99.9% | 1536 |
Noema can also export a mem0-like predict/proxy JSON file for the next
answerer/judge phase, plus a per-question directory that matches mem0
`--evaluate-only` expectations:
```bash
cargo run -p noema -- bench \
--locomo-evidence /tmp/locomo10.json \
--top-k 200 \
--locomo-memory-source raw-plus-fact-layer \
--locomo-predict-output /tmp/noema-locomo-raw-plus-fact-layer-top200.json \
--locomo-predict-dir /tmp/noema-locomo-mem0-predict-top200 \
--locomo-answer-prompt-char-budget 96000 \
--locomo-answer-tasks-output /tmp/noema-locomo-answer-tasks-top200-budget96k.jsonl
```
The current top-200 unified export is about 323 MB and contains all 1540 mem0
evaluated category 1-4 questions. Questions whose evidence IDs cannot be mapped
to a Noema memory are kept in the export and counted as proxy misses, so mem0
`--evaluate-only` can run without missing per-question files.
LOCOMO evidence fields that pack multiple `dia_id`s into one semicolon- or
space-separated string are split before scoring and task export, and leading
zero forms such as `D30:05` are normalized to the matching turn id.
| Artifact | Source | Top K | Proxy score | Correct / total |
| --- | --- | ---: | ---: | ---: |
| `/tmp/noema-locomo-raw-plus-fact-layer-top200.json` | raw-plus-fact-layer | 200 | 99.7% | 1536 / 1540 |
| `/tmp/noema-locomo-mem0-predict-top200/` | raw-plus-fact-layer | 200 | 99.7% | 1540 question files + summary |
| `/tmp/noema-locomo-answer-tasks-top200-budget96k.jsonl` | raw-plus-fact-layer | 200 | answerer input | 1540 tasks |
The unbudgeted answer-task JSONL is about 201 MB after query-aware episode
compaction. With `--locomo-answer-prompt-char-budget 96000`, the top-200
answer-task JSONL is about 148 MB and keeps the answerer prompt under roughly
96k characters per question without changing the predict/evidence-proxy export.
Each line contains a `custom_id`, the
LOCOMO question metadata, and a host-LLM `messages` payload built from the same
top-k retrieval context. It is intended for zode, Codex, Claude Code, or an
external batch runner to generate answers before a judge pass computes the final
LOCOMO accuracy.
Prompt budgets are a cost/retention tradeoff. Noema can audit a budgeted answer
task file against the full predict JSON:
```bash
cargo run -p noema -- bench \
--locomo-predict-input /tmp/noema-locomo-raw-plus-fact-layer-top200.json \
--top-k 200 \
--locomo-answer-tasks-input /tmp/noema-locomo-answer-tasks-top200-budget96k.jsonl \
--locomo-retention-output /tmp/noema-locomo-answer-retention-top200-budget96k.json
```
The conservative evidence-ID audit maps each budgeted task back to the first
`retrieval_results_in_prompt` records and checks whether LOCOMO `D...` evidence
IDs still appear in those retained memories. It is stricter than the evidence
proxy because fact-layer summary hits do not always carry raw `D...` IDs. The
same retention JSON includes `prompt_summary`, so prompt character totals,
estimated prompt tokens, retained retrieval counts, omitted retrieval counts,
and truncation counts can be reproduced without a separate zode dry-run.
| Answer prompt budget | Task file bytes | Estimated prompt tokens | Avg retained retrievals | ID any-hit / evaluable | ID all-hit / evaluable | Baseline any/all lost |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 48k chars | 72,455,091 | 17,805,319 | 32.8 | 1515 / 1536 | 1440 / 1536 | 19 / 91 |
| 64k chars | 98,329,446 | 24,239,274 | 58.2 | 1529 / 1536 | 1487 / 1536 | 5 / 44 |
| 96k chars, v7 | 148,381,415 | 36,611,760 | 155.3 | 1534 / 1536 | 1529 / 1536 | 0 / 2 |
With zode as the host LLM runner:
```bash
python3 /Users/kayshen/Workspace/ZSeven-W/zode/benchmarks/noema_locomo.py \
--tasks /tmp/noema-locomo-answer-tasks-top200-budget96k.jsonl \
--output /tmp/noema-locomo-answer-results.jsonl \
--zode-config-dir ~/.zode \
--jobs 1 \
--resume \
--retry-empty \
--retry-failed \
--summary-output /tmp/noema-locomo-answer-results.summary.json \
--manifest-output /tmp/noema-locomo-answer-run.manifest.json \
--stop-on-provider-blocker \
--retries 2
```
For long runs, Noema can also emit a compact retry-only answer task file from
the current append-only result JSONL:
```bash
cargo run -p noema -- bench \
--locomo-predict-input /tmp/noema-locomo-raw-plus-fact-layer-top200.json \
--top-k 200 \
--locomo-answer-prompt-char-budget 96000 \
--locomo-answer-results /tmp/noema-locomo-answer-results.jsonl \
--locomo-retry-answer-tasks-output /tmp/noema-locomo-answer-tasks-retry-top200-budget96k.jsonl
```
That retry file preserves the original `custom_id`s and contains only missing,
empty, or host-failed answers, so zode can append recovered results back into
the same answer-results file. In the final v7 96k run, the first full answer
pass produced 16 retryable rows; a resume pass with
`--retry-empty --retry-failed` recovered all of them, leaving 1540 valid answers
and zero retryable answer rows.
Noema can also combine the current proxy, prompt-retention audit, and
answer/judge readiness into a single run report:
```bash
cargo run -p noema -- bench \
--locomo-predict-input /tmp/noema-locomo-raw-plus-fact-layer-top200.json \
--top-k 200 \
--locomo-answer-tasks-input /tmp/noema-locomo-answer-tasks-top200-budget96k.jsonl \
--locomo-answer-results /tmp/noema-locomo-answer-results-zode-top200.jsonl \
--locomo-judge-results /tmp/noema-locomo-judge-results-empty.jsonl \
--locomo-host-manifest-input /tmp/noema-locomo-answer-run.manifest.json \
--locomo-report-output /tmp/noema-locomo-run-report-top200-budget96k.json
```
The current v7 96k full report records
`predict_proxy.overall.accuracy=99.74025974025976`,
`prompt_retention.overall.retained_any_evidence_hits=1534`,
`prompt_retention.overall.retained_all_evidence_hits=1529`,
`completion.final_ready=true`, `completion.blocked_reason=ready`, and a
`target_verdict` with `score=92.66233766233766`,
`correct=1427`, `total=1540`, `exceeds_mem0=true`, and
`meets_noema_target=true`.
When `--locomo-host-manifest-input` is provided, the report embeds the zode
runner manifest under `host_runner`, adding provider-run provenance such as
`unrun_due_to_provider_blocker`.
After answer generation, Noema can convert answer results into judge-task JSONL:
```bash
cargo run -p noema -- bench \
--locomo-predict-input /tmp/noema-locomo-raw-plus-fact-layer-top200.json \
--top-k 200 \
--locomo-answer-results /tmp/noema-locomo-answer-results.jsonl \
--locomo-judge-tasks-output /tmp/noema-locomo-judge-tasks-top200.jsonl
```
Answer result lines can use `{ "custom_id": "...", "answer": "..." }`,
`generated_answer`, or a common batch response shape with
`response.body.choices[0].message.content`. The generated judge tasks ask for a
JSON `{ "reasoning": "...", "label": "CORRECT|WRONG" }` response and preserve
the original `question_id`/cutoff in `custom_id`.
Run the judge tasks through zode the same way:
```bash
python3 /Users/kayshen/Workspace/ZSeven-W/zode/benchmarks/noema_locomo.py \
--tasks /tmp/noema-locomo-judge-tasks-top200.jsonl \
--output /tmp/noema-locomo-judge-results.jsonl \
--zode-config-dir ~/.zode \
--jobs 1 \
--resume \
--retry-empty \
--retry-failed \
--summary-output /tmp/noema-locomo-judge-results.summary.json \
--manifest-output /tmp/noema-locomo-judge-run.manifest.json \
--stop-on-provider-blocker \
--retries 2
```
The zode runner summary is a quick host-side count of latest answer/judge rows.
It can inspect a task file without an output file or host LLM call:
```bash
python3 /Users/kayshen/Workspace/ZSeven-W/zode/benchmarks/noema_locomo.py \
--dry-run \
--tasks /tmp/noema-locomo-answer-tasks-top200-budget96k.jsonl \
--summary-output /tmp/noema-locomo-answer-dry-run.summary.json \
--manifest-output /tmp/noema-locomo-answer-dry-run.manifest.json
```
It can also summarize existing result files without a task file or host LLM call:
```bash
python3 /Users/kayshen/Workspace/ZSeven-W/zode/benchmarks/noema_locomo.py \
--summary-only \
--fail-on-retryable \
--output /tmp/noema-locomo-answer-results.jsonl \
--summary-output /tmp/noema-locomo-answer-results.summary.json \
--manifest-output /tmp/noema-locomo-answer-summary.manifest.json
```
With `--fail-on-retryable`, the runner exits with status 2 when latest results
still contain ordinary retryable answer or judge rows; if those retryables include
a provider blocker such as `http_402_payment_required`, summary-only exits with
status 3.
For provider-balance or billing-sensitive retry batches, run the zode runner
with `--jobs 1 --stop-on-provider-blocker`; it stops at the first blocker such
as `http_402_payment_required`, writes the partial JSONL and manifest, and exits
with status 3.
Noema can gate the full offline benchmark state across predict, answer, and
judge artifacts:
```bash
cargo run -p noema -- bench \
--locomo-predict-input /tmp/noema-locomo-raw-plus-fact-layer-top200.json \
--top-k 200 \
--locomo-answer-tasks-input /tmp/noema-locomo-answer-tasks-top200-budget96k.jsonl \
--locomo-answer-results /tmp/noema-locomo-answer-results-zode-top200.jsonl \
--locomo-judge-results /tmp/noema-locomo-judge-results-empty.jsonl \
--locomo-host-manifest-input /tmp/noema-locomo-answer-summary-zode-top200.manifest.json \
--locomo-fail-if-incomplete
```
Before answer and judge artifacts are complete, the gate returns an error such as
`LOCOMO run incomplete: blocked_reason=host_provider_blocked
next_action=resolve_provider_blocker retryable=0
provider_blocker_reason=http_402_payment_required` while a provider balance
blocker is unresolved. Once answer and judge artifacts are complete, the same
gate prints that the run is ready for final scoring.
`--manifest-output` records the host-run provenance needed for reproducible
benchmark reports: task/result paths, zode binary, provider/model names, retry
settings, run/skipped counts, `tasks_total`, `pending_before_run`,
`unrun_due_to_provider_blocker`, task input size stats, prompt character
distribution, a rough `ceil(prompt_chars / 4)` prompt-token estimate, the same
latest-row summary, provider-blocker status, and answer failure-reason buckets
such as `http_402_payment_required`. It intentionally does not include provider
API keys.
Use `--zode-config-dir` to run against an isolated zode config directory instead
of whichever global config is active in the shell.
The current v7 96k-budget answer run manifest records
`task_file_bytes=148381415`, `tasks_loaded=1540`, and prompt chars
`p50=95738`, `p95=95959`, `max=96000`. It estimates prompt input at `36611760`
tokens total, with `p50=23935`, `p95=23990`, and `max=24000` per task using the
runner's 4 chars/token estimate. It also records Noema prompt-retention stats:
`retrieval_results_in_prompt.mean=155.3422077922078`, `p95=192`, and
`truncated_memories.total=0`. For comparison, the unbudgeted compacted top-200
task file records `task_file_bytes=200747951`, prompt chars `p50=130719`,
`p95=147523`, `max=162186`, and about `49778930` estimated prompt tokens total.
The current full judge-task artifact is
`/tmp/noema-locomo-judge-tasks-zode-top200-v7-96k-full.jsonl`; its zode
manifest records `task_file_bytes=2608433`, `tasks_loaded=1540`, prompt chars
`total=1683047`, `p50=1070`, `p95=1323`, `max=2110`, and estimated prompt input
tokens `total=421327`, `p50=268`, `p95=331`, `max=528`.
Use Noema's status JSON below as the authoritative readiness check because it
also knows the predict set and expected task ids.
If a judge run is interrupted or returns malformed host output, emit retry-only
judge tasks from the current answer and judge results:
```bash
cargo run -p noema -- bench \
--locomo-predict-input /tmp/noema-locomo-raw-plus-fact-layer-top200.json \
--top-k 200 \
--locomo-answer-results /tmp/noema-locomo-answer-results.jsonl \
--locomo-judge-results /tmp/noema-locomo-judge-results.jsonl \
--locomo-retry-judge-tasks-output /tmp/noema-locomo-judge-tasks-retry-top200.jsonl
```
At any point, write a benchmark-status JSON report from the current predict,
answer, and optional judge outputs:
```bash
cargo run -p noema -- bench \
--locomo-predict-input /tmp/noema-locomo-raw-plus-fact-layer-top200.json \
--top-k 200 \
--locomo-answer-results /tmp/noema-locomo-answer-results.jsonl \
--locomo-status-output /tmp/noema-locomo-status-top200.json
```
The status report uses append-only/latest-row semantics for result JSONL, matching
the zode runner's `--resume` behavior. It counts valid answers, empty answers,
host failures, missing answers, valid judge labels, malformed/host-failed judge
rows, and whether the run is ready for final scoring. Add
`--locomo-judge-results <file>` once judge results exist.
After the judge pass finishes, Noema can recompute the final LOCOMO
`metrics_by_cutoff` from the judge labels:
```bash
cargo run -p noema -- bench \
--locomo-predict-input /tmp/noema-locomo-raw-plus-fact-layer-top200.json \
--top-k 200 \
--locomo-answer-results /tmp/noema-locomo-answer-results.jsonl \
--locomo-judge-results /tmp/noema-locomo-judge-results.jsonl \
--locomo-final-output /tmp/noema-locomo-final-results-top200.json
```
The final-output step requires complete answer and judge results for every
exported evaluation. It preserves the original retrieval context, replaces the
proxy cutoff result with `CORRECT`/`WRONG` judge labels, and marks the output as
`eval_mode = "answerer_judge_offline"`.
When the judged result is ready, Noema can write and enforce the Mem0 target
verdict:
```bash
cargo run -p noema -- bench \
--locomo-predict-input /tmp/noema-locomo-raw-plus-fact-layer-top200.json \
--top-k 200 \
--locomo-answer-results /tmp/noema-locomo-answer-results.jsonl \
--locomo-judge-results /tmp/noema-locomo-judge-results.jsonl \
--locomo-target-output /tmp/noema-locomo-target-verdict-top200.json \
--locomo-require-beats-mem0
```
The verdict records the judged score, Mem0's LoCoMo target (`92.5`),
Noema's strict target (`92.6`), and booleans for `exceeds_mem0` and
`meets_noema_target`. The combined run report embeds the same verdict under
`target_verdict` whenever `completion.final_ready=true`.
Current zode-hosted full run status for the top-200 export:
| Artifact | Total | Valid answers | Valid judge labels | Correct | Wrong | Retryable answers | Retryable judges | Final ready |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- |
| `/tmp/noema-locomo-target-verdict-zode-top200-v7-96k-full.json` | 1540 | 1540 | 1540 | 1427 | 113 | 0 | 0 | yes |
The result files are append-only/latest-row JSONL. The final answer summary has
`rows=1556` because it includes the initial 16 retryable answer rows plus the
successful resume rows, but `unique=1540`, `answers.valid=1540`, and
`answers.retryable=0`. The judge summary has `rows=1540`, `unique=1540`,
`judges.correct=1427`, `judges.wrong=113`, and `judges.retryable=0`.
The `raw-plus-fact-layer` evidence proxy remains useful for retrieval debugging,
but the reported LoCoMo score above is the final answerer/judge offline score
for the exported 1540 questions. It is directly compared against the Mem0
LoCoMo top-200 reference score of 92.5 from the referenced benchmark commit.
### Local Recall Latency
Noema includes a small synthetic benchmark runner for recall-path tracking:
```bash
cargo run --release -p noema -- bench --memories 1000 --queries 8 --iterations 50
```
The benchmark creates a temporary Noema root, writes synthetic Markdown memory
records, and measures two paths:
- `noema_engine_recall`: reuse one `NoemaEngine` and call `recall`.
- `zode_turn_injection_equivalent`: mirror zode's current memory injection path
by constructing an engine from `NOEMA_ROOT`, recalling, and rendering the
`MemoryPack` to Markdown for each turn. This does not include provider/model
latency.
Measured on 2026-06-21 with a release build on macOS 26.5.1, Apple M2 Pro,
`rustc 1.95.0`, `cargo 1.95.0`.
Dataset: 1000 generated memories, 8 queries, 50 iterations, 400 measured
operations per scenario, 150262 generated memory-body bytes.
| Scenario | Operations | Total ms | Mean us/op | p50 us | p95 us |
| --- | ---: | ---: | ---: | ---: | ---: |
| noema_engine_recall | 400 | 9391.788 | 23479.470 | 22976.084 | 25944.042 |
| zode_turn_injection_equivalent | 400 | 10524.228 | 26310.570 | 23319.334 | 46983.917 |
Phase breakdown:
| Scenario | Phase | Operations | Total ms | Mean us/op |
| --- | --- | ---: | ---: | ---: |
| noema_engine_recall | load_memories | 400 | 8153.373 | 20383.432 |
| noema_engine_recall | score_memories | 400 | 1143.644 | 2859.110 |
| noema_engine_recall | build_pack | 400 | 43.103 | 107.756 |
| zode_turn_injection_equivalent | create_engine | 400 | 0.101 | 0.252 |
| zode_turn_injection_equivalent | load_memories | 400 | 9255.372 | 23138.431 |
| zode_turn_injection_equivalent | score_memories | 400 | 1170.879 | 2927.197 |
| zode_turn_injection_equivalent | build_pack | 400 | 43.847 | 109.619 |
| zode_turn_injection_equivalent | render_markdown | 400 | 2.085 | 5.211 |
These numbers measure the current file-backed lexical recall path. They are a
baseline for future work on cached indexes, zode-native turn integration, and
cold-store hydration behavior.
## Development
Run the standard checks:
```bash
cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
```
Run a local smoke test:
```bash
export NOEMA_ROOT="$(mktemp -d)"
cargo run -p noema -- init
cargo run -p noema -- remember "Prefer Rust for Noema."
cargo run -p noema -- review
cargo run -p noema-mcp <<< '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}'
```
## Roadmap
- Stabilize the file protocol and frontmatter schema.
- Harden MCP and host integration contracts.
- Expand zode-native review and recall workflows.
- Complete production S3-compatible offload configuration.
- Add richer enterprise policy management and tenant administration.
- Publish crate/package releases after the storage contract is stable.
## License
MIT. See the workspace package metadata in `Cargo.toml`.
MCP Config
Below is the configuration for this MCP Server. You can copy it directly to Cursor or other MCP clients.
mcp.json
Connection Info
You Might Also Like
No recommendations available