OpenPersona

<h1 align="center">OpenPersona</h1> <p align="center"> <b>An AI that remembers your people for you.</b> </p> <p align="center"> The open-source, AI-native relationship memory layer —<br/> across <b>iMessage</b> · <b>WhatsApp</b> · <b>Telegram</b> · <b>WeChat</b> · <b>Slack</b> · <b>Gmail</b> ·<br/> and any chat you live in. </p> <p align="center"> <code>v0.4.5</code> · macOS · <b>private alpha</b> · public source ~v0.5 </p> --- ## Why this exists I built a feature graveyard. Two weeks ago, OpenPersona was a *"local-first commitment graph"* — extract every promise, fact, and event from your IM history into a queryable database, then visualise it. I built 30+ CLI commands. 5 dashboard views (Today / Inbox / Grid / Self View / Search). 5 collectors. A launchd daemon. A menubar app. A 15-tool MCP server. SvelteKit + FastAPI + SQLite + Markdown round-trip. 700+ tests. It looked impressive on paper. I barely opened it. Two weeks of dogfood is short, but the signal was already loud: of 30+ CLI commands, I used 4. I never voluntarily opened Inbox or Grid. Of 200 extracted promises, half were noise. The product's most damning flaw: **the author wasn't using it.** The bug wasn't quality. It was *category*. I had built a SaaS-style CRM for a job that doesn't have a SaaS shape. Real relationships aren't promises and facts in a table — they're attention, texture, time. The "open the dashboard daily" pattern is the graveyard of wellness apps and BI tools. I had walked straight into it. So I deleted half the codebase and pivoted the frame. The data layer stayed (SQLite + Markdown round-trip is the one piece worth keeping). Everything else got rebuilt around one sentence: > **OpenPersona doesn't ask you to open it. It remembers your people for you.** The product is now mostly invisible. It lives in surfaces you already use: - Your **lock screen** — a one-line glance, all day - Your **push notifications** — voice-matched, AI-written, only when it matters - Your **Apple Calendar event description** — pre-meeting brief auto-injected before you arrive - A **natural-language chat hotkey** (⌘⇧Space) — ask anything about anyone Dashboards still exist as backstop. They're hidden by default. **If you find yourself opening this app daily, I've failed at the job.** --- ## What it does Four surfaces. That's the product. ### 🔒 Lock screen widget A single line of text on your lock screen. *"Maya · silent 21d."* *"Vincent at 14:00."* Glanceable. Updated continuously. Native macOS WidgetKit (Swift). ### 📱 Push notifications The product speaks when it has something to say. - **Morning push (8 am)** — *"Today · 3 things."* Concrete, actionable, ranked deterministically. - **Pre-meeting brief (5 min before)** — *"Vincent @ Cathay · last talked 9d · 'intro to Eileen' open."* - **Drift detection** — when an inner-circle relationship goes silent past your baseline. Every notification is **voice-matched**: written by AI in your tone, generated per-recipient, never templated. One tap reveals a draft reply ready to send through the platform's native app. ### 📅 Apple Calendar event auto-prep For any calendar event with a known person, OpenPersona writes a pre-meeting brief into the **event's description field** N hours before the meeting. You open Calendar.app (your existing workflow); the brief is already there. No new app to check. No tab to remember. ### 🔍 Natural-language chat (⌘⇧Space) A global hotkey opens an LLM chat anchored to your relationship graph. Ask anything: ``` > who's at Cathay? > what did Bob say about the deck last week? > draft a reply to Maya in my voice > who should I reach out to this week? ``` Replaces every traditional search bar, filter, and dashboard query. --- ## Where the data comes from OpenPersona reads conversations from the IM platforms you actually live in. | Platform | Live API | Offline export | Notes | |---|---|---|---| | **iMessage** | ✅ shipping | — | Direct `chat.db` read · SMS spam filter · service-aware | | **WeChat** | ✅ shipping | — | Via [`wechat-cli`](https://github.com/Chen17-sq/wechat-cli) subprocess | | **WhatsApp** | beta (`wacrawl`) | ✅ shipping | `OPENPERSONA_WHATSAPP_EXPORT_DIR` for Export-Chat .txt | | **Telegram** | beta (Telethon) | ✅ shipping | `OPENPERSONA_TELEGRAM_EXPORT_JSON` for Desktop export | | **Gmail** | ✅ shipping (IMAP + OAuth) | — | `op auth gmail` runs the loopback flow; both API + IMAP paths live | | **Outlook** | beta (Microsoft Graph) | ✅ shipping | `OPENPERSONA_OUTLOOK_EML_DIR` for .eml directory | | **Slack** | — | ✅ shipping | `OPENPERSONA_SLACK_EXPORT_DIR` for `slackdump` / workspace export | | **Discord** | — | ✅ shipping | `OPENPERSONA_DISCORD_EXPORT_DIR` for official export or DiscordChatExporter JSON | | **Apple Health** | — | ✅ shipping | Export ZIP → `health:*` facts on `me` (`op apple-health-sync`) | | **Your platform** | 🤝 anytime | 🤝 anytime | [Collector Protocol](docs/architecture.md) — 3 record types, 2 functions, ship a PR | > **Honest reality column** (paperboy P2 #31): "shipping" means > the code path runs end-to-end with a synthetic fixture. Beta-flagged > live paths need user tokens / a desktop binary install + a real > account to validate; they're code-complete but only iMessage + > WeChat have been dogfooded against real data daily by the > maintainer. Run `op collectors-status` to see what's wired up > on your machine. The collector layer is **one protocol** — three record types (`Source`, `Message`, `Event`), two functions to implement (`iter_messages`, `iter_events`). Anything that emits messages can plug in. Every new collector enriches every OpenPersona user's graph; we compete on the layer above. --- ## Honesty: cost, privacy, what we send to which AI OpenPersona is local-first **storage**. LLM extraction calls a cloud provider you choose. We're explicit about this because the alternative is dishonest. - **Storage**: every byte of your graph stays on your Mac. SQLite + per-person Markdown, `0700` directory permissions, atomic writes. - **LLM calls**: per-message extraction sends the message text to the provider you've configured (DeepSeek default, OpenAI / Anthropic / Ollama supported). - **Cost** (DeepSeek V4 Flash, default): roughly **$0.50–$5 per month** depending on IM volume + AI-native depth (per-meeting briefs, voice-matched pushes, weekly persona refresh). - **Fully local mode** *(experimental — see caveats)*: pass `--local-only` to route everything through Ollama (Qwen 32B or similar). Your data never leaves your Mac. Real-world tradeoffs (audit P2 #52): - **Hardware**: Qwen 32B uses ~32GB RAM. M3+ recommended. - **Speed**: 30-day WeChat extract = 4–6 hours vs ~5 minutes on cloud. - **Quality**: Chinese commitment-extraction precision drops ~20% vs DeepSeek; verify on your data first. - **Recommended use case**: legal / medical / financial-context conversations. Hybrid mode (local for voice, cloud for extraction) is roadmapped. - **Sensitive content filter**: credit cards, OTP codes, password formats are dropped at the collector layer before any LLM sees them. - **Per-person killswitch + bulk forget**: `op forget --person <id>` atomically removes a person from every table, audit-logged. **Posture (private alpha)**: 0 CVEs across 90+ transitive deps · `bandit` baseline clean · `ruff` green on every commit · **1015 unit tests** across collectors / extractors / store / API / MCP / CLI / agent layer / SPA / CSP / inline-edit / widget xcodeproj / schema index audit / Gmail OAuth flow / MCP write-gate / schema versioning / WeChat group classification / mutations CDC / lock-screen widget quality gate / a11y boundary / markdown round-trip / cold-start attention_score baseline. Full threat model ships in the alpha bundle; moves to `docs/security.md` at public source open. > **Test scope honesty** (paperboy P2 #33): the 1015 number is unit > + integration tests with mocked LLM. Real-data smoke (DeepSeek > calls + actual chat.db + EventKit + Swift widget) is manual > dogfood by the maintainer — not in CI. PRs adding e2e fixtures > are very welcome once source opens. --- ## Try it — for private alpha users If you have a private alpha invite, you got a `.whl` plus an `install.sh` — drop both in a folder, run: ```bash bash install.sh op quickstart ``` Within 30 seconds of `quickstart` finishing, you'll get your first push. Day-to-day commands (full set in `op --help`): | Command | What it does | |---|---| | `op remember` | Run extraction (incremental — picks up only new messages) | | `op recall <person>` | Generate a brief for any person on demand | | `op forget --person <id>` | Atomic delete across every table | | `op merge <src> <dst>` | Merge two person rows when entity resolution split one | | `op dedupe` | Find and merge duplicate people / promises | | `op clean` | Sweep noise out of the graph | | `op doctor` | Comprehensive readiness diagnostic | For Claude Desktop / Cursor integration: ```bash op mcp # stdio MCP server — see docs/mcp.md ``` --- ## For developers — OpenPersona as a layer OpenPersona is **not just a product, it's a layer others can plug into**. Three integration shapes are first-class: ### MCP (recommended for AI agents) The MCP server (**17 tools** — 8 read + 6 write + 1 brief synth + 2 archive) gives Claude Desktop / Cursor / any MCP-compatible agent direct access to your relationship graph + the local IM message corpus via stdio. See [`docs/mcp.md`](docs/mcp.md). ### HTTP API (Agent Native, for everything else) | Endpoint | Purpose | |---|---| | `GET /api/changes?since=<iso>` | Poll the change feed since cursor | | `GET /api/messages?person_id=…` | Query the **local IM archive** — read full history without re-pulling chat.db | | `GET /api/schema/{entity}` | Per-entity JSON Schema for code-gen | | `POST /api/webhooks` | Register a callback URL — delivery daemon now active | | `POST /api/bulk/promises` | Multi-row insert with retry-safe `idempotency_key` | | `GET /api/openapi.json` | Full OpenAPI 3.1 spec | | `GET /api/docs` | Interactive Swagger UI | ### Collector Protocol (for new IM platforms) Three record types, two functions to implement. Anything that emits messages can plug in. See [`docs/architecture.md`](docs/architecture.md). --- ## Why private alpha — and what flips it public Source is currently closed because the v1 pivot is in active flight. The shape stabilises **~v1.1** (~1 month), at which point this README's repo opens. What's gating it: - **Real-data validation of v1 surfaces** — pre-meeting brief / push voice / chat hotkey need a week of someone's real iMessage + WeChat history to tune. Closed alpha lets me iterate without churning a public commit log. - **Lock screen widget** — Swift / WidgetKit work in progress. v0.4 ships the Xcode project generator (`op widget-xcodeproj`); signing / notarisation is the remaining gate. - **Gmail OAuth completion** — IMAP path ships in v0.4; the OAuth refresh-token flow is the polish gate for full Gmail API. WhatsApp / Telegram / Outlook / Slack / Discord all shipped v0.4 offline-export paths. What's already public: - This repo (`Chen17-sq/OpenPersona`) tracks **strategy + roadmap + contact**. Everything in [`docs/`](docs/) is real and up to date. - Daily commits visible on [@Chen17-sq](https://github.com/Chen17-sq)'s public contribution graph (private repo commits surface there). What's gated is real-data validation, not the build itself. **Source opens when the extractors hold up against a week of someone's real iMessage / WeChat / Gmail history.** ### Want early access? - **Email** — [schen.aldrich@gmail.com](mailto:schen.aldrich@gmail.com) with one line on how you'd use it - **LinkedIn** — [Aldrich Chen](https://www.linkedin.com/in/aldrich17siqi/) - **WeChat** — `18574843907` (mention OpenPersona) - **Open an issue** here describing your use case — I read every one and it shapes the v1.1 cut --- ## Architecture ``` surface Lock screen widget (Swift, v0.4 xcodeproj generator) · Push (macOS native) · Calendar event injection · ⌘⇧Space chat hotkey · Person page (Bauhaus, hidden by default) · MCP server (17 tools) intelligence Streaming extract (per-message LLM judgment) · Per-person tone calibration · Voice-matched AI push generation · Behavioural inference (last contact, cadence, drift) · Implicit feedback observer (workflow interception, no buttons) · Wave E contact metadata (bio / relationship / preference / tag / date / school) persona SQLite + per-person Markdown — atomic, lockable, git-diffable, round-trip safe · raw_messages archive (every message persisted, agents query via /api/messages or MCP query_messages) · inner-circle tier persistence (top-20 / next-30, recomputed nightly from attention_score) ────── Collector Protocol ────── collectors iMessage · WeChat · WhatsApp (live + offline) · Telegram (live + offline) · Gmail (IMAP + API stub) · Outlook (Graph + .eml) · Slack (offline) · Discord (offline) · Apple Health (export ZIP) · macOS Calendar (bidirectional) · iCal URL (Google / iCloud / Outlook) · your own ``` We extract structured rows from conversations. We **don't** capture screens, OCR images, or transcribe audio — those plug in via the Collector Protocol. Bring your own meeting AI or wearable. The MCP server exposes the graph to Claude Desktop / Cursor / any MCP-compatible agent — ask *"what did I promise Bob?"* without leaving the chat. See [`docs/mcp.md`](docs/mcp.md) for the tool inventory. --- ## Roadmap | | What | Status | |---|---|---| | **v0** *(retired)* | 5 dashboard surfaces · Promise Grid · Calendar push/pull · Inbox edit · 15-tool MCP · auto-extract daemon | superseded by v0.1 pivot | | **v0.1** | Pivot — lock-widget plumbing · push · calendar injection · chat hotkey backend · streaming extract · commitment-strength three-tier · AI-narrated portrait · voice-matched drafts · observer + correlator | shipped | | **v0.2** *(now, private alpha)* | UX polish (Toast / EmptyState / `?` / optimistic UI / Person-page Svelte rewrite) · setup wizard + sources table · `op morning-push` daemon · terminal-notifier backend · **Agent Native HTTP API** (/changes, /schema, /webhooks, /bulk + idempotency) · status banner · **Master mode** (audit panel · conflict detection · staleness scorer · `op identity-merge` cross-source phone match · groups + group_members schema · conflict-resolve endpoint with master-review UI) | shipped | | **v0.3** *(shipped)* | **raw_messages archive** (persistent local IM corpus) · `op backfill` for full-history pulls · `/api/messages` agent retrieval · `op storage` report · webhook delivery daemon (v0.2 registrations now fire) · Ollama local-mode (`op setup --only ollama`) · group-chat LLM extraction (topic / convener / lurkers / mutual-intro) | shipped | | **v0.4** *(shipped)* | **Wave E contact metadata extractor** (bio / relationship / preference / tag / date / school / note with hallucination guard) · **inner-circle tier persistence** (top-20 inner / next-30 close from `attention_score`, daily 04:00 recompute) · **Person Page raw-messages viewer** (lazy-loaded archive timeline) · **MCP archive tools** (`query_messages` + `message_archive_stats` for external agents) · **6 new collector offline paths** (WhatsApp Export-Chat .txt / Telegram result.json / Gmail IMAP / Outlook .eml dir / Slack export / Discord export) · **schema index audit** (+6 hot-path indexes) · **Apple Health sync CLI** (`op apple-health-sync`) · **widget xcodeproj generator** (`op widget-xcodeproj`) | shipped | | **v0.4.1** *(shipped)* | Wave E **wired into weekly tick** (Sundays 04:30 over inner-circle) · `op metadata-extract` / `op tier-recompute` / `op doctor-schema` / `op collectors-status` CLI commands · **Gmail OAuth** end-to-end (`op auth gmail` runs the loopback flow; Gmail API path live, stdlib only — no `google-auth-oauthlib` dep) · IMAP pagination + max-threads warning · `scripts/publish-pypi.sh` with pre-flight CHANGELOG check | shipped | | **v0.4.2** *(shipped)* | External-audit round 1: **WeChat `@chatroom` group detection** (root-cause for 16k msgs all stamped chat_kind='private') · commit-pipeline **quality floor** (drops `what`<5 chars, intent-without-time, audit-curated reject set) · **noise-people regex** (catches `刚刚`/`吴+`/test-fixture leaks/embedded group hints) · **MCP write-tool security gate** (`OPENPERSONA_MCP_WRITES_REQUIRE_REVIEW=1` short-circuits direct writes; audit.log forensic trail always-on) · **dedicated daemon-writer connection** (`db.daemon_writer()` with explicit BEGIN IMMEDIATE) · **schema-versioning framework** (`schema_versions` + linear migrations) | shipped | | **v0.4.3** *(shipped)* | External-audit round 2: **Lock-screen widget quality gate** (≥1 fact + tier preference; no more zero-context placeholder names on the hero surface) · **ingest-time noise filter** (BROADCAST_BLOCKLIST checked at raw_messages.upsert AND in auto_extract before LLM call) · **voice calibration from raw_messages** (real conversation, not task-y promise excerpts) · **tier exposed end-to-end** (PersonOut.tier · `?tier=inner` filter · Inner/Close badges · /people pill row) · **`window.confirm` → ConfirmDialog** (Bauhaus modal, focus trap, role=alertdialog) · **a11y boundary** (svelte:boundary + skip-link, WCAG 2.4.1) · **mutations CDC log** (real /api/changes deletions; fixes the GDPR violation where `op forget` never reached webhook subscribers) · **`op doctor --report`** (redacted bug-bundle: schema + state + audit-tail, zero PII) | shipped | | **v0.4.4** *(shipped)* | External-audit round 3: **inner-circle dead-loop fix** (interactions logged at /accept /dismiss /page-open /push-fired; cold-start activity baseline so day-1 ranking is meaningful) · **secret hygiene** (backup excludes `llm.env` + `auth/` by default; audit.log chmod 0600) · **UTC timestamps** at every store write (stops cross-tz drift on first_seen_at vs last_seen_at) · **past-tense + self-reminder filter** (drops "发了X" / "I sent Y" + committer==committee==me in commit pipeline + prompt) · **promise IDs in markdown** `<!-- pr_xxx -->` for lossless round-trip · **rate limit split** {llm, read, mutate} buckets (no more inbox-bulk-accept self-DOS) · **strip date suffix** from display_name (no more `p_刘若琳-12-18`) · **`op cache --clear`** · **`op repair`** (orphan source backfill) · **`op uninstall --everything`** · **`op demo-mode`** (env-var swap to .demo for safe live demos) · **docs/README.md navigation** · **KNOWN_ISSUES.md → KNOWN_LIMITATIONS.md** (reframed as design choices) · **README Ollama reality** (32GB RAM, 4-6h, -20% recall caveats) | shipped | | **v0.4.5** *(now, shipped)* | External-audit round 4 — live production bugs: **lock-screen widget diff-aware writes** (was rewriting same JSON 11x in 100s; now skips rewrite when content unchanged via `os.utime` mtime bump) · **widget cooldown + freshness gates** (rotates instead of camping on one item; drops promises with when_iso >6h in past — no more "明天" rendered for a 10-day-old date) · **wechat-cli known-failed-chat blocklist** (24h cache after "找不到" so "公众号" virtual chat isn't retried every tick) · **audit.log durability** (`fcntl.LOCK_EX` + `os.fsync` so concurrent MCP clients don't corrupt + crashes don't lose tail) · **MCP review SECURE BY DEFAULT** (flipped semantics: review always-on; user opts INTO direct writes via `OPENPERSONA_MCP_TRUST_AGENT=1`; old `WRITES_REQUIRE_REVIEW=0` still works as explicit opt-out) · **MCP add_fact prompt-injection hardening** (key prefix whitelist {bio, relationship, preference, tag, date, school, note, health, read, work}; 500-char value cap) | shipped | | **v1.0** *(+3 mo)* | Cross-device sync · plugin marketplace · Lock-screen widget signing & distribution · PyPI publish · **source opens here** | — | [`docs/roadmap.md`](docs/roadmap.md) for the day-by-day breakdown. [`docs/product-spec.md`](docs/product-spec.md) for the canonical product definition. --- ## Contributing Once source opens (~v1.0), the three highest-leverage contributions will be: 1. **A new collector** — implement [`Collector Protocol`](docs/architecture.md) for any IM / email / chat platform you live in. Three record types, two functions, real test coverage already in place. v0.4 ships offline-export paths for WhatsApp / Telegram / Gmail / Outlook / Slack / Discord — extending these to live API mode (e.g. Gmail OAuth, Slack Web API) is a great next step. 2. **An agent integration** — OpenPersona ships an Agent Native HTTP API (`/api/changes` polling, `/api/schema` introspection, `/api/webhooks` push, `/api/bulk` retry-safe writes). Build a CLI / browser extension / mobile app on top, link it back here. 3. **AI-native voice tuning** — prompts living in the codebase. PRs improving voice consistency, language detection, per-person tone matching, or commitment-strength classification welcomed. If you maintain a context-aware product (meeting AI, wearable, custom internal tool) and want to feed OpenPersona's graph: open an issue, we'll co-design. ## License MIT (applies once source opens).